recuedMNN Error: C stack usage 7978148 is too close to the limit.

[wanghao98]

Hello, thank you for the amazing package! However we ran into some issue when I was running reducedMNN() with my own PCA embeddings. I hit an error saying ERROR: C stack usage 7978148 is too close to the limit.
When we looked into the function we realized for reducedMNN, ultimately .create_tree_predefined is called and that calls .binarize_tree and .fill_tree. Both of those two functions are recursive.
Any thought on fixing this problem? Thanks!

[LTLA]

Do you have a minimal reproducible example? Looking at the code, the recursion seems fine to me; I can't think of what you might provide in merge.order= that causes an infinite recursion.

[wanghao98]

Hi, Thank you for the response. Here is the code I used:
mnn_sample <- reducedMNN(pca_embedding_sample), here pca_embedding_sample is a list of 336 pca matrix. We didn't specify any other argument for that function. When I specify auto.merge= TRUE to skip the recursive code, it didn't hit the error but has been running for several days. Hope the info might be helpful.
Thanks again for the help!

[LTLA]

I have no idea. You're going to have to provide a MRE, even if it is just with simulated data.

[wanghao98]

Hi, sorry for the late reply. Here is the link to the data we used. We saved it into a Rda file. https://drive.google.com/file/d/1m6W4lPsMc5ehkwGIsm93F2SGws7PPm9L/view?usp=sharing
After you load the data, you could run mnn_sample <- reducedMNN(pca_embedding_sample)
This is where we hit the error Error: C stack usage 7973380 is too close to the limit
Thank you for looking into that.

[wanghao98]

Hello, just want to follow up on this. Thanks!

[LTLA]

Will try to get to it tomorrow, but I'm afraid I can't promise anything...

[LTLA]

Huh, works fine for me:

X <- reducedMNN(pca_embedding_sample)
X
## DataFrame with 153216 rows and 2 columns
##                                corrected                  batch
##                                 <matrix>            <character>
## 1229878 1.333930: 3.877292: 1.725218:... 0a1148dc-356f-4dc4-9..
## 645643  0.543186: 4.240856: 1.139626:... 0a1148dc-356f-4dc4-9..
## 558085  2.683715: 1.127521: 0.915093:... 0a1148dc-356f-4dc4-9..
## 828841  3.399150:-7.506363:-0.339844:... 0a1148dc-356f-4dc4-9..
## 1136033 2.125245: 0.630467:-1.438122:... 0a1148dc-356f-4dc4-9..
## ...                                  ...                    ...
## 1022646  -0.272424:2.74660:-4.610657:... f09bc3b1-4818-4240-b..
## 935845    2.619467:1.25032:-0.804194:... f09bc3b1-4818-4240-b..
## 885645   -0.134671:3.30904:-2.640632:... f09bc3b1-4818-4240-b..
## 544186    2.254283:3.16733: 1.023467:... f09bc3b1-4818-4240-b..
## 1136882  -0.145355:3.31120:-3.357116:... f09bc3b1-4818-4240-b..

A bit slower than I'd like; the underlying algorithm is an old version that is quadratic with respect to the number of batches (cubic, for auto-merging!) and I haven't had the chance/funding to update it to the latest version here.