Question about memory usage

[BDumphart]

Hi again!

This is more of a general question because i don't know where else something could be wrong.

I need to open .c3d files of over 2k patients, each with ~5 .c3d files in a loop. And i run into a RAM problem.. i had the same issue when working with the btk tool - there i just had to close the connection to the .c3d file in thr end and the RAM was fine, but I can't find a close function for ezc3d - am I missing something here or should ezc3d close the file automatically? I am using the python version, if this changes anything.

Thanks,

Bernhard

[pariterre]

Hi @BDumphart !

Unlike BTK, there is no "connexion" to a backend in c3d. The output of the file is the whole story. This means that overriding the output variable should be enough UNLESS there is a memory leak in the code. If it is the case please let me know!

[BDumphart]

Hi @pariterre!

Thank you for the quick response.
How would I know if it is a memory leak in the code? The programmed ended with the message: "Process finished with the exit code -1073740791 (0xC0000409)" nothing more and from what I could find online this has something to do with memory.
I already tried to save my important variables every 1000 iteration to a pickle file and re-declare all the used variables and nothing changed error wise (The error occurs at iteration 2300).
If you need any more information let me know!

Cheers,

Bernhard

[pariterre]

You should monitor the RAM usage. If you are on Windows, this can be done from the task manager. From UNIX system, you can use top from the command lines (or the equivalent of the task manager).
Then, your code should look like that :

from ezc3d import c3d
for file in files:
    c = c3d(file)
    # Do Something

and NOT:

from ezc3d import c3d
c = []
for file in files:
    c.append(c3d(file))
    # Do Something

if the memory usage increase overtime in a linear fashion, then there is a memory leak. If not, then it means it crashes for another reason. Maybe one of the file is corrupted or something

[BDumphart]

The memory increases in a linear fashion - and also, it got further then the last time - so the file is not corrupt.
I implement it like the first version you mentioned here yes - only using and appending data from the variable c (as in your little snippet).

I also have a smaller sample at hand with only 20 participants and I ran the code twice on my laptop in the same python console. This added up to 70% memory usage from 55%. Afterwards I did globals().clear() - all variables were gone memory usage stayed pretty much the same up to 68% (also did a gc.collect - nothing changed).

What do you suggest?

[pariterre]

Oh! That definitely looks like a memory leak. There is nothing you can do for now. I'll have to have a look a the source code to find the bug and fix it.
The only workaround you have for the short-term is to splice analysis if you can (that is completely shuting Python after some amount of work and restarting from there). This may or may not be possible...

I'll keep you posted

[BDumphart]

Yeah, we are doing that right now - thank you for the information :)
If you need anything from my side, please let me know I am happy to help.

[pariterre]

That's allright, I could replicate the issue on my side, it is in the force platform module. If you do not need it you can add extract_forceplat_data=False when you read the c3d and it does not leak!

[BDumphart]

That´s great that you found it so fast! Unfortunately, we also need the force plate data - splitting works fine just now, so it is ok!
Should I close the Issue or leave it open?

[pariterre]

Fixed in #315
I pushed a new release, it usually takes about an hour to get the latest version up and running! Please update to 1.5.9 when it is uploaded to conda to test this new version

[BDumphart]

I tried it, works fine for me now - thank you for fixing and creating this nice tool! :)

Cheers,

Bernhard