Reading E2E, memory error?

[micceo]

Hi again!

You're never to old to learn something new, first time out with python here. So some cut and paste in my try to read an e2e, used the example.

I got an error.

X:\m1>my-e2e.py

Traceback (most recent call last):
  File "X:\m1\my-e2e.py", line 13, in <module>
    file.read_fundus_image()
  File "C:\prg\dev\Python\Python311\Lib\site-packages\oct_converter\readers\e2e.py", line 342, in read_fundus_image
    raw_volume = np.frombuffer(f.read(count), dtype=np.uint8)
                               ^^^^^^^^^^^^^

The my-eye.py from the example.

from oct_converter.readers import E2E

filepath = "x:\m1\EMRorder20230217.e2e"
file = E2E(filepath)
oct_volumes = (
    file.read_oct_volume()
)  # returns a list of all OCT volumes with additional metadata if available
for volume in oct_volumes:
    volume.peek(show_contours=True)  # plots a montage of the volume
    volume.save("{}_{}.avi".format(volume.volume_id, volume.laterality))

fundus_images = (
    file.read_fundus_image()
)  # returns a list of all fundus images with additional metadata if available
for image in fundus_images:
    image.save("{}+{}.png".format(image.image_id, image.laterality))

The file I'm trying to read comes from a HEYEX2 (Heidelberg), 351368 bytes, EMRorder20230217.e2e
Here is the first rows from that file.

43 4D 44 62 00 00 00 00 00 00 00 00 64 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 4D 44 62 4D 44 69 72 00 00 00 00 00
64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 02 00 00 58 00 00 00
C8 00 00 00 DC 9D 78 56 4D 44 62 44 69 72 00 00
00 00 00 00 64 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 02 00 00
58 00 00 00 00 00 00 00 F2 7A 56 34 8C 00 00 00
8C 58 00 00 83 00 00 00 00 00 00 00 11 00 00 00
FF FF FF FF FF FF FF FF FF FF FF FF FF FF 00 00
09 00 00 00 D2 9C 65 87 B8 00 00 00 4B 59 00 00

Anyone available to get me on the right track here?

Thanks in advance!

[Oli4]

Dear @micceo,

I do not know what the problem here is and I think it is hard to figure out without the full Error message and traceback. But you might want to try the E2E Reader in eyepy. Because I wrote the code over there it would be easier to help for me. You can find documentation here

[micceo]

Hi and thanks for the answer.
I can't even open the e2e with imagej which is documented to be able to open these files. Strongly suspect this file, that there is something up with format or the way I exported it. I'm going to try to get more samples from the HEYEX2 software (which is more than odd when it comes to exporting data).
The format doesn't correlate to formats specification what I can figure. https://bitbucket.org/uocte/uocte/wiki/Heidelberg%20File%20Format

I'll give eyepy a try too.

Thanks again. I'll report back my findings.

[Oli4]

Correct me if I am wrong but I always thought the ImageJ Plugin reads the .vol export but not .e2e

[micceo]

I've got my fair share of new things here :). vc compile ... and it all ends with this (I'm in over my head).

INFO: building 'skimage.feature.brief_cy' extension
      INFO: compiling C sources
      INFO: clang-cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_NONE -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages/pythran -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages\numpy\core\include -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages\numpy\core\include -Ibuild\src.win-amd64-3.11\numpy\distutils\include -IC:\prg\dev\Python\Python311\include -IC:\prg\dev\Python\Python311\include -IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\include -IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt /EHsc /Tpskimage/feature/brief_pythran.cpp /Fobuild\temp.win-amd64-3.11\Release\skimage/feature/brief_pythran.obj /std:c++14 /w /openmp /Zm1000


      [WinError 2] The system cannot find the file specified


      error: Command "clang-cl.exe /c /nologo /O2 /W3 /GL /DNDEBUG /MD -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_NONE -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages/pythran -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages\numpy\core\include -IC:\Users\micce\AppData\Local\Temp\pip-build-env-4nq_5drz\overlay\Lib\site-packages\numpy\core\include -Ibuild\src.win-amd64-3.11\numpy\distutils\include -IC:\prg\dev\Python\Python311\include -IC:\prg\dev\Python\Python311\include -IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.34.31933\include -IC:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Auxiliary\VS\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.22000.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\um -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\shared -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\winrt -IC:\Program Files (x86)\Windows Kits\10\\include\10.0.22000.0\\cppwinrt /EHsc /Tpskimage/feature/brief_pythran.cpp /Fobuild\temp.win-amd64-3.11\Release\skimage/feature/brief_pythran.obj /std:c++14 /w /openmp /Zm1000" failed with exit status 127
      INFO:
      ########### EXT COMPILER OPTIMIZATION ###########
      INFO: Platform      :
        Architecture: x64
        Compiler    : msvc

      CPU baseline  :
        Requested   : 'min'
        Enabled     : SSE SSE2 SSE3
        Flags       : none
        Extra checks: none

      CPU dispatch  :
        Requested   : 'max -xop -fma4'
        Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
        Generated   : none
      INFO: CCompilerOpt.cache_flush[857] : write cache to path -> C:\Users\micce\AppData\Local\Temp\pip-install-kt0afuot\scikit-image_bf17377afd3041c4be66bb2eef5d0be4\build\temp.win-amd64-3.11\Release\ccompiler_opt_cache_ext.py
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for scikit-image

[micceo]

"It's me again", the build process is not yet working with Python 3.11, which I have. Sorry for spamming the thread here before doing my g**gle-fu, "file not found vc Release\skimage/feature/brief_pythran.obj" leads to the answer.
https://forum.image.sc/t/pip-install-scikit-image-fails-windows11-python3-11/73311/4

[marksgraham]

HI @micceo

Would you be able to testing with an earlier version of Python - 3.7-3.9? I've not managed to test with 3.11 yet.

Also, it looks like your script manages to read the oct volume and fails when it gets round to reading the fundus. Does any output get saved?

Currently the output should get saved as a video file but these can be hard to view. You can get it to save as a series of .pngs by changing to

for volume in oct_volumes:
    volume.save("{}_{}.png".format(volume.volume_id, volume.laterality))

[micceo]

Passes without error. But no .png seems to be saved. They should go into the same folder I'm running from I presume.

I've tired all my e2e, also the one I know contains all the types: seqeuences, volumes and sections

from oct_converter.readers import E2E

filepath = "EMRorder20230217.e2e"
file = E2E(filepath)
oct_volumes = file.read_oct_volume()

for volume in oct_volumes:
    volume.save("{}_{}.png".format(volume.volume_id, volume.laterality))

[micceo]

For what it's worth. Tried some more cut and paste from the example. file.read_oct_volume works. file.read_fundus_image does give the out of memory error. Commented out that section below, then it runs with out error, but no avi or png are produced.

print('running...')
from oct_converter.readers import E2E

#filepath = "SinaEftek_Generic Order _20220830_000.e2e"
filepath = "DavidssonElsa_Generic Order _20211122_000.e2e"
file = E2E(filepath)
oct_volumes = (file.read_oct_volume())

for volume in oct_volumes:
    volume.peek(show_contours=True)  # plots a montage of the volume
    volume.save("{}_{}.avi".format(volume.volume_id, volume.laterality))

for volume in oct_volumes:
    volume.save("{}_{}.png".format(volume.volume_id, volume.laterality))

# fundus_images = file.read_fundus_image()
# 
# for image in fundus_images:
#     image.save("{}+{}.png".format(image.image_id, image.laterality))

print('done.')

If it's in your interest I can anonymize a file and share it with you.

[marksgraham]

It looks like it doesn't find any volumes. Would be great if you could share an anonymous file for me to take a look at

[micceo]

I've sent you a mail.

[Hazim-T]

@micceo @marksgraham I am currently working on a similar project where I have to read E2E files and process them in a machine learning library, could you let me know how your project went?