[BUG] Running out of CmaMemory

[mgineer85]

Describe the bug
Hello @davidplowman, as per request, I create a separate issue about my issue running out of CmaMemory.

With reference to this issue comment in #1102 that is dedicated to the Pi 5. I am on a Pi 4 currently.

To Reproduce
Following script dies after about 10 loops:

import io

from picamera2 import Picamera2
from picamera2.encoders import MJPEGEncoder
from picamera2.outputs import FileOutput

buffer = io.BytesIO()
_picamera2: Picamera2 = Picamera2()


def _switch_mode(new_config):
    _picamera2.stop_encoder()

    try:
        # in try-catch because switch_mode can fail if picamera cannot allocate buffers.
        # if this happens, backend signals error and shall be restarted.
        _picamera2.switch_mode(new_config)
    except RuntimeError as exc:
        print(exc)
        print(f"error switching mode in picamera2 due to {exc}")

    else:
        _picamera2.start_encoder(MJPEGEncoder(), FileOutput(buffer))
        print("switchmode finished successfully")


# config HQ mode (used for picture capture and live preview on countdown)
_capture_config = _picamera2.create_still_configuration(
    main={"size": (4608, 2592)},
    lores={"size": (2304, 1296)},
    encode="lores",
    buffer_count=3,
    display="lores",
)

# config preview mode (used for permanent live view)
_preview_config = _picamera2.create_video_configuration(
    main={"size": (2304, 1296)},
    lores={"size": (2304, 1296)},
    encode="lores",
    buffer_count=3,
    display="lores",
)


# configure; camera needs to be stopped before
_picamera2.configure(_preview_config)

# start camera
_picamera2.start()


i = 0
while True:
    i += 1

    _switch_mode(_capture_config)

    # in capture mode, get frame and metadata

    data = io.BytesIO()
    _picamera2.capture_file(data, format="jpeg")
    _ = data.getbuffer()
    # comment to run out of cma memory slower, but script died for me after ~40 loops.
    _metadata = _picamera2.capture_metadata(wait=1.5)  # with this line the script works about 10 loops only.
    print(f"got capture frame #{i}")

    _switch_mode(_preview_config)

    # in preview mode, get frame indirect via filewriter and metadata here
    _metadata = _picamera2.capture_metadata(wait=1.5)

During script runtime, you see the Cma is getting lower and lower, finally the script fails and the CmaMemory is free again:

pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          102612 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          205392 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          102612 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          102612 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:           48664 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          166900 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:           48664 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:           54700 kB
pi@photobooth-chameleon:~ $ cat /proc/meminfo | grep Cma
CmaTotal:         524288 kB
CmaFree:          496948 kB

Expected behaviour
Script not dying ;)

Console Output, Screenshots

got capture frame #7
[0:46:57.446386148] [6911]  INFO Camera camera.cpp:1197 configuring streams: (0) 2304x1296-XBGR8888 (1) 2304x1296-YUV420 (2) 2304x1296-SBGGR10_CSI2P
[0:46:57.460423467] [6907]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA
switchmode finished successfully
[0:46:57.791064542] [6911]  INFO Camera camera.cpp:1197 configuring streams: (0) 4608x2592-BGR888 (1) 2304x1296-YUV420 (2) 4608x2592-SBGGR10_CSI2P
[0:46:57.805056991] [6907]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected unicam format: 4608x2592-pBAA
switchmode finished successfully
got capture frame #8
[0:46:58.984504323] [6911]  INFO Camera camera.cpp:1197 configuring streams: (0) 2304x1296-XBGR8888 (1) 2304x1296-YUV420 (2) 2304x1296-SBGGR10_CSI2P
[0:46:58.998391012] [6907]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 2304x1296-SBGGR10_1X10 - Selected unicam format: 2304x1296-pBAA
switchmode finished successfully
[0:46:59.261707673] [6911]  INFO Camera camera.cpp:1197 configuring streams: (0) 4608x2592-BGR888 (1) 2304x1296-YUV420 (2) 4608x2592-SBGGR10_CSI2P
[0:46:59.267449575] [6907]  INFO RPI vc4.cpp:622 Sensor: /base/soc/i2c0mux/i2c@1/imx708@1a - Selected sensor format: 4608x2592-SBGGR10_1X10 - Selected unicam format: 4608x2592-pBAA
Traceback (most recent call last):
  File "/home/pi/picam_test_cma_minimal.py", line 57, in <module>
    _switch_mode(_capture_config)
  File "/home/pi/picam_test_cma_minimal.py", line 17, in _switch_mode
    _picamera2.switch_mode(new_config)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1428, in switch_mode
    return self.dispatch_functions(functions, wait, signal_function, immediate=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1354, in dispatch_functions
    return job.get_result(timeout=timeout) if wait else job
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/picamera2/job.py", line 79, in get_result
    return self._future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/picamera2/job.py", line 48, in execute
    done, result = self._functions[0]()
                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1421, in switch_mode_
    self.configure_(camera_config)
  File "/usr/lib/python3/dist-packages/picamera2/picamera2.py", line 1109, in configure_
    self.allocator.allocate(libcamera_config, camera_config.get("use_case"))
  File "/usr/lib/python3/dist-packages/picamera2/allocators/dmaallocator.py", line 43, in allocate
    fd = self.dmaHeap.alloc(f"picamera2-{i}", stream_config.frame_size)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/picamera2/dma_heap.py", line 98, in alloc
    ret = fcntl.ioctl(self.__dmaHeapHandle.get(), DMA_HEAP_IOCTL_ALLOC, alloc)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [Errno 12] Cannot allocate memory
pi@photobooth-chameleon:~ $ 

Hardware :
Raspberry Pi 4 and IMX708 original camera module.

[mgineer85]

When changing the while loop to the following, it will work endlessly for me. Endless translates to 10 minutes currently ;)

i = 0
while True:
    i += 1

    _switch_mode(_capture_config)

    # in capture mode, get frame and metadata

    data = io.BytesIO()
    with _picamera2.captured_request(wait=1.5) as request:
        # Do something with the request.
        print(request)
        request.save("main", data, format="jpeg")
        print(request.get_metadata())
        # _picamera2.capture_file(data, format="jpeg")
        _ = data.getbuffer()
        # comment to run out of cma memory slower, but script died for me after ~40 loops.
        # _metadata = _picamera2.capture_metadata(wait=1.5)  # with this line the script works about 10 loops only.
        # The request is released automatically when we leave the scope of the "with".
    print(f"got capture frame #{i}")

    _switch_mode(_preview_config)

    # in preview mode, get frame indirect via filewriter and metadata here
    _metadata = _picamera2.capture_metadata(wait=1.5)

Did I do something illegal by requesting the metadata right after a capture separately?

[davidplowman]

Hi, thanks for reporting this.

Yes, I think there might be something leaking there, so we'll have to look into that. In the meantime, a workaround (which may actually be better regardless in this sort of scenario) might be to use the "persistent allocator". This will allocate the buffers for each use case ("preview" or "still") just once, and hang on to them.

To do this, add:

from picamera2.allocators import PersistentAllocator

and create the Picamera2 object with

_picamera2 = Picamera2(allocator=PersistentAllocator())

@will-v-pi Could I ask you to have a look at the regular DmaAllocator in this particular case? Thanks!

[mgineer85]

Can confirm that it works with the PersistentAllocator continuously.
The CmaFree never drops below ~250MB.

Will use this until this issue is solved. Thank you very much!