Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot kill picam #185

Open
bostiko opened this issue Oct 15, 2022 · 10 comments
Open

Cannot kill picam #185

bostiko opened this issue Oct 15, 2022 · 10 comments

Comments

@bostiko
Copy link

bostiko commented Oct 15, 2022

Hallo,
sorry to bother you again. I have problem with one of my setup - usb sound card, HQ camera, picam and ffmpeg streaming to youtube.
At the prevously working instalaltion I made an update with the fresh install of raspian with latest picam. When I run streaming, the picam shows warning: Record buffer is starving. Recorded file may not start from keyframe. Try reducing the value of --gopsize. after some minutes. When I try to restart the picam, the picam got stuck in state. I cannot end the picam even with pkill -9 picam.
Can you help me, please?

My "run" script:
url="rtmp://a.rtmp.youtube.com/live2/"
bitrate=192000
samplerate=48000
log_ffmpeg="/var/log/ffmpeg.log"
log_picam="/var/log/picam.log"

if [ "$1" != "" ]; then
url="$1"
fi

ffmpeg -i tcp://127.0.0.1:8181?listen -c:v copy -c:a aac
-ar $samplerate -ab $bitrate -f flv $url 2>> $log_ffmpeg &
/home/pi/picam/picam --alsadev hw:CODEC
--volume 2 -r $samplerate -a $bitrate --avclevel 4.2
--hdmi 1 --preview
--verbose --tcpout tcp://127.0.0.1:8181 >> $log_picam

And the picam.log file:
video_width=1920
video_height=1080
video_fps=30.000000
video_pts_step=3000
video_gop_size=30
video_hflip=0
video_vflip=0
video_bitrate=3000000
video_avc_profile=high
video_avc_level=4.2
video_qp_min=-1
video_qp_max=-1
video_qp_initial=-1
video_slice_dquant=-1
video_brightness=0
video_contrast=0
video_saturation=0
video_sharpness=0
alsa_dev=hw:CODEC
audio_channels=1
audio_sample_rate=48000
audio_bitrate=192000
audio_volume_multiply=2.000000
is_hlsout_enabled=0
is_hls_encryption_enabled=0
hls_keyframes_per_segment=1
hls_number_of_segments=3
hls_encryption_key_uri=stream.key
hls_encryption_key=0x75b0a81de17487c88a47507a7e1fdf73
hls_encryption_iv=0x000102030405060708090a0b0c0d0e0f
hls_output_dir=/run/shm/video
rtsp_enabled=0
rtsp_video_control_path=/tmp/node_rtsp_rtmp_videoControl
rtsp_audio_control_path=/tmp/node_rtsp_rtmp_audioControl
rtsp_video_data_path=/tmp/node_rtsp_rtmp_videoData
rtsp_audio_data_path=/tmp/node_rtsp_rtmp_audioData
tcp_enabled=1
tcp_output_dest=tcp://127.0.0.1:8181
auto_exposure_enabled=0
auto_exposure_threshold=5.000000
is_vfr_enabled=0
white_balance=auto
exposure_control=auto
awb_red_gain=0.000000
awb_blue_gain=0.000000
metering=average
manual_exposure_compensation=0
exposure_compensation=0.000000
manual_exposure_aperture=0
exposure_aperture=0
manual_exposure_shutter_speed=0
exposure_shutter_speed=0
manual_exposure_sensitivity=0
exposure_sensitivity=0
roi_left=0.000000
roi_top=0.000000
roi_width=1.000000
roi_height=1.000000
min_fps=-1.000000
max_fps=-1.000000
is_timestamp_enabled=0
timestamp_format=%a %b %d %l:%M:%S %p
timestamp_layout=15
timestamp_horizontal_margin=10
timestamp_vertical_margin=10
is_timestamp_abs_pos_enabled=0
timestamp_pos_x=0
timestamp_pos_y=0
timestamp_text_align=1
timestamp_font_name=FreeMono:style=Bold
timestamp_font_file=
timestamp_font_face_index=(null)
timestamp_font_points=14.000000
timestamp_font_dpi=96
timestamp_color=ffffff
timestamp_stroke_color=000000
timestamp_stroke_width=1
timestamp_letter_spacing=0
is_preview_enabled=1
is_previewrect_enabled=0
preview_x=0
preview_y=0
preview_width=0
preview_height=0
preview_hdmi=1
is_audio_preview_enabled=0
audio_preview_dev=plughw:0,0
record_buffer_keyframes=5
state_dir=state
hooks_dir=hooks
created directory: ./rec/tmp
opening ALSA device for capture: hw:CODEC
disable_audio_capturing: 0
final microphone channels: -1
configuring devices
make_egl_preview error: Couldn't open X display
preview: ctor: x_=0 y_=0 width_=0 height_=0
preview: count_crtcs=6 count_encoders=3 count_connectors=2 count_fbs=0 max_image_width=7680 max_image_height=7680
preview: CRTC connector 0: skipped because preview_hdmi=1 is specified
preview: CRTC connector 1: chosen
preview: inspecting connector: 1
Connector 41 (crtc 0): type 11, 0x0 Running without preview window
Opening camera...
Acquired camera -557509536
Configuring video...
Camera streams configured
Available controls:
AeEnable : [false..true]
ExposureTime : [60..127156999]
NoiseReductionMode : [0..4]
FrameDurationLimits : [19989..127157333]
ColourCorrectionMatrix : [-16.000000..16.000000]
Saturation : [0.000000..32.000000]
AeMeteringMode : [0..3]
Contrast : [0.000000..32.000000]
ExposureValue : [-8.000000..8.000000]
AwbMode : [0..7]
AeConstraintMode : [0..3]
ScalerCrop : [(0, 0)/128x128..(0, 0)/4056x2160]
AnalogueGain : [1.000000..22.260870]
ColourGains : [0.000000..32.000000]
Brightness : [-1.000000..1.000000]
AeExposureMode : [0..3]
AwbEnable : [false..true]
Sharpness : [0.000000..16.000000]
Buffers allocated and mapped
preview: startPreview begin
preview: startPreview end
Video setup complete
Opened VideoEncoder on /dev/video11 as fd 27
Got 6 output buffers
Got 12 capture buffers
Codec streaming started
Requests created
Using crop €ůN´U
cfr frame_time=33333
camera_set_metering_mode: average
camera_set_exposure_control: auto
camera_set_white_balance: auto
enable AWB: 0
camera_set_custom_awb_gains: red=0.0, blue=0.0
camera_set_brightness: 0.0
camera_set_contrast: 1.0
camera_set_saturation: 1.0
camera_set_sharpness: 0.0
Camera started!
audio setup
sample_rate: 48000
nb_samples: 1024
format: 1
audio_codec_ctx->ch_layout: 549198304304
av_frame->channel_layout: 549198304304
buffer_size=2048
allocated 2048 bytes for audio samples
audio_pts_step: 1920
configuring audio capture device
audio_sample_rate: 48000 (48000)
actual sample rate=48000 dir=0
setting microphone buffer size to 102400 (buffer_size=2048 alsa_buffer_multiply=50)
failed to set buffer size for microphone: buffer_size=2048 multiply=50
trying smaller buffer size for microphone: buffer_size=2048 multiply=25
microphone: buffer size: 51200 frames (channels=1 buffer_size=2048 multiply=25)
microphone: setting period size to 1024
actual_period_size=1024 dir=0
audio capture device:
Hardware PCM card 1 'USB Audio CODEC' device 0 subdevice 0
Its setup is:
stream : CAPTURE
access : MMAP_INTERLEAVED
format : S16_LE
subformat : STD
channels : 1
rate : 48000
exact rate : 48000 (48000/1)
msbits : 16
buffer_size : 51200
period_size : 1024
period_time : 21333
tstamp_mode : NONE
tstamp_type : MONOTONIC
period_step : 1
avail_min : 1024
period_event : 0
start_threshold : 1
stop_threshold : 51200
silence_threshold: 0
silence_size : 0
boundary : 7205759403792793600
appl_ptr : 0
hw_ptr : 0
audio device configured
[microphone started]capturing started
............................... 31.09 fps k=1 a-v=-1680 c-a=-21599 u=0 d=0 pts=113040
.............................. 29.95 fps k=2 a-v=-1440 c-a=-21668 u=0 d=0 pts=203040
.............................. 30.07 fps k=3 a-v=-1200 c-a=-22116 u=0 d=0 pts=293040

.............................. 29.35 fps k=863 a-v=-1620 c-a=-11216 u=0 d=6 pts=77657940
.............................. 30.71 fps k=864 a-v=-3300 c-a=-11608 u=0 d=6 pts=77747940
.............................. 30.02 fps k=865 a-v=-3060 c-a=-11904 u=0 d=6 pts=77837940
...................warning: Record buffer is starving. Recorded file may not start from keyframe. Try reducing the value of --gopsize.
warning: Record buffer is starving. Recorded file may not start from keyframe. Try reducing the value of --gopsize.

@iizukanao
Copy link
Owner

Does it happen 100% of the time (is it always reproducible)?
What camera and microphone hardware are you using?

@bostiko
Copy link
Author

bostiko commented Oct 16, 2022

Hallo, I was trying to make several videos one after another withe the length of 1 hour. It looks like after restarting the RPi, the first video is fine, but the second video stops after 10 minutes. And then the picam freeze and I need to restart the device.

I am using RPi's original HQ camera and external USB soundcard Miditech Audiolink light.

@iizukanao
Copy link
Owner

I don't have the HQ camera or Miditech Audiolink light, but if picam outputs the warning "Record buffer is starving", I think the camera stops working for some reason. Be aware that the camera is very sensitive to electrical interference, such as static noise, which can cause the camera to stop working. Please see if both libcamera-vid and arecord are working when the problem occurs.

@bostiko
Copy link
Author

bostiko commented Oct 17, 2022

Today I run a lot of test videos. I record and upload to youtube 7pcs of 1hour videos succesfully. During recording the 8th video at around 20 minutes into the recording the "Record buffer starving" starts to occure. I kill the picam and it stuck in state.

I run libcamera-vid -t 10000 -o test.h264 with response:

Preview window unavailable
[19:38:42.352817397] [3189]  INFO Camera camera_manager.cpp:293 libcamera v0.0.0+3866-0c55e522
[19:38:42.372484121] [3190] ERROR V4L2 v4l2_device.cpp:349 'imx477 10-001a': Unable to set controls: Device or resource busy
[19:38:42.392059161] [3190]  INFO RPI raspberrypi.cpp:1374 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media0
[19:38:42.392871779] [3189]  INFO Camera camera.cpp:1035 configuring streams: (0) 640x480-YUV420
[19:38:42.392963685] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video0[13:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393010851] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video1[14:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393048721] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video13[15:out]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393084702] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video14[16:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393121646] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video15[17:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393157219] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video16[18:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.393202533] [3190] ERROR V4L2 v4l2_device.cpp:349 'imx477 10-001a': Unable to set controls: Device or resource busy
[19:38:42.394303481] [3190] ERROR V4L2 v4l2_videodevice.cpp:1040 /dev/video0[13:cap]: Unable to set format: Device or resource busy
[19:38:42.394536829] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video0[13:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394576792] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video1[14:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394611051] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video13[15:out]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394644995] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video14[16:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394679013] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video15[17:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394713401] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video16[18:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394769715] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video16[18:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.394901806] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video15[17:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.395000471] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video14[16:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.395092748] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video13[15:out]: Unable to request 0 buffers: Device or resource busy
[19:38:42.395199913] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video1[14:cap]: Unable to request 0 buffers: Device or resource busy
[19:38:42.395291893] [3190] ERROR V4L2 v4l2_videodevice.cpp:1234 /dev/video0[13:cap]: Unable to request 0 buffers: Device or resource busy
ERROR: *** failed to configure streams ***

And also arecord -D hw:CODEC -d 5 test.wav
arecord: main:830: audio open error: Device or resource busy
Do you have any suggestions please?

Sidenote: As for the HW setup, camera is connected via 2m long camera cable, but it is in the old church with little interference. Prevouse version of SW run for almost two years without any problems. The problems starts when I update the SW with a fresh instalation of rasbian and a new picam.

@iizukanao
Copy link
Owner

I think there is a hardware problem in your environment. Replacing the 2m cable with a shorter one, as you commented before, would help. #150 (comment)

If you think it is a software problem, can you run libcamera-vid and arecord for days without problems?

@bostiko
Copy link
Author

bostiko commented Oct 18, 2022

This was a different church, I have arround 10 instalation in my country in different churches. In the one mentioned in #150, the RPi and camera was located on the pipe organs with the electric relays. This caused a lot of troubles from the day I install that RPi. After some time we moved the RPi far from the organs and it start to work.

In different church with exactly the same HW and SW as in #150 this setup work without issues. Until I update the SW as I mentioned earlier. In this church the whole setup is located at the "noise-free" place far from the source of interference. For two years there was no problems with the setup. Until I update the whole setup :(

*Footnote: To be sure I have the same setup in all of my 10 churches, I save SD card image and copy that image to all of my installations. I only change the credentials.

As for the libcamera-vid and arecord, I will try...

@iizukanao
Copy link
Owner

I see. Maybe the change in the camera stack (from OpenMAX to libcamera) is causing the problem. Please check if the device error only occurs when using picam.

If the problem persists, you may continue to use the old version of picam.

@bostiko
Copy link
Author

bostiko commented Nov 7, 2022

Hallo, I changed the camera cable for the shorter one (from 2m to 0.5m) and it now works for a week without any problems. Sorry for the troubles and thank you for your advices.

If you have a spare time, can you try to introduce some interferances to your picam ribbon cable and try to repeat my issue? It would be great if the picam cannot stuck in state after some interferences and can carry on the transmition.

And lastly, thank you for the excellent work with this project and your ongoing support :)
Have a nice day

@bostiko bostiko closed this as completed Nov 7, 2022
@iizukanao
Copy link
Owner

@bostiko Good to hear that the problem has been resolved.
I have encountered such a phenomenon (camera hangs) many times when I have physically affected the camera. When that happens, the problem is occurring at the lower firmware level and cannot be addressed at the picam level (same goes for libcamera-vid) and the Raspberry Pi needs to be rebooted.

@bostiko
Copy link
Author

bostiko commented May 3, 2023

Hallo,
thank you for your ongoing support. I must hovewer report the problem is back :(

After several months of service (3-5 streams with 1 hour each per week) device stops working reliably. Now approximatelly 95% of streams stops before sheduled end with a log message "warning: Record buffer is starving. Recorded file may not start from keyframe. Try reducing the value of --gopsize.". Event more, approximately 50% of streams stops within first 5 minutes :(

I tried several versions of picam from 2.0.5 to 2.0.12, but the situation is the same. The version 2.0.5 works on 4 other site without any problems. For about a week I am trying to resolve problem remotely, but I am not able to solve this.
After the warning, when I try to kill picam with "sudo pkill -9 picam", the picam freeze and stuck in state. After that I am not able to start picam without restarting rpi. Running picam in this state resulted with:
[0:20:03.945331280] [1218] INFO Camera camera_manager.cpp:299 libcamera v0.0.4+22-923f5d70 [0:20:03.964856379] [1220] ERROR V4L2 v4l2_device.cpp:352 'imx477 10-001a': Unable to set controls: Device or resource busy [0:20:03.979117201] [1220] INFO RPI raspberrypi.cpp:1476 Registered camera /base/soc/i2c0mux/i2c@1/imx477@1a to Unicam device /dev/media4 and ISP device /dev/media1 [0:20:03.979933538] [1218] INFO Camera camera.cpp:1028 configuring streams: (0) 1920x1080-YUV420 [0:20:03.980121146] [1220] ERROR V4L2 v4l2_device.cpp:352 'imx477 10-001a': Unable to set controls: Device or resource busy

I have two questions:

  1. Is the "warning: Record buffer is starving..." related only with video, or this problem can relate also with the sound inputs?
  2. Can you make some failsafe within the code for picam so it does not stuck in state?

Thank you :)

@bostiko bostiko reopened this May 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants