Inconsistent behavior with encoding profiles and pixel format support recognization

[wu4]

GPU is a Radeon RX 7600 XT, using Mesa 23.3.6.

When I try to run wl-screenrec with no arguments, I receive:

Using output DP-3
Opening libva device from DRM device /dev/dri/renderD128
[h264_vaapi @ 0x5588af718e00] No usable encoding profile found.
failed to open encoder in low_power mode (Function not implemented), trying non low_power mode. if you have an intel iGPU, set enable_guc=2 in the i915 module to use the fixed function encoder. pass --low-power=off to suppress this warning
[h264_vaapi @ 0x5588af718e00] No usable encoding profile found.
failed to create encoder: Function not implemented

From a lot of research and trial-and-error, however, I've managed to make wl-screenrec work with these options:

wl-screenrec --ffmpeg-encoder av1_vaapi --low-power=off -f out.mp4

When recording with these options, I receive no errors, and it successfully records to the file with hardware acceleration. When supplying the -v flag, I can tell it's using the vaapi pixfmt:

Using output DP-3
Opening libva device from DRM device /dev/dri/renderD128
+----------+
|    in    |default--[1920x1080 0:1 vaapi]--Parsed_crop_0:default
| (buffer) |
+----------+

                                                            +--------------+
Parsed_scale_vaapi_1:default--[1920x1080 0:1 vaapi]--default|     out      |
                                                            | (buffersink) |
                                                            +--------------+

                                          +---------------+
in:default--[1920x1080 0:1 vaapi]--default| Parsed_crop_0 |default--[1920x1080 0:1 vaapi]--Parsed_scale_vaapi_1:default
                                          |    (crop)     |
                                          +---------------+

                                                     +----------------------+
Parsed_crop_0:default--[1920x1080 0:1 vaapi]--default| Parsed_scale_vaapi_1 |default--[1920x1080 0:1 vaapi]--out:default
                                                     |    (scale_vaapi)     |
                                                     +----------------------+


Output #0, mp4, to 'out.mp4':
  Metadata:
    encoder         : Lavf60.16.100
  Stream #0:0: Video: av1 (Main) (av01 / 0x31307661), vaapi, 1920x1080, q=2-31, 40000 kb/s, 90k tbn
...

However, if I try to manually supply it with --encode-pixfmt vaapi, I receive this:

Using output DP-3
Opening libva device from DRM device /dev/dri/renderD128
[Parsed_scale_vaapi_1 @ 0x5635cb41aac0] Hardware does not support output format vaapi.
[Parsed_scale_vaapi_1 @ 0x5635cb41aac0] Failed to configure output pad on Parsed_scale_vaapi_1
thread 'main' panicked at /home/wuffie/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wl-screenrec-0.1.3/src/main.rs:1707:18:
called `Result::unwrap()` on an `Err` value: ffmpeg::Error(22: Invalid argument)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[wu4]

Here is the output of vainfo -a:

Trying display: wayland
libva info: VA-API version 1.20.0
libva info: Trying to open /usr/lib64/dri/radeonsi_drv_video.so
libva info: Found init function __vaDriverInit_1_20
libva info: va_openDriver() returns 0
vainfo: VA-API version: 1.20 (libva 2.20.1)
vainfo: Driver version: Mesa Gallium driver 23.3.6 for AMD Radeon Graphics (radeonsi, navi33, LLVM 17.0.6, DRM 3.57, 6.8.8-200.fc39.x86_64)
vainfo: Supported config attributes per profile/entrypoint pair
VAProfileJPEGBaseline/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_YUV444
                                             VA_RT_FORMAT_YUV400
    VAConfigAttribMaxPictureWidth          : 4096
    VAConfigAttribMaxPictureHeight         : 4096

VAProfileVP9Profile0/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
    VAConfigAttribMaxPictureWidth          : 8192
    VAConfigAttribMaxPictureHeight         : 4352

VAProfileVP9Profile2/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_YUV420_10BPP
    VAConfigAttribMaxPictureWidth          : 8192
    VAConfigAttribMaxPictureHeight         : 4352

VAProfileAV1Profile0/VAEntrypointVLD
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_YUV420_10BPP
    VAConfigAttribMaxPictureWidth          : 8192
    VAConfigAttribMaxPictureHeight         : 4352

VAProfileAV1Profile0/VAEntrypointEncSlice
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_YUV420_10BPP
    VAConfigAttribRateControl              : VA_RC_CBR
                                             VA_RC_VBR
                                             VA_RC_CQP
    VAConfigAttribEncPackedHeaders         : VA_ENC_PACKED_HEADER_SEQUENCE
                                             VA_ENC_PACKED_HEADER_PICTURE
    VAConfigAttribEncMaxRefFrames          : l0=1
                                             l1=0
    VAConfigAttribEncMaxSlices             : 128
    VAConfigAttribEncSliceStructure        : VA_ENC_SLICE_STRUCTURE_POWER_OF_TWO_ROWS
                                             VA_ENC_SLICE_STRUCTURE_EQUAL_ROWS
    VAConfigAttribMaxPictureWidth          : 8192
    VAConfigAttribMaxPictureHeight         : 4352
    VAConfigAttribEncQualityRange          : number of supported quality levels is 32
    VAConfigAttribEncTileSupport           : supported
    VAConfigAttribMaxFrameSize             : max_frame_size=1
                                             multiple_pass=0

VAProfileNone/VAEntrypointVideoProc
    VAConfigAttribRTFormat                 : VA_RT_FORMAT_YUV420
                                             VA_RT_FORMAT_YUV422
                                             VA_RT_FORMAT_YUV444
                                             VA_RT_FORMAT_YUV400
                                             VA_RT_FORMAT_YUV420_10
                                             VA_RT_FORMAT_RGB32
                                             VA_RT_FORMAT_RGBP
                                             VA_RT_FORMAT_YUV420_10BPP

[russelltg]

--encode-pixfmt is a bit tricky and maybe a bit misleading. This is the pixel format passed to the encoder, regardless of if it's using vaapi memory or not.

Also, fyi, you can probably use --codec av1 instead of --ffmpeg-encoder, which does the same thing but doesn't require knowledge of ffmpeg codec names.

It's unclear from the question what exactly the problem is, you say you got it working....

I just pushed a commit that adds an error message for this situation