Struggling to correctly set up fisheye reduction using ffmpeg lenscorrection

[rohrbrecher]

Disclaimer: I initially posted this here in the frigate github: blakeblackshear/frigate#20635 (comment)

Hi,
I'm running Frigate using Docker within a Proxmox LXC on a NUC with an Intel Ultra 125H CPU (incl. Arc iGPU). I have tried to configure everything as per the manual and if I simply use the camera stream everything works pretty nicely. I have both motion detection as well as HwAcceleration on and my CPU is idling at 1-2%.

Now my Reolink Doorbell has a pretty annoying fisheye and I would like to use the ffmpeg lenscorrection filter to get rid of it. I searched and tried for several hours but I couldn't get it to work.

Whenever I configure the filter my CPU usage goes up to >10%, indicating that it's running entirely on CPU, ignoring the GPU.

This is how my current configuration looks like:

ffmpeg:rtsp://admin:password@192.168.178.105:554#video=h264#raw=-vf "lenscorrection=k1=-0.45:k2=0.2"
ffmpeg:rtsp://admin:password@192.168.178.105:554#video=h264#raw=-vf "lenscorrection=k1=-0.45:k2=0.2"#hardware

Adding the camera stream called "filtered" this way leads to the increase in CPU strain. I've tried adding #hardware to the same line but then the filter stopped working entirely. It seems like having both the "-vf" as well as "#hardware" in the same ffmpeg stream isn't working.

So can anyone tell me how I can achieve this?

[AlexxIT]

It's very dangerous to use any raw arguments with built-in transcoding params. They could be incompatible. As you see.

You need to build all ffmpeg params manually with your changes. And use exec source for that.

It's hard, but possible, to use -vf with hardware transcoding. Problem that vf is software changes. FFmpeg should do hardware decoding, software changes and hardware encoding. This will cost a lot of CPU.