what if we don't rectify the stereo images?

[qingzhengzhuma]

I am wondering if i don't rectify the stereo images, will it influence the cost calculation? because the orientation of the real matched points is very likely not the same, there is always a rotation or other transformation in the surounding content of the real matched points. Additionally, in this configuration, the MAX_DISPARITY might be the width of the right image, which could be over 1000 pixels, because the right image might actually be in the left of the refference image, the global memory could not be enough, but we could use tiling of the refference image to deal with it. However, the shared memory usage is very diffcult to eliminate, can anyone give me some advice to handle it?

[qingzhengzhuma]

After reading more about the cencus transform feature, whithout stereo rectification, i think the precision will decrease, because this feature isn't roboust the rotation. In my situation, the motion between two cameras is very large so it's diffcult to rectify the stereo images or maybe i wrongly did it with opencv.

[initialneil]

When the baseline is large or the rotation is large, you might need to see plane sweeping. Use plane sweeping to setup the cost volume and solve with SGM.
Meshroom/AliceVision's depth map generation module seems to do it this way.