I have looked at the KITTI Stereo 2012 dataset, which features parts of the KITTI VO dataset. I conclude that they are part of the same set. This means that I can just use the pretrained 2012 model.
Now, when I evaluate it on sequence 04 of the VO dataset, my disparity values seem to be off compared to ELAS and other stereo algorithms:

Also I tried out the other pretrained models. I compared the velodyne projected to the color image. Also, I altered/improved the velodyne to camera calibration, which is somewhat off for the VO dataset.

COMMAND USED: python submission.py --KITTI 2012 --datapath C:\KittiOdometry\data_odometry_color\dataset\sequences\04\ --loadmodel pretrained_model_KITTI2012.tar
Note that I altered the KITTI_submission_loader2012.py to work with KITTI VO instead.

Here's my resulting image of the first pair:

Here are the two input pictures. It would be really nice if somebody would post their disparity results of these two images with the pretrained KITTI 2012 model and hourglass. Then we could have a comparison and identify the source of the error faster.