Improved chroma key for point tracker #1665
Conversation
|
The original chroma key support was written by https://github.com/a1k0n. He should weight in if he wants. If he doesn't, I'm going to assume that he agreed, and reviewing this pull request. |
|
I have implemented a performance improvement. Are you okay if I push further changes? Also, it occurs to me that it could be helpful to introduce a new slider to specify how narrow the chroma key band is. (At least it would be useful to me!) This would work with or without the new overexposed logic, but would be especially beneficial in further tuning the case with overexposure. I presently have whitish backgrounds in my room which, depending on ambient light levels, still pass the threshold because they are bright enough. I have experimented and found that a tighter definition of "green" for lower light level areas fixes this. If others think such a new slider would be helpful I could bundle with with this PR or submit another. Thoughts? |
|
Feel free to push into the PR's branch. It would help with testing and debugging if you made some sample videos demonstrating color key tracking, that can be replayed using virtual webcam software. |
|
Okay, change pushed. The replay idea is brilliant. Will record some videos. |
|
Test video: chromatest.mp4The shiny highlights on the door in the background present a fun challenge. |
|
I can't see the video. |
|
Doesn't play in my web browser either. Plays fine on VLC, though. Must be some codec thing. Hang on... |
chromatest.mp4Right, different video format. |
|
FWIW, this software allows video input without having to pirate it or run as a trial version. Be sure to install the 32-bit version.
When adding the output method, don't specify a format but add it directly (or else output won't work). You might get a Vulkan-related error on startup which can be safely ignored. The joys of open-source. |
Thanks. I have added the slider for "chroma key strength" now and I reckon I have done all I set out to do with this PR (potential bugs, etc. aside of course!). |
| { | ||
| // get the intensity of the key color (i.e. +ve coefficients) | ||
| uchar blue = orig_ptr[j][0], green = orig_ptr[j][1], red = orig_ptr[j][2]; | ||
| float key = std::max(b, 0.0f) * blue + std::max(g, 0.0f) * green + std::max(r, 0.0f) * red; |
There was a problem hiding this comment.
You want std::fmaxf and std::fminf.
There was a problem hiding this comment.
std::fmax was a whole load slower than std::max in a debug build. I'll have a look and see how it compares in release.
| @@ -146,32 +167,38 @@ void PointExtractor::color_to_grayscale(const cv::Mat& frame, cv::Mat1b& output) | |||
| } | |||
| case pt_color_red_chromakey: | |||
| { | |||
| filter_single_channel(frame, 1, -0.5, -0.5, output); | |||
| float non_key_coeff = -0.5 * *s.chroma_key_strength; | |||
There was a problem hiding this comment.
You can hoist non_key_coeff out of the switch.
| break; | ||
| } | ||
| case pt_color_cyan_chromakey: | ||
| { | ||
| filter_single_channel(frame, -1, 0.5, 0.5, output); | ||
| float non_key_coeff = -1.0 * *s.chroma_key_strength; |
There was a problem hiding this comment.
and to save on typing, replace this with 2 * non_key_coeff.
There was a problem hiding this comment.
I think it's a bit less readable that way but it's your project. I'll do what you prefer.
|
In the UI on the model tab you inadvertedly switched the tab to the custom model. |
|
Edit the widget tab order and fix the new chroma key options appearing out of order. |
Thanks.
Would you believe I already spent ages on trying to get tab order right? Anyway I am fairly confident about it now. |
|
I have removed the duplication around I have checked |
This shouldn't be happening. There's nothing left for me to complain about. Let's merge this. |
Fab. Thanks. |
I have been trying to use chroma key to isolate the light from green LEDs on my headset clip. However the middle of the LEDs is so bright that it is overexposed on the camera and the color fades to white. i.e. I get white blobs with green halos. The chroma key logic sees this as dark blobs surrounded by light halos. This confuses the subsequent thresholding logic. I can turn down the camera exposure but this results in very small blobs and these decrease point tracker's accuracy.
This PR adds an option for chroma key to detect overexposed pixels as the requested color when using any of the chroma key colors. This works really well.
Using green as an example, the logic change is...
Existing logic: every pixel is transformed to a grayscale value with the formula
G - 0.5 * (R + B). (WhereR,GandBand the red, green and blue channel intensities.) Subsequently a threshold is applied. The greener a pixel, the more likely the transformed value is to exceed the threshold.New logic: The green channel intensity is used to calculate a weighted average between the green channel and the grayscale value calculated above. So the new formula is
G/255 * G + (1 - G/255) * (G - 0.5 * (R + B)). This simplifies toG - 0.5 * (R + B) + G/255 * 0.5 * (R + B), which is the existing chroma grayscale value plusG/255 * 0.5 * (R + B).With the new logic, the closer G is to 255, the less color saturation is expected from the pixel.
I have also added a checkbox to the options which is enabled when chroma key is selected. This allows either the new or the existing logic to be selected. It defaults to the existing logic so that a new version of opentrack would not change the behaviour without the user specifically requesting it.