-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Closed
Milestone
Description
Consider the following program:
#include <stdio.h>
#include <stdint.h>
#include <SDL2/SDL.h>
// Run under guard malloc to ensure you hit the bug.
int main(int argc, const char * argv[]) {
size_t w = 1920;
size_t h = 1088;
size_t stride = w;
// calculate size of NV12 texture with above params:
size_t ySize = stride * h; // plane0
size_t uvSize = stride * (h / 2); // plane1
size_t nv12Size = ySize + uvSize;
printf("w: %zu h: %zu stride: %zu ySize: %zu uvSize: %zu nv12Size: %zu\n",
w, h, stride, ySize, uvSize, nv12Size);
// Allocate texture
uint8_t *nv12 = calloc(1, nv12Size /* +1 here would prevent the crash */);
printf("nv12: %p last valid address: %p first invalid address: %p\n",
nv12,
nv12 + (nv12Size - 1),
nv12 + nv12Size);
SDL_Surface *rgb = SDL_CreateRGBSurfaceWithFormat(0,
(int)w,
(int)h,
24,
SDL_PIXELFORMAT_RGB24);
// SDL's YUV=>RGB SSE2 conversion code can read 1 byte too far leading to a possible crash.
// For instance, if buffer is 0x13b674000, frameWidth = 1920, pitch=1920, frameHeight = 1088,
// then bufferSize = 3133440 and the last valid byte in buffer is at address 0x000000013b970FFF,
// but SDL tries to read 0x000000013b971000 which is one byte too far.
SDL_ConvertPixels((int)w,
(int)h,
SDL_PIXELFORMAT_NV12,
nv12,
(int)stride,
rgb->format->format,
rgb->pixels,
rgb->pitch);
printf("SDL_ConvertPixels didn't crash\n");
return 0;
}
Running this program crashes here:
Thread 1: EXC_BAD_ACCESS (code=1, address=0x10e50f000)
#0 0x000000010071a737 in yuvnv12_rgb24_sseu at /Users/jahoward/git/ext/SDL/src/video/yuv2rgb/yuv_rgb_sse_func.h:432
#1 0x00000001006e9d7f in yuv_rgb_sse [inlined] at /Users/jahoward/git/ext/SDL/src/video/SDL_yuv.c:267
#2 0x00000001006e9b7a in SDL_ConvertPixels_YUV_to_RGB at /Users/jahoward/git/ext/SDL/src/video/SDL_yuv.c:416
#3 0x00000001007b3c61 in SDL_ConvertPixels_REAL at /Users/jahoward/git/ext/SDL/src/video/SDL_surface.c:1396
#4 0x0000000100003eb5 in main at /Users/jahoward/sandbox/SDLConvertPixelsBug/SDLConvertPixelsBug/main.c:44
Program output prior to the crash is:
w: 1920 h: 1088 stride: 1920 ySize: 2088960 uvSize: 1044480 nv12Size: 3133440
nv12: 0x10e212000 last valid address: 0x10e50efff first invalid address: 0x10e50f000
I don't have a patch because I don't totally understand how yuvnv12_rgb24_sseu
works.
git history shows this YUV => RGB conversion code was contributed by Adrien Descamps.