Skip to content

Out of bounds read (by 1 byte) in yuvnv12_rgb24_sseu #4841

@james-howard

Description

@james-howard

Consider the following program:

#include <stdio.h>
#include <stdint.h>
#include <SDL2/SDL.h>

// Run under guard malloc to ensure you hit the bug.
int main(int argc, const char * argv[]) {
    size_t w = 1920;
    size_t h = 1088;
    size_t stride = w;

    // calculate size of NV12 texture with above params:
    size_t ySize = stride * h; // plane0
    size_t uvSize = stride * (h / 2); // plane1
    size_t nv12Size = ySize + uvSize;

    printf("w: %zu h: %zu stride: %zu ySize: %zu uvSize: %zu nv12Size: %zu\n",
           w, h, stride, ySize, uvSize, nv12Size);

    // Allocate texture
    uint8_t *nv12 = calloc(1, nv12Size /* +1 here would prevent the crash */);

    printf("nv12: %p last valid address: %p first invalid address: %p\n",
           nv12,
           nv12 + (nv12Size - 1),
           nv12 + nv12Size);

    SDL_Surface *rgb = SDL_CreateRGBSurfaceWithFormat(0,
                                                      (int)w,
                                                      (int)h,
                                                      24,
                                                      SDL_PIXELFORMAT_RGB24);

    // SDL's YUV=>RGB SSE2 conversion code can read 1 byte too far leading to a possible crash.
    // For instance, if buffer is 0x13b674000, frameWidth = 1920, pitch=1920, frameHeight = 1088,
    // then bufferSize = 3133440 and the last valid byte in buffer is at address 0x000000013b970FFF,
    // but SDL tries to read 0x000000013b971000 which is one byte too far.
    SDL_ConvertPixels((int)w,
                      (int)h,
                      SDL_PIXELFORMAT_NV12,
                      nv12,
                      (int)stride,
                      rgb->format->format,
                      rgb->pixels,
                      rgb->pitch);

    printf("SDL_ConvertPixels didn't crash\n");
    return 0;
}

Running this program crashes here:

Thread 1: EXC_BAD_ACCESS (code=1, address=0x10e50f000)
#0	0x000000010071a737 in yuvnv12_rgb24_sseu at /Users/jahoward/git/ext/SDL/src/video/yuv2rgb/yuv_rgb_sse_func.h:432
#1	0x00000001006e9d7f in yuv_rgb_sse [inlined] at /Users/jahoward/git/ext/SDL/src/video/SDL_yuv.c:267
#2	0x00000001006e9b7a in SDL_ConvertPixels_YUV_to_RGB at /Users/jahoward/git/ext/SDL/src/video/SDL_yuv.c:416
#3	0x00000001007b3c61 in SDL_ConvertPixels_REAL at /Users/jahoward/git/ext/SDL/src/video/SDL_surface.c:1396
#4	0x0000000100003eb5 in main at /Users/jahoward/sandbox/SDLConvertPixelsBug/SDLConvertPixelsBug/main.c:44

Program output prior to the crash is:

w: 1920 h: 1088 stride: 1920 ySize: 2088960 uvSize: 1044480 nv12Size: 3133440
nv12: 0x10e212000 last valid address: 0x10e50efff first invalid address: 0x10e50f000

I don't have a patch because I don't totally understand how yuvnv12_rgb24_sseu works.

git history shows this YUV => RGB conversion code was contributed by Adrien Descamps.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions