Skip to content

Error with StaxRip 2.48.2 using AV1 and RTX 5080 "Failed to map frame: CUDA_ERROR_MAP_FAILED" #710

@jonrodrigos

Description

@jonrodrigos

I'd like some help.
Every time I'm converting videos, whether AV1 or H.265, I keep encountering a problem. Some videos convert normally until the end, but others show an error after about 4 minutes. If I try converting the same video again, sometimes it goes all the way through, or it fails again when it’s close to finishing — like 5 minutes before the end.
So the issue seems random, and I don't know the cause.
I'm using StaxRip version 2.48.2. I've already updated Rigaya's NVENC from 8.0.8 to 8.0.9 — same error. I also tried older versions of the Nvidia driver and I'm currently on the latest one.

Error Video encoding (v2.48.2)

NVEncC 8.09

C:\StaxRip-v2.48.1-x64\Apps\Encoders\NVEncC\NVEncC64.exe --avhw --qvbr 33 --codec av1 --preset p7 --output-depth 10 --level 5.1 --qp-init 1 --qp-max 132 --vbv-bufsize 60000 --max-bitrate 30000 --aq --aq-strength 8 --aq-temporal --bframes 4 --ref 7 --lookahead 32 --lookahead-level 3 --vpp-ngx-truehdr middlegray=47 --vpp-resize algo=ngx-vsr,vsr-quality=3 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,01)" --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --split-enc forced_2 --max-cll "1000,400" --chromaloc 2 --cuda-schedule spin --output-buf 128 --mv-precision q-pel --crop 0,136,0,132 --output-res 3840x1624 -i F:\Downloads\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360.mkv -o C:\Stax_Temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_new_out.mkv


C:\Stax_Temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_new_out.mkv

Max bitrate is lowered 30000 -> 26666 due to level 5.1 restriction.
NVEncC (x64) 8.09 (r3369) by rigaya, Jul 16 2025 12:08:21 (VC 1944/Win)
OS Version Windows 11 x64 (26100) [UTF-8]
CPU 12th Gen Intel Core i7-12700K [4.81GHz] (8P+4E,12C/20T)
GPU #0: NVIDIA GeForce RTX 5080 (10752 cores, 2625 MHz)[PCIe5x16][577.00]
NVENC / CUDA NVENC API 13.0, CUDA 12.9, schedule mode: spin
Input Buffers CUDA, 45 frames
Input Info avcuvid: H.264/AVC, 1920x1080, 24000/1001 fps
Vpp Filters crop: 0,136,0,132/cspconv(nv12 -> yv12(16bit))
resize: ngx-vsr 1920x812 -> 3840x1624
ngx-vsr: colorspace: cspconv(yv12(16bit) -> yuv444(16bit))
matrix:bt709->GBR
cspconv(rgb(fp32) -> rgb32)
nvsdk-ngx vsr: quality: 3
cspconv(rgb32 -> rgb(fp32))
colorspace: matrix:GBR->bt709
ngx-truehdr: colorspace: matrix:bt709->GBR
cspconv(rgb(fp32) -> rgb32)
ngx-truehdr: contrast: 125, saturation: 75, middleGray: 47, maxLuminance: 1000
cspconv(rgba(fp16) -> rgb(fp32))
colorspace: prim:bt709->bt2020
transfer:linear->smpte2084
matrix:GBR->bt2020nc
cspconv(yuv444(16bit) -> p010)
Output Info AV1 main 10bit @ Level 5.1
3840x1624p 0:0 23.976fps (24000/1001fps)
avwriter: av1 => matroska
Encoder Preset quality
Rate Control VBR
Multipass none
Bitrate 0 kbps (Max: 26666 kbps)
Target Quality 33.00
Initial QP I:1 P:1 B:1
QP range I:0-132 P:0-132 B:0-132
QP Offset cb:0 cr:0
VBV buf size 60000 kbit
Split Enc Mode forced_2
Lookahead on, 32 frames, Adaptive I, B Insert
GOP length 240 frames
B frames 4 frames [ref mode: middle]
Ref frames 7 frames, MultiRef L0:auto L1:auto
AQ on (spatial, temporal, strength 8)
Part size max auto / min auto
Tile num columns auto / rows auto
TemporalLayers max 0
Refs forward auto, backward auto
VUI matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
WP(0.312700 0.329000) L(1000.000000 0.000100)
MaxCLL/MaxFALL 1000/400
Others mv:Q-pel
CUDA: Failed to map frame: CUDA_ERROR_MAP_FAILED: This indicates that a map or register operation has failed..
Break in task CUDA: CUDA_ERROR_MAP_FAILED: This indicates that a map or register operation has failed..
NVDEC: in 0, out 154654, outQeueue size: 0.
CHECKPTS: in 154654, out 154654, outQeueue size: 0.
CUDA: in 0, out 154652, outQeueue size: 1, frame release: 0.
NVENC: in 154652, out 154618, outQeueue size: 1.
encoded 154618 frames, 49.90 fps, 7090.81 kbps, 5451.16 MB
encode time 0:51:38, CPU: 1.8, GPU: 100.0, VE: 57.9, VD: 1.6, GPUClock: 2852MHz, VEClock: 2270MHz
frame type IDR 645
frame type I 645, avgQP 77.46, total size 49.26 MB
frame type P 153973, avgQP 72.06, total size 5401.90 MB

Video encoding returned exit code: 1 (0x1)


StaxRip.ErrorAbortException: NVEncC 8.09

C:\StaxRip-v2.48.1-x64\Apps\Encoders\NVEncC\NVEncC64.exe --avhw --qvbr 33 --codec av1 --preset p7 --output-depth 10 --level 5.1 --qp-init 1 --qp-max 132 --vbv-bufsize 60000 --max-bitrate 30000 --aq --aq-strength 8 --aq-temporal --bframes 4 --ref 7 --lookahead 32 --lookahead-level 3 --vpp-ngx-truehdr middlegray=47 --vpp-resize algo=ngx-vsr,vsr-quality=3 --master-display "G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,01)" --colormatrix bt2020nc --colorprim bt2020 --transfer smpte2084 --split-enc forced_2 --max-cll "1000,400" --chromaloc 2 --cuda-schedule spin --output-buf 128 --mv-precision q-pel --crop 0,136,0,132 --output-res 3840x1624 -i F:\Downloads\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360.mkv -o C:\Stax_Temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_new_out.mkv

---------------------------------------------------------------------------------------------------------------------------------------------------------------------
C:\Stax_Temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_temp\Alien.3.Special.Edition.Bluray.1080p.Remux.DTS-HD.5.1.Dual.Hiro360_new_out.mkv
---------------------------------------------------------------------------------------------------------------------------------------------------------------------
Max bitrate is lowered 30000 -> 26666 due to level 5.1 restriction.
NVEncC (x64) 8.09 (r3369) by rigaya, Jul 16 2025 12:08:21 (VC 1944/Win)
OS Version     Windows 11 x64 (26100) [UTF-8]
CPU            12th Gen Intel Core i7-12700K [4.81GHz] (8P+4E,12C/20T)
GPU            #0: NVIDIA GeForce RTX 5080 (10752 cores, 2625 MHz)[PCIe5x16][577.00]
NVENC / CUDA   NVENC API 13.0, CUDA 12.9, schedule mode: spin
Input Buffers  CUDA, 45 frames
Input Info     avcuvid: H.264/AVC, 1920x1080, 24000/1001 fps
Vpp Filters    crop: 0,136,0,132/cspconv(nv12 -> yv12(16bit))
               resize: ngx-vsr 1920x812 -> 3840x1624
                           ngx-vsr: colorspace: cspconv(yv12(16bit) -> yuv444(16bit))
                                       matrix:bt709->GBR
                                    cspconv(rgb(fp32) -> rgb32)
                                    nvsdk-ngx vsr: quality: 3
                                    cspconv(rgb32 -> rgb(fp32))
                                    colorspace: matrix:GBR->bt709
               ngx-truehdr: colorspace: matrix:bt709->GBR
                            cspconv(rgb(fp32) -> rgb32)
                            ngx-truehdr: contrast: 125, saturation: 75, middleGray: 47, maxLuminance: 1000
                            cspconv(rgba(fp16) -> rgb(fp32))
                            colorspace: prim:bt709->bt2020
                           transfer:linear->smpte2084
                           matrix:GBR->bt2020nc
               cspconv(yuv444(16bit) -> p010)
Output Info    AV1 main 10bit @ Level 5.1
               3840x1624p 0:0 23.976fps (24000/1001fps)
               avwriter: av1 => matroska
Encoder Preset quality
Rate Control   VBR
Multipass      none
Bitrate        0 kbps (Max: 26666 kbps)
Target Quality 33.00
Initial QP     I:1  P:1  B:1
QP range       I:0-132  P:0-132  B:0-132
QP Offset      cb:0  cr:0
VBV buf size   60000 kbit
Split Enc Mode forced_2
Lookahead      on, 32 frames, Adaptive I, B Insert
GOP length     240 frames
B frames       4 frames [ref mode: middle]
Ref frames     7 frames, MultiRef L0:auto L1:auto
AQ             on (spatial, temporal, strength 8)
Part size      max auto / min auto
Tile num       columns auto / rows auto
TemporalLayers max 0
Refs           forward auto, backward auto
VUI            matrix:bt2020nc,colorprim:bt2020,transfer:smpte2084,chromaloc:topleft
MasteringDisp  G(0.265000 0.690000) B(0.150000 0.060000) R(0.680000 0.320000)
               WP(0.312700 0.329000) L(1000.000000 0.000100)
MaxCLL/MaxFALL 1000/400
Others         mv:Q-pel 
CUDA: Failed to map frame: CUDA_ERROR_MAP_FAILED: This indicates that a map or register operation has failed..
Break in task CUDA: CUDA_ERROR_MAP_FAILED: This indicates that a map or register operation has failed..
NVDEC: in 0, out 154654, outQeueue size: 0.
CHECKPTS: in 154654, out 154654, outQeueue size: 0.
CUDA: in 0, out 154652, outQeueue size: 1, frame release: 0.
NVENC: in 154652, out 154618, outQeueue size: 1.
encoded 154618 frames, 49.90 fps, 7090.81 kbps, 5451.16 MB
encode time 0:51:38, CPU: 1.8, GPU: 100.0, VE: 57.9, VD: 1.6, GPUClock: 2852MHz, VEClock: 2270MHz
frame type IDR    645
frame type I      645,  avgQP  77.46,  total size    49.26 MB
frame type P   153973,  avgQP  72.06,  total size  5401.90 MB


Video encoding returned exit code: 1 (0x1)

em StaxRip.Proc.Start() na D:\Code\Github\staxrip\staxrip\Source\General\Proc.vb:linha 367
em StaxRip.NVEnc.Encode() na D:\Code\Github\staxrip\staxrip\Source\Encoding\NVEnc.vb:linha 165
em StaxRip.GlobalClass._Closure$__._Lambda$__156-7() na D:\Code\Github\staxrip\staxrip\Source\General\GlobalClass.vb:linha 439
em System.Threading.Tasks.Parallel.<>c__DisplayClass4_0.b__0()
--- Fim do rastreamento de pilha do local anterior onde a exceção foi gerada ---
em System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
em StaxRip.GlobalClass.ProcessJob(String jobPath) na D:\Code\Github\staxrip\staxrip\Source\General\GlobalClass.vb:linha 4

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions