Complex dirty region causes overflow of xrdp comms buffer #318
Description
Version: xrdp v0.10.0 and xorgxrdp v0.10.1
Steps to reproduce:
- Use
xtermas WM for simplicity - Connect from mstsc.exe with 4K screen (3840x2160)
- Run
xeyes -geometry 3840x2160(exactly the same screen size with xrandr reports) - Xorg hangs up
I see the following xorgxrdp log. This hang-up doesn't happen v0.9. So it is related to recent GFX change.
I recalled #64 from the log with the following log line. It looks like a buffer overrun. I also recalled #5 and #62.
[ 2341.417] rdpClientConSendMsg: overrun error len, 35029 stream size 32868, client count 2
It looks like a buffer overrun, I tried to increase buffer size to 64k. I don't see any obvious buffer overrun with 64k buffer but Xorg still hangs up.
[ 2320.111] rdpClientConProcessMsgClientInput: invalidate x 0 y 0 cx 5759 cy 2163
[ 2329.463] rdpRRGetInfo:
[ 2329.464] rdpRRCrtcGetGamma: 0x55993aa226f0 0x55993aa3ebf0 0x55993aa3eff0 0x55993aa3edf0
[ 2329.464] rdpRRCrtcGetGamma: 0x55993aa226f0 0x55993aa3ebf0 0x55993aa3eff0 0x55993aa3edf0
[ 2329.465] rdpRRCrtcGetGamma: 0x55993aa3f200 0x55993aa2ec50 0x55993aa2f050 0x55993aa2ee50
[ 2329.465] rdpRRCrtcGetGamma: 0x55993aa3f200 0x55993aa2ec50 0x55993aa2f050 0x55993aa2ee50
[ 2341.417] rdpClientConSendMsg: overrun error len, 35029 stream size 32868, client count 2
[ 2398.054] rdpClientConProcessMsgClientInput: invalidate x 0 y 0 cx 5759 cy 2163
[ 2529.347] rdpClientConProcessMsgClientInput: invalidate x 0 y 0 cx 5759 cy 2163
[ 2534.024] rdpClientConRecv: g_sck_recv failed(returned 0)
[ 2534.024] rdpClientConRecvMsg: error
[ 2534.024] rdpClientConCheck: rdpClientConGotData failed
[ 2534.024] rdpClientConDisconnect:
[ 2534.024] rdpRemoveClientConFromDev: removing clientCon 0x55993aa3fcf0
[ 2534.024] (EE)
[ 2534.024] (EE) Backtrace:
[ 2534.025] (EE) 0: /usr/libexec/Xorg (xorg_backtrace+0x89) [0x559939442619]
[ 2534.025] (EE) 1: /usr/libexec/Xorg (0x55993926c000+0x1df149) [0x55993944b149]
[ 2534.025] (EE) 2: /lib64/libc.so.6 (0x7fac2bc00000+0x54db0) [0x7fac2bc54db0]
[ 2534.025] (EE) 3: /lib64/libc.so.6 (0x7fac2bc00000+0xa154c) [0x7fac2bca154c]
[ 2534.025] (EE) 4: /lib64/libc.so.6 (raise+0x16) [0x7fac2bc54d06]
[ 2534.025] (EE) 5: /lib64/libc.so.6 (abort+0xd3) [0x7fac2bc287f3]
[ 2534.025] (EE) 6: /lib64/libc.so.6 (0x7fac2bc00000+0x29130) [0x7fac2bc29130]
[ 2534.026] (EE) 7: /lib64/libc.so.6 (0x7fac2bc00000+0xab617) [0x7fac2bcab617]
[ 2534.026] (EE) 8: /lib64/libc.so.6 (0x7fac2bc00000+0xad30c) [0x7fac2bcad30c]
[ 2534.026] (EE) 9: /lib64/libc.so.6 (free+0x55) [0x7fac2bcaf955]
[ 2534.026] (EE) 10: /usr/lib64/xorg/modules/libxorgxrdp.so (0x7fac2b24c000+0xc12b) [0x7fac2b25812b]
[ 2534.026] (EE) 11: /usr/lib64/xorg/modules/libxorgxrdp.so (rdpClientConCheck+0x13c) [0x7fac2b25a14c]
[ 2534.026] (EE) 12: /usr/libexec/Xorg (WakeupHandler+0xba) [0x5599392d01ca]
[ 2534.026] (EE) 13: /usr/libexec/Xorg (WaitForSomething+0x1ef) [0x55993944484f]
[ 2534.026] (EE) 14: /usr/libexec/Xorg (0x55993926c000+0x4c3dd) [0x5599392b83dd]
[ 2534.027] (EE) 15: /lib64/libc.so.6 (0x7fac2bc00000+0x3feb0) [0x7fac2bc3feb0]
[ 2534.027] (EE) 16: /lib64/libc.so.6 (__libc_start_main+0x80) [0x7fac2bc3ff60]
[ 2534.027] (EE) 17: /usr/libexec/Xorg (_start+0x25) [0x5599392b8ea5]
[ 2534.027] (EE)
[ 2534.027] (EE)
Fatal server error:
[ 2534.027] (EE) Caught signal 6 (Aborted). Server aborting
[ 2534.027] (EE)
[ 2534.027] (EE)
Please consult the The X.Org Foundation support
at http://wiki.x.org
for help.
[ 2534.027] (EE) Please also check the log file at ".xorgxrdp.10.log" for additional information.
[ 2534.027] (EE)
[ 2534.027] rdpmouseControl: what 4
[ 2534.027] rdpkeybControl: what 4
[ 2534.027] rdpLeaveVT: