RTMP downloader does not support unicode filenames #23765
Description
Checklist
- I'm reporting a broken site support issue
- I've verified that I'm running youtube-dl version 2020.01.15
- I've checked that all provided URLs are alive and playable in a browser
- I've checked that all URLs and arguments with special characters are properly quoted or escaped
- I've searched the bugtracker for similar bug reports including closed ones
- I've read bugs section in FAQ
Verbose log
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['--write-info-json', '--write-sub', '--all-subs', '--username', 'PRIVATE', '--password', 'PRIVATE', 'https://live.nicovideo.jp/watch/lv309042194', '--verbose', '-i']
[debug] Encodings: locale cp1252, fs utf-8, out utf-8, pref cp1252
[debug] youtube-dl version 2020.01.15
[debug] Git HEAD: 702405f05
[debug] Python version 3.6.6 (CPython) - Windows-10-10.0.17134-SP0
[debug] exe versions: ffmpeg git-2019-12-23-5b42d33, ffprobe git-2019-12-23-5b42d33, rtmpdump 2.3
[debug] Proxy map: {}
[NiconicoLive] Logging in
[NiconicoLive] lv309042194: Downloading webpage
[debug] Default format spec: bestvideo+bestaudio/best
[info] Writing video description metadata as JSON to: ロックマン30周年記念生放送-lv309042194.info.json
[debug] Invoking downloader on 'rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04/mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v'
[download] Destination: ロックマン30周年記念生放送-lv309042194.flv
[debug] rtmpdump command line: rtmpdump --verbose -r "rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04/mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v" -o ロックマン30周年記念生放送-lv309042194.flv.part --conn "S:90858899:lv309042194:0:1579272301:305f6ffbe8ac5033" --resume --skip 1
[rtmpdump] RTMPDump v2.3
[rtmpdump] (c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
[rtmpdump] DEBUG: Parsing...
[rtmpdump] DEBUG: Parsed protocol: 0
[rtmpdump] DEBUG: Parsed host : nlaoe113.live.nicovideo.jp
[rtmpdump] DEBUG: Parsed app : fileorigin/04
[rtmpdump] DEBUG: Number of skipped key frames for resume: 1
[rtmpdump] DEBUG: Protocol : RTMP
[rtmpdump] DEBUG: Hostname : nlaoe113.live.nicovideo.jp
[rtmpdump] DEBUG: Port : 1935
[rtmpdump] DEBUG: Playpath : mp4:/content/20171213/lv309042194_184426134000_1_4d071f.f4v
[rtmpdump] DEBUG: tcUrl : rtmp://nlaoe113.live.nicovideo.jp:1935/fileorigin/04
[rtmpdump] DEBUG: app : fileorigin/04
[rtmpdump] DEBUG: live : no
[rtmpdump] DEBUG: timeout : 30 sec
[rtmpdump] Failed to open file! ?????30???????-lv309042194.flv.part
ERROR: unable to download video data: [WinError 2] The system cannot find the file specified: 'ロックマン30周年記念生放送-lv309042194.flv.part'
Traceback (most recent call last):
File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\YoutubeDL.py", line 1948, in process_info
success = dl(filename, info_dict)
File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\YoutubeDL.py", line 1887, in dl
return fd.download(name, info)
File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\downloader\common.py", line 395, in download
return self.real_download(filename, info_dict)
File "D:\Sourcecode\private\youtube-dl-branch\youtube_dl\downloader\rtmp.py", line 183, in real_download
prevsize = os.path.getsize(encodeFilename(tmpfilename))
File "C:\Program Files (x86)\Microsoft Visual Studio\Shared\Python36_64\lib\genericpath.py", line 50, in getsize
return os.stat(filename).st_size
FileNotFoundError: [WinError 2] The system cannot find the file specified: 'ロックマン30周年記念生放送-lv309042194.flv.part'
Description
I've been working on implementing #17415 which uses RTMP, but I've discovered that it doesn't support unicode filenames. This can be tested with any extractor that uses the RTMP downloader, and specifying the -f format flag with a filename that uses unicode. Inversely, my particular issue with the above log is fixed if I specify an ASCII filename.
After some investigation, this seems to be an issue with rtmpdump itself as its code uses char instead of wchar for output filenames, meaning that it's incapable of actually using unicode.
There are multiple fixes that could be used for this:
- Using a different temporary filename
- Manually piping stdout from the process to a file handle created by python
- Using an altered version of rtmpdump/librtmp that has support for unicode filenames
- Using a different application entirely, such as ffmpeg. I am aware that the zeranoe builds do not ship with librtmp by default, so this may be limited