Skip to content

[youtube] Support --download-sections with formats --live-from-start #6498

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 73 commits into
base: master
Choose a base branch
from

Conversation

elyse0
Copy link
Contributor

@elyse0 elyse0 commented Mar 10, 2023

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Like #5822, this PR adds support for downloading a specific section of a YT livestream format with --live-from-start, but with a different approach.

The Dash parser now extracts the start and end timestamps of each fragment and the fragment_generator code uses them to filter based on the selected range. A big advantage over #5822 is that these changes could allow support for --download-sections on any Dash livestream, if/when that downloader is implemented.

Section selector

The current approach for specifying a range is using a new # syntax and negative durations, where these durations are relative to "now".

--download-sections "#-2hours - -1hour"  // Start downloading from 2 hours ago and stop until 1 hour ago
--download-sections "#-1hour - 0" // Start downloading from 1 hour ago and stop until now
--download-sections "#-1hour - 1hour" // Start downloading from 1 hour ago and stop until 1 hour from now

Fixes #3451

yt-dlp "https://www.youtube.com/watch?v=w_DfTc7F5oQ" --verbose --live-from-start -f '140+134' --download-sections "#-15minutes - -5minutes" -o venice
[debug] Command-line config: ['https://www.youtube.com/watch?v=w_DfTc7F5oQ', '--verbose', '--live-from-start', '-f', '140+134', '--download-sections', '#-15minutes - -5minutes', '-o', 'venice']
[debug] Encodings: locale UTF-8, fs utf-8, pref UTF-8, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version stable@2023.03.04 [392389b7d] (source)
[debug] Lazy loading extractors is disabled
[debug] Git HEAD: 4e93198ae
[debug] Python 3.10.7 (CPython x86_64 64bit) - Linux-5.19.0-41-generic-x86_64-with-glibc2.36 (OpenSSL 3.0.5 5 Jul 2022, glibc 2.36)
[debug] exe versions: ffmpeg N-108931-g4dda3b1653-20221104 (setts), ffprobe N-108931-g4dda3b1653-20221104, phantomjs broken, rtmpdump 2.4
[debug] Optional libraries: Cryptodome-3.15.0, certifi-2022.09.24, mutagen-1.46.0, sqlite3-2.6.0, websockets-10.4
[debug] Proxy map: {}
[debug] Extractor Plugins: ElyseNebulaChannelIE, ElyseNebulaIE, ElyseNebulaSubscriptionsIE, OnlineRadioBoxIE
[debug] Plugin directories: ['/home/amish/.yt-dlp/plugins/elyse0/yt_dlp_plugins']
[debug] Loaded 1790 extractors
[youtube] Extracting URL: https://www.youtube.com/watch?v=w_DfTc7F5oQ
[youtube] w_DfTc7F5oQ: Downloading webpage
[youtube] w_DfTc7F5oQ: Downloading android player API JSON
[youtube] w_DfTc7F5oQ: Downloading MPD manifest
[youtube] w_DfTc7F5oQ: Downloading MPD manifest
[debug] Sort order given by extractor: quality, res, fps, hdr:12, source, vcodec:vp9.2, channels, acodec, lang, proto
[debug] Formats sorted by: hasvid, ie_pref, quality, res, fps, hdr:12(7), source, vcodec:vp9.2(10), channels, acodec, lang, proto, filesize, fs_approx, tbr, vbr, abr, asr, vext, aext, hasaud, id
[info] w_DfTc7F5oQ: Downloading 1 format(s): 140+134
[info] w_DfTc7F5oQ: Downloading 1 time ranges: 1683437142.1-1683437742.1
[debug] Invoking dashsegments downloader on "https://manifest.googlevideo.com/api/manifest/dash/expire/1683459643/ei/2zlXZPjBC_q_lu8PuYuUiAk/ip/xxx/id/w_DfTc7F5oQ.1/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/spc/qEK7B547rhoxXCE0iglM36M7YD7ipy8/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24007246/beids/24512781/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cspc%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRgIhAOCkqQCbdOaAIMN4LCSjBVodFi1xP6kQFxMWOTjxSlreAiEA9xqSRAoPw0nuZ1K0ZpcEsO-7RCt3VrKZ1DLrpXq_F3A%3D", "https://manifest.googlevideo.com/api/manifest/dash/expire/1683459643/ei/2zlXZPjBC_q_lu8PuYuUiAk/ip/xxx/id/w_DfTc7F5oQ.1/source/yt_live_broadcast/requiressl/yes/as/fmp4_audio_clear%2Cwebm_audio_clear%2Cwebm2_audio_clear%2Cfmp4_sd_hd_clear%2Cwebm2_sd_hd_clear/spc/qEK7B547rhoxXCE0iglM36M7YD7ipy8/vprv/1/pacing/0/itag_bl/376%2C377%2C384%2C385%2C612%2C613%2C617%2C619%2C623%2C628%2C655%2C656%2C660%2C662%2C666%2C671/keepalive/yes/fexp/24007246/beids/24512781/itag/0/playlist_type/DVR/sparams/expire%2Cei%2Cip%2Cid%2Csource%2Crequiressl%2Cas%2Cspc%2Cvprv%2Citag%2Cplaylist_type/sig/AOq0QJ8wRgIhAOCkqQCbdOaAIMN4LCSjBVodFi1xP6kQFxMWOTjxSlreAiEA9xqSRAoPw0nuZ1K0ZpcEsO-7RCt3VrKZ1DLrpXq_F3A%3D"
[dashsegments] Total fragments: unknown (live)
[download] Destination: venice.f140.mp4
[dashsegments] Total fragments: unknown (live)
[download] Destination: venice.f134.mp4
[debug] [youtube] Selected section: 1683437142.123764 -> 1683437742.123764
WARNING: [youtube] Starting download from the last 120 hours of the live stream since YouTube does not have data before that. If you think this is wrong, please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
[debug] [youtube] [w_DfTc7F5oQ] Generating fragments for format 140
[debug] [youtube] Selected section: 1683437142.123764 -> 1683437742.123764
[debug] [youtube] [w_DfTc7F5oQ] Generating fragments for format 134
[download] 100% of   12.17MiB in 00:00:57 at 216.43KiB/s
[download] 100% of   33.73MiB in 00:01:13 at 468.67KiB/s
[Merger] Merging formats into "venice.mp4"
[debug] ffmpeg command line: ffmpeg -y -loglevel repeat+info -i file:venice.f140.mp4 -i file:venice.f134.mp4 -c copy -map 0:a:0 -map 1:v:0 -movflags +faststart file:venice.temp.mp4
Deleting original file venice.f134.mp4 (pass -k to keep)
Deleting original file venice.f140.mp4 (pass -k to keep)
Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check all of the following options that apply:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

@pukkandan
Copy link
Member

pukkandan commented Mar 10, 2023

The approach looks good. There are a few part that I don't understand, but I'll do a full review when this is undrafted

PS: It would be great if someone can improve --download-sections, because I started using .* just to get it working but it fails in a lot of cases, like *(now-1hour)-(now-45minutes)

We had the plan to support negative values as timestamp from end of stream. So, this would be *-1h--45m. This is trivial to add support for by modifying download_range_func and the validation in __init__.py. If you can implement in this PR, go ahead. Otherwise I'll add it in before merging. I don't particularly like the -- that appears in this syntax, but not much we can do about it. Supporting now-x-now-y would create ambiguity and supporting brackets properly is non-trivial.

D0LLYNH0 and others added 18 commits March 11, 2023 19:51
and related cleanup

Thanks @AudricV for the finding
Authored by: makeworld-the-better-one
Closes yt-dlp#6395
Building fragment list for all formats take significant time for large videos
@azatoth
Copy link

azatoth commented Aug 17, 2024

@bashonly @pukkandan Any progress in getting this PR merged?

@cyrus01337
Copy link

I'd love to help trial this if needed as I'm curious what's holding this PR. Are there any specific instructions for installing this version? Happy to compile from fork repo if not.

@bashonly
Copy link
Member

Are there any specific instructions for installing this version? Are there any specific instructions for installing this version? Happy to compile from fork repo if not.

if you want to use an executable, you can just run yt-dlp --update-to bashonly/yt-dlp@live-from-sections. Otherwise install from PR author's fork branch

@cyrus01337
Copy link

Testing with a random news livestream, was able to get a 59m 50s livestream rather than the rounded 1h. Below is the command I ran:

./yt-dlp \
    --live-from-start \
    --verbose \
    "https://www.youtube.com/watch?v=8WX6YL9JnLw" \
    --cookies ./cookies.txt \
    --concurrent-fragments 8 \
    --download-sections "#-1h - 0"

Note that this isn't a dealbreaker for me, as a workaround to this is to convert the time to seconds then tack on an extra 10s, for 1h before the livestream started: --download-sections "#-3610s - 0"

Here's a simple and verbose log file:

@yt-dlp yt-dlp locked and limited conversation to collaborators Jan 31, 2025
@Ulmo
Copy link

Ulmo commented Aug 6, 2025

Any news on this?

@Ulmo
Copy link

Ulmo commented Aug 6, 2025

I'm getting the following response when I try to "gh pr checkout 6498" on the latest yt-dlp:

error: Your local changes to the following files would be overwritten by checkout:
	README.md
	supportedsites.md
	yt_dlp/extractor/lbry.py
Please commit your changes or stash them before you switch branches.
Aborting
failed to run git: exit status 1

I'm good with understanding the patch command, but I've never understood the Github system since it always seemed backwards to me (the nomenclature is incomprehensible to me). Can someone explain how to merge this to the latest version? There's a lot of code fixes and updates in the latest version that I want to preserve.

@aswad546
Copy link

Please merge this in soon, its a very useful change.

@joshrivers
Copy link

I'm not a maintainer here, but I've talked to them as well as done a little bit of work looking into it. This PR will not be merged in it's current form for good reasons, including that the authors are no longer involved, that this change makes brittle changes to the core of yt-dlp, and it only benefits YouTube streams but might break many other sites.

I am extremely grateful that the maintainers here regularly update this branch with fixes from master and produce an updated build which continues to work.

Using this branch is very easy thanks to that hard work. For linux:

wget https://github.com/yt-dlp/yt-dlp/releases/latest/download/yt-dlp -O /usr/local/bin/yt-dlp
chmod a+rx /usr/local/bin/yt-dlp
/usr/local/bin/yt-dlp --update-to 'bashonly/yt-dlp@live-from-sections'

Personally, I'd like to find some time to do the work to get this into mergable shape, but time is short for everyone. If anyone wants to contribute, the maintainers here are friendly and welcoming. Any help for this feature will require significant effort and real technical skills, though. If it was easy, it would already have been done.

(P.S. @Ulmo this thread isn't the right place to ask Git questions, but there are friendly tutorials available if you want to learn)

Mainly, I'd ask anybody who comes by here to not add pressure or demands because I'd prefer the maintainers to keep updating this branch and build. If it becomes a hassle it is more likely that they'll just close this PR and the available build will not work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
site:youtube site-enhancement Feature request for some website
Projects
Status: youtube livestream
Development

Successfully merging this pull request may close these issues.

[Youtube] --live-from-start support for sequence / range download