Skip to content

[youtube] Options for begin and end of recording live streams #5822

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from

Conversation

azatoth
Copy link

@azatoth azatoth commented Dec 18, 2022

IMPORTANT: PRs without the template will be CLOSED

Description of your pull request and other information

Options for begin and end of recording live streams

Extend --live-from-start with two options -live-from-start-beginand -live-from-start-end which will limit the recording to between these values.

--live-from-start-end is optional and if omitted will continously record as normal.

While as generic as the existing --live-from-start option, it's limited to youtube for now.

Example:

yt-dlp https://www.youtube.com/watch?v=nbBeoReu12E --live-from-start --live-from-start-begin 2022-12-18T10:35:00+01:00 --live-from-start-end 2022-12-18T10:35:16+01:00

Fixes #3451

Template

Before submitting a pull request make sure you have:

In order to be accepted and merged into yt-dlp each piece of code must be in public domain or released under Unlicense. Check one of the following options:

  • I am the original author of this code and I am willing to release it under Unlicense
  • I am not the original author of this code but it is in public domain or released under Unlicense (provide reliable evidence)

What is the purpose of your pull request?

@pukkandan
Copy link
Member

Don't add new switches. Instead, make the feature obey --download-sections *start-end. The downloader can access the timestamps at section_start/end keys in the info_dict

@pukkandan pukkandan added enhancement New feature or request site-enhancement Feature request for some website pending-fixes PR has had changes requested labels Dec 18, 2022
@azatoth
Copy link
Author

azatoth commented Dec 18, 2022

Don't add new switches. Instead, make the feature obey --download-sections *start-end. The downloader can access the timestamps at section_start/end keys in the info_dict

I was thinking about using that one but as it's explicitly prohibited to be used when live I thought it was better to not try to attempt that.

@pukkandan
Copy link
Member

it's explicitly prohibited to be used when live

This is because we don't have an implementation for it

@azatoth
Copy link
Author

azatoth commented Dec 18, 2022

it's explicitly prohibited to be used when live

This is because we don't have an implementation for it

I assume you have a totally different implementation in mind as I can't see it possible to utilize --download-sections for this. neither to be able to specify chapters, nor having multiple separate sections.

@azatoth
Copy link
Author

azatoth commented Dec 18, 2022

Don't add new switches. Instead, make the feature obey --download-sections *start-end. The downloader can access the timestamps at section_start/end keys in the info_dict

So I can see --download-sections to work if we add following restrictions when we are under a live stream:

  1. make end optional
  2. have it count backwards from time of start downloading
  3. disallow specifying chapter(s)
  4. disallow multiple ranges

imo making these stipulations makes a totally separate function than what we currently have.

Extend `--live-from-start` with two options `-live-from-start-begin`and
`-live-from-start-end` which will limit the recording to between these
values.

`--live-from-start-end` is optional and if omitted will continously
record as normal.
@azatoth azatoth force-pushed the live_from_start_range branch from adadd3d to 7a8a226 Compare December 18, 2022 21:38
@pukkandan
Copy link
Member

pukkandan commented Dec 18, 2022

  • make end optional

You can already provide infinity as end

  • have it count backwards from time of start downloading

Is there no way around this? If we can determine the current length of the stream (or the start time), we can translate b/w the two systems. This will also allow to implement #4554 and #5608 in future

  • disallow specifying chapter(s)

Chapter to timestamp translation is done by core code. You don't need to be concerned about it. Besides, I believe live streams cannot have chapters anyway (maybe premiers can have it?)

  • disallow multiple ranges

This is a problem only if we cannot solve (2)

@azatoth
Copy link
Author

azatoth commented Dec 22, 2022

Is there no way around this? If we can determine the current length of the stream (or the start time), we can translate b/w the two systems. This will also allow to implement #4554 and #5608 in future

While it would be possible to count from start of live stream, I think that would be impractical for most usecases as it would require the user to know exactly when the live stream started.
Counting from the MAX_DURATION would make it practically impossible to use as the window would move.

I can think of two compromises:

  • Allow specifying start/end as unixtime, i.e. --download-sections "@1671698944-1671698964"
  • Allow negative values, i.e. --download-sections *-start--end

@pukkandan
Copy link
Member

pukkandan commented Dec 22, 2022

  • Allow negative values, i.e. --download-sections *-start--end

Allowing negative values is trivial and I can add it in core code. Just make the downloader handle positive values, and make sure the extractor returns correct timestamp/release_timestamp

@azatoth
Copy link
Author

azatoth commented Dec 23, 2022

Allowing negative values is trivial and I can add it in core code. Just make the downloader handle +ive values, and make sure the extractor returns correct timestamp/release_timestamp

Can you explain what you mean by "+ive values"?

I must confess I've problem navigate the code; And I'm not sure what you want me to accomplish :/

@pukkandan pukkandan removed the pending-fixes PR has had changes requested label Dec 23, 2022
@kadamwhite
Copy link

Could "+ive" mean "positive"?

@SnowSquire

This comment was marked as spam.

@halimselim

This comment was marked as spam.

@mrienstra
Copy link

Working for me as well, installed using:

python3 -m pip install -U pip setuptools wheel
python3 -m pip install --force-reinstall https://github.com/azatoth/yt-dlp/archive/live_from_start_range.tar.gz

There seems to be a variable offset between the time requested and the time downloaded, I tried 3 different livestreams, each had a different offset (-0h32m15s, -3h12m16s, +2h52m53s). This offset may drift slowly as well, still testing.

The "Starting download from the last 120 hours of the live stream since YouTube does not have data before that" warning should probably be suppressed when using these flags. See extractor/youtube.py#L2763-L2765.

Would be nice to have an accurate fragment_count shown, right now it shows something like (frag 406/222451).

It would be great to have the option to specify the start & end using either:

  • Same offset as used by the &t= URL param, which is in seconds, though you can also pass it as &t=38h20m03s. Time elapsed since stream started.
  • Same offset as used by clips, the initial HTML payload when loading a clip contains (in JS) ytInitialPlayerResponse.clipConfig.startTimeMs & ytInitialPlayerResponse.clipConfig.endTimeMs, these values are the same as &t= URL param except in milliseconds rather than seconds.

But more importantly, it would be great to get this "good enough" and merged. @pukkandan, what do you need to see addressed at this point?

@pukkandan
Copy link
Member

pukkandan commented Mar 9, 2023

The PR is on my radar, but so is many others. I believe I have explained the changes needed in above comments (mainly #5822 (comment)). If OP (or anyone else) cannot address them, I plan to get around to make the changes myself eventually. But it will take time - like I said, so many stuff on my plate and only so much time I can spare. If you want to expedite the process, try and help with making the changes, or even spin up your own PR.

@pukkandan
Copy link
Member

pukkandan commented Mar 9, 2023

I see I missed this comment #5822 (comment)

Can you explain what you mean by "+ive values"?

positive values. As opposed to the negative ones discussed here - #5822 (comment). I'm saying if you can make positive values work in download code, I'll make the necessary changes in core code to also make negative values work. The downloader does not need to worry about it.

And I'm not sure what you want me to accomplish :/

Which part is unclear? Instead of adding two new switches for this extremely specific use case, I just want the downloader code to look at section_start/section_end in infodict and make cuts accordingly.

@mrienstra
Copy link

Thanks @pukkandan, appreciate the response!

@azatoth, any thoughts on how to proceed? I've started poking around to try to figure out how to proceed, but I'm new to this codebase.

@pukkandan
Copy link
Member

#6498 is a nicer implementation and superseeds this

@azatoth
Copy link
Author

azatoth commented Mar 15, 2023

#6498 is a nicer implementation and superseeds this

Fair enough, should I close this one?

@mrienstra
Copy link

@azatoth, if you close this, I'd appreciate it if you could delay deleting your branch until #6498 has been merged. Otherwise I'd be unable to return to using your branch after testing #6498 (assuming I do so again), which would mean no way to download stream segments!

... Well, I guess I can just download the archive -- done! -- but still, might be nice to have your work as a reference while #6498 is still in the oven.

@pukkandan pukkandan force-pushed the master branch 2 times, most recently from ee280c7 to 7aeda6c Compare May 24, 2023 18:09
@Grub4K
Copy link
Member

Grub4K commented Nov 17, 2024

Closed bc superseeded

@Grub4K Grub4K closed this Nov 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
do-not-merge enhancement New feature or request site-enhancement Feature request for some website
Projects
Status: youtube livestream
Development

Successfully merging this pull request may close these issues.

[Youtube] --live-from-start support for sequence / range download
7 participants