feat(workers): allows videoWorker to use ytdlp command line arguments… #1117

erik-nilcoast · 2025-03-12T15:18:46Z

… specified in the config

MohamedBassem

Thanks! Looks good to me, but we need to update the documentation in docs/docs/03-configuration.md.

MohamedBassem · 2025-03-12T22:13:07Z

apps/workers/videoWorker.ts

  const ytDlpArguments = [url];
  if (serverConfig.crawler.maxVideoDownloadSize > 0) {
    ytDlpArguments.push(
      "-f",
      `best[filesize<${serverConfig.crawler.maxVideoDownloadSize}M]`,
    );
  }
+
+  ytDlpArguments.push(...serverConfig.crawler.ytDlpArguments.split(","));
  ytDlpArguments.push("-o", assetPath);
  ytDlpArguments.push("--no-playlist");


Should we maybe move this to the default of the command?

game for whatever works for y'all. if there's a good way to represent an array with the config library you're using, i'm all for it.

MohamedBassem · 2025-03-12T22:13:53Z

packages/shared/config.ts

@@ -49,6 +49,7 @@ const allEnv = z.object({
  CRAWLER_VIDEO_DOWNLOAD_MAX_SIZE: z.coerce.number().default(50),
  CRAWLER_VIDEO_DOWNLOAD_TIMEOUT_SEC: z.coerce.number().default(10 * 60),
  CRAWLER_ENABLE_ADBLOCKER: stringBool("true"),
+  YTDLP_ARGS: z.string().default(""),


We can do the parsing here as well.

Suggested change

YTDLP_ARGS: z.string().default(""),

YTDLP_ARGS: z.string().transform(t => t.split(",").filter(a => a)).default([]),

works for me. i'll make those changes

actually that won't work if the default is empty array as zod is going to always expect that to be a string.

YTDLP_ARGS: z.string().transform(t => t.split(",").filter(a => a)).default('') // works but now the type is ambiguous

How about:

YTDLP_ARGS: z.string().default('').transform(t => t.split(",").filter(a => a))

?

that'll do it.

MohamedBassem · 2025-03-12T22:19:00Z

Btw, did you happen to check if any of the ytldb arguments can contain commas? Because that's the delimiter we're using and it might end up being problematic if they do.

erik-nilcoast · 2025-03-12T22:20:56Z

Btw, did you happen to check if any of the ytldb arguments can contain commas? Because that's the delimiter we're using and it might end up being problematic if they do.

I'll check, I'm also happy to pass certain ones through. I really only need subtitles and login.

erik-nilcoast · 2025-03-12T22:23:25Z

Btw, did you happen to check if any of the ytldb arguments can contain commas? Because that's the delimiter we're using and it might end up being problematic if they do.

I'll check, I'm also happy to pass certain ones through. I really only need subtitles and login.

Yeah unfortunately there are.

-h, --help                      Print this help text and exit
--version                       Print program version and exit
-U, --update                    Update this program to the latest version
--no-update                     Do not check for updates (default)
--update-to [CHANNEL]@[TAG]     Upgrade/downgrade to a specific version.
                                CHANNEL can be a repository as well. CHANNEL
                                and TAG default to "stable" and "latest"
                                respectively if omitted; See "UPDATE" for
                                details. Supported channels: stable,
                                nightly, master
-i, --ignore-errors             Ignore download and postprocessing errors.
                                The download will be considered successful
                                even if the postprocessing fails
--no-abort-on-error             Continue with next video on download errors;
                                e.g. to skip unavailable videos in a
                                playlist (default)
--abort-on-error                Abort downloading of further videos if an
                                error occurs (Alias: --no-ignore-errors)
--dump-user-agent               Display the current user-agent and exit
--list-extractors               List all supported extractors and exit
--extractor-descriptions        Output descriptions of all supported
                                extractors and exit
--use-extractors NAMES          Extractor names to use separated by commas.
                                You can also use regexes, "all", "default"
                                and "end" (end URL matching); e.g. --ies
                                "holodex.*,end,youtube". Prefix the name
                                with a "-" to exclude it, e.g. --ies
                                default,-generic. Use --list-extractors for
                                a list of extractor names. (Alias: --ies)
--default-search PREFIX         Use this prefix for unqualified URLs. E.g.
                                "gvsearch2:python" downloads two videos from
                                google videos for the search term "python".
                                Use the value "auto" to let yt-dlp guess
                                ("auto_warning" to emit a warning when
                                guessing). "error" just throws an error. The
                                default value "fixup_error" repairs broken
                                URLs, but emits an error if this is not
                                possible instead of searching
--ignore-config                 Don't load any more configuration files
                                except those given to --config-locations.
                                For backward compatibility, if this option
                                is found inside the system configuration
                                file, the user configuration is not loaded.
                                (Alias: --no-config)
--no-config-locations           Do not load any custom configuration files
                                (default). When given inside a configuration
                                file, ignore all previous --config-locations
                                defined in the current file
--config-locations PATH         Location of the main configuration file;
                                either the path to the config or its
                                containing directory ("-" for stdin). Can be
                                used multiple times and inside other
                                configuration files
--plugin-dirs PATH              Path to an additional directory to search
                                for plugins. This option can be used
                                multiple times to add multiple directories.
                                Use "default" to search the default plugin
                                directories (default)
--no-plugin-dirs                Clear plugin directories to search,
                                including defaults and those provided by
                                previous --plugin-dirs
--flat-playlist                 Do not extract a playlist's URL result
                                entries; some entry metadata may be missing
                                and downloading may be bypassed
--no-flat-playlist              Fully extract the videos of a playlist
                                (default)
--live-from-start               Download livestreams from the start.
                                Currently only supported for YouTube
                                (Experimental)
--no-live-from-start            Download livestreams from the current time
                                (default)
--wait-for-video MIN[-MAX]      Wait for scheduled streams to become
                                available. Pass the minimum number of
                                seconds (or range) to wait between retries
--no-wait-for-video             Do not wait for scheduled streams (default)
--mark-watched                  Mark videos watched (even with --simulate)
--no-mark-watched               Do not mark videos watched (default)
--color [STREAM:]POLICY         Whether to emit color codes in output,
                                optionally prefixed by the STREAM (stdout or
                                stderr) to apply the setting to. Can be one
                                of "always", "auto" (default), "never", or
                                "no_color" (use non color terminal
                                sequences). Use "auto-tty" or "no_color-tty"
                                to decide based on terminal support only.
                                Can be used multiple times
--compat-options OPTS           Options that can help keep compatibility
                                with youtube-dl or youtube-dlc
                                configurations by reverting some of the
                                changes made in yt-dlp. See "Differences in
                                default behavior" for details
--alias ALIASES OPTIONS         Create aliases for an option string. Unless
                                an alias starts with a dash "-", it is
                                prefixed with "--". Arguments are parsed
                                according to the Python string formatting
                                mini-language. E.g. --alias get-audio,-X
                                "-S=aext:{0},abr -x --audio-format {0}"
                                creates options "--get-audio" and "-X" that
                                takes an argument (ARG0) and expands to
                                "-S=aext:ARG0,abr -x --audio-format ARG0".
                                All defined aliases are listed in the --help
                                output. Alias options can trigger more
                                aliases; so be careful to avoid defining
                                recursive options. As a safety measure, each
                                alias may be triggered a maximum of 100
                                times. This option can be used multiple times

which would be problematic for my approach :(

MohamedBassem · 2025-03-12T22:27:32Z

which would be problematic for my approach

Let's try to find another delimiter then? Maybe %% or something?

erik-nilcoast · 2025-03-12T22:32:16Z

Let's try to find another delimiter then? Maybe %% or something?

Happy to code that up. I'll see if I can find a common pattern. Parsing the flags isn't too hard:

-I 1,2,3 --foo bar,baz

It's always going to be a -(\w+)\s(.*)-? or something similar. There might be a simple library we could use that'd be better than any regexp I'm going to write, but it's probably a lot more than you need for this.

MohamedBassem · 2025-03-12T22:46:43Z

Let's just keep it simple and instead of splitting by commas, we can split by %%.

erik-nilcoast · 2025-03-12T23:07:50Z

Fixed up!

erik-nilcoast · 2025-03-12T23:08:13Z

Thanks! Looks good to me, but we need to update the documentation in docs/docs/03-configuration.md.

Added!

MohamedBassem

You don't allow maintainers to edit the PR? Could have sent those small changes myself :D But in this case, I'll have to ask for a couple of small changes before merging.

MohamedBassem · 2025-03-16T10:12:50Z

packages/shared/config.ts

@@ -49,6 +49,10 @@ const allEnv = z.object({
  CRAWLER_VIDEO_DOWNLOAD_MAX_SIZE: z.coerce.number().default(50),
  CRAWLER_VIDEO_DOWNLOAD_TIMEOUT_SEC: z.coerce.number().default(10 * 60),
  CRAWLER_ENABLE_ADBLOCKER: stringBool("true"),
+  YTDLP_ARGS: z


Let's add the CRAWLER_ prefix here for consistency

docs/docs/03-configuration.md

erik-nilcoast · 2025-03-16T17:41:56Z

You don't allow maintainers to edit the PR? Could have sent those small changes myself :D But in this case, I'll have to ask for a couple of small changes before merging.

Not intentional, I'm happy to open it up.

… specified in the config

MohamedBassem reviewed Mar 12, 2025

View reviewed changes

ebenoist force-pushed the erik/ytdlp-config branch 2 times, most recently from 3d8a742 to 0182110 Compare March 12, 2025 23:07

MohamedBassem reviewed Mar 16, 2025

View reviewed changes

feat(workers): allows videoWorker to use ytdlp command line arguments…

dd44d44

… specified in the config

ebenoist force-pushed the erik/ytdlp-config branch from 8c3a123 to dd44d44 Compare March 16, 2025 17:45

MohamedBassem merged commit bffbd43 into karakeep-app:main Mar 16, 2025
5 checks passed

	YTDLP_ARGS: z.string().default(""),
	YTDLP_ARGS: z.string().transform(t => t.split(",").filter(a => a)).default([]),

Uh oh!

feat(workers): allows videoWorker to use ytdlp command line arguments… #1117

feat(workers): allows videoWorker to use ytdlp command line arguments… #1117

Uh oh!

Conversation

erik-nilcoast commented Mar 12, 2025

Uh oh!

MohamedBassem left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MohamedBassem commented Mar 12, 2025

Uh oh!

erik-nilcoast commented Mar 12, 2025

Uh oh!

erik-nilcoast commented Mar 12, 2025

Uh oh!

MohamedBassem commented Mar 12, 2025

Uh oh!

erik-nilcoast commented Mar 12, 2025

Uh oh!

MohamedBassem commented Mar 12, 2025

Uh oh!

erik-nilcoast commented Mar 12, 2025

Uh oh!

erik-nilcoast commented Mar 12, 2025

Uh oh!

MohamedBassem left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

erik-nilcoast commented Mar 16, 2025

Uh oh!

Uh oh!

Uh oh!