-
Notifications
You must be signed in to change notification settings - Fork 572
Allow bake files to be specified via environment variable #3242
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
The environment variable `BUILDX_BAKE_FILE` (and optional variable `BUILDX_BAKE_FILE_SEPARATOR`) can be used to specify one or more bake files (similar to `compose`). This is mutually exclusive with`--file` (which takes precedence). This is done very early to ensure the values are treated just like `--file`, e.g., participate in telemetry. This includes leaving relative paths as-is, which deviates from `compose` (which makes them absolute). Signed-off-by: Roberto Villarreal <rrjjvv@yahoo.com>
@@ -143,6 +143,11 @@ Use the `-f` / `--file` option to specify the build definition file to use. | |||
The file can be an HCL, JSON or Compose file. If multiple files are specified, | |||
all are read and the build configurations are combined. | |||
|
|||
Alternatively, the environment variable `BUILDX_BAKE_FILE` can be used to specify the build definition to use. | |||
This is mutually exclusive with `-f` / `--file`; if both are specified, the environment variable is ignored. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok seems it follows compose behavior https://docs.docker.com/reference/cli/docker/compose/#set-up-environment-variables but I would tend to error out in such case. Although with bake we print the loaded definitions during build so at least user is aware of performed definitions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would tend to error out in such case
That would be pretty unusual and unexpected I would think. If it contradicted with another variable/flag that would be one thing. Otherwise I wouldn't see it as any different than BUILDKIT_PROGRESS
vs. --progress
, BUILDX_BUILDER
vs. --builder
, etc., where a flag always silently wins over the environment. I guess in those cases it's a little different in that those are single value options, whereas these files are multi-value, and maybe you're concerned about user expectations regarding whether flags and environment are additive in this case?
Let me know if I misunderstood or misinterpreted your comment (or if I did read it correctly and you still feel it should be an error).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise I wouldn't see it as any different than
BUILDKIT_PROGRESS
vs.--progress
,BUILDX_BUILDER
vs.--builder
, etc., where a flag always silently wins over the environment.
Yes indeed, hence why we print builder used and bake definitions loaded in the logs as "hints". So agreed to stay consistent.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we print builder used and bake definitions loaded in the logs as "hints"
Incidentally, that is one of the main reasons I deviated from compose's behavior of converting environment-provided files to absolute paths (which I noted in the commit comment). An absolute path made the output very ugly. And after seeing the input files were added as trace attributes, I figured not forcing an absolute path would simply avoid the question of whether it would be (or needed to be) scrubbed or not.
Just wanted to call that out in case it triggers any thoughts. I thought/hoped by putting the processing where I did, the questions/rules about where relative paths are anchored would be no different than for --file
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
And after seeing the input files were added as trace attributes, I figured not forcing an absolute path would simply avoid the question of whether it would be (or needed to be) scrubbed or not.
You mean through the traced command in
Lines 68 to 72 in d09eb75
ctx, end, err := tracing.TraceCurrentCommand(ctx, append([]string{"bake"}, targets...), | |
attribute.String("builder", in.builder), | |
attribute.StringSlice("targets", targets), | |
attribute.StringSlice("files", in.files), | |
) |
What I meant by "hence why we print builder used and bake definitions loaded in the logs" is when invoking the command you have the loaded bake definitions displayed in progress output:
$ docker buildx bake --print
#1 [internal] load local bake definitions
#1 reading docker-bake.hcl 8.03kB / 8.03kB done
#1 DONE 0.0s
I was thinking that maybe we could mark files loaded from an env var with another message like:
$ docker buildx bake --print
#1 [internal] load bake definitions from BUILDX_BAKE_FILE env
#1 reading docker-bake.hcl 8.03kB / 8.03kB done
#1 DONE 0.0s
So user is not confused.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You mean through the traced command in
Yes, exactly; I didn't follow the entire code path, but that is exactly the code I saw as I was trying to figure out where best to add my new code. I know many folks consider an absolute path leakage (e.g., seeing a user's directory structure and username). I don't know if a full path would have actually been passed through or if this community would have cared even if it was. Of course if a user specifies absolute paths that's a different story. But I saw no technical reason/advantage to converting relative paths to absolute, and a few downsides.
What I meant by [...]
I was thinking that maybe we could mark files loaded from an env var with another message like
Yup, I knew what you meant (the UI portion), but I didn't realize your concern was around communicating which mechanism was used to specify the file; I thought it was around that a file was used (vs. ignored). Sorry I missed that!
So yes, I can do that. Quickly looking at the code, I think there's going to be a choice between doing it:
- "simply"; doing a second lookup in
readBakeFiles
where that printing occurs to see if that environment variable exists, and if does, assume it was respected. This is obviously the case today, but not really future-safe. - "correctly"; tracking the fact it was specified via environment, as opposed to the simple sleight-of-hand I'm currently doing (directly populating
in.files
from the environment as if they had been specified via-f
)
I'll try to find an unobtrusive way to do this "correctly", but if you actually prefer "simply" in this case, let me know.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let me know what you think of what I pushed. Most of the 'good' solutions I could see entailed adding code disproportionate to the feature itself or changing the flow (which you already said you liked). I left it as a separate commit anticipating you might want (possibly substantial) changes or even opt to drop it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I assumed that your suggestion to to specify the full message on the one test was intended to cover all the tests, so I did just that.
While it would make sense to add "from file" to complement "from env," (in the common case of `--file` or using the default), it wouldn't provide any real value. A simpler solution would have been looking for the existence of the variable at the point where printing happens. It felt wrong duplicating the logic. Executing the same logic (if it was extracted) wouldn't be as bad, but still not ideal. A 'correct' solution would be to explicitly track the source of each definition, which would be clearer and more future-proof. It didn't seem like this feature warranted that amount of engineering (with no known features that might make use of it). This implementation seemed like a fair compromise; none of the functions are exported, and all have only one caller. I also considered converting prefixing environment values with `env://` so they could be thought of (and processed like) `cmd://` values. I didn't think it would be viewed as a good solution. Co-authored-by: CrazyMax <1951866+crazy-max@users.noreply.github.com> Signed-off-by: Roberto Villarreal <rrjjvv@yahoo.com>
6e08bb0
to
d44ffb4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM thanks!
PTAL @tonistiigi
@ArthurFlag Would need additional docs follow-up in https://github.com/docker/docs/blob/main/content/manuals/build/building/variables.md#build-tool-configuration-variables |
Fixes #3218
The environment variable
BUILDX_BAKE_FILE
(and optional variableBUILDX_BAKE_FILE_SEPARATOR
) can be used to specify one or more bake files (similar tocompose
). This is mutually exclusive with--file
(which takes precedence).This is done very early to ensure the values are treated just like
--file
, e.g., participate in telemetry. This includes leaving relative paths as-is, which deviates fromcompose
(which makes them absolute).This was intended to functionally be just like compose, but there was no opportunity to reuse code and the patterns/flows used were different enough that I went with something simple rather than trying to make them look the same.
When I created the feature request, it didn't occur to me that there could be potential confusion/ambiguity between
COMPOSE_FILE
(not respected by bake to my knowledge),--file
(which can be used to specify compose files), and whether the newBUILDX_BAKE_FILE
would/should allow compose files. The issue is ignored/unaddressed in this initial implementation, as it wasn't immediately obvious whetherCOMPOSE_FILE
should be supported and what its relationship toBUILDX_BAKE_FILE
would be. The focus is solely onBUILDX_BAKE_FILE
is equivalent to--file
.