Skip to content

Clean up task file/config merging semantics #286

@vito

Description

@vito

Today to specify a task config file and override things like params and inputs, you do:

- get: concrete-input
- task: foo
  file: input/foo.yml
  config:
    params: {FOO: 1}
    inputs:
    - name: concrete-input
      path: generic-input

This is all well and good but things get weird when we add more things to task configs. For example, we don't override outputs since we're not sure what would even mean. Turns out we do support overriding inputs, which is done via a wholesale merge, and is occasionally used to remap a generic input name to a concrete one, while preserving the path.

It's also weird that config contains essentially a partially instantiated task config, meaning if config and file are both present, it doesn't have to be valid, but if only config is present, it has to be valid.

Today we merge the following fields:

  • platform - not sure why, seems like the task .yml should know this ahead of time
  • image - same, don't know when you'd want to override this
  • params - the most common use case (passing in secrets via pipeline config)
  • inputs - this is used for remapping input names while preserving the path expected by the .yml, but it feels pretty inelegant
  • run.path - also not sure why this is overrideable

So, instead, we should:

  • no longer allow specifying both config and file
  • expect config to always be a fully-defined task config, if specified
  • no longer support overriding platform, image, inputs, and run.path
  • promote params to a separate field (maybe just reuse params and ensure the values are always strings on a task step)
  • add configuration for remapping task inputs by name, e.g. input_names: {generic-name: concrete-name}

Additionally folks have tried overriding outputs, similar to how we support overriding inputs, but that's just not implemented (part of the risk of overloading task config). We could add an output_names mapping similar to input_names mentioned above.

The full result would be something like:

- get: concrete-input
- task: foo
  file: input/foo.yml
  params: {FOO: 1}
  input_names: {generic-input: concrete-input}
  output_names: {generic-output: concrete-output}

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions