Motivation

Users are looking for the ability to customize pipeline node inputs without the need to edit the pipeline in the pipeline editor.

Example:

• User creates a pipeline that executes one node. The node contains an input property (say a threshold) that impacts the output of the node.
• The user wants to run the pipeline multiple times (e.g. using the Elyra CLI or from the runtime environment's GUI) and use a different value for each run.

As of version 3.12 the user has to modify the pipeline file using the pipeline editor to accomplish the goal.

Pipeline parameters would enable us to provide the means to customize certain properties, which are evaluated at the time the pipeline is processed (exported/submitted for execution), which can happen outside the pipeline editor.

Example how pipeline parameters are exposed in the Kubeflow Central Dashboard:

Creating pipelines with parameters

Expose pipeline parameters in pipeline editor

• If the runtime supports pipeline parameters, the properties slide-out includes an additional tab (e.g. pipeline parameters).

Define pipeline parameters

• User can define a list of parameters.
• By default a pipeline does not include any parameters.
• A pipeline parameter definition comprises
  1. A unique key, identifying the parameter name, e.g. PARM_1
  2. An optional default value. A default value makes it possible to process ('submit'/'export') a pipeline using the pipeline editor without the need for the user to specify a custom value. (Refer to later section on usage.)
  3. A parameter type, which can be initially only 'string', with support for other primitive types added in the future. Typing allows for basic validation and is only applicable if applicable node properties are typed. For example, if a node property PROP1 is typed number, only compatible parameter can be assigned to that property.
  4. An attribute that defines whether a parameter value is required. If a parameter is required, a parameter default value must be defined or a user-provided value exist or the pipeline will not be processed by Elyra.

Runtime parameter definitions are persisted in the .pipeline file, e.g. as siblings of pipeline_defaults as shown in this mockup:

     "app_data": {
        "ui_data": {
          "comments": []
        },
        "version": 8,
        "properties": {
          "pipeline_parameters": [
              {
                "name": "PARM_1",
                "default_value": "default_value_1",
                ...
              }
           ],
          "pipeline_defaults": {
              ...
          }

Add 'pipeline parameter' as property value input option

• This only applies to pipeline editors for runtime environments that support pipeline parameters.
• Only node properties that serve as component inputs should accept parameters as input.
• For those properties the input selector list is extended to include a new source: pipeline parameter.

Assign pipeline parameter to property

• If 'pipeline parameter' is selected as property input source, the user can choose an entry from the list of available type-compatible parameters.

  

The pipeline editor stores the association between the property and the parameter name in the .pipeline file, as shown in the last line of this mockup:

            "component_parameters": {
              "dependencies": [],
              "include_subdirectories": "false",
              "outputs": [],
              "env_vars": [],
              "kubernetes_pod_annotations": [],
              "kubernetes_pod_labels": [],
              "kubernetes_secrets": [],
              "kubernetes_tolerations": [],
              "mounted_volumes": [],
              "algorithm": "...",
              "threshold": "<REF-TO-PARM_1>"

Processing pipelines with parameters

When Elyra processes a pipeline today the following happens:

• User invokes export or execute.

• User chooses target runtime environment.
  

• Pipeline is validated and then exported or executed.

  

The following applies only to runtime editors that support pipeline parameters.

The steps remain the same if a pipeline does not include any node properties that are configured to use pipeline parameters as input. If at least one pipeline parameter is assigned as an input, the process changes as follows:

• User invokes export or execute.

• User chooses target runtime environment.

• User assigns a custom value to each referenced pipeline parameter. The default is rendered as placeholder and an explicit or implicit value must be assigned if the parameter was defined as required.

  

• Pipeline is validated and then exported or executed.

CLI

• 'run'/'submit'/'export' commands are extended to allow for parameter value override (e.g. parm_1=custom_value_1)
• 'describe' command is extended to list pipeline parameters

Runtime env tooling

• parameters are exposed as runtime env specific 'run parameters' and can be customized