Skip to content

make generic pipelines to DAG rendering work with Airflow 2.x #3166

@shalberd

Description

@shalberd

Describe the issue
Currently, even when using generic pipeline editor with python files and notebooks, the rendered DAG is not compatible with Airflow 2..6.0 or higher, or any Airflow 2.x, for that matter :-)
Broken DAG: [/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py", line 2, in <module> from airflow.contrib.kubernetes.volume_mount import VolumeMount ModuleNotFoundError: No module named 'airflow.contrib.kubernetes'

To Reproduce
Steps to reproduce the behavior:

  1. Create a generic pipeline
  2. Submit it to the Airflow runtime with Airflow 2.x

Screenshots or log output
If applicable, add screenshots or log output to help explain your problem.

Log Output
Paste the log output here.

Expected behavior
DAg is rendered correctly for use in Airflow 2.x

Starting with jinja template
https://github.com/elyra-ai/elyra/blob/main/elyra/templates/airflow/airflow_template.jinja2

and elyra/elyra/pipeline/airflow/processor_airflow.py

possible hints
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html

https://airflow.apache.org/docs/apache-airflow/2.5.3/howto/upgrading-from-1-10/index.html

https://airflow.apache.org/docs/apache-airflow/2.5.3/core-concepts/dags.html

apache/airflow#21653

Deployment information
Describe what you've deployed and how:

  • Elyra version: latest
  • Installation source: [e.g. PyPI, conda, from source, official container image, custom container image]
  • Deployment type: Open Data Hub pre-RedHat takeover
  • Operating system: [e.g. macos, linux]

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions