-
Notifications
You must be signed in to change notification settings - Fork 358
Description
Describe the issue
Currently, even when using generic pipeline editor with python files and notebooks, the rendered DAG is not compatible with Airflow 2..6.0 or higher, or any Airflow 2.x, for that matter :-)
Broken DAG: [/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py] Traceback (most recent call last): File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed File "/opt/airflow/dags/repo/SvensWetterapprootpvc-0202162036.py", line 2, in <module> from airflow.contrib.kubernetes.volume_mount import VolumeMount ModuleNotFoundError: No module named 'airflow.contrib.kubernetes'
To Reproduce
Steps to reproduce the behavior:
- Create a generic pipeline
- Submit it to the Airflow runtime with Airflow 2.x
Screenshots or log output
If applicable, add screenshots or log output to help explain your problem.
Log Output
Paste the log output here.
Expected behavior
DAg is rendered correctly for use in Airflow 2.x
Starting with jinja template
https://github.com/elyra-ai/elyra/blob/main/elyra/templates/airflow/airflow_template.jinja2
and elyra/elyra/pipeline/airflow/processor_airflow.py
possible hints
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/operators.html
https://airflow.apache.org/docs/apache-airflow/2.5.3/howto/upgrading-from-1-10/index.html
https://airflow.apache.org/docs/apache-airflow/2.5.3/core-concepts/dags.html
Deployment information
Describe what you've deployed and how:
- Elyra version: latest
- Installation source: [e.g. PyPI, conda, from source, official container image, custom container image]
- Deployment type: Open Data Hub pre-RedHat takeover
- Operating system: [e.g. macos, linux]