10,760 questions
0
votes
1
answer
37
views
Accessing Docker Host's Docker Socket from a Containerized Airflow Setup (Permission Denied Issue) [closed]
I'm running Apache Airflow in Docker using the official docker-compose setup, and I wanted to use the DockerOperator to launch additional containers using the host’s Docker daemon.
To do this, I:
...
- 117
0
votes
1
answer
31
views
Is there DAG dependecy page or analog in Airflow 3.0
In the Airflow 2+ there is "DAG dependencies" page (Browse menu item) that shows all DAG dependcies from other DAGs. Are there any analog in Airflow 3.0? I have not found any similar page.
...
- 33.9k
0
votes
0
answers
112
views
Airflow + Keycloak SSO: User with "Admin" role gets "Viewer" role after login [closed]
We’re using Apache Airflow 3.0.2 deployed on Kubernetes with the official Airflow Helm chart (1.17.0). We’ve integrated Keycloak for SSO using OAuth. Users are able to log in successfully via SSO, but ...
- 1,552
1
vote
1
answer
34
views
How to set starting condition for airflow task group
I have the following dag structure:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.utils.task_group import TaskGroup
from airflow.providers.common.sql.operators.sql ...
- 423
0
votes
0
answers
33
views
dependencies between assets in airflow
I define two assets in airflow 3.0.3 that after running first asset (user) second asset (user_location) run too. the code is like this:
from airflow.sdk import asset, Asset, Context
from airflow....
- 1,423
0
votes
0
answers
73
views
Spring Boot NullPointerException and ClassNotFoundException at application termination
Overview
We have spring boot code. Code is using Java NIO to write the json content on GCS. We are writing many small files. Java jar code is invoked by the python code.
In between code throws ...
- 155
0
votes
0
answers
28
views
How to correctly handle Airflow DAG go-live
We're migrating some scheduled jobs from to Airflow. The DAGs are written at some point in time, tested and deployed to the production environment. Another team is responsible for enabling these DAGs ...
- 14.9k
1
vote
0
answers
38
views
setting up airflow 2.4.0 in docker
i am trying to setup airflow 2.4.0 as its the tested version of airflow that works with ZenML.
i downloaded the docker-compose yaml from https://airflow.apache.org/docs/apache-airflow/2.4.0/docker-...
- 994
-1
votes
1
answer
49
views
Setup task in airflow DAG
I have a service which generates DAG according to a template and uploads it to a git repo.
There's an Airflow container, using that repo as its DAGs source.
How can I, as efficiently as possible, run ...
- 539
2
votes
0
answers
107
views
Airflow 3.0.2 + Helm + Keycloak SSO: User role changes from "Admin" to "Viewer" after login
We are using Apache Airflow 3.0.2 with the official Helm chart version 1.17.0, deployed on Kubernetes via Terraform. We're integrating SSO using Keycloak.
Problem
After successful SSO login, users ...
- 1,552
-1
votes
0
answers
55
views
Dockerized Airflow DAG fails to persist Pandas-parsed API data to mounted volume during multi-container orchestration -- intermittent write issue [closed]
I'm running a Dockerized data pipeline using Apache Airflow to periodically fetch structured data from a third-party API, clean it using Pandas, and persist the cleaned DataFrame as a .parquet file to ...
1
vote
0
answers
33
views
Airflow ExternalTaskSensor waits forever even though the task has finished
Airflow ExternalTaskSensor waits forever after TriggerDagRunOperator
I just started learning Airflow and I have a problem with ExternalTaskSensor.
I have 2 DAGs:
A trigger_dag that waits for a file, ...
0
votes
0
answers
36
views
airflow dag runs twice a day, 2nd one queues up five seconds after first run... dangerous behaviour
I am at my wits end with airflow running twice per day. it seems to decide to run a second dag run 5 seconds after the first. the first dag takes 4 hours or so and then it starts the 2nd one.
I have ...
- 1,432
1
vote
1
answer
91
views
Apache Airflow DAG not running
I have installed the Airflow Docker environment as described in this guide (3.0.2)and I'm able to run a very simple DAG using the EmptyOperator.
However, when I create another DAG using, for example, ...
- 707
0
votes
1
answer
120
views
Airflow Executor CeleryExecutor(parallelism=48) reported that the task instance [queued]> finished with state failed
I'm looking for help/advise on understanding error we are facing in our airflow environment couple of times a day.
We are hosting airflow in azure kubernetes , for backend database we were using ...
- 125