Pre-requisite: Airflow
The user interface (UI) feature in Airflow helps us understand, monitor, and troubleshoot our data pipelines. UI serves to be a great tool that gives us insights into our DAGs and DAG runs.
Important Views in Airflow UI
DAG View
The DAG view in Airflow UI gives us a list of all the DAGs that we are having in our Airflow Instance.
Components of DAG View
- Toggle: Using this toggle button we can either pause running data pipelines or trigger the paused data pipelines.
- DAG: DAGs help us to categorize our data pipelines according to their functionality and teams.
- Owner: Here, the owner is the owner of our data – pipelines, which we have specified in our code in the default_args section.
- Run – The runs give us the states of the current and the past run DAGs, which can be either queued, success, running, or failed.
- Schedule:– Schedule defines the frequency at which our data pipeline is going to be triggered. It can be daily, monthly, yearly, and so on.
- Last Run:– It specifies the last period of time at which our DAG has been triggered.
- Next Run:– It specifies the next period of time at which the data pipeline will be triggered.
- Recent Tasks:– Gives us the current status of our data pipelines which can be queued, successful, running, skipped, shutdown, failure, and so on.
- Actions:– Using actions, we can trigger our DAG with or without configuration and also delete the metadata related to our DAGs, such as task instances or DAG runs
- Links:– Enables us to access other useful views in the airflow UI
Tree View
The tree view helps us get an overview of the history and the current DAG runs along with the status of the tasks. It is extremely helpful for spotting any errors in the tasks, by just looking at the colors of the squares. We can also check if any DAG is running late or for any failures in the task
Here, the circles correspond to the DAG Runs and the squares correspond to the tasks.
Graph View
The Graph View in Airflow UI is perfect to check the dependencies of our data pipeline and also, get the status of the tasks for the latest DAG run
Here, the rectangles correspond to the tasks and the border colors correspond to the status of the tasks.
Gantt View
Gantt View allows us to analyze task duration as well as the overlaps. Taking a look at the Gantt view, we can quickly identify the bottlenecks and also, where the majority of the time is spent on a specific diagram. The larger the rectangle in Gantt view, the longer it takes to complete the task
Calendar View
The calendar view gives a sketch of the complete history of the DAG over the time period, which can be either months or years. With the help of the calendar view, we can see the drift of the success/failure run rate within a specific time period.
Task Duration View
This view gives us an overview of the duration of the tasks run over the past N years. It helps us to understand where the bulk of the time is spent.
Code View
With the help of the code view, we are able to access the code of our data pipeline. Also, it is an important view to make sure that the changes made to the code are actually applied to the DAG in airflow
Interaction with the Tasks in Airflow UI
In Airflow UI, if we click on a particular task, then we will land on the task context menu, from there, we can take the following actions on a given task
Instance Details: This gives us additional information related to our task instance, like the dag_id, dag_model, dag_run, etc.
Rendered: It tells us what will the output of the data that we injected in our task at run time. This is related to templates.
Log: It fetches the log of the DAG, which shows the output as well as tells whether the DAG run was a success or failure
All Instances: It permits us to check all task instances for given tasks across all of the task runs.
File Upstream: By clicking on the file Upstream, we can get our tasks as well as the upstream tasks on the graph view.
Task Actions: With the help of task actions, we can manually run our DAGs, clear the previous runs, or mark success or failure to a particular task instance.