Tasks in Airflow - Sheet1 (1)

Question 1

1. What is a Task in Airflow?

Answer 1

A Task is the basic unit of execution in Airflow. They are arranged into DAGs and then have upstream and downstream dependencies set between them to express the order they should run in.

Question 2

2. What are the three basic kinds of Task in Airflow?

Answer 2

Operators, Sensors, and a TaskFlow-decorated @task. Operators are predefined task templates, Sensors wait for an external event to happen, and a TaskFlow-decorated @task is a custom Python function packaged up as a Task.

Question 3

3. How are tasks related to each other in Airflow?

Answer 3

Tasks in Airflow are related to each other through their dependencies. These dependencies can be upstream or downstream tasks. Upstream tasks directly precede other tasks, while downstream tasks need to be a direct child of the other task.

Question 4

4. How can dependencies between tasks be declared in Airflow?

Answer 4

Dependencies can be declared using the&raquo_space; and &laquo_space;(bitshift) operators, or the set_upstream and set_downstream methods.

Question 5

5. How does information pass between Tasks in Airflow?

Answer 5

Tasks don’t pass information to each other by default and run independently. If you want to pass information from one Task to another, you should use XComs.

Question 6

6. What is a Task Instance in Airflow?

Answer 6

An instance of a Task is a specific run of that task for a given DAG (and thus for a given data interval). It represents the stage of the lifecycle the task is in.

Question 7

7. What are the possible states for a Task Instance in Airflow?

Answer 7

The states are none, scheduled, queued, running, success, shutdown, restarting, failed, skipped, upstream_failed, up_for_retry, up_for_reschedule, deferred, and removed.

Question 8

8. What is a timeout in Airflow?

Answer 8

A timeout is a maximum runtime for a task. If it’s breached, the task times out and AirflowTaskTimeout is raised. Sensors also have a timeout parameter which is relevant for sensors in reschedule mode.

Question 9

9. What is an SLA in Airflow?

Answer 9

An SLA, or a Service Level Agreement, is an expectation for the maximum time a Task should be completed relative to the Dag Run start time. If a task takes longer than this to run, it is then visible in the “SLA Misses” part of the user interface, as well as going out in an email of all tasks that missed their SLA.

Question 10

10. What are special exceptions in Airflow?

Answer 10

Airflow provides two special exceptions that can be raised from within custom Task/Operator code: AirflowSkipException will mark the current task as skipped and AirflowFailException will mark the current task as failed ignoring any remaining retry attempts.

Question 11

11. What are Zombie/Undead Tasks in Airflow?

Answer 11

Zombie tasks are tasks that are supposed to be running but suddenly died. Undead tasks are tasks that are not supposed to be running but are, often caused when you manually edit Task Instances via the UI.

Question 12

12. What is Executor Configuration in Airflow?

Answer 12

Some Executors allow optional per-task configuration. This is achieved via the executor_config argument to a Task or Operator.

Question 13

13. How can dependencies between tasks be declared using the bitshift operators?

Answer 13

You can declare dependencies between tasks using the bitshift operators (» and «) in Airflow. For example: first_task&raquo_space; second_task&raquo_space; [third_task, fourth_task]

Question 14

14. What is the recommended way to declare dependencies between tasks in Airflow?

Answer 14

The recommended way to declare dependencies between tasks in Airflow is to use the bitshift operators (» and «), as they are easier to read in most cases.

Question 15

15. What is the default behavior of a Task in Airflow regarding upstream tasks?

Answer 15

By default, a Task will run when all of its upstream (parent) tasks have succeeded in Airflow.

Question 16

16. How can you modify the behavior of a Task in Airflow to add branching or change its dependencies?

Answer 16

You can modify the behavior of a Task in Airflow by using Control Flow techniques. These techniques allow you to add branching, specify which upstream tasks to wait for, or change behavior based on the current run’s history.

Question 17

17. What are the two ways to declare dependencies between tasks in Airflow?

Answer 17

The two ways to declare dependencies between tasks in Airflow are using the bitshift operators (» and «) or the more explicit set_upstream and set_downstream methods.

Question 18

18. What is the purpose of XComs in Airflow?

Answer 18

XComs are used in Airflow to pass information between tasks. They provide a way to share data, such as results or intermediate values, from one task to another within a workflow.

Question 19

19. What is an SLA miss in Airflow?

Answer 19

An SLA miss occurs when a task in Airflow takes longer to complete than the defined SLA (Service Level Agreement) timeframe. It is tracked and can be seen in the Airflow UI and can trigger notifications or alerts.

Question 20

20. How can you set an SLA for a task in Airflow?

Answer 20

To set an SLA for a task in Airflow, you can pass a datetime.timedelta object to the task’s “sla” parameter. This defines the maximum time allowed for the task to complete relative to the DAG Run start time.

Question 21

21. What are AirflowSkipException and AirflowFailException used for?

Answer 21

AirflowSkipException is used to mark the current task as skipped, while AirflowFailException is used to mark the current task as failed, ignoring any remaining retry attempts. These exceptions provide control over the task’s state from within custom Task/Operator code.

Question 22

22. What are Zombie tasks and Undead tasks in Airflow?

Answer 22

Zombie tasks in Airflow are tasks that are supposed to be running but suddenly died or became inactive. Undead tasks are tasks that are not supposed to be running but are active due to manual editing of Task Instances via the Airflow UI.

Question 23

23. How can you configure task-specific settings for certain Executors in Airflow?

Answer 23

You can configure task-specific settings for certain Executors in Airflow using the “executor_config” argument provided to a Task or Operator. This allows you to specify executor-specific parameters or configurations for individual tasks.

Question 24

24. What is the purpose of a Task in Airflow?

Answer 24

A Task in Airflow represents a unit of work or a specific action that needs to be executed as part of a workflow. It can be an operator, a sensor, or a custom Python function decorated as a Task.

Question 25

25. What is the relationship between Tasks and Operators in Airflow?

Answer 25

In Airflow, Tasks and Operators are somewhat interchangeable concepts. Operators are predefined task templates that can be used to build most parts of a DAG (Directed Acyclic Graph), and when called in a DAG file, they become Tasks.

Question 26

26. What is the role of Sensors in Airflow?

Answer 26

Sensors are a special subclass of Operators in Airflow that are designed for waiting and monitoring external events or conditions before proceeding with the execution of downstream tasks. They help synchronize the workflow with external dependencies.

Question 27

27. How are Tasks and Operators related to the BaseOperator class in Airflow?

Answer 27

In Airflow, Tasks and Operators are subclasses of Airflow’s BaseOperator class. The BaseOperator provides a foundation for defining and executing tasks, and it encapsulates common functionality and attributes that are shared among tasks and operators.

Question 28

28. What is the purpose of upstream and downstream tasks in Airflow?

Answer 28

Upstream tasks in Airflow are the tasks that directly precede a given task. Downstream tasks are the tasks that are directly dependent on a given task. These relationships define the order and dependencies between tasks within a DAG.

Question 29

29. How can you declare dependencies between tasks using the set_upstream and set_downstream methods in Airflow?

Answer 29

To declare dependencies between tasks using the set_upstream and set_downstream methods in Airflow, you can use the following syntax: first_task.set_downstream(second_task), third_task.set_upstream(second_task). These methods explicitly define the upstream and downstream dependencies between tasks.

Question 30

30. What is the purpose of the execution_timeout attribute in Airflow?

Answer 30

The execution_timeout attribute in Airflow allows you to set a maximum runtime for a task. If the task exceeds this duration, it will time out and AirflowTaskTimeout will be raised.

Question 31

31. What is an SLA miss callback in Airflow?

Answer 31

An SLA miss callback is a function that can be defined in Airflow to run custom logic when a task misses its SLA (Service Level Agreement) timeframe. It receives parameters such as the DAG, task list, blocking task list, and SLA information.

Question 32

32. How can you configure per-task settings for specific Executors in Airflow?

Answer 32

In Airflow, per-task settings for specific Executors can be configured using the executor_config argument provided to a Task or Operator. This allows you to specify executor-specific configurations, such as the Docker image for tasks running on the KubernetesExecutor.

Question 33

33. What is the purpose of Control Flow in Airflow?

Answer 33

Control Flow in Airflow allows you to modify the behavior of tasks and express complex workflow logic. It enables you to add branching, specify conditional dependencies, and control task execution based on the history or context of the current run.
34. How can you pass information between tasks in Airflow?

Question 34

43. What is the purpose of TaskFlow in Airflow?

Answer 34

TaskFlow is a feature in Airflow that allows you to define tasks as custom Python functions decorated with @task. It provides a more flexible and expressive way to create tasks compared to using predefined operators.

Question 35

44. What is the relationship between TaskFlow and Task in Airflow?

Answer 35

TaskFlow tasks, decorated with @task, are a type of Task in Airflow. They represent custom Python functions packaged as tasks within a DAG. TaskFlow tasks can be used alongside other types of tasks and operators in a workflow.

Question 36

45. What are the possible states of a Task Instance in Airflow?

Answer 36

The possible states of a Task Instance in Airflow are: none, scheduled, queued, running, success, shutdown, restarting, failed, skipped, upstream_failed, up_for_retry, up_for_reschedule, deferred, and removed. These states represent the various stages in the lifecycle of a task during execution.

Question 37

46. How can you handle task timeouts in Airflow?

Answer 37

To handle task timeouts in Airflow, you can set the “execution_timeout” attribute of a task to a datetime.timedelta value. This defines the maximum allowed runtime for the task. If the task exceeds this duration, it times out and AirflowTaskTimeout is raised.

Question 38

47. What is the purpose of SLAs in Airflow?

Answer 38

SLAs (Service Level Agreements) in Airflow define the maximum time a task should be completed relative to the DAG Run start time. They provide a measure of performance and can trigger alerts or notifications when a task exceeds its defined SLA timeframe.

Question 39

48. What are Zombie tasks in Airflow?

Answer 39

Zombie tasks in Airflow are tasks that are supposed to be running but suddenly died or became inactive. They are typically detected during periodic checks performed by Airflow, and the necessary actions are taken to clean them up or handle their failures.

Question 40

49. What are Undead tasks in Airflow?

Answer 40

Undead tasks in Airflow are tasks that are not supposed to be running but are active. They often occur when Task Instances are manually edited via the Airflow user interface, resulting in tasks being in an active state even when they shouldn’t be.

Question 41

50. What is Executor Configuration in Airflow?

Answer 41

Executor Configuration in Airflow allows you to set per-task configurations for specific Executors. It provides a way to customize the execution environment and behavior of tasks by specifying executor-specific parameters or settings.

Question 42

51. How can you set an SLA for a task in Airflow?

Answer 42

To set an SLA for a task in Airflow, you can use the “sla” parameter of the Task or Operator. By specifying a datetime.timedelta value, you define the maximum time allowed for the task to complete relative to the DAG Run start time.

Question 43

52. What are the two ways to declare task dependencies in Airflow?

Answer 43

In Airflow, task dependencies can be declared using the bitshift operators (» and «) or the set_upstream and set_downstream methods. Both approaches allow you to establish upstream and downstream relationships between tasks.

Question 44

53. What is the purpose of an SLA miss callback in Airflow?

Answer 44

An SLA miss callback in Airflow is a function that gets triggered when a task misses its SLA timeframe. It provides a way to handle and react to SLA violations, allowing you to customize actions such as sending notifications, logging, or performing specific tasks.