Apache Airflow allows you to maximize the value of any type of big data in a company. As a powerful big data orchestration tool, Apache Airflow refines data from an unrefined state into a reliable, more useful state. What’s more, Airflow spreads the capacity to do this across your organization, thus creating a movement towards data excellence.
We believe that getting data into motion by building pipelines and refineries, running it through machine-learning algorithms, and processing it to drive dashboards is becoming the lifeblood of leading organizations. Most organizations struggle to run their data products in a uniform manner, which causes a mismatch between their talents and their requirements.
In automating your data pipelines, Apache Airflow allows your organization to turn its dormant assets into dynamic resources that provide a competitive edge.
Q: Is Apache Airflow: a tool to orchestrate big data right for me?
- Yes – if you are a data scientist
- Yes – if you are a data engineer
- Yes – if you want to bring data products to production
Q: What will I achieve by completing this training?
You will learn:
- A general introduction into Airflow, and the situations in which it will benefit you
- A hands-on exercise using Airflow, writing a DAG that interacts with components on Google Cloud, for instance BigQuery, DataFlow and Cloud Storage
- How to write customized operators, hooks and sensors using Airflow internals
You will gain hands-on experience in:
- The Airflow ecosystem
- Analyzing data types that are suitable for using Airflow
- Building and automating data pipelines
You will develop the skills to:
- Use and manage Apache Airflow
- Build and assess data pipelines
- Integrate Airflow seamlessly into your data landscape using custom operators
Q: What else should I know?
- In order to benefit from this course participants are expected to have general knowledge of Python programming concepts (e.g. be familiar with concepts mentioned under Basics on learnpython.org
- Knowledge of basic data engineering principles
- Bring a laptop with connectivity to the internet