Workflow Orchestration with Cloud Composer

Workflow Orchestration with Cloud ComposerWOCCGOGoogleGO-WOCC1.0<ul> <li>Explore Apache Airflow and Cloud Composer as workflow orchestration solutions.</li><li>Create and manage Airflow DAGs following best practices.</li><li>Test and debug Airflow DAGs.</li><li>Monitor and observe Airflow DAGs on Cloud Composer</li></ul>Completion of "<a class="fl-href-prod" href="/swisscom/fr/course/google-bbdp"><svg role="img" aria-hidden="true" focusable="false" data-nosnippet class="cms-linkmark"><use xlink:href="/css/img/icnset-linkmarks.svg#linkmark"></use></svg>Building Batch Data Pipelines on Google Cloud (BBDP)</a>" or equivalent knowledge of data analytics and engineering on Google Cloud.Data Engineers who wish to learn how to use Apache Airflow and Cloud Composer to orchestrate their data engineering workflows.<h4>Module 1 - Introduction to Cloud Composer</h4> Topics <ul> <li>Data Engineer's need for Workflow Orchestration</li><li>Introduction to Apache Airflow</li><li>Cloud Composer</li><li>Environment Setup</li><li>Using the Composer and Airflow</li></ul> Objectives <ul> <li>Explore Apache Airflow and Cloud Composer.</li><li>Provision Cloud Composer instances.</li><li>Explore the Airflow and Composer UIs.</li></ul> Activities <ul> <li>Lab: Provisioning Cloud Composer</li></ul><h4>Module 2 - Creating and managing DAGs</h4> Topics <ul> <li>DAG structure and best practices</li><li>Common operators</li><li>Dependencies, trigger rules, and flow control</li><li>Integration of Airflow and Google Cloud Services</li></ul> Objectives <ul> <li>Write DAGs.</li><li>Explore common Airflow operators.</li><li>Manage triggers, dependencies, and flow control.</li><li>Integrate Airflow with Google Cloud Services.</li></ul> Activities <ul> <li>Lab: Assembling a Data Processing Workflow</li></ul><h4>Module 3 - Advanced Airflow techniques and best practices</h4> Topics <ul> <li>Advanced Airflow features</li><li>Debugging DAGs</li><li>Performance and scalability</li><li>Security and Access Control</li><li>Observability and monitoring</li></ul> Objectives <ul> <li>Leverage advanced Airflow features.</li><li>Debug DAGs.</li><li>Observe and monitor your running DAGs</li></ul> Activities <ul> <li>Lab: Extending and Monitoring DAGs</li></ul>- Explore Apache Airflow and Cloud Composer as workflow orchestration solutions. - Create and manage Airflow DAGs following best practices. - Test and debug Airflow DAGs. - Monitor and observe Airflow DAGs on Cloud ComposerCompletion of "Building Batch Data Pipelines on Google Cloud (BBDP)" or equivalent knowledge of data analytics and engineering on Google Cloud.Data Engineers who wish to learn how to use Apache Airflow and Cloud Composer to orchestrate their data engineering workflows.Module 1 - Introduction to Cloud Composer Topics - Data Engineer's need for Workflow Orchestration - Introduction to Apache Airflow - Cloud Composer - Environment Setup - Using the Composer and Airflow Objectives - Explore Apache Airflow and Cloud Composer. - Provision Cloud Composer instances. - Explore the Airflow and Composer UIs. Activities - Lab: Provisioning Cloud Composer Module 2 - Creating and managing DAGs Topics - DAG structure and best practices - Common operators - Dependencies, trigger rules, and flow control - Integration of Airflow and Google Cloud Services Objectives - Write DAGs. - Explore common Airflow operators. - Manage triggers, dependencies, and flow control. - Integrate Airflow with Google Cloud Services. Activities - Lab: Assembling a Data Processing Workflow Module 3 - Advanced Airflow techniques and best practices Topics - Advanced Airflow features - Debugging DAGs - Performance and scalability - Security and Access Control - Observability and monitoring Objectives - Leverage advanced Airflow features. - Debug DAGs. - Observe and monitor your running DAGs Activities - Lab: Extending and Monitoring DAGs1 jour595.00820.00660.00650.00950.00950.00950.00790.00950.00950.00