Introduction to Data Engineering on Google Cloud

Introduction to Data Engineering on Google CloudIDEGGOGoogleGO-IDEG1.0<ul> <li>Understand the role of a data engineer.</li><li>Identify data engineering tasks and core components used on Google Cloud.</li><li>Understand how to create and deploy data pipelines of varying patterns on Google Cloud.</li><li>Identify and utilize various automation techniques on Google Cloud.</li></ul><ul> <li>Prior Google Cloud experience at the fundamental level using Cloud Shell and accessing products from the Google Cloud console.</li><li>Basic proficiency with a common query language such as SQL.</li><li>Experience with data modeling and ETL (extract, transform, load) activities.</li><li>Experience developing applications using a common programming language such as Python.</li></ul><ul> <li>Data engineers</li><li>Database administrators</li><li>System administrators</li></ul><h4>Module 1 - Data Engineering Tasks and Components</h4> Topics: <ul> <li>The role of a data engineer</li><li>Data sources versus data sinks</li><li>Data formats</li><li>Storage solution options on Google Cloud</li><li>Metadata management options on Google Cloud</li><li>Sharing datasets using Analytics Hub</li></ul>Objectives: <ul> <li>Explain the role of a data engineer.</li><li>Understand the differences between a data source and a data sink.</li><li>Explain the different types of data formats.</li><li>Explain the storage solution options on Google Cloud.</li><li>Learn about the metadata management options on Google Cloud.</li><li>Understand how to share datasets with ease using Analytics Hub.</li><li>Understand how to load data into BigQuery using the Google Cloud console or the gcloud CLI.</li></ul>Activities: <ul> <li>Lab: Loading Data into BigQuery</li><li>Quiz</li></ul><h4>Module 2 - Data Replication and Migration</h4> Topics: <ul> <li>Replication and migration architecture</li><li>The gcloud command-line tool</li><li>Moving datasets</li><li>Datastream</li></ul>Objectives: <ul> <li>Explain the baseline Google Cloud data replication and migration architecture.</li><li>Understand the options and use cases for the gcloud command-line tool.</li><li>Explain the functionality and use cases for Storage Transfer Service.</li><li>Explain the functionality and use cases for Transfer Appliance.</li><li>Understand the features and deployment of Datastream.</li></ul>Activities: <ul> <li>Lab: Datastream: PostgreSQL Replication to BigQuery (optional for ILT)</li><li>Quiz</li></ul><h4>Module 3 - The Extract and Load Data Pipeline Pattern</h4> Topics: <ul> <li>Extract and load architecture</li><li>The bq command-line tool</li><li>BigQuery Data Transfer Service</li><li>BigLake</li></ul>Objectives: <ul> <li>Explain the baseline extract and load architecture diagram.</li><li>Understand the options of the bq command-line tool.</li><li>Explain the functionality and use cases for BigQuery Data Transfer Service.</li><li>Explain the functionality and use cases for BigLake as a non-extract-load pattern.</li></ul>Activities: <ul> <li>Lab: BigLake: Qwik Start</li><li>Quiz</li></ul><h4>Module 4 - The Extract, Load, and Transform Data Pipeline Pattern</h4> Topics: <ul> <li>Extract, load, and transform (ELT) architecture</li><li>SQL scripting and scheduling with BigQuery</li><li>Dataform</li></ul>Objectives: <ul> <li>Explain the baseline extract, load, and transform architecture diagram.</li><li>Understand a common ELT pipeline on Google Cloud.</li><li>Learn about BigQuery’s SQL scripting and scheduling capabilities.</li><li>Explain the functionality and use cases for Dataform.</li></ul>Activities: <ul> <li>Lab: Create and Execute a SQL Workflow in Dataform</li><li>Quiz</li></ul><h4>Module 5 - The Extract, Transform, and Load Data Pipeline Pattern</h4> Topics: <ul> <li>Extract, transform, and load (ETL) architecture</li><li>Google Cloud GUI tools for ETL data pipelines</li><li>Batch data processing using Dataproc</li><li>Streaming data processing options</li><li>Bigtable and data pipelines</li></ul>Objectives: <ul> <li>Explain the baseline extract, transform, and load architecture diagram.</li><li>Learn about the GUI tools on Google Cloud used for ETL data pipelines.</li><li>Explain batch data processing using Dataproc.</li><li>Learn how to use Dataproc Serverless for Spark for ETL.</li><li>Explain streaming data processing options.</li><li>Explain the role Bigtable plays in data pipelines.</li></ul>Activities: <ul> <li>Lab: Use Dataproc Serverless for Spark to Load BigQuery (optional for ILT)</li><li>Lab: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow</li><li>Quiz</li></ul><h4>Module 6 - Automation Techniques</h4> Topics: <ul> <li>Automation patterns and options for pipelines</li><li>Cloud Scheduler and Workflows</li><li>Cloud Composer</li><li>Cloud Run Functions</li><li>Eventarc</li></ul>Objectives: <ul> <li>Explain the automation patterns and options available for pipelines.</li><li>Learn about Cloud Scheduler and Workflows.</li><li>Learn about Cloud Composer.</li><li>Learn about Cloud Run functions.</li><li>Explain the functionality and automation use cases for Eventarc.</li></ul>Activities: <ul> <li>Lab: Use Cloud Run Functions to Load BigQuery (optional for ILT)</li><li>Quiz</li></ul>- Understand the role of a data engineer. - Identify data engineering tasks and core components used on Google Cloud. - Understand how to create and deploy data pipelines of varying patterns on Google Cloud. - Identify and utilize various automation techniques on Google Cloud.- Prior Google Cloud experience at the fundamental level using Cloud Shell and accessing products from the Google Cloud console. - Basic proficiency with a common query language such as SQL. - Experience with data modeling and ETL (extract, transform, load) activities. - Experience developing applications using a common programming language such as Python.- Data engineers - Database administrators - System administratorsModule 1 - Data Engineering Tasks and Components Topics: - The role of a data engineer - Data sources versus data sinks - Data formats - Storage solution options on Google Cloud - Metadata management options on Google Cloud - Sharing datasets using Analytics Hub Objectives: - Explain the role of a data engineer. - Understand the differences between a data source and a data sink. - Explain the different types of data formats. - Explain the storage solution options on Google Cloud. - Learn about the metadata management options on Google Cloud. - Understand how to share datasets with ease using Analytics Hub. - Understand how to load data into BigQuery using the Google Cloud console or the gcloud CLI. Activities: - Lab: Loading Data into BigQuery - Quiz Module 2 - Data Replication and Migration Topics: - Replication and migration architecture - The gcloud command-line tool - Moving datasets - Datastream Objectives: - Explain the baseline Google Cloud data replication and migration architecture. - Understand the options and use cases for the gcloud command-line tool. - Explain the functionality and use cases for Storage Transfer Service. - Explain the functionality and use cases for Transfer Appliance. - Understand the features and deployment of Datastream. Activities: - Lab: Datastream: PostgreSQL Replication to BigQuery (optional for ILT) - Quiz Module 3 - The Extract and Load Data Pipeline Pattern Topics: - Extract and load architecture - The bq command-line tool - BigQuery Data Transfer Service - BigLake Objectives: - Explain the baseline extract and load architecture diagram. - Understand the options of the bq command-line tool. - Explain the functionality and use cases for BigQuery Data Transfer Service. - Explain the functionality and use cases for BigLake as a non-extract-load pattern. Activities: - Lab: BigLake: Qwik Start - Quiz Module 4 - The Extract, Load, and Transform Data Pipeline Pattern Topics: - Extract, load, and transform (ELT) architecture - SQL scripting and scheduling with BigQuery - Dataform Objectives: - Explain the baseline extract, load, and transform architecture diagram. - Understand a common ELT pipeline on Google Cloud. - Learn about BigQuery’s SQL scripting and scheduling capabilities. - Explain the functionality and use cases for Dataform. Activities: - Lab: Create and Execute a SQL Workflow in Dataform - Quiz Module 5 - The Extract, Transform, and Load Data Pipeline Pattern Topics: - Extract, transform, and load (ETL) architecture - Google Cloud GUI tools for ETL data pipelines - Batch data processing using Dataproc - Streaming data processing options - Bigtable and data pipelines Objectives: - Explain the baseline extract, transform, and load architecture diagram. - Learn about the GUI tools on Google Cloud used for ETL data pipelines. - Explain batch data processing using Dataproc. - Learn how to use Dataproc Serverless for Spark for ETL. - Explain streaming data processing options. - Explain the role Bigtable plays in data pipelines. Activities: - Lab: Use Dataproc Serverless for Spark to Load BigQuery (optional for ILT) - Lab: Creating a Streaming Data Pipeline for a Real-Time Dashboard with Dataflow - Quiz Module 6 - Automation Techniques Topics: - Automation patterns and options for pipelines - Cloud Scheduler and Workflows - Cloud Composer - Cloud Run Functions - Eventarc Objectives: - Explain the automation patterns and options available for pipelines. - Learn about Cloud Scheduler and Workflows. - Learn about Cloud Composer. - Learn about Cloud Run functions. - Explain the functionality and automation use cases for Eventarc. Activities: - Lab: Use Cloud Run Functions to Load BigQuery (optional for ILT) - Quiz1 day595.00660.00650.00820.00950.00950.00950.00790.00950.00