Enroll Now for latest Courses ENROLL NOW

Azure Data Engineering

Azure Data Engineering encompasses designing, implementing, and managing data solutions on Microsoft Azure, involving tasks like data ingestion, storage, processing, and analytics using various Azure services. Participants in the training gain hands-on experience in building scalable data pipelines, implementing data warehousing solutions, and performing advanced analytics tasks. Through lectures, exercises, and projects, attendees develop skills to architect, implement, and manage end-to-end data solutions on Azure, enabling them to tackle data engineering challenges effectively.

What You gain from Us

Python Programming

Azure Data Factory

Databricks

Curricula for Data Engineering

Cloud Basics (Coverage on AZ-900) 4hrs

▼

Introduction to Cloud (AWS/Azure/GCP)
Azure Data Engineering (Databricks,ADF, ADLS)
Core cloud concepts
Overview of cloud networking concepts
Lab on Cloud

SQL 8hrs

▼

Database vs Datawarehouse vs Data Lake
SQL basics (DDL/DML/DQL)
Advanced SQL (windowing-Rank,denserank,rownum etc)
lab on SQL

Python Programming 8hrs

▼

Data Types, Operators, Control statements, Functions, Modules, Lists, Tuples, Dictionaries, object oriented programming

Bigdata Basics 4hrs

▼

Introduction to Bigdata/Hadoop(Map Reduce).
Common file formats-(csv,json,xml,parquet,avro)
Rows vs Columnar Format.

Pyspark 6hrs

▼

Introduction to RDD/Dataset/Dataframes
Dataframe API in detail.
Commonly used dataframe methods(display/lit/withColumnRenamed/withColumn/col/cast).
Read/Write data in Pyspark.
User Defined Functions in Spark.
SparkSql

Databricks Delta 2hrs

▼

Datalakes
ACID properties & Time Travel
External and Managed Tables
Reading and Writing delta tables.
Lab on delta.

Databricks Workspace understanding 3hrs

▼

Creating an Azure Databricks Service
Creating a Databricks worspace in the Azure Portal
Databricks service using the Azure CLI(command-line interface)
Databricks service using Azure Resource Manager(ARM) templates
Ading users and groups to the workspace
Creating a cluster from the user interface(UI)
Getting started with notebooks and jobs Azure Databricks
Authenticating to Databricks using a PAT

Databricks Cluster Management 3hrs

▼

Creating and configuring clusters
Managing Clusters
Displaying clusters
Starting a cluster
Terminating a cluster
Delete a cluster
Cluster Information
Cluster logs
Cluster access control
Types of Clusters
All pupose clusters
Job cluster
Databricks Pools
Databricks without pools
Databricks with Pools
Clusters Mode
Standard
High Concurrency
Single Node
Autoscalling
Databricks runtime versions
Multiuser Clusters

Databricks notebook core functionalities 6hrs

▼

Creating and managing notebooks
Exporting notebooks
Importing notebooks
Attaching a notebook to a cluster
Spark environment variables
SparkContext(sc)
SQLContext/HiveContext(sqlContext)
SparkSession(spark)
Scheduling a notebook
Default Languege
Notebook permissions
Folder permissions
Cloning notebook
Renaming notebook

Databricks Utilities and Notebook Parameters 4hrs

▼

Dbutils commands on files, directories
Notebooks and libraries
Databricks Variables
Widget Types
Databricks notebook parameters

Working with batch processing in Databricks using SQL 3hrs

▼

Reading data
Checking row count
Selecting columns
Filtering data
Droping columns
Adding or replacing columns
Printing schema
Renaming a column
Droping duplicate rows
Limiting output rows
Sorting rows
Grouping data
Visualizing data
Writing data to a sink

Azure DataFactory 8hrs

▼

Integration Runtime
Linked Service
Data Set
Source and Sink
Activity- Copy,Conditional,Filter, Databricks Notebook,Lookup, Stored Procedure, Set Variable,Delete
Parameters
Data Flow
Pipeline
ADF Monitoring

Testing Tool: Great Expectations 2hrs

▼

Comparing two tables, Profiling on Tables, Need for tesing framework.

Project + Interview Preparation 4hrs

▼

MCQ's, Usecases, Project description.

Demo Project usecases and other technical discussions 5hrs

▼

SCD implementations, ETL usecases,Simple ML usecases.