Hands on lab for real time analytics in Azure.
This is a one day hands-on lab on Real Time Analytics in Azure.
Azure services used in this lab:
- Azure Data Explorer
- Azure Event Hub
- Azure Stream Analytics
- Power BI
- Azure SQL Database
- Azure Monitor
Getting familiar with Kusto, Azure Data Explorer query language. We suggest the following resources:
Look at the business scenario and requirements of the hands on lab.
Break into teams of 3 to 8 individuals to design a target solution on Azure.
The solution should address each of the business requirements from the previous section. Draw a diagram with the main components.
Each team will then present their solution.
The instructor will present the prefered solution.
The hands on lab modules are based on that solution.
Each module contains a challenge and objectives.
They also contain a suggested solution.
We recommend trying the modules without looking at the suggested solution first. This would maximize the challenges and therefore the benefits.
In this first module, we'll explore the data we are going to work with.
Objectives:
- Provision an Azure Data Explorer (Kusto) cluster
- Query telemetry samples
- Prepare ingestion by developing queries to transform the raw data
Go to module 1 instructions.
In this module we will setup the real time ingestion of data into our Kusto Cluster.
Objectives:
- Setup a simulator of IoT data for Azure Event Hub
- Continuously ingest raw data in Azure Data Explorer
Go to module 2 instructions.
In this module we will take the raw JSON data we setup for continuous ingestion in Module 2 and continuously transform it into strongly-typed data, using the queries we developed in module 1.
Objectives:
- Get strong-type table populated in near-real time
Go to module 3 instructions.
Now that we have data ingested in real time, we are going to query it to get insights.
Objectives:
- Look at ingestion latency
- Query and chart data
- Look at Azure Data Explorer metrics (monitoring)
Go to module 4 instructions.
In this module, we'll query to the next level. We'll look at each drone's telemetry as its own time series and try to find anomalies.
Objectives:
- Get comfortable with time series
- Analyse failures in drones using time series tooling
Go to module 5 instructions.
- Changing upstream process to reveal "real" telemetry with late arrivals + duplicates
- Introduce ASA to the rescue
In this module we are going to integrate external data from Azure SQL DB to enrich the ingested data.
- Ingest a couple of reference data tables (or reference?)
- Author update policies to transform the data using reference tables
- Query some more
- Setup continuous exporting
Go to module 5 instructions.
- Setup Power BI to query near real time data