このエピソードについて
Welcome back to Episode 8! Today, we're talking about data that simply cannot wait for an overnight batch run: Real-Time Intelligence.
Think about cases like live IoT factory sensors, server error logs, user clicks on a shopping app, or payment transactions. If you run a data pipeline only once an hour or
once a day, you will find out about problems way too late. You need to see and react to what is happening right now.
Fabric manages this live streaming data using two main tools:
1. Eventstreams: This is a simple, no-code way to catch data while it’s moving. You just point a feed from tools like Apache Kafka, Azure Event Hubs, or a
webhook directly into Fabric. You can filter out bad rows or clean incoming JSON data on the fly before it even hits the disk.
2. KQL Databases: This uses the Kusto engine—the same tech behind Azure Data Explorer. Standard SQL databases often choke when you dump millions of log
entries into them every minute. KQL is designed specifically to scan billions of timestamped rows in less than a second. You can write simple KQL queries, or if
you don't know KQL, you can even use regular SQL syntax to search your logs.
From there, you have two quick ways to use the data:
• Drop the feed directly into a live dashboard so numbers update on your screen in real time.
• Use Data Activator (Fabric Activator) to set up automated alerts—for example, sending a Teams ping or triggering a workflow the moment server errors cross a
danger limit.
And the best part? All of that incoming live data automatically settles into OneLake alongside your normal daily batch tables.
In Episode 9, we’ll look at how security, access permissions, and governance work with Microsoft Purview.
About the Host
Rajnish Pandey is a Senior Data Engineer with over decade of experience in the data industry. Through QueryZens, he shares practical insights, real-world experiences, and conversations around Data Engineering and modern data platforms.
Hosted by Rajnish Pandey | QueryZens
Making Data Engineering easier to understand, one conversation at a time.
