Microsoft-Fabric

When working with real-time data, it’s easy for your streams to become cluttered — especially as events come from multiple sources with different structures. That’s where the Manage Fields option becomes essential.

It gives you control over the shape of your data stream. You can choose which fields to keep and drop those you don’t need, which helps reduce noise and improves performance. For example, you might remove diagnostic fields or metadata columns that aren’t needed for your analytics.

2026-04-03

When we work with streaming data, one of the first transformations we often apply is filtering. The reality is that not every event is useful. In fact, in many scenarios, the vast majority of events are just background noise. Filtering gives us a way to narrow the stream down to only the events that actually matter for the business problem we’re solving.

Take IoT telemetry as an example. A device might send thousands of readings every hour, but if 99% of them show perfectly normal operating conditions, storing and analyzing them all just adds cost and complexity. By applying a filter, we could say: only keep the events where the temperature rises above 80 degrees Celsius, or only process readings where a vibration level exceeds a set threshold. This way, we’re focusing on signals of interest instead of wasting resources on noise.

2026-04-01

When we talk about ingestion modes in Fabric Real-Time Intelligence, we’re really thinking about how events move from their source into a destination, such as an Eventhouse table or another downstream system.

The pattern you choose affects latency, cost, data quality, and the flexibility of your real-time architecture. There are two broad models that students will encounter: direct ingestion and processing before ingestion.

Direct ingestion is the simplest path. Events arrive from a source such as IoT devices, applications, or an event broker, and they are immediately written into the target system without any intermediate shaping. This mode gives the lowest latency because nothing happens in between. It is most useful when you want to preserve raw events for later analysis, replay, or transformations that happen downstream. It is also the right choice when your first priority is freshness and the consumers are able to handle any necessary cleaning or shaping themselves.

2026-03-30

Event Schema Sets in Fabric Real-Time Intelligence are essentially a way to standardize the shape of the data coming into your real-time environment. When events are flowing in from a variety of sources—such as IoT devices, applications, APIs, or logs—you typically see a lot of variation: different fields, inconsistent casing, unexpected nested structures, or additional attributes that drift over time. Schema Sets give you a central place to define the expected structure for those events.

2026-03-28

One of the biggest differences between batch ETL pipelines and real-time pipelines is how you manage data quality. In a batch world, you often have long, multi-step processes that validate, clean, and enrich data before it ever reaches your reports. You can afford those extra passes because the data isn’t needed instantly.

In a real-time system, you don’t have that luxury. Events arrive continuously, and you need to deal with problems on the fly. That means data quality checks have to be fast, lightweight, and built directly into your ingestion or processing stage.

2026-03-26

While on site this week, another common problem that I see everywhere arose again.

When you need to format currency, you use the “c” format right? It’s in nearly every set of course materials I’ve ever seen. And people do it in almost every demonstration.

But so often, that’s wrong!

When you do this, you’re telling the system to display the monetary value using the local currency.

Is that correct though?

2026-03-21

Another key part of event processing in Fabric is mapping. Mapping is all about shaping the raw events into the structure you actually want to work with downstream.

When data first arrives, it often comes in the schema defined by the producer system. That might not match what your analytics tools, your warehouse, or your business users expect. For example, a device might send a field called ’tempC’ when you really want it named ‘TemperatureCelsius’.

2026-03-20

When working with real-time data, one of the biggest challenges is signal versus noise. Not every event that arrives is valuable for analysis or action. For example, IoT devices may send thousands of telemetry points per second, but only a small fraction of those actually represent unusual or meaningful behavior.

That’s where filtering comes in. Filtering lets us apply simple conditions to events right at the ingestion or processing stage. For instance, imagine we have a stream of temperature readings coming from industrial sensors. Most readings might sit between 20 and 50 degrees Celsius — perfectly normal. But maybe we only care if the temperature goes above 80°C, because that indicates a possible overheating issue. With a filter, we can discard all the normal events and only pass through the ones that require attention.

2026-03-18

Once events are flowing through an Eventstream, the next decision is: where should they go? This is where outputs come into play.

Fabric supports several key destinations. You can send events into a Lakehouse, which is ideal for combining real-time streams with historical data and keeping a permanent record for later analysis. You can push data into a Warehouse for structured reporting and BI queries. Or you can use a KQL database if your focus is on fast, interactive queries over logs, telemetry, or time-series data.

2026-03-16

When we look at event processing inputs, the first thing to know is that Fabric supports a broad range of streaming sources. The big four are Kafka, Azure Event Hubs, Azure IoT Hub, and any system that speaks AMQP. These cover most of the event-driven architectures you’ll see in the real world, from enterprise message brokers to IoT device fleets and large-scale cloud-native streaming pipelines.

Another important point is that inputs can come from both cloud and on-premises environments. Many organizations are in hybrid mode — perhaps you’ve got a Kafka cluster still running in your datacenter, while also using Event Hubs in Azure for new workloads. Fabric Eventstreams can connect to both, allowing you to bring all those events into a unified pipeline without needing to modernize everything at once.

2026-03-14