The Bit Bucket

One of the biggest differences between batch ETL pipelines and real-time pipelines is how you manage data quality. In a batch world, you often have long, multi-step processes that validate, clean, and enrich data before it ever reaches your reports. You can afford those extra passes because the data isn’t needed instantly.

In a real-time system, you don’t have that luxury. Events arrive continuously, and you need to deal with problems on the fly. That means data quality checks have to be fast, lightweight, and built directly into your ingestion or processing stage.

2026-03-26

Whenever we’re talking about an amount of data, it’s important to understand the units that are used. In all the early days of computing, it all seemed pretty simple. We had KB for 1024 bytes, MB for 1024 * 1024 bytes, etc.

The first people I saw messing that up were the hard drive manufacturers. Originally, they followed the standard units that we had been using in computing. But somewhere along the way, they changed how this worked. The vendors decided that if they had 10,000,000 bytes of storage, they would call that a 10MB hard drive, but of course it wasn’t, at least not in how we used to measure them. Some of the vendors even started talking about hard drive megabytes like it was some other new unit. That meant that suddenly a 128MB hard drive (128 * 1024 * 1024 * 1024 or 137,438,953,472 bytes) became a 137MB hard drive.

2026-03-25

Show 3 with Guest Dr Neil Roodyn

2026-03-24

I was part of a discussion the other day where the topic was the most misleading error message you’ve ever seen. I’ve been in the industry long enough that it’s a pretty tall list of error messages that I need to consider.

The winner for me

But I finally decided on one:

Back in the VB6 days, there was a common error message that said Out of Memory.

There were many issues that could lead to that error message, but running out of memory was probably the least likely.

2026-03-23

I’ve previously written about how I like to always have a certification exam that I’m working on.

Recently, I saw the new AB-731 exam for AI Transformation Leader. It’s the only exam required for the Microsoft Certified: AI Transformation Leader certification.

What interested me is that it was one of the first exams that are now designed to target the business people who are involved in making decisions within an organization. In this case, it’s about deciding how AI might impact the organization, how to plan and carry out the transformation, and which tools to use.

2026-03-22

While on site this week, another common problem that I see everywhere arose again.

When you need to format currency, you use the “c” format right? It’s in nearly every set of course materials I’ve ever seen. And people do it in almost every demonstration.

But so often, that’s wrong!

When you do this, you’re telling the system to display the monetary value using the local currency.

Is that correct though?

2026-03-21

Another key part of event processing in Fabric is mapping. Mapping is all about shaping the raw events into the structure you actually want to work with downstream.

When data first arrives, it often comes in the schema defined by the producer system. That might not match what your analytics tools, your warehouse, or your business users expect. For example, a device might send a field called ’tempC’ when you really want it named ‘TemperatureCelsius’.

2026-03-20

Show 2 with Guest Steve Stedman

2026-03-19

When working with real-time data, one of the biggest challenges is signal versus noise. Not every event that arrives is valuable for analysis or action. For example, IoT devices may send thousands of telemetry points per second, but only a small fraction of those actually represent unusual or meaningful behavior.

That’s where filtering comes in. Filtering lets us apply simple conditions to events right at the ingestion or processing stage. For instance, imagine we have a stream of temperature readings coming from industrial sensors. Most readings might sit between 20 and 50 degrees Celsius — perfectly normal. But maybe we only care if the temperature goes above 80°C, because that indicates a possible overheating issue. With a filter, we can discard all the normal events and only pass through the ones that require attention.

2026-03-18

Over the years, I’ve seen so many discussions regarding the certification process and exams. I’ve seen posts from many people that are very experienced with products saying they can’t see any point in the certification exams and also argue that competencies in the Microsoft Partner program shouldn’t be based on exams. They feel these people should somehow just be recognised for their other contributions.

Grandfathering

Regarding the certification process, I don’t agree that anyone should be just grandfathered in. Any of the people that have a great deal of knowledge and experience really should be able to just take the exams and be done with it.

2026-03-17