The Bit Bucket

When it comes to designing an Eventstream pipeline in Fabric, the process generally follows a clear, three-step flow. First, you start with the inputs — the data sources. This might include Kafka, Event Hubs, IoT Hub, or other streaming systems. At this stage, you define the connections and schemas so Fabric knows how to interpret incoming events.

The second step is where you apply transformations. These are the operations that make raw data more usable and more valuable. You might apply filtering to reduce noise and drop irrelevant events. You might use mapping to rename fields, adjust types, or flatten JSON into something cleaner. And you might use routing to branch different types of events to different destinations. Together, these transformations ensure that events are shaped, cleaned, and directed properly before they move downstream.

2026-05-05

It was great to catch up with Jess Pomfret today and to have her on a SQL Down Under podcast.

Jess is a Data Platform Engineer and a dual Microsoft MVP. She started working with SQL Server in 2011, and she says she enjoys the problem-solving aspects of automating processes with PowerShell.

Jess also enjoys contributing to dbatools and dbachecks, two open source PowerShell modules that aid DBAs with automating the management of SQL Server instances.

2026-05-05

I’ve recently been talking to clients about SQL CLR objects. When these were first introduced in SQL Server 2005, many of us had high hopes for them. SQL Server has never been great in regard to extensibility and this provided some way to extend the product.

Nowadays, I avoid SQL CLR. And that’s a real pity. But it’s no longer supported in Azure SQL Database, apart from the system CLR objects of geometry, geography, and hierarchyid. (Note: I’m also not a fan of hierarchyid). I need to use extensibility methods that are available in the different environments that I work in, and Azure SQL Database is one of those. The same applies to Fabric SQL Database.

2026-05-04

Routing is one of the features that makes Eventstreams especially powerful in Fabric. It gives you fine-grained control over where different types of events end up. Instead of building separate ingestion pipelines for every scenario, you can manage multiple flows in one stream and branch the data based on rules you define.

For example, imagine a company that handles both IoT telemetry and financial transactions. IoT data, like sensor readings, is most valuable in a KQL database, where you can run fast time-series analysis and detect anomalies. At the same time, financial transactions are better suited to a warehouse, where structured schemas and BI reporting tools can shine. With routing, both event types can enter the same Eventstream, but the system automatically directs them to the most appropriate destination.

2026-05-03

I was wondering today how often EULAs (end user license agreements) get tested in courts, and in particular, EULAs that appear in consumer-grade applications.

While most sound quite official, it’s hard to imagine most of them being very enforceable. Does anyone EVER read them?

Fair Cop !

I was amused a few years back when I was installing an application, clicked over the EULA and the application said “how could you possibly have read that in 1.076 seconds?”. Yep, got me there; that’s a fair cop.

2026-05-02

KQL databases are a specialized storage option in Fabric designed specifically for high-volume event and telemetry data. KQL stands for Kusto Query Language, which comes from the Azure Data Explorer (ADX) engine.

These databases are optimized to handle workloads where you might have billions of small events — like application logs, IoT telemetry, or time-series data — and you need to query them at speed.

Where a warehouse is optimized for structured, relational data and a lakehouse is great for mixing structured and semi-structured data, a KQL database shines when you need to scan and aggregate across massive volumes of events very quickly. You can run queries that look back over millions of log entries or thousands of IoT readings and get sub-second responses. That kind of responsiveness is what makes it possible to power real-time dashboards, alerting systems, and anomaly detection workflows.

2026-05-01

Increasingly, developers are using tools that try to automate code generation when dealing with databases. Stored procedures have been a thorn in the side of this. Mostly that’s because it’s difficult to obtain the metadata that is really needed.

You should not need to be able to read the code of a stored procedure to know how to use it. And that includes the common exceptions it might throw. None of the existing tools for dealing with this currently do what’s needed.

2026-04-30

Apache Kafka is one of the most widely used event streaming systems in the world, and for good reason. At its core, Kafka is a distributed, open-source platform that makes it possible to capture, process, and deliver millions of events per second with high reliability.

Kafka organizes data into topics, which you can think of as named channels. A producer writes events into a topic — this could be an application logging user activity, or a payment system recording transactions.

2026-04-29

I was doing some varied reading this morning and stumbled across an article by Paul Graham. I want to highlight this passage:

We now have several examples to prove that amateurs can surpass professionals, when they have the right kind of system to channel their efforts. Wikipedia may be the most famous. Experts have given Wikipedia middling reviews, but they miss the critical point: it’s good enough. And it’s free, which means people actually read it. On the web, articles you have to pay for might as well not exist. Even if you were willing to pay to read them yourself, you can’t link to them. They’re not part of the conversation.

2026-04-28

In most Eventstream transformations, we use the drag-and-drop UI — managing fields, filters, joins, and aggregations visually. But for more advanced scenarios, you can switch to Custom SQL Code mode.

This option allows you to write SQL-style expressions directly against your streaming data.

It’s particularly useful when the visual editor can’t express the full logic you need — for example, if you want to perform complex conditional transformations, apply multi-step calculations, or combine filters and aggregations in a single operation.

2026-04-27