The Bit Bucket

KQL databases are a specialized storage option in Fabric designed specifically for high-volume event and telemetry data. KQL stands for Kusto Query Language, which comes from the Azure Data Explorer (ADX) engine.

These databases are optimized to handle workloads where you might have billions of small events — like application logs, IoT telemetry, or time-series data — and you need to query them at speed.

Where a warehouse is optimized for structured, relational data and a lakehouse is great for mixing structured and semi-structured data, a KQL database shines when you need to scan and aggregate across massive volumes of events very quickly. You can run queries that look back over millions of log entries or thousands of IoT readings and get sub-second responses. That kind of responsiveness is what makes it possible to power real-time dashboards, alerting systems, and anomaly detection workflows.

2026-05-01

Increasingly, developers are using tools that try to automate code generation when dealing with databases. Stored procedures have been a thorn in the side of this. Mostly that’s because it’s difficult to obtain the metadata that is really needed.

You should not need to be able to read the code of a stored procedure to know how to use it. And that includes the common exceptions it might throw. None of the existing tools for dealing with this currently do what’s needed.

2026-04-30

Apache Kafka is one of the most widely used event streaming systems in the world, and for good reason. At its core, Kafka is a distributed, open-source platform that makes it possible to capture, process, and deliver millions of events per second with high reliability.

Kafka organizes data into topics, which you can think of as named channels. A producer writes events into a topic — this could be an application logging user activity, or a payment system recording transactions.

2026-04-29

I was doing some varied reading this morning and stumbled across an article by Paul Graham. I want to highlight this passage:

We now have several examples to prove that amateurs can surpass professionals, when they have the right kind of system to channel their efforts. Wikipedia may be the most famous. Experts have given Wikipedia middling reviews, but they miss the critical point: it’s good enough. And it’s free, which means people actually read it. On the web, articles you have to pay for might as well not exist. Even if you were willing to pay to read them yourself, you can’t link to them. They’re not part of the conversation.

2026-04-28

In most Eventstream transformations, we use the drag-and-drop UI — managing fields, filters, joins, and aggregations visually. But for more advanced scenarios, you can switch to Custom SQL Code mode.

This option allows you to write SQL-style expressions directly against your streaming data.

It’s particularly useful when the visual editor can’t express the full logic you need — for example, if you want to perform complex conditional transformations, apply multi-step calculations, or combine filters and aggregations in a single operation.

2026-04-27

A while back, I wrote about my experiences with trying to see if ChatGPT could answer questions as though it was a baseball umpire. It did quite poorly, and I came to the conclusion that the problem was what it learned from. Curiously, it got hard questions correct and easy questions wrong. It seemed to me that people who knew what they were talking about were the only ones to ever discuss the hard questions, while everyone had an opinion (often wrong) about the easy questions.

2026-04-26

Expanding is the opposite of aggregation. Instead of rolling up many events into a single summary, expansion takes a single complex event and splits it into multiple simpler events. This is really important when one incoming record carries more than one piece of information.

A good example is a log entry that contains multiple error codes in a single message. If we treat that as one event, we lose the ability to analyze each error code individually. By expanding, we break it into separate events — one per error code — which makes it much easier to count, group, and detect patterns across error types.

2026-04-25

On a client site some years back, I came across a situation (unfortunately too common) where a column in a table was being used for two purposes. It could either hold an integer value or a string. Only about 100 rows out of many millions had the integer value. Some of the client code needed to calculate the maximum value when it was an integer. First step I tried was to add a persisted calculated column like so:

2026-04-24

Another powerful transformation in real-time processing is the union. A union allows us to take multiple input event streams and merge them into a single, consolidated stream. This is especially useful when we have similar data sources that we want to treat consistently in downstream processing.

A common scenario is application logs. You might have several applications, each producing its own stream of log events. Instead of maintaining separate pipelines for each one, we can use a union to combine them all into a single log stream. From there, we can apply filters, mappings, or aggregations in a single place, which makes management much easier.

2026-04-23

I fell for this one this week. If you execute the following code in SQLCMD mode, what would you expect the output to be?

:SETVAR PrincipalServer WINSERVERBASE
:SETVAR MirrorServer WINSERVERBASE\\SQLDEV02
:SETVAR WitnessServer WINSERVERBASE\\SQLDEV03

:CONNECT $(PrincipalServer)
SELECT @@SERVERNAME;

:CONNECT $(MirrorServer)
SELECT @@SERVERNAME;

:CONNECT $(WitnessServer)
SELECT @@SERVERNAME;

I’m guessing you might not have expected:

WINSERVERBASE\SQLDEV03

WINSERVERBASE\SQLDEV03

WINSERVERBASE\SQLDEV03

The problem is the lack of a batch separator. What I should have written was this:

:SETVAR PrincipalServer WINSERVERBASE
:SETVAR MirrorServer WINSERVERBASE\\SQLDEV02
:SETVAR WitnessServer WINSERVERBASE\\SQLDEV03

:CONNECT $(PrincipalServer)
SELECT @@SERVERNAME;
GO

:CONNECT $(MirrorServer)
SELECT @@SERVERNAME;
GO

:CONNECT $(WitnessServer)
SELECT @@SERVERNAME;
GO

While this may be strictly correct, I can’t imagine it’s the behaviour anyone would wish for. Do you think that a :CONNECT statement in a SQLCMD batch should also be treated as a batch separator? Does it ever make sense for it not to?

2026-04-22