The Bit Bucket

In most Eventstream transformations, we use the drag-and-drop UI — managing fields, filters, joins, and aggregations visually. But for more advanced scenarios, you can switch to Custom SQL Code mode.

This option allows you to write SQL-style expressions directly against your streaming data.

It’s particularly useful when the visual editor can’t express the full logic you need — for example, if you want to perform complex conditional transformations, apply multi-step calculations, or combine filters and aggregations in a single operation.

2026-04-27

A while back, I wrote about my experiences with trying to see if ChatGPT could answer questions as though it was a baseball umpire. It did quite poorly, and I came to the conclusion that the problem was what it learned from. Curiously, it got hard questions correct and easy questions wrong. It seemed to me that people who knew what they were talking about were the only ones to ever discuss the hard questions, while everyone had an opinion (often wrong) about the easy questions.

2026-04-26

Expanding is the opposite of aggregation. Instead of rolling up many events into a single summary, expansion takes a single complex event and splits it into multiple simpler events. This is really important when one incoming record carries more than one piece of information.

A good example is a log entry that contains multiple error codes in a single message. If we treat that as one event, we lose the ability to analyze each error code individually. By expanding, we break it into separate events — one per error code — which makes it much easier to count, group, and detect patterns across error types.

2026-04-25

On a client site some years back, I came across a situation (unfortunately too common) where a column in a table was being used for two purposes. It could either hold an integer value or a string. Only about 100 rows out of many millions had the integer value. Some of the client code needed to calculate the maximum value when it was an integer. First step I tried was to add a persisted calculated column like so:

2026-04-24

Another powerful transformation in real-time processing is the union. A union allows us to take multiple input event streams and merge them into a single, consolidated stream. This is especially useful when we have similar data sources that we want to treat consistently in downstream processing.

A common scenario is application logs. You might have several applications, each producing its own stream of log events. Instead of maintaining separate pipelines for each one, we can use a union to combine them all into a single log stream. From there, we can apply filters, mappings, or aggregations in a single place, which makes management much easier.

2026-04-23

I fell for this one this week. If you execute the following code in SQLCMD mode, what would you expect the output to be?

:SETVAR PrincipalServer WINSERVERBASE
:SETVAR MirrorServer WINSERVERBASE\\SQLDEV02
:SETVAR WitnessServer WINSERVERBASE\\SQLDEV03

:CONNECT $(PrincipalServer)
SELECT @@SERVERNAME;

:CONNECT $(MirrorServer)
SELECT @@SERVERNAME;

:CONNECT $(WitnessServer)
SELECT @@SERVERNAME;

I’m guessing you might not have expected:

WINSERVERBASE\SQLDEV03

WINSERVERBASE\SQLDEV03

WINSERVERBASE\SQLDEV03

The problem is the lack of a batch separator. What I should have written was this:

:SETVAR PrincipalServer WINSERVERBASE
:SETVAR MirrorServer WINSERVERBASE\\SQLDEV02
:SETVAR WitnessServer WINSERVERBASE\\SQLDEV03

:CONNECT $(PrincipalServer)
SELECT @@SERVERNAME;
GO

:CONNECT $(MirrorServer)
SELECT @@SERVERNAME;
GO

:CONNECT $(WitnessServer)
SELECT @@SERVERNAME;
GO

While this may be strictly correct, I can’t imagine it’s the behaviour anyone would wish for. Do you think that a :CONNECT statement in a SQLCMD batch should also be treated as a batch separator? Does it ever make sense for it not to?

2026-04-22

A snapshot transformation is about capturing the current state of something from a continuous stream of events. Instead of storing every single historical record, we keep just the most recent update for each entity.

Think of IoT devices as an example. Each device may send a temperature reading every few seconds. Over time, that becomes a huge stream of data. But often, what the business really wants to know is simple: what’s the latest temperature reading from each device right now? That’s where a snapshot comes in. It continuously updates a table of ‘current values’ as new events arrive, so at any given moment, you can query the most recent state without reprocessing the entire history.

2026-04-21

A while back, I managed to catch the tail end of the reptiles series that Sir David Attenborough created. If you have a spare 3 1/2 minutes, take a look at the end of his Life in Cold Blood series episode about crocodiles.

People seem to think crocodiles are cold, unintelligent eating machines. I have a major respect for them. They’ve been around since the time of the dinosaurs, and anything that can do that, is doing something right. In fact, until we arrived with guns, etc. they really didn’t have much to worry about. They were a top-level predator.

2026-04-20

A hopping window is a flexible windowing strategy that combines elements of both tumbling and sliding. Like tumbling, hopping windows have a fixed length — for example, 10 minutes. But unlike tumbling, they are allowed to overlap, because they advance by a smaller step size than their length.

For example, let’s say we configure a 10-minute window that hops forward every 5 minutes. That means between 12:00 and 12:10, you get one window, and between 12:05 and 12:15, you get another. Each window is 10 minutes long, but because they’re starting 5 minutes apart, they overlap. This means that every event can contribute to more than one window.

2026-04-19

I often work with reports either in SQL Server Reporting Services (SSRS or now Power BI Reporting Services) and Fabric Paginated Reports. At a site this week, the complexity of the reports we were working on reminded me that I really, really don’t like seeing T-SQL code (or really any complex business logic) embedded in reports.

Don’t make refactoring difficult or impossible

DBAs tend to be considered a conservative bunch. One thing they’re usually conservative about is refactoring their databases. In many cases, this is because they have little idea what they will break when they make database changes.

2026-04-18