The Bit Bucket

When designing real-time data systems, it’s important to consider the cost differences between streaming and batch workloads.

Streaming pipelines tend to be more expensive because they require always-on compute — data is constantly being ingested, transformed, and monitored, even during quieter periods. That means the compute capacity behind your Eventstreams, KQL queries, or Activator rules is active all the time.

Batch processing, on the other hand, is usually cheaper and easier to control. It runs on a schedule — for example, hourly or nightly — so compute resources are only used when the job is running. This makes batch ideal for large volumes of data that don’t need to be processed instantly.

2026-08-21

I’ve spent some time lately reading some older and more famous books. One that seemed fascinating was a short story by James Joyce called The Dead .

Author

James Joyce was an Irish novelist and poet. He was also well-known as a literary critic. Wikipedia says he is regarded as one of the most influential and important writers of the 20th century. I’ve spent time in Ireland over the years, and I’ve seen a lot of tributes to him both up in Dublin, and also around Kildare. He’s treated as somewhat of a national treasure.

2026-08-20

Even more SQL love !

Another popular SQL dialect is for DuckDB. And we’ve just completed our first course using it.

Creating reports, analytics, or applications? And need to get data out of DuckDB? Or are you building applications using DuckDB.

Learn to write SQL queries like a pro !

We now have very popular SQL courses, for all the most important SQL dialects. There’s our flagship course for T-SQL, then already courses for PostgreSQL, Snowflake, Oracle, DB2, MySQL, Azure HorizonDB, and Google BigQuery. We’ve just added our new course Writing SQL Queries for DuckDB and you can enrol in it now. It’s just $95 USD.

2026-08-20

Governance and compliance are critical considerations in real-time systems — they apply every bit as much as they do in traditional batch workloads. The difference is that in streaming environments, everything happens continuously, so your controls and monitoring must operate in real time as well.

The first step is to track data lineage — understanding where each event originated, how it was transformed, and what downstream systems or actions it affected. Lineage visibility helps you answer questions like Where did this number come from? or Which events led to this automated decision? It also supports root cause analysis when problems occur.

2026-08-19

SQL Server databases have a reputation for being hard to test, or at least hard to test appropriately.

For good testing, and particularly for unit tests, you really want the following:

• Database in a known state before each test
• Database containing large amounts of (preferably masked) data (production-sized)
• Quick restore after each test before the next test

For most databases, this is hard to achieve. The restore after each test means that a normal database restore can’t be used. What I often see instead, is people using transactions to try to achieve this i.e. the process becomes:

2026-08-18

Security and governance are just as important in real-time streaming systems as they are in traditional batch pipelines — and in some cases, even more so. Because streaming data is continuous, a single misconfiguration can expose sensitive data for long periods before it’s noticed.

The first principle is to secure event sources and destinations.

Every data connection — whether it’s an IoT device, API, or event broker — should use authentication to verify its identity and encryption in transit to protect data as it moves. In Microsoft Fabric, this typically means using TLS for secure transmission and managed identities for authentication, avoiding hard-coded credentials wherever they are available.

2026-08-17

Many years back, I was invited to chair a course accreditation panel for a local TAFE (Technical and Further Education) course. They had started to offer a computing-related 3-year diploma, and the hope was that it wasn’t too far below the 3-year degrees offered at local universities. One part of that accreditation process involved me discussing the course with the staff members who were teaching it.

After talking to almost all the staff, what struck me was how similar they all were. In the requirements for the course, there was a standard that each staff member needed to meet, but there was also a requirement for the group of staff to be diverse enough to have broad knowledge of the industry. There was no individual staff member that you could identify as not being at the appropriate standard, but almost all of them had exactly the same background, career progression, etc.

2026-08-16

In a real-time data system, it’s not enough to process events once and move on. There are many cases where you need to replay or reprocess event data that has already passed through the system.

Replay and reprocessing allow you to go back in time — to re-run data through your pipelines as if it were arriving again in real time.

This capability is especially useful for debugging issues, conducting compliance audits, or retraining machine learning models with historical data.

2026-08-15

I recently received a review copy of Causal Inference with Bayesian Networks by Yousri El Fattah and Reza Bagheri from my friends at PackT.

Authors

Yousri El Fattah is the CEO of Causal Computing and an expert in machine intelligence, causal modelling, control systems engineering, and data science.

Reza Bagheri is a working data scientist at Ipsos. He has written extensively on data science and machine learning, and has spoken at substantial conferences.

2026-08-14

Fault tolerance means designing systems that can keep running even when parts of them fail. In real-time data pipelines, something will eventually go wrong — a node might go offline, a network connection could drop, or a consumer might crash. The key is to plan for those failures from the start.

The first principle is to expect failure and aim for graceful degradation rather than total outage. Your system should continue operating, perhaps with reduced functionality or performance, while recovery takes place.

2026-08-13