The Bit Bucket

Governance and compliance are critical considerations in real-time systems — they apply every bit as much as they do in traditional batch workloads. The difference is that in streaming environments, everything happens continuously, so your controls and monitoring must operate in real time as well.

The first step is to track data lineage — understanding where each event originated, how it was transformed, and what downstream systems or actions it affected. Lineage visibility helps you answer questions like Where did this number come from? or Which events led to this automated decision? It also supports root cause analysis when problems occur.

2026-08-19

SQL Server databases have a reputation for being hard to test, or at least hard to test appropriately.

For good testing, and particularly for unit tests, you really want the following:

• Database in a known state before each test
• Database containing large amounts of (preferably masked) data (production-sized)
• Quick restore after each test before the next test

For most databases, this is hard to achieve. The restore after each test means that a normal database restore can’t be used. What I often see instead, is people using transactions to try to achieve this i.e. the process becomes:

2026-08-18

Security and governance are just as important in real-time streaming systems as they are in traditional batch pipelines — and in some cases, even more so. Because streaming data is continuous, a single misconfiguration can expose sensitive data for long periods before it’s noticed.

The first principle is to secure event sources and destinations.

Every data connection — whether it’s an IoT device, API, or event broker — should use authentication to verify its identity and encryption in transit to protect data as it moves. In Microsoft Fabric, this typically means using TLS for secure transmission and managed identities for authentication, avoiding hard-coded credentials wherever they are available.

2026-08-17

Many years back, I was invited to chair a course accreditation panel for a local TAFE (Technical and Further Education) course. They had started to offer a computing-related 3-year diploma, and the hope was that it wasn’t too far below the 3-year degrees offered at local universities. One part of that accreditation process involved me discussing the course with the staff members who were teaching it.

After talking to almost all the staff, what struck me was how similar they all were. In the requirements for the course, there was a standard that each staff member needed to meet, but there was also a requirement for the group of staff to be diverse enough to have broad knowledge of the industry. There was no individual staff member that you could identify as not being at the appropriate standard, but almost all of them had exactly the same background, career progression, etc.

2026-08-16

In a real-time data system, it’s not enough to process events once and move on. There are many cases where you need to replay or reprocess event data that has already passed through the system.

Replay and reprocessing allow you to go back in time — to re-run data through your pipelines as if it were arriving again in real time.

This capability is especially useful for debugging issues, conducting compliance audits, or retraining machine learning models with historical data.

2026-08-15

I recently received a review copy of Causal Inference with Bayesian Networks by Yousri El Fattah and Reza Bagheri from my friends at PackT.

Authors

Yousri El Fattah is the CEO of Causal Computing and an expert in machine intelligence, causal modelling, control systems engineering, and data science.

Reza Bagheri is a working data scientist at Ipsos. He has written extensively on data science and machine learning, and has spoken at substantial conferences.

2026-08-14

Fault tolerance means designing systems that can keep running even when parts of them fail. In real-time data pipelines, something will eventually go wrong — a node might go offline, a network connection could drop, or a consumer might crash. The key is to plan for those failures from the start.

The first principle is to expect failure and aim for graceful degradation rather than total outage. Your system should continue operating, perhaps with reduced functionality or performance, while recovery takes place.

2026-08-13

Here’s a quick T-SQL test for you.

Without looking below to see the answer first, try to guess what each of these statements will produce as output:

SELECT TRY_CAST('' AS int);
SELECT TRY_CAST('    ' AS int);
SELECT TRY_CAST('' AS date);
SELECT TRY_CAST('' AS decimal(18, 2));
SELECT TRY_CONVERT(date, '', 103);

And to slightly distract you from checking out the answers yet, here is another wise-looking owl who is thinking about the answers, and warning you not to look further down the page yet:

2026-08-12

In any real-time system, we can’t assume that every event will be delivered successfully on the first attempt. Network interruptions, temporary service outages, or throttling limits can all cause event delivery failures.

That’s why retry logic is a fundamental part of reliable streaming architecture.

The most common approach is to use exponential backoff — meaning that the system waits progressively longer between retries. This avoids overwhelming the destination service during outages. For example, retries might occur after 1 second, then 2 seconds, then 4, and so on, up to a maximum delay.

2026-08-11

Had a client today asking about this error message. They were working away on a machine and suddenly they got the message Failed to update the database because the database is read-only.

• The user hadn’t changed anything that they were aware of.
• Based on the user’s permissions (ie: what they could see), everything in SSMS looked normal.
• When they checked the sys.databases view, the database showed MULTI_USER.
• There was enough disk space.
• Folder permissions had not changed.
• The user was puzzled.

The issue was caused by the database being part of an availability group, and the AG had failed over. So suddenly, the database the user was connected to, was now a read-only replica, not the primary replica. That’s why the database said it was read-only.

2026-08-10