The Bit Bucket

In a real-time data system, it’s not enough to process events once and move on. There are many cases where you need to replay or reprocess event data that has already passed through the system.

Replay and reprocessing allow you to go back in time — to re-run data through your pipelines as if it were arriving again in real time.

This capability is especially useful for debugging issues, conducting compliance audits, or retraining machine learning models with historical data.

2026-08-15

I recently received a review copy of Causal Inference with Bayesian Networks by Yousri El Fattah and Reza Bagheri from my friends at PackT.

Authors

Yousri El Fattah is the CEO of Causal Computing and an expert in machine intelligence, causal modelling, control systems engineering, and data science.

Reza Bagheri is a working data scientist at Ipsos. He has written extensively on data science and machine learning, and has spoken at substantial conferences.

2026-08-14

Fault tolerance means designing systems that can keep running even when parts of them fail. In real-time data pipelines, something will eventually go wrong — a node might go offline, a network connection could drop, or a consumer might crash. The key is to plan for those failures from the start.

The first principle is to expect failure and aim for graceful degradation rather than total outage. Your system should continue operating, perhaps with reduced functionality or performance, while recovery takes place.

2026-08-13

Here’s a quick T-SQL test for you.

Without looking below to see the answer first, try to guess what each of these statements will produce as output:

SELECT TRY_CAST('' AS int);
SELECT TRY_CAST('    ' AS int);
SELECT TRY_CAST('' AS date);
SELECT TRY_CAST('' AS decimal(18, 2));
SELECT TRY_CONVERT(date, '', 103);

And to slightly distract you from checking out the answers yet, here is another wise-looking owl who is thinking about the answers, and warning you not to look further down the page yet:

2026-08-12

In any real-time system, we can’t assume that every event will be delivered successfully on the first attempt. Network interruptions, temporary service outages, or throttling limits can all cause event delivery failures.

That’s why retry logic is a fundamental part of reliable streaming architecture.

The most common approach is to use exponential backoff — meaning that the system waits progressively longer between retries. This avoids overwhelming the destination service during outages. For example, retries might occur after 1 second, then 2 seconds, then 4, and so on, up to a maximum delay.

2026-08-11

Had a client today asking about this error message. They were working away on a machine and suddenly they got the message Failed to update the database because the database is read-only.

• The user hadn’t changed anything that they were aware of.
• Based on the user’s permissions (ie: what they could see), everything in SSMS looked normal.
• When they checked the sys.databases view, the database showed MULTI_USER.
• There was enough disk space.
• Folder permissions had not changed.
• The user was puzzled.

The issue was caused by the database being part of an availability group, and the AG had failed over. So suddenly, the database the user was connected to, was now a read-only replica, not the primary replica. That’s why the database said it was read-only.

2026-08-10

In any real-time data system, there’s a point where the incoming event rate can exceed what the system can process. This condition is known as backpressure.

Backpressure can occur for a few reasons — a sudden spike in data volume, slow or overloaded consumers, or limited throughput in one part of the pipeline. If it’s not handled properly, it can cascade through the system, eventually causing delays or even a complete stall in event processing.

2026-08-09

I recently received a review copy of Python for Algorithmic Trading Cookbook (2nd Edition): Recipes for designing, building, and deploying algorithmic trading strategies with Python by Jason Strimpel from my friends at PackT.

Author

Jason Strimpel is the founder of PyQuant News, co-founder of Quant Science, and Managing Director of Global AI and Advanced Analytics at a top-tier consulting firm.

Content

This book is an ambitious, highly practical guide to building the complete research-to-execution workflow for systematic trading. It treats algorithmic trading as an engineering discipline involving data acquisition, storage, analysis, backtesting, risk measurement, execution, and deployment. I was surprised by the breadth of modern tools that are discussed: 68 recipes, 51 Jupyter notebooks, 17 modular trading applications, and four GPU-focused scripts.

2026-08-08

In real-time systems, reliability and resilience are essential. Unlike batch workloads, where you can retry a job later, streaming pipelines are continuous — they must keep running even when parts of the system fail.

The first design principle is to ensure that streams keep flowing.

That often means building redundancy into both your data sources and destinations. For example, you might configure multiple input connections or have failover routes so that data can still be processed if one stream or endpoint goes offline.

2026-08-07

I recently posted about rebuilding clustered columnstore indexes when they needed maintenance . One of the questions that a reader asked was:

Which is best in performance and CCI packing for fast query:

• Truncate table, drop CCI, insert data, create CCI.
• Truncate table, insert data and therefore keep CCI.

And of course, that’s a great question. Here are the issues:

If I had to just pick one of those, for best query performance and best CCI packing, the first option is usually better:

2026-08-06