The Bit Bucket

Time to read

I was on a flight from Houston to Washington via Dallas a while back. We seemed to sit forever on the tarmac in Dallas, in fact just over 1.5 hours from leaving gate to taking off. It was amazingly silly. We were 18th in the takeoff queue and the airport required the planes to keep turning their engines off after moving each time. The captain (who was clearly fed up) warned us that if he had to stop and restart the engines one more time, we’d have had to return to the terminal for more fuel. And we hadn’t even left Houston.

2026-09-13

Partitioning is one of the most effective ways to improve scalability and performance in event-driven systems. It works by dividing a single event stream into multiple independent subsets, or partitions, that can be processed in parallel.

Each partition operates independently, which allows multiple consumers or processing nodes to work simultaneously without interfering with one another. This parallelism increases throughput and helps maintain low latency even as data volumes grow.

The partition key determines how events are distributed among partitions. Common choices include device ID, user ID, or region, depending on the data model and access patterns. The key should be selected carefully:

2026-09-12

If you are aware of Betteridge’s law of headlines , you already know the answer, but let me explain.

There are a lot of odd myths that surround SQL Server. One of the more persistent ones is related to indexes on columns that hold bit values.

A SQL Server column that’s a bit can be in three states. It can have the value zero; it can have the value one; and it can be NULL (ie: have no value at all).

2026-09-11

In Kusto Query Language (KQL), materialized views are a key optimization feature that improve the performance of repeated or computationally expensive queries on streaming data. Normally, when you query a real-time data stream, the system must process the incoming events and apply all calculations each time the query runs. As data volumes increase, this can become inefficient and introduce latency.

A materialized view solves this by precomputing and storing the results of a defined query in a physical table. This table is automatically updated as new events arrive.

2026-09-10

Was 2018 a leap year?

What about the year 2000?

Will 2100 be a leap year?

When I was a young student at school, we learned that leap years were every four years. Then as I got older, I learned that there was more to it than that.

It’s important to understand that any calendar is just an approximation, and there is a lot more to calendars than most people realize.

2026-09-09

Scaling an Eventstream involves more than simply increasing capacity — it’s about ensuring that each stage of the data pipeline can handle higher volumes efficiently and reliably.

The first approach to scaling is by adding more inputs and outputs. As the number of data sources or destinations grows, Eventstream can be configured to handle multiple concurrent data feeds, such as several IoT hubs, Kafka topics, or application logs.

You can also tune the Eventstream’s capacity to handle greater throughput. This involves adjusting the compute and memory resources allocated to the Eventstream so that it can process more events per second without increasing latency.

2026-09-08

There’s a UK game show that I like to watch. It’s called Lingo and it’s the standard word game turned into a full TV show. It’s based on a US game show of the same name.

While watching it, I realized that I’ve come to form an immediate opinion on the education level of the contestants, not based on their appearance, not based on their accents, not based on their assortment of tattoos, but on how they pronounce the letter H.

2026-09-07

Streaming systems produce large and continuous data flows, so it’s essential to have a clear data retention strategy. Without one, storage costs and data management complexity can increase very quickly.

The first decision is what type of data to keep. In many cases, raw event data is valuable for troubleshooting or forensic analysis but may not need to be retained indefinitely. Aggregated or summarized data (for example, hourly or daily averages) is often sufficient for long-term trend analysis.

2026-09-06

I recently read the 2nd edition of a book on serverless ETL. Serverless ETL and Analytics with AWS Glue (2nd Edition) . I hadn’t had the chance to read the first edition of this book, but my friends at PackT sent me the 2nd edition to check out.

Authors

Noritaka Sekiyama is a big data engineer working at a data and AI company.

Albert Quiroga is a senior solutions architect at Amazon.

2026-09-05

In practice, most real-time systems use a combination of streaming and batch processing rather than relying entirely on one or the other. This approach is often described as a hybrid architecture, where each path serves a different purpose.

The hot path (also called the streaming path) handles data as it arrives. It’s designed for immediacy, enabling rapid decisions such as triggering alerts or applying automated actions. The trade-off is that it usually works with limited context and may rely on approximations or rolling summaries.

2026-09-04