The Bit Bucket

I recently listened (via Audibla) to Timothy Snyder’s book On Tyranny: Twenty Lessons from the Twentieth Century . I wasn’t sure what I’d make of it but I’d heard it was interesting, and I noticed that on a show led by Rachel Maddow.

Author

Timothy Snyder is a well-known author, historian and intellectual. He’s an expert on eastern Europe and on the Second World War.

On Tyranny

I decided to go through this book for two reasons. I was fascinated by his thoughts on tyranny, and I liked the fact that it was a short book. I’m also a fan of books that are narrated by their authors, and Snyder is the narrator for this book.

2026-09-01

IoT environments typically involve large numbers of devices — sensors, controllers, and machinery — each continuously generating telemetry data. Common examples include temperature, vibration, pressure, voltage, and power consumption readings. These data streams provide valuable insight into device health and operating conditions.

Problem to solve

When a component starts to degrade or operate outside normal ranges, early signs often appear in the telemetry before a full failure occurs.

The challenge is that traditional reporting systems tend to work on batch cycles — for example, generating summaries every hour or day. That delay means early warning signs can be missed, especially in high-frequency data streams where conditions can change in seconds.

2026-08-31

I recently mentioned that I’ve spent some time lately reading some older and more famous books. I’m particularly keen to re-read books that I remember strongly from when I was young. I love John Steinkbeck’s books and Of Mice and Men particularly stuck in my mind.

Author

John Steinbeck was an American writer and novelist who won the 1962 Nobel Prize in Literature for his realistic and imaginative writings. I’ve read many of his books and have loved them.

2026-08-30

Credit card fraud continues to be a major challenge for financial institutions. It can happen within seconds—often before traditional systems even notice. Once a fraudulent transaction is processed, recovering the funds is difficult or impossible.

Problem to solve

Most legacy fraud detection systems rely on batch processing, where transactions are analyzed in hourly or daily cycles. That delay makes them ineffective for real-time protection.

The business need here is clear: detect and respond to fraud as it happens. This means evaluating transaction streams in real time, identifying anomalies within seconds, and automatically taking action—such as flagging or blocking suspicious activity.

2026-08-29

There’s a nasty trend that I’ve seen at a number of sites in recent years. It’s the tendency to try to block and or censor anything that the company thinks could be detected as an issue. Some companies are so concerned about their IP (intellectual property) that they try to stop any potential leak of that property.

While on the surface, that all might seem to make sense, but often it ends up being not sensible and unproductive.

2026-08-28

Many real-time data projects fail not because of the technology, but because of design assumptions and unnecessary complexity.

One of the most common mistakes is over-engineering for ultra-low latency. Teams sometimes aim for sub-second response times even when the use case doesn’t require it. That often adds complexity and cost without meaningful benefit. Instead, define what ‘real time’ truly means for your scenario — it might be seconds or even minutes, depending on the business need.

2026-08-27

I see so much written about how to scale SQL Server systems, and this generally starts with needing to improve SQL Server database performance. When I read articles from the SQL Server field support teams with titles like Top 10 Performance Problems for SQL Server, I often just smile.

The problem is one of perspective. If you are looking at the performance problems that are brought to the support teams to solve, you get a very, very skewed view of what’s typical.

2026-08-26

Monitoring and observability are critical to maintaining reliable real-time data pipelines. Without them, even small issues can cascade into missed events, delayed alerts, or inconsistent data.

Start by monitoring ingestion metrics — how many events are coming in per second — and processing latency, which measures how long it takes for data to move through your pipeline. High latency often indicates backpressure or insufficient compute capacity. Also, track error rates to identify components that may be dropping or rejecting events.

2026-08-25

Way back in SQL Server 2012, a whole raft of new T-SQL functions were introduced. As I’ve mentioned before, I get excited when there are new T-SQL functions.

Some of the ones added back then, I wasn’t so excited about. EOMONTH was in that category, not because of the functionality, but because of the name (wish it was ENDOFMONTH or better, END_OF_MONTH), and lack of symmetry (lack of a STARTOFMONTH or BEGINNINGOFMONTH).

2026-08-24

In Microsoft Fabric, cost and performance optimization comes from being selective about what you process in real time and how you manage it afterward.

Streaming everything is rarely necessary — and can quickly become expensive. Instead, focus on hot, time-critical data that genuinely requires immediate insight or action. Examples include operational metrics, device telemetry, or fraud-detection signals. Less urgent data can flow through scheduled or downstream processes.

You can also reduce load by pre-aggregating or filtering events upstream. For example, summarizing counts or averages before sending data into an Eventstream or KQL database can dramatically lower both volume and compute cost without sacrificing analytic value.

2026-08-23