The Bit Bucket

Monitoring and observability are critical to maintaining reliable real-time data pipelines. Without them, even small issues can cascade into missed events, delayed alerts, or inconsistent data.

Start by monitoring ingestion metrics — how many events are coming in per second — and processing latency, which measures how long it takes for data to move through your pipeline. High latency often indicates backpressure or insufficient compute capacity. Also, track error rates to identify components that may be dropping or rejecting events.

2026-08-25

Way back in SQL Server 2012, a whole raft of new T-SQL functions were introduced. As I’ve mentioned before, I get excited when there are new T-SQL functions.

Some of the ones added back then, I wasn’t so excited about. EOMONTH was in that category, not because of the functionality, but because of the name (wish it was ENDOFMONTH or better, END_OF_MONTH), and lack of symmetry (lack of a STARTOFMONTH or BEGINNINGOFMONTH).

2026-08-24

In Microsoft Fabric, cost and performance optimization comes from being selective about what you process in real time and how you manage it afterward.

Streaming everything is rarely necessary — and can quickly become expensive. Instead, focus on hot, time-critical data that genuinely requires immediate insight or action. Examples include operational metrics, device telemetry, or fraud-detection signals. Less urgent data can flow through scheduled or downstream processes.

You can also reduce load by pre-aggregating or filtering events upstream. For example, summarizing counts or averages before sending data into an Eventstream or KQL database can dramatically lower both volume and compute cost without sacrificing analytic value.

2026-08-23

Have you heard someone recently say Drive Safe ? If you’re my age, I’m sure that grates on you somewhat. In the end, safe is an adjective, but here it’s being used as an adverb. If I’d said that at school, I would have been corrected and told it should have been Drive Safely.

Similarly, today I was watching the UK Lingo TV show . I really like it and generally, I hear quite correct English on it, particularly from the presenter. But I then heard him say He did brilliant. And immediately I had the same guttural reaction to it.

2026-08-22

When designing real-time data systems, it’s important to consider the cost differences between streaming and batch workloads.

Streaming pipelines tend to be more expensive because they require always-on compute — data is constantly being ingested, transformed, and monitored, even during quieter periods. That means the compute capacity behind your Eventstreams, KQL queries, or Activator rules is active all the time.

Batch processing, on the other hand, is usually cheaper and easier to control. It runs on a schedule — for example, hourly or nightly — so compute resources are only used when the job is running. This makes batch ideal for large volumes of data that don’t need to be processed instantly.

2026-08-21

I’ve spent some time lately reading some older and more famous books. One that seemed fascinating was a short story by James Joyce called The Dead .

Author

James Joyce was an Irish novelist and poet. He was also well-known as a literary critic. Wikipedia says he is regarded as one of the most influential and important writers of the 20th century. I’ve spent time in Ireland over the years, and I’ve seen a lot of tributes to him both up in Dublin, and also around Kildare. He’s treated as somewhat of a national treasure.

2026-08-20

Even more SQL love !

Another popular SQL dialect is for DuckDB. And we’ve just completed our first course using it.

Creating reports, analytics, or applications? And need to get data out of DuckDB? Or are you building applications using DuckDB.

Learn to write SQL queries like a pro !

We now have very popular SQL courses, for all the most important SQL dialects. There’s our flagship course for T-SQL, then already courses for PostgreSQL, Snowflake, Oracle, DB2, MySQL, Azure HorizonDB, and Google BigQuery. We’ve just added our new course Writing SQL Queries for DuckDB and you can enrol in it now. It’s just $95 USD.

2026-08-20

Governance and compliance are critical considerations in real-time systems — they apply every bit as much as they do in traditional batch workloads. The difference is that in streaming environments, everything happens continuously, so your controls and monitoring must operate in real time as well.

The first step is to track data lineage — understanding where each event originated, how it was transformed, and what downstream systems or actions it affected. Lineage visibility helps you answer questions like Where did this number come from? or Which events led to this automated decision? It also supports root cause analysis when problems occur.

2026-08-19

SQL Server databases have a reputation for being hard to test, or at least hard to test appropriately.

For good testing, and particularly for unit tests, you really want the following:

• Database in a known state before each test
• Database containing large amounts of (preferably masked) data (production-sized)
• Quick restore after each test before the next test

For most databases, this is hard to achieve. The restore after each test means that a normal database restore can’t be used. What I often see instead, is people using transactions to try to achieve this i.e. the process becomes:

2026-08-18

Security and governance are just as important in real-time streaming systems as they are in traditional batch pipelines — and in some cases, even more so. Because streaming data is continuous, a single misconfiguration can expose sensitive data for long periods before it’s noticed.

The first principle is to secure event sources and destinations.

Every data connection — whether it’s an IoT device, API, or event broker — should use authentication to verify its identity and encryption in transit to protect data as it moves. In Microsoft Fabric, this typically means using TLS for secure transmission and managed identities for authentication, avoiding hard-coded credentials wherever they are available.

2026-08-17