Microsoft-Fabric

OneLake is really the foundation of Fabric’s storage model. The idea is simple but powerful: instead of having separate storage systems for each analytics tool or service, Fabric provides a single, unified data lake. This gives you one logical place where all of your data lives, and all the workloads in Fabric can share it.

Technically, OneLake is built on the open Delta Lake format. Underneath, it uses Parquet files for efficient columnar storage, but it adds transactional support on top. That means multiple processes can read and write to the same data in a consistent way, with guarantees around reliability and performance. It’s open, it’s proven, and it avoids the pitfalls of closed, proprietary formats.

2026-02-28

One of the most powerful aspects of Fabric’s Real-Time Intelligence is how it connects to external sources. The mechanism for doing this is through Eventstreams. Eventstreams are essentially the pipelines that define where your data is coming in — the inputs — and where it’s going out — the outputs. In a later post, we’ll explore Eventstreams.

Fabric comes with a range of native connectors. These include direct connections to industry-standard technologies like Kafka, Azure Event Hubs, Azure IoT Hub, and other AMQP-based systems. That means if you already have investments in streaming infrastructure — whether in the cloud or on-premises — Fabric can plug into them without a lot of custom development.

2026-02-26

We’ve looked at events generated by applications, databases, and storage systems — but it’s important to remember that Azure and Fabric themselves also generate events. These are sometimes called platform events, because they come from the infrastructure and services you’re running rather than from your business data directly.

Some examples are really practical. Azure might emit events when resources are scaled up or down — say, when a cluster automatically adds nodes to handle increased load. Fabric might generate events when a pipeline starts, completes, or fails. You’ll also see events related to service health, configuration changes, or security alerts.

2026-02-24

Another important category of real-time sources comes from cloud storage platforms like Azure Blob Storage and AWS S3. These aren’t just passive data stores — they can actually generate events whenever something happens to the data inside them.

For example, when a new file is uploaded into a data lake, the storage service can immediately raise an event. That event might then trigger an ingestion pipeline, start a transformation process, or kick off machine learning model scoring. The key here is that you don’t have to wait for a scheduled scan or a batch job to check for new files. The storage system itself notifies you the instant a change occurs.

2026-02-22

This diagram highlights the difference between Change Data Capture (CDC) and Change Event Streams (CES) — two ways of turning databases into real-time event sources.

On the left side, we see CDC. Here the focus is on row-level changes. Every time a row is inserted, updated, or deleted in a table, that operation is captured and streamed out as an event. Those events typically feed into real-time replication or analytics pipelines. In other words, CDC makes sure your downstream systems always have fresh copies of your data. It’s like having a live feed of every transaction at the cash register. That’s why we use the analogy here of cash register receipts — you get a record of each sale exactly as it happens.

2026-02-20

When we talk about Change Event Streams, or CES, we’re looking at a different layer of database activity than CDC.

CDC — Change Data Capture — focuses on row-level changes: inserts, updates, and deletes in your transactional tables. That’s very useful for analytics and replication. But there’s a lot more going on inside a database than just row changes.

CES captures the broader set of events that affect the structure and governance of the database itself. This includes things like schema changes — if someone adds a new column, drops a table, or creates an index. It also includes permission updates — for example, when access rights are granted or revoked. And it can even capture configuration changes or other metadata-level modifications.

2026-02-18

Many organizations already have huge amounts of business-critical data sitting in relational databases — SQL Server, PostgreSQL, MySQL, and so on. But by default, these databases aren’t real-time data sources. Traditionally, if you wanted to know what changed, you’d run queries at intervals — maybe once an hour, or once a day — and look for differences. That approach works, but it introduces delay.

This is where Change Data Capture, or CDC, comes in. CDC is a technique that turns a traditional database into a real-time data source. Instead of polling for differences, CDC actually streams the changes themselves as they occur. Anytime a row is inserted, updated, or deleted, that operation is captured and emitted as an event.

2026-02-16

The following diagram shows how different messaging technologies — Kafka, Event Hubs, and RabbitMQ — fit into a real-time pipeline.

At the top, we have our event producers: applications generating user activity, IoT sensors sending telemetry, and financial systems pushing transactions. These are the raw sources of events.

Now, those events need to get transported reliably to downstream consumers. That’s where our three options come in:

• Kafka, on the left, is like a freight train. It’s open source, distributed, and designed to move massive volumes of events continuously. It’s ideal for streaming pipelines at very large scale.
• Event Hubs, in the middle, is essentially the managed version of Kafka on Azure. You get the same freight train power, but as a managed service. You don’t worry about the tracks or the engines — Azure handles that for you. It integrates directly with Fabric, which makes it especially attractive if you’re already in the Microsoft ecosystem.
• RabbitMQ, on the right, is like a courier van. It’s not designed to move massive volumes, but it’s incredibly reliable and flexible for routing and guaranteed delivery. If you need every message delivered exactly once — even if the consumer is offline for a bit — RabbitMQ is your best option.

Finally, at the bottom, we have the consumers: dashboards, analytics systems, or applications that depend on this event data. The key is that each broker delivers data in a way that best matches its design philosophy: high throughput (Kafka), managed scale (Event Hubs), or guaranteed precision (RabbitMQ).

2026-02-14

RabbitMQ is another widely used message broker, but it fills a slightly different role compared to Kafka or Event Hubs. It’s an open-source broker that primarily implements the AMQP protocol, though it supports other protocols as well.

Where RabbitMQ shines is in reliable delivery and flexible routing of messages. It’s extremely good at guaranteeing that each message gets to the right destination, even if the consumer isn’t available at the time. You can define queues, exchanges, and routing rules to handle very complex message delivery patterns.

2026-02-12

When we talk about protocols for sending and receiving streams of event data, two of the most common you’ll come across are HTTP and AMQP.

HTTP

HTTP is the workhorse of the web. It’s everywhere, it’s simple, and it’s supported by almost every platform and device. The model is request/response: the client asks for something, the server replies, and then the connection is done. HTTP is stateless, meaning every request is independent. That simplicity makes it easy to use, but it also makes it less suitable for continuous, real-time data flows. If you want to stream updates constantly, you either have to keep making new HTTP requests or hold the connection open in ways that HTTP wasn’t originally designed for.

2026-02-08