Let’s take a look at the windowing options available. The first of these is the tumbling window, and it’s the simplest type of temporal window. It slices the event stream into fixed-length, adjacent blocks of time, with no overlap and no gaps. Think of it like dividing a timeline into perfectly equal buckets — each window starts as soon as the previous one ends.

For example, let’s say we’re monitoring sales transactions. If we set up a tumbling window of five minutes, then every five minutes the system calculates totals or averages. At the end of each window, you get a new result, and then the process resets for the next five-minute slice.

This approach is ideal when you need consistent, regular reporting intervals. Dashboards and monitoring systems often use tumbling windows because they make it easy to compare one period to the next — for example, sales in the last 5 minutes or number of logins in the last 10 minutes.

The trade-off is that because windows don’t overlap, you may miss changes happening between buckets. If a spike occurs at the boundary between two windows, each one will only show part of it. That’s where other window types like sliding or hopping come in, but for simple, predictable metrics, tumbling windows are often the best choice.
 

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Most SQL Server developers are aware that the EXEC statement can be used to:

• Execute a stored procedure (system, user-defined, extended)
• Execute some dynamic SQL

And most understand that you can SELECT from a scalar user-defined function.

But the option that many people don’t seem to be aware of, is that you can also use EXEC to call a scalar function.

I remember noticing this in the documentation for the EXEC command some years back. Prior to that, it had never dawned on me that you could use EXEC to call a scalar UDF. It’s also in the oldest documentation that I was able to check, so I’d say it’s worked for a long time.

Here’s a trivial example of a function and calling it as I would have in the past:

USE tempdb;  
GO

CREATE FUNCTION dbo.SayHello(@WhoTo NVARCHAR(100))  
RETURNS NVARCHAR(120)  
AS  
BEGIN  
  RETURN N'Hello ' + @WhoTo;  
END;  
GO

SELECT dbo.SayHello(N'Greg');  
GO

Now clearly I’d never use a function like that. The performance impacts would be horrid. But I wanted to keep this very simple.

Now, notice that you can also call it like this:

DECLARE @ReturnValue NVARCHAR(120);

EXEC @ReturnValue = dbo.SayHello N'Greg';

SELECT @ReturnValue;

Prior to the day back then when I read the documentation for EXEC, I would have guessed this wouldn’t work.

Learn more about Advanced T-SQL

If you really want to learn about SQL Server Advanced T-SQL, we have an online on-demand course that you can enrol in, right now. You’ll find it at SQL Server Advanced T-SQL for Developers and DBAs

One of the challenges with streaming data is that it never ends — it just keeps flowing. If we tried to calculate totals or averages across the entire stream, the numbers would just keep growing forever, and we’d never get a meaningful result. That’s why we use temporal windows.

A temporal window lets us break the continuous stream into slices of time, so we can apply aggregations within those boundaries. For example, instead of calculating total transactions forever, we might calculate total transactions every minute, or average sensor readings every five seconds. Each window produces a result that can be stored, visualized, or acted upon in real time.

Currently, these temporal options are supported in the Group By and the Custom SQL operations.

There are several windowing strategies, and each one is designed for different business needs. Tumbling windows divide time into fixed, non-overlapping slices — great for reporting metrics per minute or per hour. Sliding windows move forward in small steps, overlapping slightly, which makes them useful for spotting trends or detecting spikes. Session windows group events into sessions of activity separated by idle gaps, which is very useful in customer behavior analysis. And hopping windows combine the advantages of overlapping and fixed intervals to detect patterns at multiple granularities.

Windows turn an infinite, continuous stream into manageable chunks that we can analyze. They give us control over how aggregations are applied, and they allow business users to see insights in a way that makes sense — whether that’s every few seconds for system monitoring, every few minutes for operational dashboards, or every hour for business reporting.

Without windows, streaming analytics would feel overwhelming — just a firehose of data. With windows, we can organize that firehose into buckets, making it possible to compute meaningful metrics, detect anomalies, and power real-time decisions.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

The WITH EXECUTE AS clause was a great addition for defining stored procedures and functions, to change the execution context, just for the duration of the stored procedure or function. For example:

CREATE PROC SomeSchema.SomeProc
WITH EXECUTE AS USER = 'Fred'
AS
... 

Mostly I use this with the OWNER option:

CREATE PROC SomeSchema.SomeProc
WITH EXECUTE AS OWNER
AS
... 

It’s also useful during testing, where I can temporarily change my execution context during testing. For example:

EXEC AS USER = 'Fred';

\-- Try some code here while running as Fred

REVERT;

But the option that most people don’t realize is possible, is that you can set the execution context for a single execution like this:

EXEC ('Some command to be executed as Fred') AS USER = 'Fred';

This could be particularly useful when you need to call dynamic SQL code with a specific execution context.

I hope that helps to simplify your code.

Learn more about SQL Server Administration

If you really want to learn about SQL Server administration right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at SQL Server Administration for Developers and DBAs

When we talk about grouping in real-time data processing, we’re talking about organizing events according to attributes that are meaningful for analysis. Instead of treating the event stream as one giant firehose, grouping lets us carve it up into categories that align with how the business thinks about its data.

For example, in an IoT scenario, we might group telemetry events by device ID. This way, instead of calculating one global average temperature across thousands of sensors, we can calculate average per device, giving us much more useful insights. In a retail scenario, we could group transactions by customer ID to analyze individual purchasing patterns, or by region to monitor performance across different locations. Grouping is what enables those per-customer, per-device, or per-region dashboards that managers and operators rely on.

Grouping also acts as the foundation for aggregation. Once we’ve organized events into groups, we can apply aggregation functions — like count, sum, or average — within each group. For instance, we might calculate ’total sales per customer per hour,’ or ‘maximum vibration reading per machine in the last 5 minutes.’ Without grouping, those kinds of breakdowns wouldn’t be possible.

Another key aspect is scalability. When streams are grouped, they can also be partitioned. This means different processors can handle different groups in parallel — one node might handle customer IDs starting with A–M, while another handles N–Z. That parallelism is essential for handling very high event volumes in real time. If we tried to process everything in one monolithic stream, the system would quickly become overloaded.

So grouping really gives us two big advantages: it makes analytics more relevant and precise by breaking data down into meaningful categories, and it makes pipelines more scalable and efficient by enabling parallel processing.

When you design real-time pipelines in Fabric, think carefully about the grouping keys you choose. They need to line up with how the business wants to slice its data, but they also need to be practical for performance and scale.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

It’s common advice that most SQL Server tables should have a clustered index. There are some exceptions to this but it’s a pretty general rule, and if in doubt, you should follow it. (Note that this is not the same as having a primary key).

I regularly come across tables without clustered indexes for all the wrong reasons. So, in our free SDU Tools for developers and DBAs, we added a tool that can look for user tables that don’t have a clustered index. No surprise, it’s called ListUserHeapTables because a table without a clustered index is a heap.

It takes three parameters:

@DatabaseName sysname - the database to look into
@SchemasToList nvarchar(max) - a comma-delimited list of schemas to check (or ‘ALL’)
@TablesToList nvarchar(max) - a comma-delimited list of table s to check (or ‘ALL’)

This procedure was added in an early version of SDU Tools but in version 27, we’ve made it more useful. As well as the schema name and table name, it now provides a page count, and a forwarded record count.

Forwarded records occur when the heap has nonclustered indexes and the underlying row is modified and moves to a different location. They avoid the need to update all the related indexes as well.

Find out more

You can see it in action in the main image above, and in the updated video here:

Access to SDU Tools is one of the benefits of being an SDU Insider, along with access to our other free tools and eBooks. Please just visit here for more info:

http://sdutools.sqldownunder.com

The Join operation allows you to bring together data from multiple sources in real time — much like performing a SQL join on continuously arriving data.

Imagine you have one stream of telemetry from IoT devices and another stream of configuration updates.

By joining them on a common key, such as a device ID, you can enrich your telemetry with configuration or location context in real time.

Eventstream supports different types of joins, including both inner joins (matching only overlapping records), and outer joins (keeping all events from one stream, even when there’s no match).

These joins can happen on fields like transaction IDs, timestamps, or other logical keys.

In practice, joining is what enables correlation — connecting events from different systems to form a complete picture. For example, matching a purchase transaction with its fraud score stream, or aligning telemetry with an alert feed.

By carefully defining your join conditions, you can unlock much richer insights from your real-time pipelines — turning isolated event streams into integrated, contextualized data flows.

Note that early on, it appeared that eventstream joins would directly support reference datasets, but at present, joins can only combine events from input streams.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Our free SDU Tools for developers and DBAs, now includes a very large number of tools, with procedures, functions, and views. Version 27 adds the first set of views and functions for working with Nepali dates. These are useful in Nepal and in a number of Buddhist-related areas.

The first tool added is a view called NepaliMonths. It returns the Nepali names for months. You can see it in the main image above.

The next tool added is a view called NepaliMonthDays. It maps year numbers and month numbers to the length of each month. It is a complex pattern in Nepali date processing, so a limited range of dates is supported.

They also provide options to convert between English dates and Nepali dates (EnglishDateToNepaliDate) and vice-versa (NepaliDateToEnglishDate).

This function is computationally intensive. If you have large volumes of data to compare, consider modifying it to a table-valued function.

Find out more

You can see it in action in the main image above, and in the video here:

You can use our tools as a set or as a great example of how to write functions like these.

Access to SDU Tools is one of the benefits of being an SDU Insider, along with access to our other free tools and eBooks. Please just visit here for more info:

http://sdutools.sqldownunder.com

When we talk about aggregation, we’re really talking about taking huge volumes of raw events and rolling them up into something that’s usable, measurable, and actionable. A raw stream might be thousands of individual events per second — each transaction, each sensor ping, each click. By themselves, they’re useful for tracing details, but they don’t tell the bigger story.

Aggregation lets us step back and say: Instead of looking at every single reading, let’s look at the total, the average, or the maximum over a period of time. Common aggregation functions are ones you already know — COUNT, SUM, AVG, MIN, MAX.

These functions help us turn data into metrics and KPIs that can be consumed at scale.

For example, a retailer might aggregate all point-of-sale transactions into total revenue per minute. That’s much easier to track in a dashboard than trying to follow every single receipt. In an IoT scenario, we could calculate the average temperature across all devices in a building every 10 seconds, rather than monitoring each device individually. For anomaly detection, we might track rolling maximum values, like the highest CPU usage in the last five minutes, to quickly spot problems.

Aggregation also helps control data volume and cost. Imagine if every dashboard had to query billions of raw events in real time. It wouldn’t be feasible. Instead, we pre-aggregate and reduce that firehose into compact metrics, which keeps queries fast and infrastructure efficient.

In practice, aggregations are often the bridge between raw streams and the dashboards or alerts that business users interact with. They allow executives to see KPIs like orders per minute or average response time, while analysts and engineers still have the option of drilling into the raw data if needed.

Aggregation is about compression with meaning: taking floods of raw events and turning them into real-time signals that people and systems can actually act on.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Our free SDU Tools for developers and DBAs, now includes a very large number of tools, with procedures, functions, and views. The TokenSetSimilarity function calculates token set similarity for two strings.

It answers the question: Do these two strings contain mostly the same words, even if the order, spacing, or repetition differs?

It is useful where word order varies, or extra or missing words are common. It can also help where character-level typos are less important than the presence of words.

The function is order-insensitive (which is good for names and titles) and is intentionally designed to be set-based (so duplicates don’t increase similarity). For token-weighting (downplaying common words like pty, ltd, inc), you might consider adding a stop-word table and filtering tokens before insert. It returns NULL on NULL or empty string input.

This function is computationally intensive. If you have large volumes of data to compare, consider modifying it to a table-valued function.

Find out more

You can see it in action in the main image above, and in the video here:

You can use our tools as a set or as a great example of how to write functions like these.

Access to SDU Tools is one of the benefits of being an SDU Insider, along with access to our other free tools and eBooks. Please just visit here for more info:

http://sdutools.sqldownunder.com

When working with real-time data, it’s easy for your streams to become cluttered — especially as events come from multiple sources with different structures. That’s where the Manage Fields option becomes essential.

It gives you control over the shape of your data stream. You can choose which fields to keep and drop those you don’t need, which helps reduce noise and improves performance. For example, you might remove diagnostic fields or metadata columns that aren’t needed for your analytics.

You can also rename fields to make your dataset easier to understand — for instance, changing a field like dev_id to DeviceID, or aligning naming conventions across multiple sources.

Another powerful capability is creating calculated or derived fields. Maybe you want to compute a total, a ratio, or convert a timestamp into a specific format — all of that can be done within the Manage Fields transformation step.

Finally, reordering fields lets you control how the output schema looks, which makes it easier for downstream processes to work with your stream.

Think of Manage Fields as your stream’s cleanup and preparation tool — ensuring that what flows into your analytics is clean, well-structured, and ready for insights.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Our free SDU Tools for developers and DBAs, now includes a very large number of tools, with procedures, functions, and views. The NormalizeForSearch function normalizes a string to make it ready for search operations.

It makes strings comparable by stripping away differences that are usually meaningless for search or matching.

It helps to answer the question: If two strings refer to the same thing, what differences should I ignore before I even start comparing?

It returns NULL on NULL input.

This function is computationally intensive. If you have large volumes of data to compare, consider modifying it to a table-valued function.

Find out more

You can see it in action in the main image above, and in the video here:

You can use our tools as a set or as a great example of how to write functions like these.

Access to SDU Tools is one of the benefits of being an SDU Insider, along with access to our other free tools and eBooks. Please just visit here for more info:

http://sdutools.sqldownunder.com

When we work with streaming data, one of the first transformations we often apply is filtering. The reality is that not every event is useful. In fact, in many scenarios, the vast majority of events are just background noise. Filtering gives us a way to narrow the stream down to only the events that actually matter for the business problem we’re solving.

Take IoT telemetry as an example. A device might send thousands of readings every hour, but if 99% of them show perfectly normal operating conditions, storing and analyzing them all just adds cost and complexity. By applying a filter, we could say: only keep the events where the temperature rises above 80 degrees Celsius, or only process readings where a vibration level exceeds a set threshold. This way, we’re focusing on signals of interest instead of wasting resources on noise.

The same idea applies in financial systems. Instead of passing every single transaction downstream, maybe we only flag those above a certain value, or those coming from specific regions. This doesn’t mean we ignore the rest entirely — we could still archive them if needed — but by filtering upfront, we make sure that real-time analytics and alerts concentrate on what’s important.

There’s also a very practical reason for filtering: cost and performance. Every event you let into the system consumes compute, storage, and network bandwidth. The more irrelevant events you allow through, the more expensive and sluggish your pipeline becomes. By reducing the event volume early, we not only save money but also make downstream processing faster and simpler.

The good news is that in Fabric, filtering is easy to configure. You don’t need to write custom code — you can define filtering conditions visually in the Eventstream designer. For example, you can set a simple condition like temperature > 80 or transactionAmount > 1000. That means both technical teams and business analysts can work together to define the rules that keep the data stream focused on real value.

So filtering is really about precision and efficiency: cutting through the noise so that what flows through your real-time system is the data that drives action.
 

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Our free SDU Tools for developers and DBAs, now includes a very large number of tools, with procedures, functions, and views. The LevenshteinDistance function calculates the Levenshtein distance between two strings.

It essentially answers the question How far apart are these two strings in terms of character edits?, where edits are inserting, deleting, or substituting a character. In this calculation, each edit has a cost of 1.

Empty and NULL values on input return NULL.

This function is computationally intensive. If you have large volumes of data to compare, consider modifying it to a table-valued function.

Find out more

You can see it in action in the main image above, and in the video here:

You can use our tools as a set or as a great example of how to write functions like these.

Access to SDU Tools is one of the benefits of being an SDU Insider, along with access to our other free tools and eBooks. Please just visit here for more info:

http://sdutools.sqldownunder.com

When we talk about ingestion modes in Fabric Real-Time Intelligence, we’re really thinking about how events move from their source into a destination, such as an Eventhouse table or another downstream system.

The pattern you choose affects latency, cost, data quality, and the flexibility of your real-time architecture. There are two broad models that students will encounter: direct ingestion and processing before ingestion.

Direct ingestion is the simplest path. Events arrive from a source such as IoT devices, applications, or an event broker, and they are immediately written into the target system without any intermediate shaping. This mode gives the lowest latency because nothing happens in between. It is most useful when you want to preserve raw events for later analysis, replay, or transformations that happen downstream. It is also the right choice when your first priority is freshness and the consumers are able to handle any necessary cleaning or shaping themselves.

However, many real-time workloads require more than just landing raw events. That’s where processing before ingestion becomes relevant. In this mode, events flow through a processing step, often a Stream Processing job, before they land in a table or get forwarded to another service.

Processing

Processing can include filtering out unwanted events, normalising formats, joining with reference data to enrich context, detecting anomalies, or creating aggregated views such as rolling counts or metrics. This does add some latency, because transformations take time to compute, but the trade-off is usually improved data quality and reduced complexity for downstream systems.

Another benefit of processing before ingestion is being able to reduce storage and compute costs. By filtering noisy or duplicate events early, you avoid ingesting unnecessary data. At the same time, curated streams can make life easier for data consumers who expect well-structured, high-quality information.

So, neither mode is better; they serve different purposes. Direct ingestion optimizes for speed and fidelity, while processing before ingestion optimizes for quality and readiness for analytics. In most real-time solutions, you’ll see a combination of both depending on the requirements for each downstream consumer.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence