Even more SQL love !

Another popular SQL dialect is for Google BigQuery. And we’ve just completed our first course using it.

Creating reports, analytics, or applications? And need to get data out of Google BigQuery? Learn to write SQL queries like a pro !

We now have very popular SQL courses, for all the most important SQL dialects. There’s our flagship course for T-SQL, then already courses for PostgreSQL, Snowflake, Oracle, DB2, MySQL, and Azure HorizonDB. We’ve just added our new course Writing SQL Queries for BigQuery and you can enrol in it now. It’s just $95 USD.

Check it out and enrol now here: Writing SQL Queries for BigQuery

Course Summary

Do you need to learn how to write SQL queries for Google BigQuery?

• You know that the information that you need is stored in a Google BigQuery dataset
• You need to find some data to extract it
• You need to create reports with reporting tools
• You are using analytic tools like Looker, Power BI, Tableau, QlikView, Excel, or Access and need to get data from Google BigQuery
• You want to learn to write queries properly, using commercial coding standards
• You are new to writing queries, or you are self-taught and want to make sure you are doing it correctly

If so, this course is for you! And as well as detailed instruction, the course also offers optional practical exercises and quizzes to reinforce your learning.

We encourage you to complete the practical exercises. We have tried to make this as easy as possible. And a free Google BigQuery account is likely all that you’ll need.

Check it out and enrol now here: Writing SQL Queries for Google BigQuery

While Activator supports many types of actions, some of the most common and useful ones are notifications — such as sending a message to Microsoft Teams, generating an email, or updating an existing workflow.

These actions help keep people in the loop. Even when automation is in place, there are times when human awareness or approval is still essential. For example, sending an alert to a Teams channel can immediately notify support engineers that something unusual has happened, while still allowing automated responses to continue in parallel.

A good approach is to combine notifications with automated remediation. For instance, if a rule detects a service failure, Activator could both: Send a Teams message or email to IT, and trigger a workflow in Power Automate or Logic Apps to restart the affected service automatically.

This creates a hybrid response — automation takes care of what it can, while people remain aware of what’s happening.

One of the key design principles in real-time operations is to balance visibility with automation. You don’t want to rely entirely on humans for time-sensitive responses, but you also don’t want automation to run silently without oversight.

In this way, Activator helps teams create responsive, transparent systems — where actions happen automatically when needed, but people are still informed and can intervene if something unexpected occurs.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

I recently received a review copy of The Orange Book of Machine Learning: Green Edition by Carl McBride Ellis from my friends at PackT.

Author

Carl McBride Ellis wrote this book as a Green Edition of the material he had been teaching across a whole range of Spanish cities.

Content

This is the sort of technical book that feels less like a formal textbook and more like spending a few very productive days with an experienced instructor. That makes sense: the book grew out of a five-day course, and its examples are built around Python, pandas, scikit-learn and Jupyter notebooks. The result is practical, opinionated and surprisingly personable. The language is quite precise but a little less conversational than I would have liked. It’s very to the point.

I did have a chuckle when I saw it was called the Orange book but was then Green. It took a moment for the reason to hit me.

I spent a lot of time on machine learning over the years, and this book covered a lot of very familiar territory, even though I’m rusty on much of it. It focuses firmly on supervised learning with tabular data. It starts with statistical foundations, exploratory data analysis and cleaning, then moves through cross-validation, regression, classification, ensembles, hyperparameter optimisation, feature engineering and tabular foundation models. That order works well because Carl does not rush straight to the glamorous algorithms. He spends time on missing values, leakage, scaling, metrics and uncertainty i.e., the things that usually determine whether a model is genuinely useful or merely looks good in a notebook.

What I enjoyed most is the author’s willingness to take a position. He didn’t just present a neutral catalogue of algorithms, which would be really easy to do. Carl warns that repeated hyperparameter searching can overfit the validation set, questions the value of some common practices, and repeatedly argues for simple, robust models rather than unnecessary complex ones. He’s also to the point about imbalanced classification, describing undersampling as wasteful and oversampling as fundamentally questionable. You might not agree with every verdict that he presents, but the strong opinions make the book memorable and encourage you to think rather than simply copy recipes.

The explanations are usually compact and intuitive. I liked the description of overfitting, and the way it’s presented as learning the noise after the signal has already been captured, which is a much more useful mental picture than treating it as a vague warning. The book also frequently connects theory back to code, and the many diagrams help make abstract ideas such as gradient descent, calibration, ensembles and transformer architectures easier to grasp. There is enough mathematics to show what is happening underneath, but most sections get quickly back to implementation.

I also liked the character shown in the book. Chapters start with quotations, the olive-themed design gives the book a recognisable visual identity, and small asides such as turtles all the way down when discussing surrogate models, were very familiar to me. And they stop the material from feeling sterile. The linked GitHub resources are another real strength, because this is a book to read beside a laptop rather than passively from the sofa. Even when Craig is being mathematically precise, he generally sounds like a teacher anticipating the question you were about to ask, or the mistake you might make.

That said, compact occasionally becomes compressed. I’m worried that some topics arrive with equations and terminology at a pace that might scare off a beginner. The breadth is impressive, but it also means that subjects such as generalized additive models, conformal prediction and foundation models sometimes feel more like guided introductions than full treatments. The recommended-reading sections do help, although readers wanting deep proofs, production deployment or end-to-end engineering guidance will need other books.

I really liked the final chapter. It’s a great addition to the book. Rather than pretending neural networks automatically win everywhere, Craig explains why traditional multilayer perceptrons often struggle on tabular data and introduces newer models such as TabPFN and TabICL. This keeps the book current without losing its central message about understanding your data, evaluating honestly and choosing models for evidence-based reasons.

Summary

This book is an energetic, practical and refreshingly candid guide. It is best suited to readers who know a little Python and want to understand not only how to fit models, but how to avoid fooling themselves. It is not effortless reading though, yet I think it’s sufficiently approachable to be useful and full of the sort of advice that should stick.

8 out of 10

Logic Apps are closely related to Power Automate, but they’re designed for enterprise IT teams that need more control, scalability, and governance.

Like Power Automate, Logic Apps let you automate workflows across systems using a visual, low-code designer.

However, Logic Apps are built on Azure, which gives them access to more advanced capabilities — such as higher throughput, better integration with on-premises systems, and enterprise-grade reliability.

You can think of Power Automate as being focused on business users and citizen developers, while Logic Apps are aimed at IT professionals and developers who build and maintain enterprise integrations.

One of the key advantages of Logic Apps is their support for DevOps and infrastructure-as-code practices. You can manage workflows using tools like Azure Resource Manager templates, Bicep, or GitHub Actions, and deploy them consistently across multiple environments.

This makes Logic Apps a strong choice for mission-critical workloads — scenarios that require governance, version control, or automated deployment pipelines.

The good news is that Activator can trigger both Power Automate and Logic Apps. That means you can choose the right workflow tool for each situation:

• Power Automate for business-led, lightweight automation
• Logic Apps for enterprise-scale, IT-managed solutions

Together, they allow Activator to integrate seamlessly across both business and technical domains, enabling organizations to respond to real-time events with the right level of control and reliability.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Database mirroring was added way in SQL Server 2005. One of the features added to it later in SQL Server 2008 was auto page repair. When SQL Server 2012 was released, Availability Groups also offered auto page repair.

Just how useful is this feature though ?

I’ll start by saying that it can’t hurt.

When SQL Server is reading a page on the primary replica and receives an unrecoverable I/O error (typically but not always, an error 823 for a checksum error), it will try to repair the page when:

• A secondary synchronous replica exists
• The secondary replica is synchronized

It will try to recover a copy of the page from the secondary replica and rewrite it on the primary. This means that when you execute a query on the primary, and it returns an I/O error, that retrying the same query a short time later might actually work.

In the meantime the page is added to the suspect pages table in msdb, and if you try to query it while the auto page repair is being attempted, you’ll get an error 829 that tells you the page is in recovery.

Did the page get repaired ?

Maybe, but in many cases, probably not.

Let’s start by saying that if a storage subsystem is returning I/O errors, it’s likely that you have a problem that really needs fixing.

Generally, auto page repair is more like a bandage or a band-aid that’s been applied.

However, storage subsystems are often now self-healing: they recognize that an I/O error occurred in one location, and automatically map that block to another underlying location. So when SQL Server then rewrites the original data, there is a chance that it will have actually fixed the problem.

It’s also worth keeping mind, that as storage systems get larger and larger, we’re starting to test the standard error rates of many underlying I/O devices. One unrecoverable error every so often is actually deemed acceptable. The proportion is really very, very low. But as we start moving larger and larger amounts of data around, we can start to hit these numbers.

I suspect that as storage gets even larger, we will start to be aware of these issues more and more.

What about the secondary replica ?

The same logic applies. The secondary can attempt to retrieve a copy of the page from a synchronized primary replica. This could occur when the secondary replica is being read. In mirroring, this could have been if we had made the replica readable via database snapshots, and in Availability Groups, this could just be a readable secondary replica that hits an I/O error on its storage.

So is it a good thing ?

Of course. It’s a help, but it’s important to understand that if pages are appearing in the suspect pages table in msdb, you have an underlying problem that needs to be resolved.

Power Automate is Microsoft’s no-code workflow automation platform, and it integrates directly with Activator to help turn events into business actions.

When a rule in Activator is met — for example, when a sensor crosses a threshold or a transaction fails multiple times — that event can automatically trigger a Power Automate flow.

A Power Automate flow is a sequence of steps that perform an operation, often involving multiple systems. For example, a trigger from Activator could:

• Create an item in SharePoint to log the event,
• Send a message to Microsoft Teams,
• Email a service desk using Outlook.

Power Automate includes over 1,000 built-in connectors, covering popular services such as Salesforce, ServiceNow, Dynamics 365, Azure services, and many others. That means you can extend Activator events well beyond Fabric, into almost any business system in your organization.

This integration is especially useful for business process automation. It allows Activator to initiate workflows that handle approvals, notifications, or escalations — all without writing code.

Because Power Automate uses a visual, step-based interface, it’s accessible not just to developers but also to citizen developers and business analysts. They can build and modify automations that respond to real-time events without needing IT to intervene.

Integrating Power Automate gives Activator the ability to connect insights to actions across your organization — bridging the gap between real-time detection and business process automation.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

I work with a lot of SQL Server databases that are poorly normalized. One of my pet dislikes is the column to rule them all.

Here are simple tests:

• If I ask you what’s stored in a column and you can’t tell me a single answer, then you’ve got a problem.
• If you need to refer to another column to work out what’s in the first column, then you’ve got a problem.

Here are some examples:

• If you have a column (let’s call it ObjectID) that sometimes holds a TeamMemberID, sometimes it’s a CoachID, sometimes it’s a TeamID, etc. then you have a design problem.
• If you must refer to another column (let’s call is ObjectType), to work out what’s in the ObjectID column, then you have a design problem.

Instead of a combination of ObjectType and ObjectID, I’d rather see you have a TeamMemberID column that’s nullable, a CoachID column that’s nullable, a TeamID column that’s nullable, etc. And at least there’s a chance that you could one day even have foreign keys in the database, and some chance of integrity. (But that’s a topic for another day).

One of the strangest reasons that I’ve heard for this was to “try to minimize the number of columns in the table”. Please don’t say that. No sensible person is going to ever exceed the limits.

Prior to SQL Server 2008, the limit for the number of columns per table was 1024.

It’s hard to imagine what you’d use more than that for, but the SharePoint team asked to have that increased. Apparently, 10,000 columns wasn’t enough, so we ended up with 30,000 columns per table now. I struggle to think about what type of design leads to that many columns but it’s also whySPARSE columns and filtered indexes were also added in that version. (Mind you, filtered indexes were a great addition to the product on their own, unrelated to SPARSE columns). Let’s just leave that reason as “oh, SharePoint”. Can’t say I love their database design, at all.

But for the rest of us, limiting the number of columns in a table isn’t a valid reason for messing up normalization, particularly when those columns are keys from other tables.

After Activator detects that a condition or pattern has been met, the next step is to trigger an action.

Actions are what turn analytics into operational responses — they ensure that insight leads directly to something happening.

Actions can be either internal or external:

Internal actions occur within Fabric — for example, updating a dataset, refreshing a Power BI report, or writing a new event back into an Eventstream.

External actions extend beyond Fabric, such as sending an alert to Microsoft Teams, emailing a support team, or calling an external API endpoint to update another system.

This is the point where real-time data becomes real-world activity. For example, if a sensor reports a critical temperature spike, Activator could automatically trigger a Power Automate flow that disables equipment, or send a Teams message to the operations team.

One of the most important aspects of actions is that they must be both timely and reliable. Real-time automation only adds value if the response happens quickly and consistently. Activator is designed to handle these triggers as soon as they occur, minimizing latency between detection and response.

This process of taking automated action completes the cycle of Real-Time Intelligence — moving from monitoring and detection to actual, tangible response.

In practice, this is often described as closing the loop. It’s the point where insights stop being purely analytical and start becoming operational.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

Intrusions into computer systems are happening all the time now. We need to address this issue as an industry, but it’s important to understand that the way we design databases plays a big role in the impacts that occur during intrusions.

If you don’t accept that you could have an intrusion, you are living in La La Land. (See https://en.wikipedia.org/wiki/Fantasy_prone_personality )

A bug in any one of the frameworks that you use, the code that you write, the protocols that you use, the operating system or hosting services that you use can potentially expose you to an intrusion.

So do we just give up?

No, what you need to ensure is that when an intrusion occurs, the damage or impact is minimized. We do this in all other industries. For example, people working in high locations don’t expect to fall but they (generally) make sure that if they do, while something nasty might happen, it won’t be disastrous.

I routinely see web applications and middleware that can access any part of a database that it wants. The developers love this as it’s easy to do. But it exposes you to major risks. If the application suffers an intrusion, you’ve opened up everything.

I always want to put mitigation in place and to limit the damage.

If your plan is to have your application connect to the database as one user, and you make that user a database owner (dbo), or a combination of db_datareader and db_datawriter, or worse, a system administrator; then you don’t have a plan.

A better plan is this:

• Create a schema for the application – In this case, let’s call it WebApp
• In the WebApp schema, create only the views and procedures that define what you want the application to be able to do (ie: it’s basically a contract between the database and the application)
• Create a new user (from a SQL login or, better-still, a domain service account) for the application to connect through.
• Grant that user EXECUTE and SELECT permission on the WebApp schema (and nothing else)

Then if the application is trampled on, the most that it can do is the list of things that you’ve defined in that schema and nothing else.

We need to start building systems more defensively, and this is reason number 82938429 for why I’m just not a fan of most ORMs as they tend to encourage entirely the wrong behavior in this area. (Some let you do it better begrudgingly).

In many real-time systems, a single event on its own doesn’t tell the full story. Pattern detection allows you to identify sequences or combinations of events that together indicate something meaningful.

For example, a single failed login attempt is rarely an issue, but multiple failed logins within five minutes could suggest a security concern. Similarly, a single temperature reading above normal might not matter, but a rising trend over time could signal equipment failure.

These kinds of patterns help us move beyond simple thresholds to understand behaviour over time or relationships between events.

In Activator, pattern detection is handled through rules that monitor event sequences or time windows. You can define conditions that must occur in order or within a defined time range. For instance, a rule might watch for three warning events followed by one failure event within a ten-minute window.

Many of these same capabilities are also available through KQL functions such as series_decompose, series_outliers, or join kind=innerunique, which can help identify patterns or correlations directly in queries before they’re fed into Activator.

Pattern detection supports a wide range of advanced scenarios — from fraud detection, to IoT monitoring, to real-time operational analytics — anywhere you need to understand not just single data points, but how events evolve and relate over time.

Ultimately, pattern detection helps your automation respond to context, not just to isolated events.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

I had the pleasure of recording a new PG Down Under podcast today. It was great to have Charles Feddersen from Microsoft back on the show.

Charles is the Partner Director of Product Management, Postgres & MySQL on Azure at Microsoft.

He leads the product management teams for the Postgres and MySQL managed services on Azure including Azure Database for PostgreSQL, Azure Database for MySQL, and Azure HorizonDB, which is why I wanted to talk to him today.

This show

In this show, we introduce Azure HorizonDB and drill into what it provides and aspects of its architecture. We also discuss the related updates to VSCode tooling.

I hope you enjoy listening to it.

You’ll find this show along with all PG Down Under shows here:

PG Down Under Podcast

In Activator, rules are the core element that make automation possible. A rule defines the condition you want the system to look for in your data — the point at which something should happen.

Rules can be simple or complex.

A simple example might be: temperature > 90°C or error rate exceeds 5%. These are fixed thresholds — clear, measurable values that indicate when a limit has been crossed.

But rules can also be more advanced, such as detecting an anomaly in a time series, identifying patterns across multiple signals, or combining several conditions together. For example, you might define a rule that triggers only when CPU usage is high and network latency increases at the same time.

The key advantage is that these rules are configured without writing code. You can define them visually in the Activator interface, using the available signals and comparison logic, or link them to results from KQL queries that already perform the analytical work.

Rules can be applied directly to incoming event streams, where data is evaluated continuously as it arrives, or to KQL query results, where you might check for trends or statistical changes.

When a rule condition is met, Activator creates a trigger, which then launches one or more actions — for example, sending a Teams notification or starting a workflow in Power Automate.

Rules are the foundation of Activator’s automation model. They bridge the gap between monitoring and action — turning real-time insights into operational responses.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

I recently received a review copy of Python Data Analysis: Master Python Analytics with Machine Learning, Deep Learning, GenAI, LLMs, and Data Engineering by Avinash Navlani and Cornellius Yudha Wijaya from my friends at PackT. This is the fourth edition of this book.

Authors

Avinash Navlani is a senior data scientist, researcher, and educator with a PhD in data science and many years in both the industry and in academia.

Cornellius Yudha Wijaya is a data science manager leading AI initiatives and driving the development of practical data and AI solutions.

Content

This is a large book; larger than I was expecting. It’s a guide to the increasingly broad world of data work with Python. The professional experience of the authors is apparent. Although its title suggests a conventional treatment of data manipulation and visualisation, the book reaches much further, covering statistics, machine learning, deep learning, natural language processing, image analysis, generative AI, parallel computing and PySpark. The strongest part is the way that it presents these subjects as parts of an end-to-end analytical workflow rather than as a disconnected catalogue of libraries.

The book starts with the foundations. Readers are introduced to common analytical processes, the Python tooling landscape, NumPy, pandas, statistics and linear algebra before moving into visualisation and data preparation. This ordering is important. Too many introductory books rush towards predictive models without first explaining how to understand, clean and transform the underlying data. Here, the chapters on retrieving data, dealing with missing values and outliers, feature engineering and time-series preparation reinforce the less glamorous but essential work that precedes useful modelling.

As well as reading data from familiar formats such as CSV, Excel, JSON and Parquet, the discussion also extends to relational and NoSQL databases, APIs, AWS S3 and Azure Blob Storage. Visualisation progresses from Matplotlib and Seaborn to interactive Plotly charts and Dash dashboards. Later chapters introduce supervised and unsupervised learning, ensemble techniques, neural networks, text and image processing, LLMs, Dask, Modin, Ray and PySpark.

Once again, it was good to see downloadable code that supports the examples, making the book most valuable when read beside a working Jupyter or Visual Studio Code environment.

This breadth is also the book’s main limitation. No single volume can explore all these areas deeply, and the later chapters are best viewed as practical introductions rather than complete treatments. Readers hoping to master deep learning, LLM fine-tuning or distributed Spark engineering will need more specialised resources afterwards. Similarly, despite the progressive structure, this is not really a first book on programming. Basic familiarity with Python is assumed, and the advanced material will be easier for readers who already understand elementary statistics and machine-learning concepts.

The teaching style is direct and example-driven. Concepts are usually followed quickly by code, output and interpretation, which helps readers connect theory with implementation. The model-evaluation examples are particularly useful because they encourage comparison rather than merely showing how to fit an algorithm. Chapters conclude with summaries, although the book would be stronger with more structured exercises, review questions or larger projects that integrate techniques across several chapters.

I was reading an early-review copy, and it occasionally contains awkward phrasing, inconsistent capitalisation and terminology that could benefit from another editorial pass. But these issues are minor. Given the book’s size and range, readers may also prefer to treat it as both a guided course and a reference, selecting chapters according to their immediate needs.

Summary

This book provides a substantial and useful survey of modern Python analytics. It is particularly well suited to analysts, developers and students who know some Python and want to connect data preparation, visualisation, modelling and scalable processing into a solid workflow. Its best value is in showing how the many pieces of today’s Python data ecosystem fit together. For readers seeking a broad, hands-on route from pandas to machinehg learning, generative AI and big-data tools, it provides an excellent starting point and a useful desk reference.

8 out of 10

Before you start creating rules in Activator, it’s important to understand how the system models data conceptually — using Objects and Signals.

An Object in Activator represents a real-world entity that you want to monitor — for example, a sensor, a customer account, a store, or a device.

Each Object has one or more Signals, which are the measurable data points associated with it. For example, a device object might have signals like temperature, battery level, and status.

Signals can originate from Eventstreams, KQL Database queries, or even dashboard visuals — anywhere real-time values are available in Fabric.

Once the Objects and Signals are defined, rules operate on them continuously. For instance, you could create a rule that watches each sensor’s temperature signal and triggers an action if it exceeds 70°C.

This object-and-signal model is what makes Activator context-aware — it doesn’t just react to isolated numbers, it understands which entity those numbers belong to. That approach allows for flexible, scalable monitoring — whether you’re tracking a handful of assets or thousands of devices streaming data simultaneously.

Learn more about Fabric RTI

If you really want to learn about RTI right now, we have an online on-demand course that you can enrol in, right now. You’ll find it at Mastering Microsoft Fabric Real-Time Intelligence

If I want accounting advice, it’s unlikely that I’d ask my dentist for that advice.

Accountants and IT

Many years ago, I created applications for food wholesalers. When the owners of these businesses decided to get a new or better computing system, invariably they’d speak to their accountants. I understand the reasons why that might seem logical to them at first, but what I saw when these clients did this, is that they invariably ended up with the wrong systems.

Why?

If you talk to the accountants, their recommendations would often be based on how good the general ledger was. They wanted to make sure that the figures that came to them from the business were already in a good state for them to use.

But to someone selling meat or fish or small-goods, that’s not the issue. It’s far more important for the system to understand how they sell and price food, how to track both quantity and weight, not just one value, etc. It’s critical to have a system that lets them manage their warehouses properly.

Very few of the systems recommended by the accountants did that. We often gained new clients who had made an initial misstep by purchasing what their accountant recommended. (And I’ll ignore the situations where the accountant was also being paid a commission by the software vendor).

Large financial organizations

So why am I raising this today?

I spend a lot of time working in large financial organizations, and security is a big issue for them. However, what I see time and again, is that they hire large accounting firms or legal firms to perform tasks like pen-testing (penetration testing), security audits of applications and systems, etc.

It’s hard to imagine why anyone would expect their accountants or legal advisers to be at the cutting edge of computer security. And as someone who’s involved in training people from those types of firms, I know that they might try hard but I can assure you that they aren’t anywhere near the current state of the art.

Litigation target?

Perhaps they think these firms are large enough that they’d be a good litigation target if something goes wrong (even though you can be sure their terms and conditions would prevent that), or that it somehow looks good to the market to use a big name accounting or legal firm.

If I really needed to secure or test the security of a system though, I’d be looking to use a boutique consultancy that specializes in that type of work. There are many consultants who are outstanding at this type of work.

They are good at what they do, and I’ll bet they don’t offer dental advice either.