Book Review: Data Pipelines with Apache Airflow (2nd Edition)

Book Review: Data Pipelines with Apache Airflow (2nd Edition)

Another Apache technology that I come across at client sites is Airflow. I recently purchased Data Pipelines with Apache Airflow (2nd Edition): Orchestration for data and AI to get a better understanding of it. Note that even though a copy was included with this book when I purchased it, I didn’t get to read the first edition of this book. I can only comment on the 2nd edition.

Authors

Julian de Ruiter is field CTO for Data and AI at Xebia Data.

Ismael Cabral is a machine learning engineer and Airflow trainer.

Kris Geusebroek is a data engineering consultant.

Daniel van der Ende is a data enginner.

Bas Harenslak was the author of the first edition of this book and is a staff architect at Astronomer.

Data Pipelines with Apache Airflow (2nd Edition)

I often get nervous when the number of authors for a book increases. It often leads to great inconsistency, different writing styles, and duplication. This book showed no signs of that.

Data pipelines often start out looking simple. Extract some data, transform it, load it somewhere, and schedule the whole thing to run overnight. The problems usually appear later, when a task fails, data arrives late, or someone needs to reload six months of history. That’s where orchestration tools become important. This book does an excellent job of explaining how Airflow can help, but also shows that building reliable pipelines takes more thought than simply connecting a few tasks.

One of the things I like most about this book is its emphasis on doing things properly. The authors don’t just show how to make Airflow work. They spend considerable time explaining how to build pipelines that are maintainable, testable, and reliable. I appreciated that. It’s relatively easy to demonstrate a working pipeline. It’s much harder to build one that someone else can understand and support a year later.

I’m told that this book has been substantially updated for Airflow 3, and that’s important. I have read that the changes to scheduling, the user interface, and the underlying architecture are significant. So, this isn’t just the first edition with a few new screenshots. The coverage of asset-aware scheduling is particularly welcome. Being able to trigger workflows when data becomes available, rather than just at a particular time, is a much more natural approach for many data engineering problems.

I also enjoyed the explanations of time-based scheduling and backfilling. Airflow’s approach to dates and scheduling can be surprisingly confusing, even for experienced developers. The authors take time to explain the differences between trigger-based and interval-based scheduling, and why they matter. I particularly liked the discussion of idempotency: designing tasks so that running them again doesn’t produce incorrect results. For me, this is one of the most important principles in data engineering, and I’m pleased it receives proper attention. No matter what tools I’m working with, I try to design pipelines so they can be re-run without issue.

The chapters on building custom operators, hooks, and sensors are another strength. Airflow is a flexible platform, but most of that flexibility comes from writing Python code. I mostly work with Fabric Data Factory now, and the ability to easily extend it via a high-level language is something I miss. Yes, I can use notebooks and launch them from the pipeline, but it’s not the same. In this book, the authors explain how to extend the platform and how to package reusable components. I also appreciated the discussion of deferrable operators, which can avoid wasting worker resources while waiting for external processes to complete.

One of my favourite sections is the chapter on testing. Too many data engineering teams still treat testing as something that applies to application development but not to data pipelines. The authors clearly disagree, and so do I. Their coverage of unit testing, integration testing, mocking external systems, and CI/CD pipelines is practical and valuable. The later chapter on best practices reinforces this with advice about keeping DAG definitions clean, managing configuration, and avoiding unnecessary processing.

The two larger project examples help bring the material together. One uses New York City taxi and bike data to compare travel times. These are well-known datasets in the industry and are used as samples for many products, including Fabric tools that I often work with. The other example builds a generative AI application around a collection of family recipes, using retrieval-augmented generation (RAG). I found the first example particularly effective because it represents the sort of data integration problem that many teams actually face. The AI example is interesting too, although I suspect some readers would have preferred another traditional data engineering scenario.

I was also pleased that the book doesn’t stop at development. The final chapters cover production deployment, scaling, monitoring, security, and Kubernetes. These are essential topics. A pipeline that runs perfectly on a developer’s laptop is only the beginning. Choosing executors, managing the metadata database, collecting logs, and setting up alerts all matter when the system becomes business-critical.

Limitations?

I do have some reservations, but not many. First, this is very much a Python developer’s book. Readers coming from SQL Server Integration Services, Azure or Fabric Data Factory, or other graphical pipeline tools may find the amount of code intimidating. I don’t mind it, but I suspect that I’m the exception here. The authors make their expectations about Python knowledge clear, but I think that limits the book’s audience.

Second, Airflow itself introduces quite a lot of moving parts. By the time you consider the scheduler, workers, metadata database, executors, containers, and deployment infrastructure, the operational overhead can be substantial. The book explains these components well, but I would have liked more direct comparisons with simpler orchestration options, particularly for smaller teams.

I also think it’s important to remember that Airflow is primarily an orchestrator, not a data processing engine. The authors make this distinction, and they correctly point out that it isn’t designed for continuous streaming workloads. That message deserves emphasis because choosing Airflow doesn’t remove the need for other tools to do the actual data processing.

Summary

I strongly recommend this book to data engineers and developers who want to build serious pipelines with Airflow. It’s detailed, practical, and especially valuable in its treatment of testing and good engineering practices. It isn’t a quick introduction for someone who wants to avoid writing code, and not every organization needs the complexity that comes with Airflow. But for teams that have chosen Airflow, or are seriously considering it, this is an excellent guide to getting beyond simple examples and building something that will stand up in production.

8 out of 10

2026-11-17