Azure-Data-Factory-Adf

I’ve heard several people complaining that Azure Data Factory (ADF) is expensive to use, and yet I’ve seen many others talking about it as a really low cost service.

So which is true?

Our experience

We’re very careful about which features of ADF that we use as there are a few that can increase the cost substantially. And without those, it becomes a very low cost service.

None of my current clients has any issue with ADF costs. The people that I hear complaining about the cost are using one or both of:

2025-07-05

A few weeks ago, I received another book from our friends at PackT. It was the second edition of Azure Data Factory Cookbook by Dmitry Foshin, Dmitry Anoshin, Tonya Chernyshova, and Xenia Ireton.

I liked the earlier version of this book, and I was pleased to see substantial work had gone into this second edition.

It’s interesting that the book doesn’t stick directly to Azure Data Factory (ADF) but also branches into topics on Azure Synapse Analytics, Microsoft Fabric, and Databricks.

2024-06-14

The people at PackT recently sent me a book to review, and I was happy to do so as it was on a topic that’s dear to my heart: Azure Data Factory. The book was Azure Data Factory Cookbook and it’s the second edition of the book. The authors are Dmitry Foshin, Tonya Chernyshova, Dmitry Anoshin, and Xenia Ireton.

PackT

In the past, I wasn’t keen on PackT books. When they first appeared, they tended to be low cost books from unknown authors, many of whom struggled with writing in English, and pretty poor editing of the content.

2024-04-20

I was doing work for a client that uses Azure Data Factory (ADF), and noticed an unexpected Error code 2011 from a Get Metadata activity.

The pipeline was using a Get Metadata activity to read a list of files in a folder in Azure Storage, so that it could use a ForEach activity to process each of the files. Eventually it became clear that the error occurred when the folder was empty.

2024-02-07

It’s scary how much of the world’s data lives in CSVs. Yet, as a standard, it’s problematic. So many implementations just don’t work as expected. Today, I had a client asking about why Azure Data Factory wouldn’t read a CSV that, while odd, was in a valid format.

The simplest equivalent of the file that wouldn’t load, would be one like this:

First Column,Second Column,Third Column,Fourth Column
12,Terry Johnson,Paul Johnson,031-23423
13,Mary Johnson,"Paul,Johnson",031-23423
14,Mia Johnson,"Paul ""the beast"", Johnson",031-23423
16,Cherry Johnson,Paul Johnson,031-23423

There are meant to be four columns. The source system wrapped strings in quotes only when the string contained a comma, as that was the delimiter for the file.

2023-09-12

I was recently doing work for a client where I needed to replace some Python code that was being used to upload a file attachment to a REST based service using a POST command. I was trying to do it using a Web Activity in Azure Data Factory. What was surprising was how little information I could find on how to do that.

While I can’t post all the code, I did want to post the core parts here in case it helps others, and for my own future reference. The basic steps involve:

2023-04-29

This one had me stumped lately.

I had an variable in ADF where the value contained carriage returns. I needed to remove them.

I guessed that would be easy:

@replace(variables(‘fileContent’), ‘\r’, ‘’)

But no matter what variation of this I tried, it wouldn’t remove them.

Thanks to the genius of fellow MVP Simon Sabin, the answer was:

@replace(variables(‘fileContent’),base64ToString(‘DQ==’),’’)

Note: if you need a line feed (LF) it’s base64ToString(‘Cg==’) and a carriage return / line feed pair (CRLF), is base64ToString(‘DQo=’).

2023-04-24

I was working with a client who has having trouble debugging an ADF pipeline, related to an ODBC linked service not working as expected.

The user had configured the connection string property of an ODBC connection this way:

•  Set a parameter to the linked service as ServiceDSN
•  Configured the connection string as @concat(‘DSN=’,linkedService().ServiceDSN)

The Test Connection for that worked fine, but when you click Apply, it fails with an Internal Server Error. Specifically, the error was:

2023-04-24

Azure Data Factory (ADF) is great for moving data around. It often needs to connect to SQL databases. If you’re creating linked services for this, please try to avoid using SQL logins. Don’t use usernames and passwords for authentication if you can avoid them.

Azure SQL Database lets you can create a user from a data factory’s managed service identity (MSI). MSIs are a special type of Service Principal (SP). They provide an identity where you don’t need to manage the ID or any sort of password.  Azure does that for you.

2021-09-26

I’m a big fan of Azure Data Factory (ADF), but one of the things you need to get used to with tools like this, is that the UI keeps changing over time. That makes it hard for several reasons:

• It’s hard to train people. Any recorded demo you have will show them things that no longer exist, within a fairly short period of time.
• Every time a cosmetic change occurs, it immediately devalues blog posts, tutorials, etc. that are out on the Internet.

I think Microsoft don’t quite get how much blog posts, etc. supplement their own documentation.

2021-03-09