Azure-Data-Factory-Adf

A few weeks ago, I received another book from our friends at PackT. It was the second edition of Azure Data Factory Cookbook by Dmitry Foshin, Dmitry Anoshin, Tonya Chernyshova, and Xenia Ireton. I liked the earlier version of this book, and I was pleased to see substantial work had gone into this second edition. It’s interesting that the book doesn’t stick directly to Azure Data Factory (ADF) but also branches into topics on Azure Synapse Analytics, Microsoft Fabric, and Databricks.

2024-06-14

The people at PackT recently sent me a book to review, and I was happy to do so as it was on a topic that’s dear to my heart: Azure Data Factory. The book was Azure Data Factory Cookbook and it’s the second edition of the book. The authors are Dmitry Foshin, Tonya Chernyshova, Dmitry Anoshin, and Xenia Ireton. PackT In the past, I wasn’t keen on PackT books. When they first appeared, they tended to be low cost books from unknown authors, many of whom struggled with writing in English, and pretty poor editing of the content.

2024-04-20

I was doing work for a client that uses Azure Data Factory (ADF), and noticed an unexpected Error code 2011 from a Get Metadata activity. The pipeline was using a Get Metadata activity to read a list of files in a folder in Azure Storage, so that it could use a ForEach activity to process each of the files. Eventually it became clear that the error occurred when the folder was empty.

2024-02-07

It’s scary how much of the world’s data lives in CSVs. Yet, as a standard, it’s problematic. So many implementations just don’t work as expected. Today, I had a client asking about why Azure Data Factory wouldn’t read a CSV that, while odd, was in a valid format. The simplest equivalent of the file that wouldn’t load, would be one like this: First Column,Second Column,Third Column,Fourth Column 12,Terry Johnson,Paul Johnson,031-23423 13,Mary Johnson,"Paul,Johnson",031-23423 14,Mia Johnson,"Paul ""the beast"", Johnson",031-23423 16,Cherry Johnson,Paul Johnson,031-23423 There are meant to be four columns.

2023-09-12

I was recently doing work for a client where I needed to replace some Python code that was being used to upload a file attachment to a REST based service using a POST command. I was trying to do it using a Web Activity in Azure Data Factory. What was surprising was how little information I could find on how to do that. While I can’t post all the code, I did want to post the core parts here in case it helps others, and for my own future reference.

2023-04-29

This one had me stumped lately. I had an variable in ADF where the value contained carriage returns. I needed to remove them. I guessed that would be easy: @replace(variables(‘fileContent’), ‘\r’, ‘’) But no matter what variation of this I tried, it wouldn’t remove them. Thanks to the genius of fellow MVP Simon Sabin, the answer was: @replace(variables(‘fileContent’),base64ToString(‘DQ==’),’’) Note: if you need a line feed (LF) it’s base64ToString(‘Cg==’) and a carriage return / line feed pair (CRLF), is base64ToString(‘DQo=’).

2023-04-24

I was working with a client who has having trouble debugging an ADF pipeline, related to an ODBC linked service not working as expected. The user had configured the connection string property of an ODBC connection this way: Set a parameter to the linked service as ServiceDSN Configured the connection string as @concat(‘DSN=’,linkedService().ServiceDSN) The Test Connection for that worked fine, but when you click Apply, it fails with an Internal Server Error.

2023-04-24

Azure Data Factory (ADF) is great for moving data around. It often needs to connect to SQL databases. If you’re creating linked services for this, please try to avoid using SQL logins. Don’t use usernames and passwords for authentication if you can avoid them. Azure SQL Database lets you can create a user from a data factory’s managed service identity (MSI). MSIs are a special type of Service Principal (SP). They provide an identity where you don’t need to manage the ID or any sort of password.

2021-09-26

I’m a big fan of Azure Data Factory (ADF), but one of the things you need to get used to with tools like this, is that the UI keeps changing over time. That makes it hard for several reasons: It’s hard to train people. Any recorded demo you have will show them things that no longer exist, within a fairly short period of time. Every time a cosmetic change occurs, it immediately devalues blog posts, tutorials, etc.

2021-03-09

I recorded another podcast with an old friend recently. I met Kamil Nowinski in Poland when I was invited to speak at SQL Day. That event is on again this year, but as a virtual event only. In our consulting work, we use Azure Data Factory frequently. Deploying it, can be a real challenge. Microsoft has a supplied method with ARM templates but that has issues. Kamil produced free tooling that’s in the Azure Marketplace and makes it much easier to work with.

2021-02-08