Azure-Data-Factory-Adf

I’m a big fan of Azure Data Factory (ADF), but one of the things you need to get used to with tools like this, is that the UI keeps changing over time. That makes it hard for several reasons:

• It’s hard to train people. Any recorded demo you have will show them things that no longer exist, within a fairly short period of time.
• Every time a cosmetic change occurs, it immediately devalues blog posts, tutorials, etc. that are out on the Internet.

I think Microsoft don’t quite get how much blog posts, etc. supplement their own documentation.

2021-03-09

I recorded another podcast with an old friend recently. I met Kamil Nowinski in Poland when I was invited to speak at SQL Day. That event is on again this year, but as a virtual event only.

In our consulting work, we use Azure Data Factory frequently. Deploying it, can be a real challenge. Microsoft has a supplied method with ARM templates but that has issues. Kamil produced free tooling that’s in the Azure Marketplace and makes it much easier to work with.

2021-02-08

I recently recorded a podcast with Kamil Nowinsky. Kamil is well-known for creating some tools that make it easy to publish individual Azure Data Factory pipelines. It’s a far better method that the standard ARM (Azure Resource Manager) template deployment that’s provided by Microsoft.

However, if you use this method (and I recommend you do), you’ll be publishing pipelines separate to the linked services that your datasets connect to.

One mistake I’ve seen a few clients making lately, is that when they created the linked services in different environments, they did so with slightly different names. Because of that, they needed to modify the deployed objects after deployment. You don’t want to be doing that.

2021-01-27

I love working with Azure Data Factory (ADF). There are times though, when I’m stuck wondering what’s going on, and it takes me a moment to discover it.

One that had me puzzled recently was a Copy activity that just wouldn’t seem to start. The pipeline had started ok. Previous activities in the pipeline had run just fine. But when it got to the Copy activity, it said “Queued” and just kept saying that seemingly endlessly.

2021-01-06

I work with a lot of technology on a daily basis, much of it amazing. But I still get excited when relatively small enhancements are made, and they make my life or development much better.

Timezone support in schedule triggers in Azure Data Factory is one of those changes.

Schedule Triggers

In Data Factory, the most common type of trigger that we use is a schedule trigger. It’s used to run a pipeline at a specific time or series of times. But one of the most painful aspects of these triggers is that they didn’t have time zone support.

2020-11-04

I’ve been working a lot with ADF (Azure Data Factory) again lately. If you’ve started to work with ADF and aren’t using Git in conjunction with it, you need to change that now.

ADF objects are also represented as JSON objects and lend themselves nicely to being stored in Git.

Another key advantage is that if you don’t have Git connected, when you’re working with ADF, you don’t have a Save button. If you’re part way through making changes, you can either Publish them or Discard All. When you have Git in place, you get an option to Save.

2020-02-21