BI: Azure Data Factory Copy Activities Won't Start (Queued)

I love working with Azure Data Factory (ADF). There are times though, when I’m stuck wondering what’s going on, and it takes me a moment to discover it.

One that had me puzzled recently was a Copy activity that just wouldn’t seem to start. The pipeline had started ok. Previous activities in the pipeline had run just fine. But when it got to the Copy activity, it said “Queued” and just kept saying that seemingly endlessly.

Integration Runtimes

The first thing to understand is that these copy activities access datasets that are connected to via linked services:

If we look at the properties of the types of linked services that we’re copying to/from, you’ll see that they in turn are accessed using Integration Runtimes.

Concurrency

The integration runtimes have concurrency limits.

So one thing that could cause your Copy activity to queue is if the number of concurrent jobs was exceeding this limit.

However, in the case I’m talking about, this was the only Copy activity running.

Linked Shared Integration Runtimes

Integration Runtimes can be shared between data factories. The trick in this case was that the integration runtime was linked to one shared by another data factory. In fact, 9 data factories were linked to this same shared integration runtime.

What I should have checked was the Monitor for the linked shared integration runtime:

That would have shown the total of what was going on:

Previously, you couldn’t see the totals from a data factory that was linking. You needed to check from the data factory that was sharing.

The Cause

My Copy activity seemed to be queued for no reason. But of course there was a reason. Another data factory had a bug and was consuming all the available concurrency from the shared integration runtime.

2021-01-06