The Bit Bucket

ADF: Passing a file attachment via a POST to a REST service using Azure Data Factory

I was recently doing work for a client where I needed to replace some Python code that was being used to upload a file attachment to a REST based service using a POST command. I was trying to do it using a Web Activity in Azure Data Factory. What was surprising was how little information I could find on how to do that.

While I can’t post all the code, I did want to post the core parts here in case it helps others, and for my own future reference. The basic steps involve:

2023-04-29

MS Tech Summit 2023 - Discounts running out

We have so many great tech friends in Poland. I was pleased to see they are running the MS Tech Summit, and pleased to be speaking at it.

Looks like a great set of sessions : https://mstechsummit.pl/en/ They tell me there are more than 100 sessions spread over 8 tracks.

Not in Poland? Not a problem. It’s running in-person and online, and with video on demand.

The early bird discount is still available but about to end.

2023-04-28

ADF: Replacing carriage returns in Data Factory expression language

This one had me stumped lately.

I had an variable in ADF where the value contained carriage returns. I needed to remove them.

I guessed that would be easy:

@replace(variables(‘fileContent’), ‘\r’, ‘’)

But no matter what variation of this I tried, it wouldn’t remove them.

Thanks to the genius of fellow MVP Simon Sabin, the answer was:

@replace(variables(‘fileContent’),base64ToString(‘DQ==’),’’)

Note: if you need a line feed (LF) it’s base64ToString(‘Cg==’) and a carriage return / line feed pair (CRLF), is base64ToString(‘DQo=’).

2023-04-24

FIX: Data Factory ODBC linked service fails to Apply and returns Internal Server Error

I was working with a client who has having trouble debugging an ADF pipeline, related to an ODBC linked service not working as expected.

The user had configured the connection string property of an ODBC connection this way:

  •  Set a parameter to the linked service as ServiceDSN
  •  Configured the connection string as @concat(‘DSN=’,linkedService().ServiceDSN)

The Test Connection for that worked fine, but when you click Apply, it fails with an Internal Server Error. Specifically, the error was:

2023-04-24

Book Review - Make Your Data Speak - Alex Kolokolov

Over the last year, I’ve come to know Alex Kolokolov more, through involvement with his data visualization challenges. I was really pleased to see he’d written his first book Make Your Data Speak (Creating Actionable Data through Excel For Non-Technical Professionals).

Things I Liked

I really liked the conversational style of the book. It’s all structured around  an approach of “Let’s see how this happens by example”. The tone was really refreshing and should be good to hold people’s interest.

2023-03-12

SQL: Understanding Change Data Capture for Azure SQL Database - Part 2 - How does it work?

In the part 1 of this series, I discussed the positioning of Change Data Capture. In part 2, I want to cover how it works.

Log Reading

There are many ways that you can output details of changes that occur in data within SQL Server. Many of those methods require actions to occur at the time the data change is made. This can be problematic.

The first problem with this, is the performance impact on the application that’s making the change. If I update a row in a table and there is part of the process that writes details of that change to some type of audit or tracking log, I’ve now increased the work that needs to happen in the context of the application that’s making the change. Generally what this means, is that I’ve slowed the application down by at least doubling the work that needs to be performed. That might not be well-received.

2023-03-06

Opinion: Spinning Rust? Not so much

I’ve worked in this industry a long time now.

From 1981 to 1986, I worked as an engineer for Hewlett-Packard. Yep, I’m “have actually met both Bill Hewlett and Dave Packard” old. They’d retired when I met them and John Young was in charge, but they came to Palo Alto to a shareholders’ meeting and I was there at the time.

Most of my work was on HP3000 commercial mini-computers. The level of engineering on those systems was unlike anything I’d ever seen either before or since. They were really quite superb.

2023-03-05

SQL: Understanding Change Data Capture for Azure SQL Database - Part 1 - Why?

I often need to capture the changes from one database into another. The most common reason is that I’m wanting to bring changes from a transactional system across into a data warehouse that’s part of a BI setup.

So which technology is best to use for this?

That’s not a trivial question to answer but here are some thoughts:

Replication?

Unfortunately, this one’s not available for Azure SQL DB as yet. Azure SQL DB can be a subscriber in Transactional Replication. We often use it this way. If we have an on-premises SQL Server, one of our favourite ways to get data into the cloud is by using Transactional Replication. (If you need to get your head around Replication with SQL Server, just head to our course here).

2023-02-26

Cosmos Down Under podcast 6 with guest Vincent-Philippe Lauzon is now published!

I published a new Cosmos Down Under podcast today.

In the podcast, I’ve discussed the new integration between Azure Cosmos DB and Azure Data Explorer with the program manager for the feature: Vincent-Philippe Lauzon.

It’s a great new capability and worth half an hour of your time to find out about it.

I hope you enjoy it. You’ll find it, along with all Cosmos Down Under podcasts at:

Cosmos Down Under Podcast

2023-02-17