DevOps: Microsoft Professional Program for DevOps

In the second half of 2016, I enrolled in the Microsoft Professional Program for Data Science, and completed it in early 2017. I have to say that I really enjoyed it overall. It was a bit challenging at times but I don't regret doing it.

If you want to get the certification, you need to enroll in the verified option for each course. Nowadays, that's pretty much $99 USD per course. You can do it for free, and if you're tight on funds, perhaps that's what you should do. I like to support the concept, and like to support both Microsoft and edX for creating these options. They are doing amazing work, so while I hear people say to just do the courses and not contribute to them, I can't say that I agree.

edX and their partners offer an incredible range of world-class courses that you can take for free, but if you want them to continue, you should consider contributing. And that applies to the non-Microsoft ones too.

I think that programs like these are more likely to be the real future for Microsoft certification in general.

Earlier this year, Microsoft created a Professional Program for DevOps. I've had an interest in DevOps for a long time, and I got the opportunity to help create one of the courses DevOps for Databases with the inimitable Steve Jones from Redgate Software. Databases are a specifically-challenging area for DevOps.

A few months back I decided to start pursuing this professional program as well. I've got one course to go (the container one) before the final capstone project. I can finish that container course in the next three months, but unfortunately the capstone project won't be available until April.

Here's the overall program:

Over the last few weeks, I've been involved in enhancing the existing Monitoring and Testing courses, and am looking forward to seeing how people find the updated versions.

To support my continuing interest in DevOps, in the upcoming weeks, you'll see DevOps-related posts from me.

 

Opinion: If you can't type, you are writing worse code than if you could

Let me make a potentially bold statement:

People who can’t type write worse code than they could be writing

I’m sure that will upset some people (probably those who can’t type or who are two or four finger typists) but it’s a conclusion that I’ve come to over many years. Coding is clearly not the same thing as typing but the reason is simple:

To write good code, you need to be prepared to constantly refactor and rework the code that you write, and if you can’t do that quickly, you’ll be more reluctant to do it.

Time and again, I’ve seen people hanging onto code that should simply have been reworked, and it’s often because doing so it seen as too hard.

It's never been easier to learn to type. There are so many applications that can help you to learn, and there are no doubt a large number of online sites to help you do that.

When I worked at a university, I was in charge of a large amount of equipment used by students. Every day, I saw students struggling to get their work done. Worse, they were stopping other students from using the same systems.

I always thought that in the first week of first year, one of the best things we could have asked the students to do, was to learn to type. There's nothing else very useful they could have done in that week, so they might as well have done something that would benefit their whole careers during that dead time. And it would have reduced the amount of equipment that the university needed to purchase.

Sadly, it was made clear to me that universities don't get involved in things like typing.

Any why am I raising this today? It’s the day that many people make New Year’s Resolutions.

If you write code for a living (application developer, database developer or whatever), you owe it to yourself to remove friction between yourself and the computer. Invest in yourself. It’s the professional thing to do.

Happy New Year to all my readers (whether or not you can type).

澳大利亚新年快乐!

Happy Christmas to my blog readers (圣诞快乐)

It's hard to believe that we're back to Christmas time. I just wanted to take a moment to thank all those who've read my blog during the year and look forward to great interactions again next year.

It's been a big year for us. I've moved to a new blog, moved to a new website, and moved to a new house. All have been "entertaining" but I'm happy with the outcome in each case.

I hope you all have a happy Christmas period, no matter what your belief system is. For most it's a holiday period regardless.

If you are travelling, take care and travel safely. We had another tragedy in Melbourne yesterday, and it just shows how fleeting life can be.

圣诞快乐 to my Chinese readers too. I've made quite a bit of progress on my Chinese this year. The most interesting (yet sad) thing I did was to MC a memorial service for a friend's father, in both English and Mandarin. That was quite a challenge. Many of the things that I had to say were in very formal language. But I think I got through it OK. I'd never spoken to a crowd of people in Mandarin before, and particularly not a crowd of Chinese speakers. I just hope I didn't mess it up too much.

Thank you all for your support.

Opinion: Designing Databases to Minimize Damage During Application Intrusions

Intrusions into computer systems are happening all the time now. We need to address this issue as an industry, but it’s important to understand that the way we design databases plays a big role in the impacts that occur during intrusions.

If you don’t accept that you could have an intrusion, you are living in La La Land. (See https://en.wikipedia.org/wiki/Fantasy_prone_personality)

A bug in any one of the frameworks that you use, the code that you write, the protocols that you use, the operating system or hosting services that you use can potentially expose you to an intrusion.

So do we just give up?

No, what you need to ensure is that when an intrusion occurs, the damage or impact is minimized. We do this in all other industries. For example, people working in high locations don’t expect to fall but they (generally) make sure that if they do, while something nasty might happen, it won’t be disastrous.

I routinely see web applications and middleware that can access any part of a database that it wants. The developers love this as it’s easy to do. But it exposes you to major risks. If the application is trampled on, you’ve opened up everything.

I always want to put mitigation in place and to limit the damage.

If your plan is to have your application connect to the database as one user, and you make that user a database owner (db0), or a combination of db_datareader and db_datawriter, or worse, a system administrator; then you don't have a plan.

A better plan is this:

  • Create a schema for the application – let’s call it WebApp
  • In the WebApp schema, create only the views and procedures that define what you want the application to be able to do (ie: it’s basically a contract between the database and the application)
  • Create a new user (from a SQL login or, better-still, a domain service account) for the application to connect through.
  • Grant that user EXECUTE and SELECT permission on the WebApp schema (and nothing else)

Then if the application is trampled on, the most that it can do is the list of things that you’ve defined in that schema and nothing else.

We need to start building systems more defensively, and this is reason #82938429 for why I just don’t like most ORMs as they tend to encourage entirely the wrong behavior in this area. (Some let you do it better begrudgingly).

Opinion: Why ask accountants and lawyers for IT advice?

If I want accounting advice, it's unlikely that I'd ask my dentist for that advice.

Many years ago, I created applications for food wholesalers. When the owners of these businesses decided to get a new or better computing system, invariably they'd speak to their accountants. I understand the reasons why that might seem logical to them at first, but what I saw when these clients did this, is that they invariably ended up with the wrong systems.

Why?

If you talk to the accountants, their recommendations would often be based on how good the general ledger was. They wanted to make sure that the figures that came to them from the business were already in a good state.

But to someone selling meat or fish or small-goods, that's not the issue. It's far more important for the system to understand how they sell and price food, how to track both quantity and weight, not just one value, etc. It's critical to have a system that lets them manage their warehouses properly.

Very few of the systems recommended by the accountants did that. We often gained new clients who had made an initial misstep by purchasing what their accountant recommended. (And I'll ignore the situations where the accountant was also being paid a commission by the software vendor).

So why am I raising this today?

I spend a lot of time working in large financial organizations, and security is a big issue for them. However, what I see time and again, is that they hire large accounting firms or legal firms to perform pen-testing (penetration testing), security audits of applications and systems, etc.

It's hard to imagine why anyone would expect their accountants or legal advisers to be at the cutting edge of computer security. And as someone who's involved in training people from those types of firms, I know that they might try hard but I can assure you that they aren't anywhere near the current state of the art.

Perhaps they think these firms are large enough that they'd be a good litigation target if something goes wrong (even though you can be sure their terms and conditions would prevent that), or that it somehow "looks good to the market" to use a big name accounting or legal firm.

If I really needed to secure or test the security of a system though, I'd be looking to use a boutique consultancy that specializes in that type of work. There are many consultants who are outstanding at this type of work.

They are good at what they do, and I'll bet they don't offer dental advice either.

SQL: Database Design -> What's in a Name?

Just after I was born, my mother and father called me Gregory. Ever since then, everyone has called me Greg. And that included my parents. To this day, my mother calls me Greg and so did my dad while he was alive (miss you dad).

However, every time I need to fill in an official form, I have to write Gregory. I could change that to Greg if I changed my name legally but I'm not going to do that. People who have had previous names will tell you that can add even more complexity.

But I have to say that every time I get a letter from a bank, a utility company, etc. or every time I'm addressed by someone in a hospital or government office, they address me as Gregory. Each and every time they do that, at first, I end up momentarily thinking "who?".

Then it's obvious to me that as much as this person is trying to sound friendly, they haven't managed to do so. It immediately puts a barrier between us. Clearly they don't actually know me.

You might think "well what can I do about that?" or "how's that my problem?" or "what's this got to do with SQL?"

And I'll tell you.

Every time you build a computer system or database that has no option for a customer/member/client/patient/etc. to record what they'd like to be called, you add to the problem.

Please consider always having a PreferredName column or something similar in every design you create.

 

 

Opinion: Don't Design Databases for One Version of One App

I've pointed out in previous blog posts that I'm not a fan of ORMs. What I'm even less of a fan of is code-first design, particularly in combination with an ORM like the Entity Framework.

It might seem cool and shiny and if you are just whipping up a small proof of concept or test app, please feel free to do it, but the minute you think of doing it for enterprise systems, just don't. And if you have colleagues wanting to do this, please just say no.

For most organizations, the data that they own is one of the most (if not the most) valuable asset the company has. The data will generally outlive generations of applications and just be morphed from shape to shape over time.

The data will often be accessed by many different applications, often created with different technology stacks. You might think you'll be able to corral all access to the data via your app; and again you'll be wrong.

So designing the data storage to suit the needs of a single version of a single application at a single point in time, is what we baseball umpires know as "a big call".

Umpires know to make calls like this emphatically and confidently.

But this is not a call that you should be making. It's the quickest way to start building disconnected silos of information that don't represent the business or how the information in the business is inter-related.

 

Opinion: Don't just hire clones of yourself

Many years back, I was invited to chair a course accreditation panel for a local TAFE (Technical and Further Education) course. They had started to offer a computing-related 3 year diploma, and the hope was that it wasn’t too far below the 3 year degrees offered at local universities. One part of that accreditation process involved me discussing the course with the staff members who were teaching it.

After talking to almost all the staff, what struck me was how similar they all were. In the requirements for the course, there was a standard that each staff member needed to meet, but there was also a requirement for the group of staff to be diverse enough to have broad knowledge of the industry. There was no individual staff member that you could identify as not being at the appropriate standard, but almost all of them had exactly the same background, career progression, etc.

The manager had basically hired clones of himself. It’s an easy mistake to make. If you feel you are the right person for a particular type of job, then hiring more people like yourself must help correct?

A similar problem happens in areas like medical research. Taking a whole bunch of people with the same background and experience isn’t going to let you cut through tricky problems that need someone to think outside the box. Adding someone like a civil engineer into the mix might seem odd but can have surprising outcomes. At the very least, they might ask a question that leads someone else in the team to think differently.

I’m remembering this story because I see the same issue in application development groups.

I’ve done some work at a company that has over 400 developers. Data is almost all that they do, yet for most of the time the company has existed; they’ve had no-one focused on data. Everyone involved in development has a similar development background. They had many intractable data-related problems yet more and more of the same type of people wasn’t going to solve those.

Hiring a team of people who think and work like you do might seem like a good idea but it’s not. You need a mixture of people if you want to be really effective. (And that also means having gender and cultural diversity too).

Opinion: Mature consultants don't always want to tear down the house

I work with data. I understand that for most organizations, that the data they own is the most valuable asset the company owns.

One thing I've learned from working with data is that unlike application software, data generally outlives generations of applications, is often used by many different applications, and typically just morphs from shape to shape over time. It almost never gets totally refreshed.

This is a good thing.

I've been in the industry long enough to see many types of consultants. One type that I have the least time for, is the type that always wants to tear down or replace whatever is already there at an organization. It's far easier to just say "let's replace it all" than to try to work out what to do.

Many of these consultants don't really understand what's already in place, but because it looks different to what they are used to, it must be wrong, and it must be replaced.

A mature consultant might not like what they see but they take the time to consider what's already there.

The mature consultant is skilled enough to work out how to take the organization from a place that they don't want to be, to a place which is better than where they are now.

That's the real skill.

Opinion: Sticking with a plan even if you don’t like it

Something I really struggle with in this industry is when newcomers to a system want to change standards within existing systems because they think something else is better. It’s a sign of immaturity yet it often applies to people who should be senior. Many system architects fall into this category.

For example, a vendor system that I’ve been working with has single column primary keys in all tables, and all the primary key columns are named PKey_ID (I’ve changed it a bit to protect the guilty). Now I can’t say I like that naming at all, but that’s not the point. There are a large number of tables that already have that naming scheme.

Enter the new architect who has a purist view where he wants to name the columns ID instead. Again, I really dislike this naming but I don’t care what the vendor does, apart from being consistent.

Yet what the architect does is change so that a handful of tables now have ID. That is a really poor outcome. If he really wants to change them, then come up with a plan to change all of them.

image

I see the same thing in SQL Server.

We already had:

  • tinyint
  • smallint
  • int
  • bigint

And we already had:

  • smalldatetime
  • datetime

So when a larger precision version needed to be added, it wouldn’t have taken Einstein to come up with bigdatetime.

image

The answer certainly wouldn’t have been datetime2 which is what we got.

Oh, you say, but datetime2 is really a combination of the new date and time data types, and we already had a datetime data type, so it couldn’t be that.

Yes, but what then happened with datetimeoffset? It’s a datetime2 with an offset, so why isn’t it datetime2offset?

What’s the answer? I’m not 100% sure but I wish these teams had more cranky old dudes who look at proposed designs and say “nah, have another go at it”.

image