Find the thread

I don’t usually do breaking news here, but this story pushes a whole lot of my buttons. Today, VMWare announced their intent to acquire SaltStack.

I have been following the automation market closely, at least since my time at BladeLogic. With BladeLogic acquired by BMC, and arch-rival Opsware by HP, much of the action moved to the open-source realm, with Puppet, Chef, Ansible, and SaltStack. Of those four, only Puppet remains as an independent player¹; Ansible was bought by Red Hat back in 2015, and of course Red Hat were themselves snapped up by IBM a few years later.

There was a gap after that, but just this month Chef was bought by Progress (who?), and now there is this Ansible news.

While merging automation functionality in the shape of Ansible into Red Hat made a lot of sense, the reaction to the Chef acquisition was more one of bemusement. We discussed the acquisition on a recent episode of the Roll for Enterprise podcast, and the only strategic rationale any of us could see for the acquisition was a possible integration with Whats Up Gold, as part of some sort of integrated detection and remediation play. I haven’t seen any further news from that direction, but it’s only been three weeks, so based on my own experience during acquisitions, I wouldn’t necessarily expect anything for a while yet.

The Action Moves Up The Stack

That theory about the role of automation in the modern infrastructure stack explains both why automation specialists no longer have the sorts of growth prospects (and valuations) that they did fifteen years ago at the time of the BladeLogic and Opsware acquisitions, and why they are being bought up now.

As the interface to software stacks moves further and further away from the bare metal, adding more and more layers of abstraction, the role of automation becomes that of plumbing: it’s important, perhaps crucial, but it’s invisible unless it breaks or fails. Arguably, this is a positive development, signifying the maturity of the automation market. Technology that is visible is cutting-edge and unreliable. There is a reason it’s called the bleeding edge; given the choice, I’d rather it be someone else’s blood getting spilled, while I hold back and learn from their mistakes.

Once that exciting technology settles down and becomes better understood, it disappears from our attention. We don’t think about what happens when we flip a switch, because we simply expect the light to come on. Intellectually we understand that there are all sorts of systems in place to make that light come on, that specialists work hard around the clock to look after those systems, and that there is a whole world of complexity around the generation and transmission of electricity, but ultimately all we care about is that it ultimately enables us to reach out and say "let there be light".

Automation is getting to that point: it’s a must-have, and because it’s a must-have, it’s no longer tenable for everyone to have to roll their own. In the dawn of personal computing, it was reasonable to expect every computer owner to bring their own soldering iron. That was obviously not a setup that could drive mass adoption, and these days, our computers are sealed shut, with no moving parts, let alone user-serviceable ones.

In the same way, back in the dog days of the last millennium, it was reasonable and even expected of me, as a junior sysadmin in training, to bang out a script that would let an Apache web server running on HP-UX authenticate users from a Windows NT domain — because there was no off-the-shelf way to do it. When I had to do add single sign-on to a project of mine last year, the SSO part took me one line of config, and I was done with that task and could move on to something more interesting and value-additive.

Automation is no longer something the CIO will care about. It’s expected and built-in, and the action has moved elsewhere. This is a victory: it’s not every software category that lasts long enough to become legacy!

🖼️ Photo by Yung Chang on Unsplash

One of the topics that currently exercise the more forward-looking among us is the potential negative impact of automation on the jobs market and the future of work in general. Comparisons are frequently made with the Industrial Age and its consequent widespread social disruption - including violent reactions, most famously the Luddite and saboteur movements.

Some cynics have pointed out that there was less concern when it was only blue-collar jobs that were being displaced, and that what made the chattering classes sit up and pay attention was the prospect of the disruption coming for their jobs too. I could not possibly comment on this view - but I can comment on what I have seen in years of selling automation software into large companies.

For more than a decade, I have been involved in pitching software that promised to automate manual tasks. My customers have always been large enterprises, usually the Global 2000 or their immediate followers. Companies like this do not buy software on a whim; rather, they build out extensive business cases and validate their assumptions in detail before committing themselves¹. There are generally three different ways of building a business case for this kind of software:

• Support a growth in demand without increasing staff levels (as much);
• Support static demand with decreasing staff;
• Quality improvement (along various different axes) and its mirror image, risk avoidance.

The first one is pretty self-evident - if you need to do more than you can manage with the existing team, you need to hire more people, and that costs money. There are some interesting second-order consequences, though. Depending on the specifics of the job to be done, it will take a certain amount of time to identify a new hire and train them up to be productive. Six months is a sensible rule of thumb, but I know of places where it takes years. If the rate of growth gets fast enough, that lag time starts to be a major issue. You can't just hire yourself out of the hole, even with endless money. The hole may also be getting deeper if other companies in the same industry and/or region are all going through the same transformation at the same time, and all competing for the same talent.

If instead you can adopt tooling that will make your existing people more efficient and let you keep up with demand, then it is worth investing some resources in doing so.

That second business case is the nasty one. In this scenario, the software will pay for itself by automating people's jobs, thus enabling the company to fire people - or in corporate talk, "reduce FTE² count". The fear of this sort of initiative is what makes rank and file employees often reflexively suspicious of new automation tools - over and above their natural suspicion that a vendor might be pitching snake-oil.

Personally I try not to build business cases around taking away people's jobs, mainly because I like being able to look myself in the mirror in the mornings (it's hard to shave any other way, for one thing). There is also a more pragmatic reason not to build a business case this way, though, and I think it is worth exploring for its wider implications.

Where Are The Results?

The thing is, in my experience business cases for automation built around FTE reduction have never been delivered successfully - if focused on automation of existing tasks. That is an important caveat, but I will come back to that.

Sure, the business case might look very persuasive - "we execute this task roughly a dozen times a day, it takes half an hour each time, and if you add that up, it's the equivalent of a full-time employee (an FTE), so we can fire one person". When you look at the details, though, it's not quite so simple.

The fact is that people rarely work at discrete tasks. Instead, they spend their time on a variety of different tasks, more or less integrated into a whole process. There is a tension between the two extremes: at the one end you have workers on a repetitive assembly line, while at the other you have people jumping around so much they can never get anything done. Most organisational functions are somewhere in between those two poles.

If automation is focused on addressing those discrete tasks, it absolutely will bring benefits, but those benefits will add up to freeing up existing employees to catch up with other tasks that were being neglected. Every IT department I have ever seen has a long tail of to-dos that keep getting pushed down the stack by higher-priority items. Automation is the force multiplier that promises to let IT catch up with its to-do list.

This sort of benefit is highly tactical, and is generally the domain of point solutions that do one thing and do it well. This will enable the first kind of business case, delivering on new requirements faster. It will not deliver the second kind of business case. The FTEs freed up through automation get redeployed, not fired, and while the organisation is receiving benefit from that, it is not what was built into the assumptions of the project, which will cause problems for its sponsors. Simply put, if someone ever checks the return on the investment (an all too rare occurrence in my experience), the expected savings will not be there.

Strategic benefits of automation, on the other hand, are delivered by bundling many of these discrete tactical tasks together into a new whole.

Realising those strategic benefits is not as straightforward as dropping a new tool into an existing process. Actually achieving the projected returns will require wholesale transformation of the process itself. This is not the sort of project that can be completed in a quarter or two (although earlier milestones should already show improvement). It should also not be confused with a technology implementation project. Rather, it is a business transformation project, and must be approached as such.

Where does this leave us?

Go Away Or I Will Replace You With A Very Small Shell Script

In my experience in the field, while tactical benefits of automation are achievable, true strategic improvement through automation can only be delivered by bundling together disparate technical tasks into a new whole. The result is that it is not skilled workers that are replaced, but rather the sorts of undifferentiated discrete tasks that many if not most large enterprises have already outsourced.

This shows who the losers of automation will be: it is the arbitrageurs and rent-seekers, the body-rental shops who provide no added value beyond cheap labour costs. The jobs that are replaced are those of operators, what used to be known as tape jockeys; people who perform repetitive tasks over and over.

The jobs that will survive and even benefit from the wave of automation are those that require interaction with other humans in order to determine how to direct the automation, plus of course the specialists required to operate the automation tools themselves. The greatest value, however, will accrue to those who can successfully navigate the interface between the two worlds. This is why it is so important to own those interfaces.

What might change is the nature of the employment contracts for those new roles. While larger organisations will continue to retain in-house skills, smaller organisations for which such capabilities are not core requirements may prefer to bring them in on a consultative basis. This will mean that many specialists will need to string together sequences of temporary contracts to replace long-duration full-time employment.

This is its own scary scenario, of course. The so-called gig economy has not been a win so far, despite its much-trumpeted potential. Perhaps the missing part to making this model work is some sort of universal basic income to provide a base and a safety net between consulting jobs? As more and more of the economy moves in this direction, at least in part due to the potential of automation, UBI or something similar will be required to bridge the gap between the assumptions of the old economy and the harsh realities of the new one.

So, the robots are not going to take our jobs - but they are going to change them, in some cases into something unrecognisable. The best thing humans can do is to plan to take care of one other.

Images by Annie Spratt, Janko Ferlic, and Jayphen Simpson via Unsplash

Automation of single tasks with point solutions is equivalent to premature local optimisation of code.

The worst part is, in both cases you may end up trapping yourself on a local optimum. It's hard to abandon these maxima and cross the valley to the higher peaks, no matter how attractive they look.

It doesn't get easier over time, though. Start thinking holistically now. Talk to your peers in different teams. Better yet, talk to your users. They'll tell you what they need.

Everyone in the trenches of IT knows that Dilbert is drawn from real life - and [today’s strip]
(
http://dilbert.com/fast/2014-04-15/) is no exception. Does your IT organisation rely on knowledge that is held only by only a few people, or maybe even one person? This is known as the bus number or bus factor - basically, the number of people who would have to be hit by a bus for the organisation to be severely affected. With slightly less black humour, let’s say they win the lottery, get a dream job elsewhere, or simply feel sick and don’t come in to work one day. Regardless of the details, most organisations have a bus number of one.

That’s right: if a single person is missing, the organisation is unable to operate normally. Note that it’s rarely just one person, old Bob who’s been there since the beginning and knows everything. It’s more likely to be Alice the database whisperer, Carol who knows which arcane options to give the batch jobs so they’ll all go through in sequence, Dave who has the admin password to the core routers, and so on.

If you’re not in IT, this may seem crazy, but trust me, this is exactly how most teams work. Everyone knows they should get around to documenting this stuff, if not outright automating it, but there are always more fires to put out than there are hours in the day… This sort of thing is a serious business risk, too, but it’s invisible to management unless they go looking for it, and few managers are inclined to look for additional problems.

Everything muddles along - until Bob wins the lottery...

On Friday I had the chance to sit down with Alf, of Alf’s Zoo fame. We had a great chat about automation, cloud and… cheese? You’ll just have to watch the show!

If you’re wondering about the art behind me, here are the two prints: Ski Pluto and Visit Mars. Both are by Steve Thomas. I had seen them linked ages ago and filed the bookmark, and when I was furnishing my home office in the new house I finally had somewhere to put them.

On a related note, every time I try to do something with Google Hangouts, I gain a better understanding of why WebEx has been so successful. Recording a ten-minute show took half an hour of futzing around. It’s one thing to do this if guest and host know each other already, but this would make a terrible first impression.