I don’t usually do breaking news here, but this story pushes a whole lot of my buttons. Today, VMWare announced their intent to acquire SaltStack.

I have been following the automation market closely, at least since my time at BladeLogic. With BladeLogic acquired by BMC, and arch-rival Opsware by HP, much of the action moved to the open-source realm, with Puppet, Chef, Ansible, and SaltStack. Of those four, only Puppet remains as an independent player¹; Ansible was bought by Red Hat back in 2015, and of course Red Hat were themselves snapped up by IBM a few years later.

There was a gap after that, but just this month Chef was bought by Progress (who?), and now there is this Ansible news.

While merging automation functionality in the shape of Ansible into Red Hat made a lot of sense, the reaction to the Chef acquisition was more one of bemusement. We discussed the acquisition on a recent episode of the Roll for Enterprise podcast, and the only strategic rationale any of us could see for the acquisition was a possible integration with Whats Up Gold, as part of some sort of integrated detection and remediation play. I haven’t seen any further news from that direction, but it’s only been three weeks, so based on my own experience during acquisitions, I wouldn’t necessarily expect anything for a while yet.

The Action Moves Up The Stack

That theory about the role of automation in the modern infrastructure stack explains both why automation specialists no longer have the sorts of growth prospects (and valuations) that they did fifteen years ago at the time of the BladeLogic and Opsware acquisitions, and why they are being bought up now.

As the interface to software stacks moves further and further away from the bare metal, adding more and more layers of abstraction, the role of automation becomes that of plumbing: it’s important, perhaps crucial, but it’s invisible unless it breaks or fails. Arguably, this is a positive development, signifying the maturity of the automation market. Technology that is visible is cutting-edge and unreliable. There is a reason it’s called the bleeding edge; given the choice, I’d rather it be someone else’s blood getting spilled, while I hold back and learn from their mistakes.

Once that exciting technology settles down and becomes better understood, it disappears from our attention. We don’t think about what happens when we flip a switch, because we simply expect the light to come on. Intellectually we understand that there are all sorts of systems in place to make that light come on, that specialists work hard around the clock to look after those systems, and that there is a whole world of complexity around the generation and transmission of electricity, but ultimately all we care about is that it ultimately enables us to reach out and say "let there be light".

Automation is getting to that point: it’s a must-have, and because it’s a must-have, it’s no longer tenable for everyone to have to roll their own. In the dawn of personal computing, it was reasonable to expect every computer owner to bring their own soldering iron. That was obviously not a setup that could drive mass adoption, and these days, our computers are sealed shut, with no moving parts, let alone user-serviceable ones.

In the same way, back in the dog days of the last millennium, it was reasonable and even expected of me, as a junior sysadmin in training, to bang out a script that would let an Apache web server running on HP-UX authenticate users from a Windows NT domain — because there was no off-the-shelf way to do it. When I had to do add single sign-on to a project of mine last year, the SSO part took me one line of config, and I was done with that task and could move on to something more interesting and value-additive.

Automation is no longer something the CIO will care about. It’s expected and built-in, and the action has moved elsewhere. This is a victory: it’s not every software category that lasts long enough to become legacy!

🖼️ Photo by Yung Chang on Unsplash

Yesterday was the LinkedIn equivalent of a birthday on Facebook: a new job announcement. I am lucky enough to have well-wishers¹ pop up from all over with congratulations, and I am grateful to all of them.

With a new job comes a new title – fortunately one that does not feature on this list of the most ridiculous job titles in tech (although I have to admit to a sneaking admiration for the sheer chutzpah of the Galactic Viceroy of Research Excellence, which is a real title that I am not at all making up).

The new gig is as Director, Field Initiatives and Readiness, EMEA at MongoDB.

Why there? Simply put, because when Dev Ittycheria comes calling, you take that call. Dev was CEO at BladeLogic when I was there, and even though I was a lowly Application Engineer, that was a tight-knit team and Dev paid attention to his people. If I have learned one thing in my years in tech, it’s that the people you work with matter more than just about anything. Dev’s uncanny knack for "catching lightning in a bottle", as he puts it, over and over again, is due in no small part to the teams he puts together around him – and I am proud to have the opportunity to join up once again.

Beyond that, MongoDB itself needs no presentation or explanation as a pick. What might need a bit more unpacking is my move from Ops, where I have spent most of my career until now, into data structures and platforms. Basically, it boils down to a need to get closer to the people actually doing and creating, and to the tools they use to do that work. Ops these days is getting more and more abstract, to the point that some people even talk about NoOps (FWIW I think that vastly oversimplifies the situation). In fact, DevOps is finally coming to fruition, not because developers got the root password, but because Ops teams started thinking like developers and treating infrastructure as code.

Between this cultural shift, and the various technological shifts (to serverless, immutable infrastructure, and infrastructure as code) that precede, follow, and go along with them, it’s less and less interesting to talk about separate Ops tooling and culture. These days, the action is in the operability of development practices, building in ways that support business agility, rather than trying to patch the dam by addressing individual sources of friction as they show up.

More specifically to me, my particular skill set works best in large organisations, where I can go between different groups and carry ideas and insights with me as I go. I’m a facilitator; when I’m doing my job right, I break information out of silos and spread it around, making sure nobody gets stuck on an island or perseveres with some activity or mode of thinking that is no longer providing value to others. Coming full circle, this fluidity in my role is why I tend to have fuzzy, non-specific job titles that make my wife’s eyes roll right back in her head – mirroring the flow I want to enable for everyone around me, whether colleagues, partners, or users.

It’s all about taking frustration and wasted effort out of the working day, which is a goal that I hope we can all get behind.

Now, time to blow away my old life…

While it may seem obvious to those of us who have been around this market for a while, it was interesting to read that at the recent Puppetconf 2016 event, Puppet still felt the need to state that "In the future code is going to be managed and deployed by other code". If you’re surprised that this sentiment still needs to be articulated explicitly in 2016, you have not been paying attention.

It is certainly true that the leading edge is all "cattle, not pets" and "automate all the things", but there’s a pretty long tail behind that head. Only 2% of workloads are currently running in "the cloud" - although the precise definition is complicated by that nebulous term. Everything else? Still running on premises, or at best in a colo.

The same goes for automation: for every fully-automated containerised full-stack deployment, there are fifty that are not automated.

Nevertheless, Puppet has built a $100M business on automation. I know a bit about this space, having worked at BladeLogic, one of the pioneers of automation. While BladeLogic and Puppet have a history, today I am wondering about whether things might have gone differently.

Luke Kanies, the founder of Puppet, was a BladeLogic employee, although he left before I ever joined. From what I gather, he was a proponent of extending BladeLogic’s foundation in Network Shell, or NSH, into a free open-source platform, on which a commercial product could be built. Instead, BladeLogic’s management preferred to shut down the open-source NSH project and just use the technology inside the commercial BladeLogic product.

For those in the know, NSH was a fantastic tool. At root it was a shell based on ZSH, but with network awareness on top. What this meant was that you could do things like this:

 host $ cp /etc/hosts //host1/etc/hosts

 host $ cd //host2/home

 host2 $ ps -ef | grep inetd

 host2 $ diff //host3/etc/passwd //host4/etc/passwd

 host2 $ iostat 2 5

 host2 $ vi //nthost/c/AUTOEXEC.BAT

 host2 $ nexec nthost reboot Let's reboot NT

You could copy files between systems, compare them or even edit them in place, and generally do all sorts of good things - including developing scripts to automate those tasks. For me at least, this was the first hint of the new world in which systems are no longer managed one by one, with admins ssh’ing into them individually, but in bulk, deploying a single config to many systems in one action. Best of all, it was multi-platform, abstracting the differences between different UNIX variants, and even working on Windows. ZSH on NT? That’s a major selling point right there!

However, even among BladeLogic employees and users, the interactive mode of NSH was a well-kept secret, with most people working exclusively within the BladeLogic GUI. What might the combination of NSH and BladeLogic have become if it had been allowed to flourish? Could a free NSH have taken the place in sysadmin’s hearts that is currently occupied by Puppet? Would this have prevented the long, quiet death of BladeLogic?

20/20 Hindsight

Of course hindsight is a wonderful thing, and what is a fairly uncontroversial strategy to propose in 2016 was not so obvious fifteen years ago. Back then, there were vanishingly few successful hybrid business models that combined an open-source platform with a commercial component. It would not be fair to criticise BladeLogic’s management at the time for not taking that route - especially since they were outstandingly successful with the strategy that they did choose. The hybrid model would have been a major strategic choice, and there is no guarantee that VCs and other investors would have gone along with it.

I just wonder sometimes - what might have been, in a world where a free download of NSH would have gained mindshare in the data center, at the same time that high-powered, PTC-trained sales people were gaining the trust of the C-suite?

Today, in 2016, Robert Stroud, a Forrester analyst at the Puppet event, is saying the following:

Businesses services now involve infrastructure, middleware, and applications, said Stroud. "Moving forward, to be a complete automation environment, the successful player in the space will have a role in all three," he said.

At BladeLogic, we were saying that ten years ago. Regardless of the commercials, this market of automated server configuration management is arguably ten years behind where it should be. Sure, we can deploy things at scale, but managing them at scale is still a challenge - although the challenge is as much one of process as of tools. The cloud has enabled all sorts of new businesses and even entire new business models, but it is still constrained by the complexity and consequent fragility of the underlying infrastructure.

What might be possible if we had solved that problem ten years ago? What new possibilities might have been enabled, that we will only find out about years from now?

I haven’t blogged in ages - which is a good thing, I hasten to add! It’s just that I have been drinking from the firehose at my new gig. It’s now more than a month since I started at Moogsoft, and I think I can begin to talk about what it all means.

I joined Moogsoft from BMC, but it’s important to note that I did not join BMC, I wound up there as part of the BladeLogic acquisition. BladeLogic was my first startup, and it was a huge amount of fun, a great learning experience, and probably my period of fastest professional development to date. Before BladeLogic I was at Mercury, but I quit to join BladeLogic, due in no small part to the acquisition by HP¹.

What is BladeLogic?

Both BladeLogic² Operations Manager (BLOM) and Incident.MOOG are innovative products in their place and time. BladeLogic, together with Opsware, redefined what server configuration management meant, and both companies went on to be acquired by larger "Big 4" IT vendors: Opsware by HP, and a year or so later, BladeLogic by BMC.

For a while both products thrived in their new environment, but in the last few years, both have been flagging. There are many reasons for this, from internal politics at both BMC and HP acting as distraction, to the rise of open-source configuration management tools such as Chef and Puppet. However, I wonder if those tools were simply the end of an era.

This is a known pattern: technologies reach their peak right before they get displaced by their successor technologies. The speed record for steam engines was set in 1938, but a diesel engine had already exceeded that speed in 1936, and by the 1950s [diesel locomotives were well on track to replace steam traction](

https://en.wikipedia.org/wiki/Diesel_locomotive#History)³.

This pattern even agrees with disruption theory: investment continues in the old technology, but it simply becomes overly complex and uneconomical compared to simpler, (initially) low-end competitors.

Pets vs Cattle

This disruption is exactly what I see happening now. The BladeLogic model of painstaking management of single servers is still relevant, but mainly for legacy or specialised systems. Much new-build development is already moving to disposable or even stateless VMs or containers, according to the classic "pets vs cattle" model.

In this brave new world, there is very little need to worry about the configuration of your "cattle" containers. Something like BladeLogic is arguably overkill, and users should instead focus on their provisioning process.

Of course it’s not quite as simple as that. Cloud zealots have been talking about replaceable short-lived cloud servers for a while, but it hasn’t really happened outside of some rather specific situations. The lifetime of VMs in the cloud has often ended up being much longer than this model would suggest, meaning that there is plenty of time for their configurations to drift and to require management in place. Part of the reason for this is that management processes and techniques that are still based on the paradigm of a persistent physical server. Much of this Weltanschauung has been adopted wholesale for virtual and cloud-based servers without much reconsideration.

There is also the topic of security and policy compliance to be considered. Given long system lifetimes, it is not sufficient to be able to validate that something was deployed correctly, as its configuration may have drifted away from that known-good state. The desired state may also change as vendors release updates and patches, or new security vulnerabilities are disclosed. In all of these cases, some mechanism is needed to check for differences between the current live configuration and the required configuration, and to bring the system into compliance with that desired state.

However, this is now. As Docker and other container-centric infrastructure technologies become more prevalent, and as business functions continue to migrate from legacy to new-build applications, I would expect that that paradigm will evolve to replaceable plug&play infrastructure components, and do so everywhere, not just at the "unicorn" companies.

What does it all mean?

Lots of smart people are working hard to enable infrastructure to be managed as code. One of the characteristics of code is that you don’t change it in production, you develop it offline, then release it and don’t change it until you overwrite with a new version. The big variables that I think will affect the speed of the transition to this new model are firstly, the rate of replacement of legacy applications, and secondly, the evolution of IT management processes and culture to take advantage of new tools.

BladeLogic itself has the opportunity to evolve to have a role in the new model, of course. Regardless, BladeLogic was a huge part of my career development - and just huge fun, if I’m honest - so I will be watching development of the IT infrastructure management market intently, but no longer from the front lines.

(╯°□°)╯︵ ┻━┻

After 7 years, it's time to move on. Today is my last day at BMC, then I get a whole one day off - and even that only because it's a public holiday - before I start my next gig.

Today

I hand in my badge and gun - er, I mean, MacBook - and on Monday morning, bright and early, I will be in San Francisco to start my new job at [Moogsoft](

http://www.moogsoft.com)

. I could not be more excited!

It's definitely a wrench to leave after so many years, and so many different roles. In particular, I have to admit that I will miss the view from BMC's Milan offices, with the Alps in the background:

Onwards to new adventures!

Or, Why BladeLogic Isn’t Puppet

Disclaimer: In case you don’t know already, I work for BMC Software, having joined with the acquisition of BladeLogic. My current role is in marketing for BMC’s cloud computing and data center automation products, including BladeLogic. In other words, if you are looking for a 100% objective take, look elsewhere. The reason this piece is here rather than on a bmc.com site is to make it clear that this is personal opinion, not official corporate communication.

At the turn of the last century, life as a sysadmin was nasty and brutish. Your choice was either to spend a lot of time doing the same few things by hand, generally at the command line, or to spend a lot of time writing scripts to automate those tasks, and then even more time maintaining the scripts, all the while being interrupted to do those same things.

Then along came two automation frameworks, Opsware and BladeLogic, that promised to make everything better. The two tools had somewhat similar origin stories: both were based around cores that came out of the hosting world, and both then built substantial frameworks around those cores. In the end both were bought by larger players, Opsware by HP, and BladeLogic by BMC.

BladeLogic and Opsware both let sysadmins automate common tasks without scripting, execute actions against multiple servers at once, and assemble sub-tasks together to do full-stack provisioning. There were (and are) any number of differences in approach, many of which can still be traced back to the very beginnings of the two companies, but it’s safe to say that they are the two most similar tools in today’s automation marketplace.

Yes, marketplace. Because in the last decade or so a huge number of automation tools have emerged (or matured to the point of usefulness), mainly in the open-source arena. If you are managing tons of OSS Linux boxes, running an OSS application stack, it makes sense to have an OSS config management tool as well.

So far, so good. For a while the OSS tools flew under the radar of the big vendors, since the sorts of people willing to download a free tool and hack Ruby to do anything with it tended not to be the same sorts of people with the six- or seven-figure budgets for the big-vendor tools. As is the way of such things, though, the two markets started to overlap, and people started to ask why one tool was free and the other was expensive. This all came to a head when Puppet Labs published a document entitled "Puppet Enterprise vs BMC BladeLogic". Matthew Zito from BMC responded with An Open Letter to PuppetLabs on BMC’s site, which led to an exchange on Twitter, storified here.

This is the longer response I promised Luke Kanies.

When I was at BladeLogic, I was in pre-sales, and one of my jobs (on top of the usual things like demos, proof-of-concept activities, and RfI/RfP responses) was to help the sales people assemble the business case for buying BladeLogic. This usually meant identifying a particular activity, measuring how much time and effort it took to complete without BladeLogic, and then proposing savings through the use of BladeLogic. Because for some unknown reason prospective customers don’t take vendors’ word for these sorts of claims, we would then arrange to prove our estimates on some mutually-agreed subset of the measured activities.

We would typically begin by talking to the sysadmins and people in related teams. They had generally spent a lot of time scripting, automating and streamlining, and were eager to tell us that there was no reason to buy what we were selling, because there were no further savings to be made. Any requests could be delivered in minutes.

The interesting thing is, we would then go around the corner to the users and ask them how long they typically had to wait for requested services to be delivered. The answers varied, but generally in the range of, not minutes, but two to eight weeks.

Weeks, not minutes

Where is that huge discrepancy coming from? Because, depressingly, it’s still the same today, a decade later.

The delay experienced by users is not caused by the fact that sysadmins are frantically flailing away at the keyboard for a full month to deliver something. Credit us sysadmins (for I am still one at heart) with a bit more sense than that. No, the problem is that there are many many different people, functions and teams involved in delivering something to users. Even if each individual step is automated and streamlined and standardised to within epsilon of perfection, the overall process is delayed by the hand-offs between the steps, and even more so when the hand-off isn’t clean and something needs to be reworked, or worse, discussed and argued about.

That is the difference between Puppet and BladeLogic. Puppet is trying to address one - or, in all fairness, several - of those steps, but BladeLogic is trying to address the entire process.

In a wider sense, this is what BMC is trying to do for all of enterprise IT. "Consumerisation of IT" has become a cliché, but it’s true that in the same way that Puppet has moved from a hobbyist market to the IT mainstream, Dropbox has moved from a home user market to a corporate one, AWS has eaten the cloud, and so on. We are living in a Cambrian explosion of tools and services.

Enterprise IT departments and the vendors that serve them cannot compete with these tools, and nor should they. The models of the new entrants give them economies - of scale, of attention, of access - that the traditional model cannot touch. The role for enterprise IT is to provide governance across the top of this extremely diverse and rapidly evolving ecosystem, and fill in the gaps between those tools so that we deliver the correct end product on time, on spec and on budget¹.

Sure, use Puppet - and Chef, and Ansible, and SaltStack, and CFEngine, and your home-grown scripts, and maybe even BladeLogic’s BLpackages. Just make sure that you are using them in a way that makes sense, and that meets the users’ needs. At the end of the day, that’s what we are all here for.