The Dev Ops Problem

What it is, and what to do about it.

As someone who spends all day, every day working with AWS technologies, I think of wheels and spaceships often.  

I’ve spent a lot of my career with my hands on the keyboard, reinventing the wheel.  It’s not unusual to write some custom code or process to solve a common IT problem, or use a tried-and-true IT tool to meet a need: Cron jobs for managing backups, custom deployment scripts, leveraging a suite of open source monitoring tools.  

It’s only natural to want to apply those same tools and processes in the cloud.  But in the cloud, we have opportunities to construct spaceships to help organizations provide a greater range of offerings and better solutions for their customers, and that’s not going to happen by making another wheel.  Building a spaceship requires standing on the shoulders of giants (who are creating flexible solutions for thousands of customers), taking the latest foundational technologies and designing from there.

To get from building wheels to erecting spaceships, we need to acknowledge three fundamental and inter-related problems with DevOps:

1. Technology:  Anyone spending any amount of time in the cloud will tell you that the pace of innovation is dizzying to keep up with, whether navigating the “New” pop outs on the AWS console, following the latest releases on blog sites, reading the latest white papers, or going to conferences such as re:Invent.  Keeping up with these changes is a full time job. 

2. Tools:  Native cloud tools compete with marketplace solutions as well as third party and open source solutions for, well, just about everything you can think of.  It’s a smorgasbord of solutions, a buffet of items to choose from, and much like a spice market in Istanbul, overpowering and at times disorienting. I always find it fascinating at re:Invent when AWS announces a ‘feature’ release, which effectively puts 20 or 30 of the vendors in the Expo hall out of business, or at least sends them back to the white board to figure out how to derive even greater value out of their products, often just to play catch up.

3. People:  The cold, hard truth is that a traditional IT mindset brings increasingly less and less benefit to the cloud infrastructure environments, where understanding the value of a cloud-native approach is critical.  In fact, it’s an incredibly scary time for IT people in general, where infrastructure as code and machine learning are replacing and enhancing so many traditional IT approaches to deployment, config management, security, governance, etc. To maintain relevance in this milieu requires a fairly exhausting mix of leaving the comfort zone, humility and study.  

IT folks who have gravitated toward the cloud, or newer generations who start working with cloud technologies often develop an overblown sense of self.  After all, it’s where all the cool kids hang out (just check out the swagger of all the swag-clad re:Inventees every year in their latest hoodies). By working in the cloud, we think we are holding on to the brass ring the makes us unique and helps us bring value to the organization.  We build automations, spin up some python or nodejs scripts in Lambda and magic happens. We are adopting the current...or somewhat recent, tools and technologies that help enable cloud.  

Meanwhile, AWS are innovating at a blistering pace, obsolescing a lot of tools, processes and ‘fixes’ implemented by DevOps teams. As a result, technical debt builds up at an increasingly rapid rate.

At the same time, at higher levels of the organization, the stakeholders - the Directors and VPs of the enterprise - want to be future-proofed.  They want to be enabled in the cloud utilizing the current and ‘next’ best practices, and be insulated from the technology decisions of individual contributors.  When those contributors leave the organization, they don’t want to be left with a mess of code, however cloudified, that solves problems that have recently been replaced with native services.

The solution is simple to post here, and of course difficult in practice:

Devops:
To stay relevant in the cloud, DevOps people need to be constantly willing to ‘level up’, keeping up with the changes to cloud service offerings.  They need to certify, re-certify, experiment and dive headfirst into the tech to understand the newest features and services broadly and deeply.  Then they need to use this knowledge to objectively decide if a solution has matured enough to meet the needs of their organization, then be humble and able enough to revise their current processes and thinking to adopt these new technologies in a well-architected manner, even if this means discarding their own, cool code, automation and previous intellectual property.   Given the pace of innovation, failing to do so will result in the enterprise absorbing far too much technical debt, which will cripple the organization’s ability to innovate and modernize.

Stakeholders:
As far as the stakeholders responsibility in this equation, the organization must be willing to accept and plan for the perpetual revision and re-factoring, and must support the individual contributor’s needs to learn and keep up with the changes to the tools and technologies by providing the time and training.  This is the price the organization must be willing to pay to play in the cloud.

This creates a natural tension between the needs of the individual contributors and stakeholders.  DevOps people keeping up with the cloud technologies, want to do things right, and the stakeholders want to get their products to market.  However, this is not a new dynamic and has existed for decades.  The same agile approaches that are currently transforming the industry need to be directed at this latest challenge and leaders need to lead, cultivating the mindset that we are all in this together.

You may not know when you are part of a well-tuned DevOps-minded organization that is ready for the next orbit, but chances are you know if you aren’t.  The organizations that adopt a cloud-native approach mindset while enabling their people to push their own technology boundaries are going to have an advantageous position in the marketplace, and in the stratosphere, over those who are just driving down another busy road.