Embedded DevOps: The Time Has Come

What DevOps Is

DevOps is a set of practices to help organizations deliver working software faster. DevOps benefits software development organizations of every size and kind. However, as of 2018 relatively few organizations developing embedded or IoT software have adopted DevOps methods and tools. We think that’s a shame, and in this and following posts we intend to lay out what DevOps is, what its benefits are, and how organizations building embedded and IoT software can implement it.

At its core, DevOps is about improving throughput for software development organizations. Every software development organization has the task of taking product concepts and turning them into shipping software. The rate at which these concepts get shipped to customers is the organization’s throughput.

The ideas behind DevOps actually come from ideas about improving manufacturing. The foundational text as applied to manufacturing is “The Goal” by Eliyahu Goldratt. Written in 1984, “The Goal” was and is a major influence on the lean manufacturing movement which transformed manufacturing productivity across the globe.

Thirty years later, Gene Kim, Kevin Behr and George Spafford wrote “The Phoenix Project” about adapting these ideas to software development. If you haven’t read “The Goal” or “The Phoenix Project”, I recommend them to you highly. In “The Phoenix Project”, the authors outline the three ways of improving productivity in software organizations.

The First Way

The First Way is all about getting work into production as fast as possible. Starting from the business stakeholders, through development and operations to the customer. Only when work is actually delivered to the customer is value created, and so accelerating the rate at which that happens is central to The First Way.

One important strategy for speeding up the flow of finished work is to reduce the amount of Work in Process (WIP). WIP doesn’t deliver any value until it is delivered to a customer. Smaller pieces of work moving through our pipeline results in a faster flow of finished work and delivered value.

It is also important to identify the bottleneck or constraint in your development pipeline. Sometimes this is a key developer or systems administrator who is so capable or so intertwined into the inner workings of your systems, or both, that he or she must be involved in all projects. Or it might be your build process, or perhaps testing and integration, or deployment.

Whatever your bottleneck is, The First Way advocates keeping your bottleneck working productively at all times. Reducing WIP helps with this because it reduces the amount of time that your developer, for example, needs to spend switching between development tasks that are currently in flight.

The Second Way

The Second Way is about increasing the amount of feedback that your team gets as they do work. This means both increasing the number of feedback loops, and making sure that the feedback reaches the folks concerned with it faster.

The most important point about The Second Way, is that the flow of work should completely stop when an error is found and the nature of the error should be communicated to those who can fix it as fast as possible.

One standard way of implementing these ideas is with a continuous integration, build and deployment process working with an automated test suite that runs quickly. Together, these two things gives quick feedback to both development and operations.

Decreasing the amount of WIP also improves the speed of the feedback loop. With a smaller amount of WIP, the time between making a change and that change being tested gets smaller, which improves the system as a whole.

The Third Way

The third way is about constant experimentation and testing. Only through experimentation can individuals and organizations truly learn and improve over time. The problem many organizations face is that the risks of experimentation are too high.

But if the First and Second Ways are followed, then you can experiment because the risks will never be too big. With low WIP, any change is a small one and the feedback is quick and constant.

DevOps Is For Embedded

The principles of the Three Ways benefit all software development organizations. Though DevOps is relatively new to the world of embedded software development, those embedded software companies that adopt it will see significant improvement in their ability to deliver value to customers.