Embedded DevOps: The Time Has Come

What DevOps Is

DevOps is a set of practices to help organizations deliver working software faster. DevOps benefits software development organizations of every size and kind. However, as of 2018 relatively few organizations developing embedded or IoT software have adopted DevOps methods and tools. We think that’s a shame, and in this and following posts we intend to lay out what DevOps is, what its benefits are, and how organizations building embedded and IoT software can implement it.

At its core, DevOps is about improving throughput for software development organizations. Every software development organization has the task of taking product concepts and turning them into shipping software. The rate at which these concepts get shipped to customers is the organization’s throughput.

The ideas behind DevOps actually come from ideas about improving manufacturing. The foundational text as applied to manufacturing is “The Goal” by Eliyahu Goldratt. Written in 1984, “The Goal” was and is a major influence on the lean manufacturing movement which transformed manufacturing productivity across the globe.

Thirty years later, Gene Kim, Kevin Behr and George Spafford wrote “The Phoenix Project” about adapting these ideas to software development. If you haven’t read “The Goal” or “The Phoenix Project”, I recommend them to you highly. In “The Phoenix Project”, the authors outline the three ways of improving productivity in software organizations.

The First Way

The First Way is all about getting work into production as fast as possible. Starting from the business stakeholders, through development and operations to the customer. Only when work is actually delivered to the customer is value created, and so accelerating the rate at which that happens is central to The First Way.

One important strategy for speeding up the flow of finished work is to reduce the amount of Work in Process (WIP). WIP doesn’t deliver any value until it is delivered to a customer. Smaller pieces of work moving through our pipeline results in a faster flow of finished work and delivered value.

It is also important to identify the bottleneck or constraint in your development pipeline. Sometimes this is a key developer or systems administrator who is so capable or so intertwined into the inner workings of your systems, or both, that he or she must be involved in all projects. Or it might be your build process, or perhaps testing and integration, or deployment.

Whatever your bottleneck is, The First Way advocates keeping your bottleneck working productively at all times. Reducing WIP helps with this because it reduces the amount of time that your developer, for example, needs to spend switching between development tasks that are currently in flight.

The Second Way

The Second Way is about increasing the amount of feedback that your team gets as they do work. This means both increasing the number of feedback loops, and making sure that the feedback reaches the folks concerned with it faster.

The most important point about The Second Way, is that the flow of work should completely stop when an error is found and the nature of the error should be communicated to those who can fix it as fast as possible.

One standard way of implementing these ideas is with a continuous integration, build and deployment process working with an automated test suite that runs quickly. Together, these two things gives quick feedback to both development and operations.

Decreasing the amount of WIP also improves the speed of the feedback loop. With a smaller amount of WIP, the time between making a change and that change being tested gets smaller, which improves the system as a whole.

The Third Way

The third way is about constant experimentation and testing. Only through experimentation can individuals and organizations truly learn and improve over time. The problem many organizations face is that the risks of experimentation are too high.

But if the First and Second Ways are followed, then you can experiment because the risks will never be too big. With low WIP, any change is a small one and the feedback is quick and constant.

DevOps Is For Embedded

The principles of the Three Ways benefit all software development organizations. Though DevOps is relatively new to the world of embedded software development, those embedded software companies that adopt it will see significant improvement in their ability to deliver value to customers.

The Business Case For The Internet Of Things

Offering Everything as a Service will make businesses twice as valuable.

The Internet of Things (IoT) allows applications to run across both the Cloud and connected devices, and that means that just about every piece of equipment that was previously bought now can be efficiently rented out.

Take an elevator, for instance.  Today a typical elevator for a commercial building costs about $112,500 to install, plus another $4,800 each year thereafter for the maintenance contract.  The IoT permits an entirely different business model.  Instead of selling the elevator to the building owner, the elevator manufacturer can instead keep ownership of the elevator and charge by the trip.  In this model, the elevator manufacturer would charge about $0.11 (eleven cents) for each elevator trip, on average.

Let’s make some assumptions in order to understand this 11 cents per trip a little better.

  • Yearly trips per elevator: 132,000 (the global average)
  • Elevator service life: 20 years (again, average)
  • Cost of capital: 7.95% (typical for an elevator manufacturer)

Under these assumptions, and at 11 cents per ride, the present value of the revenue is the same for the manufacturer under both models.  So if all else were equal, the manufacturer would be indifferent between selling the elevator upfront and getting the yearly maintenance contract, and charging 11 cents per trip, on average.

But all else is definitely not equal.  In the software industry, companies that sell software licenses upfront (and typically sign a maintenance contract) average a 3x multiple on their revenue.  That is to say that investors value companies like this at 3x their revenue.  But software companies that have recurring revenue average a 6x multiple on their revenue.  In other words software companies that sell their software as a service are twice as valuable as companies that sell upfront licenses.

Keep reading, and see how far the rabbit hole goes...

Keep reading, and see how far the rabbit hole goes...

And there’s very good reason to think that the same dynamic applies to capital equipment manufacturers like elevator companies as well as it does to software makers. 

A Boston Consulting Group analysis of performance from leading machinery companies in recent years shows that services have outperformed equipment sales in several dimensions:

  • Faster growth rates
  • Greater predictability
  • Greater resilience through economic cycles
  • Greater profitability

Any business that grows faster, more predictably, is less exposed to the economic cycle and has higher profitability will be more valuable than another business that does less well on these factors.  The Internet of Things allows companies to shift their business model to services entirely, without changing the present value of their revenue, potentially doubling the value of their business.  That’s a pretty strong business case.

And there’s a strong business case for the building owners as well.  Instead of having to finance the purchase of the elevator, the building owner’s costs instead scale up and down as the occupancy of the building (and rents) scales up and down.  Instead of needing to build a bigger balance sheet in order to rent space out profitably, the building owner reduces risk and reduces up-front cash outlays.  And, if the elevator is equipped to recognize passengers, the building owner could even pass the elevator costs on to the tenants in proportion to their elevator use.  A pretty interesting proposition for the building owner.

But really, we’re just getting started down this path.

So awesome.

So awesome.

The model we’ve just outlined is Infrastructure as a Service (IaaS) as applied to elevators.  This is the kind of business that Amazon has with Amazon Web Services in the Cloud, where they rent out servers and storage (capital equipment) by the hour.  This is a great business for Amazon. So much so that in 2018 Amazon Web Services has been growing more than 40% while also having profit margins greater than 20%!

But IaaS isn’t the only kind of “as a service” model that there is.  There are also Platforms as a Service (PaaS) in which all of the infrastructure you need to build great applications is combined into one platform that is, again, priced per use or per hour rather than sold for a lump sum up front.  In the case of a commercial building you could imagine the elevators, escalators, climate control, catering, lighting, security, common spaces (think conference room reservations for example), phones and bandwidth all being offered as a service through a common platform.

And finally there is Software as a Service.  At least, that’s what it’s called in the Cloud, where software sold as a subscription solves a problem for its user.  There are any number of examples of these (Salesforce.com is a Customer Relationship Management system as a service, Hubspot is Marketing Automation as a service, etc. etc.).  But in the Internet of Things, the last layer won’t really be software as a service.  On the Internet of Things, the last layer will be Experience as a Service (UXaaS).

UXaaS: like having these guys at your beck and call, but in the 21st century.

UXaaS: like having these guys at your beck and call, but in the 21st century.

Imagine that you have subscribed to an Experience as a Service offering that is integrated with your office building, and you’re on your way to work in the morning.  Because of that, your Uber driver knows exactly where to drop you off outside your building so that you have the shortest walk to the elevator. 

The security system checks you in and the elevator door opens without you having to even break stride on the way to the elevator.  On the way to your floor, your office lights turn on, the window shades open and the climate control ensures that it is maintaining the temperature that gives you maximum productivity. 

As you sit down in your office, a delivery person from the building’s café drops off your morning cup of coffee just as you like it.  After 15 minutes at your desk checking emails, you walk down the hall to your conference room where you’ll be connecting with a team based in Europe.  As you approach, you notice that the room ordered bagels for your local colleagues and the videoconference system has dialed out to your team in Europe automatically.

Now if your office were able to deliver that kind of experience, would it make you and your team more productive?  I think it would.  And would it influence the building that you chose to locate your office in?  Again, I think it would.

Now think back to the elevator maker.  Delivering an outstanding Experience as a Service is pretty different from making elevators, but it’s also a much more valuable and differentiated business. Experience as a Service isn’t so much about connecting a piece of equipment to the Internet, but rather is about building agile software that runs across the Cloud and multiple pieces of equipment or devices to deliver that great experience each time.  And while Experience as a Service may seem far removed from where we are today, bear the following in mind:

We always overestimate the change that will occur in the next two years and underestimate the change that will occur in the next ten. Don’t let yourself be lulled into inaction.
— Bill Gates

Everything as a Service, meaning IaaS, PaaS and UXaaS is the business opportunity for the Internet of Things.  At Apiotics, our tools help developers, manufacturers, integrators, and others quickly build great, secure IaaS, PaaS and UXaaS applications for the Internet of Things.  Come build something great with us.

- Mac Dougherty

IoT Devices: Think Cattle, Not Pets

My first job in technology was as a Unix systems administrator.  This was back in ye-olden-days when Linux was relatively new.  A lot of my job was to migrate systems off of Sun and Digital hardware and on to commodity x86 hardware running Linux.

At the time, this was a straightforward case of getting more bang for our hardware buck.

Everybody needs money. That’s why they call it ‘money’.
— David Mamet, Heist

But, though we enjoyed getting more and better hardware given the constraints of our group's budget, we didn't at first understand that in moving to Linux (and, even more so once we adopted virtual machines) we could change the way we thought about our servers in a fundamental way.

When we deployed a server back in the dark ages, we thought a lot about it.  We considered what kind of hardware should go in the system, what kind of network link it should have, what operating system and version, the applications to provision, and on and on and on.

There used to be a well established set of conventions for giving servers individual names.  Folks used to give their servers the names of Greek gods, planets or characters in a favorite novel.  The email server might be Athena, and the database server Zeus, etc. 

The point is that each of our servers were unique and they had unique names to remember them by.

Each of these servers were unique, with unique software stacks and unique capabilities.  As system administrators, our task was to set these up and then watch over them carefully so that they would remain available to our users whenever they needed them.

And they needed to be carefully watched, because each of them had unique data that was important to our users.  While we did keep backups of that data, actually restoring a system from those backups was, even in the best of cases, a lengthy process.  Moreover, since each server used a unique hardware configuration, there was no absolute guarantee that we would be able to restore that backed up data on to an appropriate new server if we needed to.

Because of this, when a system went down unexpectedly, even when things went right our users got upset, which they had every right to do.  And when our users got upset, it seriously interfered with our ongoing quest to find funny things on the Internet.

So, we monitored these servers that we named closely.  We wanted to be sure they were happy, healthy and provided with everything that a server might need.

In short, we treated these servers like pets.

Who's a good server?  You are!

Who's a good server?  You are!

At the time, treating servers like pets was simply The Way Things Are Done.  But it was also at least a little bit crazy.  It was a legacy from the time when buying a server meant buying a mainframe, or at least a very expensive system from a vendor like Sun or Digital.  If you spent your entire hardware budget for the quarter, or maybe even the year on a given system, then of course you're going to treat it with kid gloves.

But, once Linux came along there was much less reason to do that.  Now you could provision a system for a few thousand dollars.  And, because all of these Linux systems used commodity hardware, you could standardize the hardware you used.  Furthermore, once virtual machines came on to the scene, you didn't need to get any new hardware in order to provision a new server at all.  Progress!

Once enough people in technology wrapped our heads around this new paradigm, the race was on to turn servers into standardized parts that could be spun up when needed, aggregated into resource pools and put out of their misery painlessly if something went wrong.

Beef.  It's what's for server.

Beef.  It's what's for server.

With servers, this shift from treating them like pets to treating them like cattle had to happen in order for applications to scale.  When I got started in technology, a web site ran on a single web server.  Now, websites run across hundreds or even thousands of servers.

For the Internet of Things, we have to treat our connected devices like cattle instead of like pets right from the start.  There's no other way to go if we're going to run applications across tens of thousands or millions of connected devices distributed across the world.  To run applications at IoT scale, we're going to need to treat our devices like free range cattle.

It's worth noting briefly here what we mean by an IoT application.  We don't mean a piece of software that runs on a connected device.  There's nothing wrong with that, but there's nothing special about it either.  The IoT is about running software across many devices and the cloud.

The best IoT applications will run at IoT scale.  Think about an application that monitors the operations of a factory or a set of factories.  It might monitor inventory levels, manufacturing line output, any maintenance that is required, etc.  Then think, where does this application run?  It doesn't run on a server, because the server isn't connected to any of the factory systems, at least not directly.  It doesn't run on any particular connected device either.  Rather, it runs across the set of connected devices and servers that it needs.

Treating our devices like free range cattle in order to run at IoT scale means satisfying the following requirements:

  1. Devices need to be able to securely associate with our application from anywhere in the world (at the manufacturer, when the end user takes it out of the box, or anywhere in between)
  2. Devices need to be able to securely update anywhere in the world
  3. Applications need to maintain a secure and dynamic list of participating devices and cloud resources along with their versions and associated software
  4. The application needs to run no matter how many device failures (permanent or intermittent) there might be
  5. Replacement devices need to take the place of offline devices seamlessly

The first three requirements are challenging enough, but the fourth and fifth requirements are where things can really get tricky.

Ultimately, treating connected devices like cattle and not like pets isn't about the amount of special handling that each device requires (though you certainly want to minimize that).  Rather, it's about making sure that failure is always an option for any of these devices.  And, that if any of them fail, the application carries on without a hitch.

Those of you that are old enough will remember the scene in Jurassic Park when, in order to try and defend themselves against the dinosaurs run amok, our heroes decide to reboot all of the park's systems.  At that moment, Samuel L. Jackson got to utter the immortal line:

We will.  By God we will!

We will.  By God we will!

Because nobody knew what would happen when they rebooted the system.  That's a quintessential sign of systems as pets rather than systems as cattle.  At IoT scale, devices will turn on or off, get dropped, lose Internet access, catch on fire (buildings do burn down from time to time) and otherwise go off the reservation as free range cattle tend to do occasionally.  And when they do, your application infrastructure needs to treat it as an absolute non-event, and keep right on keeping on.

At Apiotics, we've spent a lot of time thinking about how to corral your herd of IoT devices.  Check out how it works at https://portal.apiotics.com/.  Keep your focus on building amazing IoT apps, let us take care of keeping the livestock in line.

-Mac Dougherty

The 11 Principles Of IoT Success

At Apiotics we're about helping web developers build great IoT apps quickly.  To help you do that, we've developed a set of 11 principles for IoT success that I'll share with you here.

For $2,000, we'll send you a 12th step of your very own.

For $2,000, we'll send you a 12th step of your very own.

  1. Work in an environment friendly to individuals and small teams
  2. Work on the frontier
  3. Embrace Conceptual Compression. With it, a small team can quickly build full solutions
  4. Don't waste time optimizing things that don't matter (yet)
  5. Don't build the wrong thing
  6. Use the time you save to talk to your users
  7. Take advantage of your small team to be responsive
  8. Use the freedom of not caring about things that don't matter (yet) to be more productive
  9. Resist the fear, uncertainty and doubt that will come your way
  10. Optimize later
  11. Head West

 

1. Work in an environment friendly to individuals and small teams

"Washington is not a place to live in. The rents are high, the food is bad, the dust is disgusting and the morals are deplorable. Go West, young man, go West and grow up with the country."

--Horace Greeley

"The wild lands of the country should be distributed so that every man should have the means and opportunity of benefitting his condition."

--Abraham Lincoln

The U.S. Homestead Act of 1862 opened up huge areas of the American West to settlement by individual families.  It gave up to 160 acres to families who were willing to settle on and improve the land for at least 5 years.  In so doing, the Homestead Act turned the American western frontier into a transformative opportunity for individuals and their families to improve their lives.

Homesteader's House

Homesteader's House

Critically, the structure of the Homestead Act advantaged small farmers rather than large corporations.  As a result, the wealth of the frontier was relatively broadly shared among those who chose to settle.  To be sure, there were injustices aplenty.  African Americans were largely de-facto excluded, and Native Americans were de-jure excluded from a process that expropriated land from them.

Nevertheless, the structure of opportunity for the settlers who settled the American West would have been very different without the Homestead Act.  One can easily imagine huge enterprises buying up the land and renting it out to tenant farmers; perhaps using the sharecropping system that emerged in the American South, where land ownership was not widely shared.

Plantation Owner's House

Plantation Owner's House

Sharecropper's Cabin

Sharecropper's Cabin

When you think about the frontiers that are available to you, it's worth thinking hard about whether they are structured to offer you a homesteader's farm, or a sharecroppers cabin.  

 

2. Work on the frontier

Now, clearly, there's not a literal frontier available in the United States anymore.  The last Homestead Act claim was filed in 1988 for 80 acres in Alaska.  But, there are frontiers to settle.

These frontiers are new technology platforms.  Platforms come on the scene, mature and some eventually fade into legacy systems.  We've seen this happen with mainframes, minicomputers, the PC, the Web, mobile, etc.

But, a platform need not necessarily be a hardware platform.  Programming languages, frameworks, cloud infrastructure all provide tooling and conceptual space for software developers to develop solutions for users.  And, really, platforms are broader than simply a set of technological tools.  They are tools plus the economic and social ecosystem (companies, people, open source projects, etc.) around those tools.

While a platform is relatively new, it represents a new frontier to be settled.  The settlers will build great things for users.  But, how those settlers prosper, or don't, has everything to do with the structure of the frontier on which they've settled.  And, just as the Homestead Act represented a deliberate choice to structure the frontier to enable settlement and development by individual families, the people and organizations that create platforms structure those platforms for or against individual developers and small teams.

For example, perhaps you've got an idea to make a new microprocessor.  The software tools you'll need to design the chip cost hundreds of thousands of dollars, getting a mask set done on a recent foundry node costs millions and you'll also need to build out an extensive set of software tools so your users can program it.  It takes a fair sized team and a reasonable amount of capital to get it done.  And, once you've done it, you'll need to earn your investors a return.  

Turns out, these are tough to make.

Turns out, these are tough to make.

Now, there's nothing wrong with that.  But, the opportunities in that space are not efficiently sized for an individual or even a team of a few individuals to tackle.

Artificial Intelligence is another field that looks to be structured against homesteaders.  While the algorithms, programming frameworks and hardware required are increasingly available and comprehensible to the individual developer, getting enough good data to train an AI is a big and expensive undertaking.  No wonder that the biggest technology companies are laser focused on the AI space.  Given the typical data requirements, it is a natural fit for players of their scale.

At this point you may be rushing to point out all of the exceptions that I am glossing over here.  There's no question that there are exceptions.  But my point is simply that these exceptions are exceptions.  As a general rule, the structure of a frontier platform has a great deal to say about the structure of the opportunities to be found building applications on it.

Perhaps you think I am incorrect.

Perhaps you think I am incorrect.

If you want to be a homesteader and not a sharecropper or a plantation owner, what you want is a frontier platform that has opportunities scaled appropriately such that a generalist developer, or a few of them can tackle the opportunity themselves.  The more you need to build a team of specialists and/or gather a large amount of resources in order to build a solution, the less homesteader friendly that platform is.

Some people think that platforms that are friendly to homesteaders must also be inhospitable to large organizations.  But that just isn't so.  The same structure that makes building solutions efficient for a single developer or a few developers makes building solutions efficient for large organizations too.

Consider the Web.  The Web is in many ways the quintessential homesteader platform.  A single developer or a small team can rapidly build an app that solves real problems for users.  But large organizations can be extremely productive building solutions for the Web too!  There's nothing that shuts big companies out of building solutions on the web.  But there are structural dynamics on other platforms that make it very difficult for an individual or a small team to bring a product to market.

I'm just about to get to it!

I'm just about to get to it!

The point is we're all better off, whether as an individual or a big company, developing solutions on platforms that work for homesteaders.  Not everything can work this way.  I'm pretty sure that Elon Musk needed to raise a bunch of money and put together a big, great, team in order to get Tesla off the ground.  You simply need a lot of resources and people to build cars.

But, wouldn't it be awesome if you didn't?

 

3. Embrace Conceptual Compression. With it, a small team can quickly build full solutions

When we started building Apiotics, I wanted to see if we could meaningfully shift the Internet of Things to be a platform friendly to homesteaders.  To do that, we needed to create a platform that would allow a single developer or a small team the ability to create a complete IoT solution.

There are a number of pieces that go into that, but perhaps the most important is Conceptual Compression.  Conceptual Compression is a term that I am shamelessly stealing from David Heinemeier Hansson (along with any number of other ideas in this post).  What I think @dhh means by that (and what I know I mean by it) is a mechanism that reduces the number of concepts developers need to have loaded up in their head in order to build a great product.  Ideally, it reduces the number of concepts down to the point that a single developer can build a great product.

Just about any technology product, and certainly any application in computing, stands on a stack of dizzying complexity.  Transistors, particularly those in microprocessors, are pretty complex.  And so are processor pipelines, caching memory hierarchies, virtual memory systems, operating systems in general, assembly language, compilers, interpreters, linkers, libraries, imperative languages, declarative languages, databases, and vim.

And then if you want your application to be able to communicate with anything, you'll need some networking.  And as it turns out, that's a fairly complicated topic as well.

Dr. Ian Malcolm describing your technology stack, probably.

Dr. Ian Malcolm describing your technology stack, probably.

Now, if you're building an application in computing and you want to understand your application at a fundamental level, then you need to understand most or perhaps all of these topics.  And if you want to understand, in particular, the performance implications of various decisions that you're making, then you need to understand most or perhaps all of these topics pretty deeply.

While there are individuals who do understand all of these topics deeply, I am pretty sure that you are not one of them, nor am I.  So if you or I want to build an application and have it be understood at a fundamental level, we need to build a team that, collectively, understands the application.

Alternatively, you could rely on the Conceptual Compression provided by your platform to just not worry about this stuff.

The only algorithm you really need.

The only algorithm you really need.

After all, the primary reason that you need or want to understand your application at a fundamental level is because you want to make decisions in light of that information.  This seems like a very reasonable thing to want to do.  And, platform builders think this is a reasonable thing for you to want to do as well.

As a result, they expose tons of different levers to pull and switches to toggle so that you can make all of these decisions in the right way for your application.  Since these platforms are technical platforms the data you need to make these decisions in an informed way is technical data.  Specifically, you need technical data about the way the performance of your particular application varies as these different levers are pulled or switches toggled.

In an ideal world, you would already have a running production instance of your application and the flexibility to simply make these changes, observe the results, and select the best set of platform settings.

In the real world, you never get to do that because the first time the application goes down under a bad set of platform settings the torches and pitchforks will be out and coming for you.

So, instead of doing that, you bring together a team that collectively understands your application at a fundamental level so that you can, as a team, Make The Right Decisions.

The thing is, most of the time it just doesn't matter.

So easy to say, so hard to do.

So easy to say, so hard to do.

4. Don't waste time optimizing things that don't matter (yet)

There's a classic book on manufacturing called The Goal and in it the author, Eliyahu M. Goldratt, describes his Theory of Constraints.  There are many great insights in this book (which I won't attempt to fully summarize here) but one of them is that the only way to improve your business is to identify what your business is currently constrained by and improve it.

A short example from manufacturing will serve to make the point.  Let's say that you have a machine that makes widgets and a truck that delivers them to your customer, who pays you when the widgets are delivered.  The truck can deliver 100 widgets per day and the machine can make 200 widgets per day. 

In this widget making operation, any effort you put into improving your widget machine is totally wasted.  In order to sell more widgets, you need to improve your truck.  Maybe you need more trucks, maybe you need a bigger truck, maybe you need a better driver.  But whatever it is that you need, until you can deliver at least 200 widgets per day you don't need an improved widget machine.  The widget machine is not the constraint.

Liam Neeson loves widget machine meetings

Liam Neeson loves widget machine meetings

And, in almost all cases, your technology stack/platform is not the constraint either.

Most of the time, the team you bring together to Make The Right Decisions is just improving the widget machine.  Yes, maybe someday it will be great to have one that can make 1,000,000,000 widgets per day, but today you're just wasting time, effort and money filling up your plant with widgets that your truck can't possibly deliver to your customers.

And why is that?  It's because that huge pile of technology you're building your application on top of is really really really awesome.  And, it gets better all the time.  The widget machine above would more accurately describe your technology stack if it more or less doubles its output every year, for free. 

Now, given the complexity of whatever stack you're on it is true that there's no end of improvements you could make if you apply sufficient talent and time.  And these improvements are often seductively objectively measurable. 10% improvement in boot time!  15% reduction in memory footprint!  But you don't need to make them.  At least, you don't need to make them yet.  Doing nothing will get you huge improvements year after year.  Your stack just isn't your constraint.

What is the constraint? Making something people (or companies) want.

This guy's shirt gets it.

This guy's shirt gets it.

5. Don't build the wrong thing

In the Internet of Things, doing this is a particularly pressing problem.

Cisco published a survey last year called IoT Value: Challenges, Breakthroughs, and Best Practices.  In it there are any number of interesting nuggets, but the one that most struck me is that nearly 75% of corporate IoT initiatives fail.

Pie charts.  Never as much pie in them as there should be.

Pie charts.  Never as much pie in them as there should be.

Digging a bit deeper, the survey shows a really interesting split between technical and non-technical responders.

If you look only at the responses from non-technical executives, and not from the technical executives, the non-technical believe that only 15% of their companies' IoT initiatives were successful.  Where the technical folks think 35% are successful.

What accounts for the split?

I think there's no question that the widget machine accounts for the difference.  You can just imagine a technical executive saying to himself as he fills out Cisco's survey: "well, it's true that nobody actually wanted our electric frankfurter, but damn if it doesn't have a bitchin back end infrastructure."

If you say so Dawson.

If you say so Dawson.

The survey also asked its respondents to list the main difficulties with the IoT, and here's what they said:

  • (Too much) time to completion
  • (Lack of) internal expertise
  • IoT integration (is too hard)
  • Budget overruns
  • (Low) quality of data

Of these, I think quality of data is the most interesting.  Normally, low quality data is a signal vs. noise problem.  There's a whole deep technical field devoted just to overcoming this problem, but I don't think that's really the issue with the IoT.  

After all, we have really great sensors and the ability to collect enormous amounts of data incredibly cheaply.  And signal processing, as complex a topic as it is, is pretty commoditized.  You can buy an incredibly sophisticated digital signal processor to extract good data from a noisy feed for literally tens of dollars.

No, low quality of data means they built the wrong thing.

The data is likely to be quite clean and comprehensible from a technical stand point, but that doesn't mean that it actually adds any value.  You could, for example, build a great weather monitoring system that provides real time data on precipitation, humidity, cloud cover, wind speed and direction, etc.  But if the people you're building it for make baseball bats... then you built the wrong thing.  Even if the data is clean, it doesn't help your users make better decisions to improve their business.  And at the end of the day, quality data is data that lets you make better quality decisions.

That's a somewhat contrived and silly example, but you see versions of that sort of thing all the time.  You could imagine having an initial meeting with your client and talking about the business they're in, making the observation that baseball games can only be played in good weather and then going off and building an awesome weather data system.  Interesting, maybe even helpful to a related business like a baseball team, but not helpful to your client the baseball bat manufacturer. 

Me too buddy.  Me too.

Me too buddy.  Me too.

Now, if you happen to know a something about baseball, this is an obvious and trivial error.  But often times when you're building an application you're building it for a group of users whose business or use case you don't know all that well, if at all.  And it's all too easy to make a relatively simple mistake like the one above and build something that is completely beside the point.

 

6. Use the time you save to talk to your users

So how do you avoid making this mistake?

You talk to your users.  A lot.  Like, all the time.  You show them what you're building while it's in process to make sure that you're aligned with their needs, and you keep showing them work in process until you deliver something they think is great.  It's not rocket surgery, but it can be surprisingly difficult to do.  There's even a whole project management philosophy, Agile, that puts this process at the forefront.

I'm not going to tell you that Agile is perfect.  There are drawbacks to any approach.  And Agile, just like all other management techniques, too often decays into cargo cultism because that's easier than actually doing the damn thing.

Regardless of how you implement it, talking to your users/customers while you're working on what you're building for them is the best way (though it's not foolproof) to build something they want.

Good question Jean-Luc.

Good question Jean-Luc.

Clearly, people aren't thinking of this when they're building IoT applications.  If they were, the success rate would be higher than 25 (or 15!) percent.  So why aren't they doing it?

Remember the main difficulties with IoT from the Cisco survey above?  No?  Ok, here they are again:

  • (Too much) time to completion
  • (Lack of) internal expertise
  • IoT integration (is too hard)
  • Budget overruns
  • (Low) quality of data

We've covered low quality of data a.k.a. building the wrong thing, but what about the other items on this list?

All of these other items are symptoms of too little Conceptual Compression and too much work on the widget machine.  Since I'm trying to stand on the shoulders of giants as much as possible with this post, I'll turn to the incomparable @internetofshit to give an example of what I mean:

BTW: can't wait for @internetofshit to make fun of us.  Then we'll know we've made it.

BTW: can't wait for @internetofshit to make fun of us.  Then we'll know we've made it.

There's just a ton of complexity in that diagram.  And, it's hard to imagine a single developer being able to really understand how all of the elements in that architecture work.  So, you need a team, and probably a big team if you want to understand it and optimize it for your use case.

And, its all too easy to spend so much time and effort on making the plumbing of your application work, that it leaves little or no time to talk to your users and find out whether you're building a weather system for a baseball bat company, or not.

But, that's not really the worst part. 

The worst part is that as you scale your team to deal with the complexity of your platform you need to change the way your team works in a way that makes it structurally impossible (or at least very difficult) to adjust what you're building to what your users want you to build.

The thing about computers is that they're dumb.  Really dumb.

We've all been there, but only xkcd had the courage to write it down.  The hero we need.

We've all been there, but only xkcd had the courage to write it down.  The hero we need.

And the way in which computers are dumb is that you as a developer have to be really super explicit about what you want them to do when you program them.  The computer never fills in the details, at least not ever in a helpful way, and you just can't rely on the sort of context clues that we rely on all the time when we talk with people.

 

7. Take advantage of your small team to be responsive

Context is super important and powerful when it comes to communication.

Different ways you communicate with other people have different amounts of bandwidth associated with them and the difference is almost entirely due to context.

Ranked from high bandwidth to low:

  • Talking in person
  • Video conferencing
  • Talking on the phone
  • Email

What's common among all of these mediums is the words you use.  You can use exactly the same words to talk with someone, video conference with them, talk on the phone and write in an email, but the amount of information that comes through to the other person will be vastly different.

The problem is that in general the scalability of these methods of communication are inversely related to their bandwidth:

Ranked from high scalability to low:

  • Email
  • Talking on the phone (conference call)
  • Video conferencing
  • Talking in person

Now you might quibble and say "but I can broadcast myself on video to millions or billions of people around the world, that's super scalable".  And I'd say "hey, this is my blog post, sit down and don't forget to tip your waitress" and then I'd remind you that I'm writing here about two-way communication, not broadcast.

Communication. It's a two way street.

Communication. It's a two way street.

The reason I'm writing about two-way communication is that's what you'll be doing when you talk with your users and customers.  Ultimately, the wants communicated in those conversations are what need to get transformed into a product that your users and customers love.  And, if you could just tell your computers to just have a conversation with your users and code up what they want, then we'd all be on easy street.

But, of course, you can't do that.  Instead, it's up to you and your team to translate your users' wants into code that your systems can run.

If you have a small team of one, two or three people, then it's easy for the whole team to have a high-bandwidth in person conversation with a user and translate the conversation directly into code.

On the other hand, if you have a large team, then it's often quite difficult for your whole team to communicate in person with your users.  In that situation, you usually appoint one or a few member(s) of the team to be the product architect(s) and have her create specification documents for the rest of the team to follow as they implement their pieces of the system.  This is fine as far as it goes, but it becomes extremely challenging if your picture of what the user wants is shifting over time.

If your picture of what you're building shifts over time, and you can't meet with your user as a team, then you're in for a lot of time and expense to shift the specifications and the architecture of your application to align with the new understanding.  Said another way, as your understanding of what the user wants changes, your architects have a large amount of work to do translating what they hear in a high-bandwidth conversation into low bandwidth documents that scale to meet the communication needs of a large team.

It's not so easy my man.

It's not so easy my man.

So why not just get it right the first time?  Because your users don't want to tell you that you've got an ugly baby.  In general, people try to be polite.  So if you go to a user and you tell them about something you're building, they'll often say "sounds great", no matter what they really think.  And often your users will just blithely assume that what you're building is exactly what they want, without actively listening to what you're actually saying it is that you're building.

The net result of this dynamic is that in most cases you can't describe what you're building to a user in order to get feedback that you can actually use.  You have to show them something.  And even then, when they're engaging with a real prototype, they may be too polite to really tell you.  But if you don't show them what you're working on, you'll almost never get to hear more about what they really want.

So you've built a prototype, and you've shown it to your user, and you've gotten useful feedback.  Achievement unlocked!  And now you need to incorporate that feedback into the next iteration of your development.  If you're lucky enough to be working with a small team, then your team can pretty much get right to coding.  They've all got the full, high bandwidth context in their heads, and can start translating that into code that the dumb computer can actually act on.

But if you have a large team, then your architect needs to spend a bunch of time translating what the user said into a new set of specifications and then bring the team up to speed on what they need to know to start using them.  And, if the user's feedback requires a fundamental change in the application's architecture then the big team is really in trouble.

Too true Billy Madison.

Too true Billy Madison.

Interestingly, this isn't because such a change is technically challenging per se.  Rather, it has to do with something called Conway's Law.  In plain English, Conway's Law says that the structure of an application mirrors the organizational chart of the engineering team assigned to build the application.  As Eric Raymond once put it "if you have four groups working on a compiler, you'll get a 4-pass compiler."

Al Shalloway's corollary to Conway's law is really what concerns us here.  This corollary states: "when development groups change how their development staff is organized, their current application structure will work against them."

What this means is that if the architecture of your application gets out of sync with the architecture of your org chart, your development effort is likely in serious trouble.  But that's exactly the sort of thing that is likely to happen if you're running a big team and trying to reshape the contours of your application in response to continuous user feedback.  The requirements change, the goalposts shift.  And this is an essential part of the process!  But it's not something that big teams are well adapted to because of the communication overhead and because of the implication of Conway's Law.

Incidentally, these dynamics are one reason why big companies can find it difficult to innovate.  If you're trying to build a new thing, that's when you most need user feedback to find your way.  And that's when small agile teams are at their most advantaged relative to big teams.  But, big companies often, quite rightly, think innovation is a top priority for them.  And, in their big company way, assign lots of resources to deal with the issue, which often attracts a big team.

Note though, that the same dynamics apply just as much to small companies as to big ones.  Big companies can get great results if they scale development efforts to be aligned with the need to be responsive to user input.  And that means that big companies, just like little ones, benefit from Conceptual Compression and homesteading platforms.

Whether big or small, all pumpkins can write great software equally well.

Whether big or small, all pumpkins can write great software equally well.

8. Use the freedom of not caring about things that don't matter (yet) to be more productive

So what is Conceptual Compression really all about?  And, what mindset do you need to bring to the table?  One of the very best distillations I've found comes from the Rails Doctrine, a manifesto of sorts that describes the mission of the Ruby on Rails web framework.  Below, I've shamelessly adopted (stolen) the parts that I find most compelling to be the heart of what we're trying to do here at Apiotics.

First: You’re not a beautiful and unique snowflake.

By giving up vain individuality, you can leapfrog the toils of mundane decisions, and make faster progress in areas that really matter.  Who cares what protocol your devices use to connect to the cloud? Does it really matter whether it's MQTT, XMPP, AMQP, HTTP or HELP? (I might have made that last one up.) Is this a decision that’s worthy of recurrent deliberation? No.  And this is actually one of the more high profile decisions that you might make as you're fine tuning your widget maker.

Apiotics' mission is to swing its machete at the thick, and ever growing, jungle of recurring decisions that face developers creating information systems for the Internet of Things. There are thousands of such decisions that just need to be made once, and if someone else can do it for you, all the better.

But beyond the productivity gains for experts, conventions also lower the barriers of entry for beginners. There are so many conventions in Apiotics that a beginner doesn’t even need to know about, but can just benefit from in ignorance. It’s possible to create great applications without knowing why everything is the way it is.

Second: Value integrated systems.

Apiotics is about making integrated IoT systems.  A whole system that addresses an entire problem.  That means we concern ourselves with everything from live updating device information on your webpage without a page refresh, to helping you use prototype device hardware that one of our contract manufacturers can most easily productize and build millions of units based on.

Apiotics specifically seeks to equip generalist individuals to make these full systems. Its purpose is not to segregate specialists into small niches and then require whole teams of such in order to build anything of enduring value.

It is this focus on empowering the individual that points to the integrated system. It’s in the integrated system we can cut out many needless abstractions and reduce the duplication between layers.

 

9. Resist the fear, uncertainty and doubt that will come your way

Now that I've gotten this out there, I can feel all of the embedded developers rising as one to shout at me, "we've been building IoT device firmware for years, it works well, why are you fixing something that isn't broken?!?"

I feel a great disturbance in the Force.  As if thousands of embedded developers cried out "this makes no sense to us".

I feel a great disturbance in the Force.  As if thousands of embedded developers cried out "this makes no sense to us".

Well, it is broken for all the reasons I've stated above, though it is also the process people have today. 

Apiotics is in the first instance a platform for web developers.  And in our first release, a platform for Ruby on Rails developers specifically (but we'll be adding support for more web frameworks).  Embedded developers have a critical role to play, which I will come back to.  But, the developer we want to empower to make an integrated IoT system using Apiotics is a web developer.

Why web developers?  Well, because many web developers already have the mind set that we think is so valuable: building things people want means rapidly building versions of the product to show to users, incorporating user feedback and repeating that process until you have a fully baked product.

Also, web developers already have powerful frameworks (like Rails) that provide enough Conceptual Compression for building web applications that a single developer can build one quickly.  Our task is to add on to that foundation to let a single developer build a web application that interacts with custom IoT devices quickly.  Still a significant task, but one that is much easier than it would be if these web frameworks did not exist.

Finally, we think that in most cases, the vast bulk of the unique business logic for an IoT application belongs in the web application.  There are certainly exceptions and cases where putting rich logic into a device or a gateway makes good sense.  But we believe that the right approach is to treat these exceptional cases as exceptions, and to focus a developer's on the right task: the unique business logic, in the right place: the web application.

If, after all this, you're curious about how we do this (and I hope you are) we've got a set of video tutorials on our site showing a web developer building a simple but full end to end IoT application in 3 minutes (and then again more slowly in about 30 minutes).

If you watch these videos, one thing that I think stands out is all of the code that you don't have to write and the pieces of the puzzle that you don't have to concern yourself with.  With Apiotics, you really can just focus on your business logic and not worry about understanding or implementing the rest.

There is no code. (That you need to think about.)

There is no code. (That you need to think about.)

Many times developers (particularly those of the experienced and perhaps somewhat grizzled variety) will harrumph that you can't expect to rely on code that you didn't write.  Now, these developers are overlooking, I believe, the benefits of Conceptual Compression.  But, they're also making a different but important mistake.

Code is a liability.

The features of your application and the benefits that they can provide to your users are real and important assets.  Many great companies have been built on the strength of such assets.  But the code that implements them is a liability.  Code needs to be implemented, tested, deployed, monitored, patched, version controlled, maintained, documented and finally retired.  That's a lot of cost and headache.  But do you know what code doesn't need any of that?

The code you don't write.

Getting functionality your users want without writing code is about the closest thing to free money that we've got in technology.  Perhaps the cryptocurrency people will disagree with me on this point.  So we'll say that it's at least one of the two closest things to free money in technology.  And this one won't bring the SEC knocking on your door.

A moment of zen.

A moment of zen.

And now I know that the embedded developers are at the point of breaking down the last gate in the fortress screaming "look this all sounds great, but pointers are real. They’re what the hardware understands. Somebody has to deal with them."

And this is very true.

So our approach to this is to encapsulate the parts of the system that interact with IoT device hardware into drivers.  If you are an embedded developer, you can develop drivers for IoT hardware and either give them away or sell them through our portal to the web developers building applications.  That way you can, as an embedded developer, focus on what you do better than anyone else, which is writing software that runs directly on the IoT device hardware.  And the web developers can select the drivers they need for the devices they want and build their applications on top.

And the way Apiotics has done this, embedded developers don't need to know anything about the particulars of the web applications that use their drivers or about the network between the devices and the web application.  They can just focus on supporting the hardware.  So, we're bringing Conceptual Compression to bear for the embedded folks as well.

Some of the people who have stayed with me this far (thank you) might be hardware people.  People who don't hesitate to gin up a new circuit board and who know that real engineers use soldering irons.  And if you are a hardware person you might at this point be ready to throw your tomato at the stage saying:

"Now that you software people have finished nattering on about whatever you software people natter on about, you're missing the main point. And the main point is that you can't possibly expect to optimize the cost of your IoT device if you don't make a careful analysis of the tradeoffs of each component of your bill of materials.  And if you're going to ship millions of these things, you damn well better do that analysis if you don't want your financial people to hunt you down with a sharp stick."

To which I say, it's a fair point as far as it goes.

 

10. Optimize later

But, remember our homesteader?  Way back at the beginning?

Home on the range.

Home on the range.

Did you notice that there are actually two houses in this picture?  The one on the left is a sod house, which is a house that is quite literally made from prairie grass.  Typically homesteaders built sod houses as their first house and then later built wood frame houses like the one on the right.  This picture captures a homestead with both structures in place.

Now a sod house isn't nearly as nice as a wood frame house.  They're damp, require frequent maintenance and don't like getting rained on, which is problematic for a house.

But, they are much quicker to build than a wood frame house and can be built by a single homesteader with simple tools.  The great virtue of the sod house is that a homesteader can build it fast and live in it while figuring out whether their homestead would actually make it as a working farm.  For a homesteader, building a wood frame house first is a great example of optimizing the widget maker rather than focusing on what matters: getting the farm up and running.

Eventually, if the farm works out, then of course there comes a time to optimize.  It's the same for the IoT.  The best thing to do is to get your application and its devices out there quickly and with minimal development cost to see if the market you're going after is a real one.  Then, if it is, you can always squeeze cost out of your device later, and keep running your Apiotics enabled application on top of it.

 

11. Head West

So that's what we're about at Apiotics.  Our mission is to transform the IoT frontier into an environment that any homesteader can build a great application, driver, or business in.  We've built some really great tools to help you bring these principles to bear on your next IoT project.  I hope you like them. 

But even more than that, I hope to see you on the trail, heading West.

- Mac Dougherty