DevOps without papercuts

(Photo is licensed under CC BY 2.0, and you can find it on Flickr)

On Papercuts

Have you ever found yourself trying to do your job - but the system is fighting you at every turn? That “simple” feature that winds up taking days, because there’s not enough test coverage, or a subsystem wasn’t factored quite right. The application is deployed, but the networking doesn’t seem to quite work the way you expect. You have to wait 30 minutes for CI to tell you if you did a good job. Or my personal nemesis: messing up the whitespace in a kubernetes yaml file.

Those are papercuts. The shortcuts we take to land some code, promising to refactor it later. The abstractions we created to make our lives easier, but that don’t quite work correctly. The fragile integration. The read-only syntax. They’re innocent enough at first - useful, powerful, functional - but as they build up over time, they grind your productivity to a halt. Everything takes forever, and the prospect of having to work on the system fills your heart with dread.

DevOps tooling is full of papercuts. 15 years of accumulated technical debt, peanut-buttered across dozens of different platforms and solutions. The tools that once gave us superpowers by removing the incredible tedium caused by doing nearly everything by hand have become the source of our pain. Death by a thousand cuts.

Imagine you’ve built a new application - it’s big, it’s complex, and it’s going to change the world. You crack your knuckles and get down to work getting it deployed to production:

1. Choose your cloud provider.
2. Pick the right mix of services.
3. Kubernetes? Terraform? Pulumi?
4. Ansible? Puppet? Chef?
5. Docker!
6. Kubernetes.
7. Draw a diagram? Draw a diagram.
8. Set up your editor.
9. Write your infrastructure as code.
10. Oh shit - should we be GitOps-ing?
11. Definitly need some continuous delivery. Maybe Github Actions?
12. Kubenetes.
13. Write some yaml.
14. Squint at yaml.
15. Go for a walk.
16. Istio?
17. Linkerd!
18. What about the database?
19. 1password for teams!
20. DataDog?
21. Honeycomb!
22. How do we test this thing? Lets set up CI.
23. Refactor. Maybe later.
24. GitOps?
25. Write some documentation for how to use this crazy machine you built. (Ha!)
26. Oh no, these IAM permissions will not do.
27. Do we need to be multicloud?
28. Take a nap.

I could go on. I know you can too.

If you work in a large organization, it’s gotten bad enough that you have almost certainly gone backwards from the original DevOps intent: you’ve got a team of dedicated DevOps Engineers. Maybe you call them SREs instead. Either way, they are the people who can deal with all of these painful, messy details. They soak up the papercuts for the rest of the team.

It’s bad enough that DevOps has become the bottleneck in our engineering flow. We stop trying new tools or approaches, because integrating them into our stack is an incredible burden. We stop working cross-functionally, because everyone who isn’t a DevOps Engineer won’t go anywhere near any of this with a 10 foot pole. It’s a twisty maze of tiny papercuts, just waiting to swallow up your days. No one tool is to blame - it’s the whole system, fighting you, all day, every day.

The DevOps experience is undoubtedly the worst part of building a modern application.

System Initiative

Removing the papercuts from DevOps is our vision. To make it collaborative again. To make it as immediate as drawing a diagram. To leverage the relationships between components to intelligently configure them. Not by restricting you, or abstracting the complexity from you - but by building the power tool you need to cut through it all.

In the coming weeks we’ll write about all the little (and big!) ways that our DevOps toolchains hurt us. We hope you’ll come talk to us - on our Discord instance, on Twitter, or in Email - and share your own stories of DevOps papercuts.

System Initiative. DevOps without papercuts. That’s the dream.