For five years, until he sold it, Matt Stratton's car sported a “DEVOPS” vanity plate.
It’s a minor detail in a bio that contains some major highlights that include co-hosting the popular Arrested DevOps podcast, plus professional stops at Chef, Microsoft and PagerDuty, the incident-monitoring platform where he currently works as a (you guessed it) DevOps advocate. But it underscores some key truths, perhaps most notably that DevOps inspires serious passion. Also, since Stratton was an early adopter, he’s well acquainted with the tools of the trade.
In explaining DevOps, another tech buzzword is helpful: “siloed.” In plainer terms, disconnected or isolated. That’s what software development and operations teams used to be in relation to one another. An organizational philosophy that unites the two (hence the portmanteau), DevOps is often visualized as an infinity loop. The dev side codes, builds and tests the software, while the ops side releases, deploys and monitors how it functions in the real world. And since they're intertwined, teams can quickly find and fix any issues that arise.
On a more granular level, DevOps includes tracking code changes, adhering to the now-standard model of continuous integration/continuous delivery (CI/CD), scheduling and orchestrating containers, managing configuration, and careful, consistent post-release monitoring. (Some of the applications on which DevOps teams rely to achieve all that are essentially standardized across the tech board; others vary depending on preference and environment.)
While that’s all as highly technical as it sounds, DevOps is more about getting as close as possible to frictionless, hand-in-glove collaboration than it is about gear-head tool obsession.
“Whenever there's organizational changes, it goes people, process then tools,” said Jeffery Smith, director of production operations at Centro and another Chicago-based DevOps advocate who’s also a fixture at the annual DevOps Days Chicago. “Only after the first two are done should you really care about the applications.”
“Some people focus on the tool, then try to backfill the rest, and that's an ass-backwards way of doing it,” Smith added.
But once the foundational buy-in is cemented and the time arrives to talk shop, an organization must bear in mind some key application distinctions. With an assist from Stratton and Smith, here are 14 DevOps tools you should know.
Source Control Management
Git is the go-to system for managing changes to source code — the “de facto standard,” as Stratton calls it.
“In source control management, there was a battle and Git basically won, for better or for worse,” he said.
There are a few likely reasons for Git's victory. The fact it was built for Linux kernel development, its rise coincided with a broader push toward open source and its distributed structure — which lets every developer on the team see the program’s full source code — all secured its pole position. And now any code repository host with hopes of real adoption had better support Git.
GitHub
Location: San Francisco
As Git became the standard, GitHub became its standard host. In software engineering circles, the GitHub icon is as ubiquitous as a Twitter or LinkedIn avatar link on a personal homepage. And once again, it came down to the open source angle.
“Social coding is where GitHub led the charge,” Stratton said. “So it's not just about versioning your file, it's about the interaction of humans. You’re able to do a pull request, note the changes you want made and write into it the whole communication structure.”
Acquired by Microsoft in 2018 for $7.5 billion, Github facilitated a culture that ultimately pervaded the tech sector. Now, Stratton said, “people are able to use the same tooling and way of thinking on open source side projects as they do in their regular enterprise or startup day jobs."
GitLab
Location: San Francisco
As you might have guessed from its name, GitLab (like GitHub) is a web-based Git repository — but with some notably different features. Along with source code repo, GitLab also boasts native support for CI/CD and containers (more on both of those things below).
"Everything you would want to do in a modern development workflow is housed in a single place in GitLab," Smith said.
That makes it a bit easier for newbies to navigate.
"For an organization that's starting off with their DevOps journey," Smith said, the less friction, the easier."
Bitbucket
Location: San Francisco
When it comes to tools, DevOps pros are often inclined towards “whatever works.” All repo applications perform the same basic function, so organizational preference holds sway.
“As long as you're using some Git-based tool, I don't really think it matters whether it's Bitbucket or GitHub or GitLab," Stratton said. “People put too much energy into that.”
Outfits that use Atlassian can keep it simple by opting for Atlassian’s offering: Bitbucket. That way, “the path will be much easier from an organizational buy-in perspective,” Smith explained.
“A lot of people use [Atlassian’s] Jira for their story management, so there's obviously tight integration [between the two]. Being able to reference Jira tickets and your Bitbucket commits is extremely nice. But if your organization is already using GitHub or GitLab, that's fine.”
Bitbucket was also an option for the few remaining holdouts against Git who preferred Mercurial, a competitor system. But that capability is about to give up the ghost. In August, Bitbucket announced that it will no longer support that relatively unpopular option for managing source code changes, and all Mercurial features and repositories will be removed in June, 2020.
Continuous Integration/Continuous Delivery
A key DevOps tenet, CI/CD refers to the pipeline that ensures quick and efficient software development. Continuous integration, or CI, essentially means that developers do a build whenever merging branches of new code changes into the master branch.
“But just building the software these days isn’t enough,” Stratton said. “You have to deploy it.”
That’s where you get continuous delivery, or the idea that, just as DevOps automates testing, it also automates the release of the final version of an application.
To accomplish that, all CI tools and CD tools operate along similar technology and have similar features. And each tool is only as effective as the programmer behind it, Smith stressed.
“The bottom line is, it executes a command that you've written in house,” he said. “So if your test suite sucks, it's going to suck in CircleCI, and suck in Bamboo, and suck in Jenkins.”
So what makes one option more attractive the next?
Jenkins CI
Location: San Francisco
Perhaps the central consideration in CI/CD is, How much should you trust the cloud? For some organizations, the answer is, Not so much.
“Maybe an organization doesn’t like pushing their proprietary information into a cloud provider,” Smith said. “Maybe it has a number of restrictions from sharing code with third parties that don't meet particular standards.”
For that kind of situation, an on-premise option makes the most sense, and Jenkins leads the market there. Open source, it boasts an ultra-robust plugin ecosystem. And because it's been around since 2011, it has established a solid performance track record.
CircleCI
Location: San Francisco
For the less cloud-averse, CircleCI is a popular standout. (Travis CI is another option, although it’s fallen by the wayside among many software engineers, said both Smith and Stratton.) With CircleCI, you can enable GitHub checks, run images from Docker’s registry and easily debug through SSH or local builds.
GoCD
Location: Chicago
Back in 2010, ThoughtWorks veterans Jez Humble (a co-author of this year’s Accelerate State of DevOps Report) and David Farley literally wrote the book on continuous delivery. Many of the ideas they first explored came to bear in a tool called Go, now known as GoCD.
“It’s intended as an implementation of a lot of those core principles,” Stratton said.
The free and open-source tool uses parallel and sequential execution to break down each pipeline into stages, then jobs, then tasks. That means developers can draw a pipeline with plenty of fine-point detail and repurpose those details across other pipelines.
Containers
If you’ve spent time around developers, you’ve probably encountered a shirt or sticker emblazoned with the shrug emoticon and this classic nugget of coding mock-resignation: “It works on my machine.” Stratton referenced that infamous eye-roller line to illustrate what a milestone containers really were.
Containers effectively eliminated that headache by standardizing the development environment and making it predictable, no matter whose eyes it happens to be under.
“Containerizing basically provides a level of isolation that says this process does this one thing, and everything that it needs to do can be moved [between environments],” he said.
It splits packaging from the actual run environment, which means developers know that what works here will also work there — be it a laptop or server, cloud or data center.
“It really puts the focus on the application and makes it a lot easier to ship."
Docker
Location: San Francisco
Containers as we now know them were a watershed, but they aren’t exactly new. Linux containers have existed for a very long time, Stratton noted, but they were difficult to use. Docker provided an accessible format and approach, igniting still-mounting (and lucrative) enthusiasm around the industry. In fact, Docker did it so effectively that it became a synonym — what Kleenex is to tissue or Xerox to photocopying.
Kubernetes
Location: San Francisco
Docker offered a way to package code so it could run the same regardless of environment. But it still needed to be operationalized, managed and scaled. This scheduling solution, “the plumbing of your infrastructure,” as Stratton put it, emerged as an open-source offshoot of Google’s internal Borg and filled that space. It soon became another “de facto standard.”
It is now commoditized by the cloud heavyweights — as EKS, AKS and GKE — but organizations still need to have an internal understanding of how to optimize Kubernetes.
“You want them to schedule for best availability and be able to scale up and down as you need to,” Stratton said.
Configuration Management
While some in software development seem ready to file configuration management alongside Dillinger, disco and the dinosaurs, rumors of its death may be exaggerated. In a post-container world, the problem of software changes not being tracked — aka configuration drift — isn't the problem it once was seeing as engineers can usually just reset the container and go back to the previously defined state.
“But you still need to get to that initial state,” Smith said. “And how you get there is configuration management.”
CM means “having the ability to test [a configuration item] and to do abstractions against it,” Stratton said. “Just like I test my application, I need to put it through a continuous delivery pipeline, so that I understand when I run, this isn't going to do the thing I expect.”
Chef
Location: Seattle
It’s useful to think about configuration management in terms of imperative vs. declarative approaches. With imperative tools, like the written-in-Ruby Chef, users build a file that reads from top to bottom and execute commands as they’re set. For those who don’t speak software development, that basically means more control under the hood.
“It gives you an incredible amount of control over what’s happening, how it's happening, and how those things are actually getting manifested,” Smith said.
At the same time, for folks who lean more toward Ops than Dev, it may be TMI.
“For some organizations that have traditional IT folks, who don't have a lot of programming backgrounds, those teams could struggle to implement tools like Chef," Smith said. "Because it's more programming focused."
Puppet
Location: Portland, Ore.
On the more streamlined (if less minutely controllable) declarative side, Puppet remains a go-to.
“You don't really need to worry about the specific implementation bits; you just define the final state of how the thing should look,” Smith said.
Instead of imperative’s top-down read, Puppet effectively figures out a dependency tree. Or, as Smith explained, "what needs to be created first before another can be. It does a lot of that magic for you.”
Other popular declarative options include Ansible and Terraform. But Puppet, like Chef, has an advantage over those two in at least one sense: it’s been around longer and has build up a robust community.
SaltStack
Location: Lehi, Utah
Beyond the imperative or declarative approach, another configuration-management consideration is what’s called remote code execution — or the ability to execute a command across a series of servers. That means every server that meets a given criteria can execute a corresponding command.
“And that’s great in an environment with a ton of servers,” Smith said. Such an organization may do well with SaltStack, which Smith lauded for its "excellent remote execution capabilities."
Monitoring and Observability
Once an application is built and released, DevOps teams must constantly ensure that it’s running as intended. A slew of monitoring tools now exist — ranging from the more traditional alert systems to dashboards that provide a lot of information at a glance — that allow engineers and operations departments to stay on top of performance.
But the most talked-about emerging development in monitoring — and perhaps DevOps as a whole —is observability. It’s a complicated, much-debated term, but the gist of it is this: now that our systems have become so complex, DevOps monitors don’t even necessarily know which issues to watch for or what alerts to set. In light of that, it's necessary to take more of a macro, bird’s-eye-view.
Prometheus
Before we dive into observability, let’s look at more traditional monitoring. In terms of the divide between basic alerts and dashboard-based visualization, Smith argue, software engineers should move toward the latter. More specifically, that means employing tools like Prometheus that "bring together metrics, learning and visualization in a single place."
That illustration of data provides a greater degree of clarity than a more binary tool, like Sensu, that simply alerts users when a certain command spits out an unexpected result. In terms of troubleshooting, however, that gets you only so far.
PagerDuty
Location: San Francisco
“Single pane of glass” has been an IT cliché for years, but there’s a reason the concept persists. Any tool that actually unites a thicket of inputs in a single dashboard is an extra-strength headache reliever. PagerDuty — which famously became the second Y Combinator alumnus to go public, after DropBox, and which counts Box and Netflix among its clients — essentially aggregates various digital signals, including monitoring tools, while also streamlining alerts. That helps prevent alarm fatigue and, ideally, allows responders to take appropriate action. However handy a tool is, though, internal best practices are a must.
“A big part of monitoring is not just getting the signals, it's also having a good incident-response culture,” Stratton said. “That's a big part of DevOps.”
Honeycomb
Location: San Francisco
Depending on whom you ask, the emergence of observability as a distinct term may or may not have a marketing component. But there’s genuine technological advancement behind it, too. The pacesetting company in this realm is Honeycomb, which lets evaluators collect data through distributed tracing across a full range of information.
“It enables us to ask new questions and look for hotspots and correlations because we do have these very complicated systems.” Stratton said.
As Smith put it, observability is “monitoring with the technical constraints that we used to have on monitoring removed” — namely the inability to add and enrich metadata to metrics.
Honeycomb is also home to several influential figures in not only DevOps, but software engineering in general, including Liz Fong-Jones and company co-founders Christine Yen and Charity Majors, the latter of whom has arguably done more than anyone to define and popularize observability.
