A clunky toolchain comes with a cost. According to a report by research firm Forrester, many development teams are seeing a 10 percent “DevOps tax” — meaning one tenth of their team was devoted to maintaining the DevOps toolchain. In a separate report, data indicated that “while DevOps tools have proliferated, release velocity remained stagnant” for the previous five years. So for all the hypothetical horsepower these tools are supposed to bring to a deployment lifecycle, a misaligned toolchain can leave the team spinning its wheels.
With the proliferation of DevOps tools, the combinations for a toolchain are now seemingly infinite. While having lots of options is great for toolchain customization, evaluating so many choices can lead to overload. In the face of this paradox, finding the right fit is critical for DevOps teams to hit their goals.
DevOps pros know a seamless toolchain can increase efficiency for a quick and reliable product development cycle, but figuring out how to put one together isn't always easy. How can DevOps teams know which tools will play nicely together and deliver tangible results?
To find out, Built In Chicago talked with DevOps team members about the technology in their toolchains, how they made product decisions and what benefits they have seen since implementation.
Discover is a direct banking and payment services company.
Give us a brief glimpse into your DevOps toolchain. What are a few of your favorite tools the team is currently using?
In recent years, our DevOps toolchain has increased security tooling availability. We leverage security tooling such as static application security testing, dynamic application security testing, interactive application security testing, credential scanning capabilities and automated open-source policy enforcement. These tools help to shift security left at very little cost to the application development teams. We’re currently looking to bring similar capabilities to regulatory compliance and company policy. We’ve also seen big improvements in automated test tooling, bringing capabilities that simplify parallel test execution and impact-based testing. Ultimately, all of these capabilities help to reduce unplanned work and lead to reduced cycle time.
What were some of the key considerations when evaluating and choosing these tools?
First and foremost we look at user experience. The aim of DevOps is to shorten delivery cycle time while maintaining quality. A big part of maintaining that balance is the ability to provide timely feedback to the development teams. During automated testing, if we discover defects and realize code needs to be reworked, the development teams need to know immediately. We want to initiate changes while the features they developed are still fresh in their minds. Otherwise, developers have to switch projects and need time to reacquaint themselves with the problem. Tools that deliver rapid feedback with a seamless user experience really benefit our team’s efficiency.
We think of it like a factory — the tools in the DevOps chain are the robots, machines and conveyor belts necessary to produce the end product. Reliability is key to keep the production line running smoothly.”
What has been the key benefit your team has seen from building out and maintaining a healthy toolchain?
Our aim has always been to provide standardized DevOps tooling and usage patterns so our software development teams don’t need to solve those problems themselves. Because our payments systems software is what differentiates Discover in the marketplace, we need our development teams to dedicate their energy to creating strategic payment systems software. They’re constantly under pressure to deliver business value through new software capabilities, so solving DevOps delivery challenges in addition to innovating on our current platform dilutes their focus. That’s why it’s vital that the DevOps toolchain can reliably support them in the background. We think of it like a factory — the tools in the DevOps chain are the robots, machines and conveyor belts necessary to produce the end product. Reliability is key to keep the production line running smoothly.
Buildout is commercial real estate broker software.
Give us a brief glimpse into your team’s current DevOps toolchain. What are a few of your favorites?
Our favorite tools in our current and prospective toolchains are Concourse CI, Argo CD and Traefik hosted by Kubernetes.
What were some of the key considerations when evaluating and choosing the tools your team would use?
It was really important that we move to a reliable deployment cycle. The inherited toolchain was slow, cumbersome and prone to high failure rates with a lot of server underutilization. We chose Concourse CI to build and test against immutable containers for reliable consistency. Argo CD is great for launching, managing and synchronizing built containers against various environment spaces. We made the decision to migrate to Kubernetes to leave behind an outdated static server design with hardly any ability to right-size during fluctuating traffic needs.
It was really important that we move to a reliable deployment cycle. The inherited toolchain was slow, cumbersome and prone to high failure rates with a lot of server underutilization.”
How has your DevOps toolchain evolved over time, and why?
My first ever full toolchain utilized static servers and Capistrano-like deployments. It used Jenkins to push code from Git with environments that were easily desynced. Local dev configuration kits set up with Puppet and VirtualBox came with the cost of long spin-up times. I still sparsely utilize Jenkins for one-off tasks that don't belong anywhere else. Most of the kit is now completely replaced by containerized environments and a more stateless approach to hosting.
