How to Build a DevOps Culture and Methodology That Works

May 26, 2020

When Andrey Zusko, a software engineering manager at Caterpillar’s digital branch Cat Digital, was first introduced to DevOps, he was immediately hooked on the philosophy, methodology and mindset. Like many engineers, he loved being empowered with full control over the complete software development lifecycle. Five years later, he only hires developers with DevOps experience or those who have a willingness to learn.  

But engineering leaders across Chicago will be the first to tell you there’s no one-size-fits-all way of incorporating DevOps into an organization. So how do team leaders build a culture that evokes an enthusiasm like Zusko’s?

The engineers we spoke with said DevOps is constantly evolving. To seam together development and operations teams requires communication, a shared vision and the right tools. From there, it’s important to step back and let the DevOps team research and ideate what to do next. Finding new ways to improve the process is what makes engineers excited.

 

John Unger
Team Lead – Systems Engineering

Ownership and common goals are essential for a strong DevOps culture, Unger said. Different teams shouldn’t have competing priorities. Instead, they need to work toward a shared vision. To help foster that mission, IMC Trading created cross-functional teams.

 

First off, how did you go about implementing a DevOps methodology within your tech organization?

DevOps was not implemented overnight. It is a methodology and mentality that was adopted over time and continues to evolve today. At first, we tried eliminating our operations team altogether. However, we realized that there is value in having a team specializing in operations. To accomplish our goals, our operations team started owning a few development tasks, and our developers took on some operational responsibilities. Now, our technology organization is less siloed and flexible enough to meet the demands of high-frequency trading. We also had to partner with end users to create workflows that facilitated DevOps. 

 For example, we use a combination of email, chat and in-person meetings to make sure issues reach their target audience as quickly as possible. Tools such as TeamCity, GoCD and GitLab helped streamline our release process and gave more control to developers. Our operations teams are empowered to make code changes themselves and work directly with developers to implement long-term improvements. With a shared sense of ownership, the right processes and helpful tools, DevOps at IMC is a success.

Shared ownership and common goals are key to a strong DevOps culture.

 

From your experiences, what are the key characteristics of a strong DevOps culture? 

Shared ownership and common goals are key to a strong DevOps culture. We need all technology teams at our organization to work together to accomplish DevOps. I’ve seen DevOps fail when different groups have competing priorities. For example, if the operations team is only responsible for uptime, they may push back on a quick-release pipeline. If developers only work on new features, bugs or stability issues can creep into the system.  

When the entire technology organization is unified and working toward a shared vision, it gives purpose and direction for DevOps. To help foster these characteristics, we frequently create cross-functional teams to work on projects and accomplish our goals. This includes members from different parts of the technology organization such as development and operations, as well as business stakeholders. 

An example is when IMC started trading on a new stock exchange. We created a cross-functional team consisting of developers, network engineers and traders. The team had a shared goal: trade on the new exchange the day it opens, as safely as possible. With a unified goal, we were able to achieve success.

 

What advice do you have for other leaders who want to adopt a DevOps model in their organization?

First, understand the problems you are trying to solve with DevOps. Are your releases slow? Are there too many bugs in production? Do your operations teams spend all day firefighting? Once you know the root issues, speak with the teams that are impacted and would see the biggest benefit from DevOps. Collaboration is key to the adoption of DevOps and you need everyone in the technology organization on board. Developers may be hesitant to take part in operations and ops teams may feel uncomfortable changing code. Listen to concerns, and build a DevOps model that works for your organization. Accept that there will be some initial setbacks and make adjustments as you learn. After all, DevOps is about being agile.

Unify your technology groups with common goals and create KPIs that can be accomplished by practicing DevOps. Form cross-functional teams to work on these projects, and share both successes and failures equally amongst development and operations to ensure the groups are truly aligned. Every organization is different.

 

Andrey Zusko
Software Engineering Manager

Zusko said he likes DevOps because engineers are empowered with full control over software development lifecycles. In order to implement DevOps successfully at Cat Digital, engineers are required to wear many hats. Not only should they be thinking about the task at hand, but also questioning how that task will be tested or deployed later in the development cycle. 

 

How did you go about implementing a DevOps methodology within your tech organization?

My first experience with a DevOps-enabled engineering team was about five years ago when I joined a new team. I was immediately hooked. Like many engineers, I love being empowered to have full control over the complete software development lifecycle. I think the methodology is easy to adopt because the pillars of DevOps — such as a deep understanding of CI/CD processes, automated testing and infrastructure deployment — all help engineers build better software.

Now, when I grow my developer teams, I always try to hire people with relevant DevOps experience or those who are curious about learning and adopting the mindset. 

 

What are the key characteristics of a strong DevOps culture? 

DevOps has to exist in a blended culture where engineers rotate and wear different hats. In my team, there is no clear distinction between developers, quality assurance, DevOps and technical support personnel. Each set of experiences contributes to a better overall outcome. 

 For example, when you have experience writing automated tests, you will naturally think about how to structure your code to make it simpler and more straightforward to test. When you participate in infrastructure design and configuration, you will gain the experience to address production issues more effectively.

Second, I believe every project should be planned and developed with DevOps in mind. On my team, we think about support and maintenance before we even start writing code. We think about expected load when we plan our infrastructure. If there’s a production issue in the middle of the night, we are the ones who get the call, so we are motivated to pay extra attention to the amount and quality of our tests. 

Develop your own performance test suite, and you’ll see fast results. 

 

What advice do you have for other leaders who want to adopt a DevOps model in their organization?

Start gradually. If your team still relies on QA, start building your own automated test suite. Build your CI/CD pipeline to run the tests. Automate your own deployment in lower environments. Design and build your own preventive monitoring and alerting. Develop your own performance test suite, and you’ll see fast results. Your teams will get excited, too. I’ve yet to see an engineer who didn’t like learning new technologies.

 

Alex Segre
DevOps Engineer

Optiver operates as a global electronic market maker. Optiver’s traders and engineers work together to design proprietary systems and execute strategies that provide liquidity and increase market efficiencies. Segre, who sits on the company’s Chicago-based team, said development teams must understand the full lifecycles of projects in order to build a successful DevOps culture.

 

How did you go about implementing a DevOps methodology within your tech organization?

Optiver U.S. has long had a strong culture of DevOps thinking throughout the organization. Our development teams are directly responsible for the full lifecycle of their projects, from design to deployment and troubleshooting, all the way to decommissioning. Our operations teams ensure consistency and stability throughout the production environment. They understand the big picture and work closely with developers throughout all stages of projects. 

This approach lets us iterate quickly without sacrificing determinism in production. Our DevOps tooling helps us scale this existing methodology. We mechanize processes in order to give our teams the leverage to do more; however, a human is always in charge. This key distinction between mechanization and automation is crucial in maintaining control and stability with so much on the line.

 

What are the key characteristics of a strong DevOps culture? 

Every problem area needs a single owner, but people must not be siloed into their roles. Every team with a stake needs to be involved in projects from the beginning so that operational concerns are a top-level part of the design. We tackle DevOps tooling challenges the same way we approach any other engineering challenge — with principles such as simple solutions and iterative design. Our culture encourages experiments in all areas, including both our technology products as well as our DevOps tooling, as long as it's done safely. We often willingly take on temporary technical debt for these experiments, starting small and iterating as they prove worthwhile. Some fail and are cleaned up, but others succeed and become permanent parts of our infrastructure.

Every problem area needs a single owner, but people must not be siloed into their roles. 

 

What advice do you have for other leaders who want to adopt a DevOps model in their organization?

Engineer the full lifecycle of every project right from the start. The development teams must understand the full depth of their projects, from implementation through operations. The operations teams must have a complete breadth-view of production and must additionally act as a control on the development teams to ensure consistency and stability of the whole environment. Both these teams need direct input at the design phase of every project, as they both contribute different points of view.

Be cautious of new technology, both in your product and in your DevOps tooling. Although new libraries and frameworks can ease the implementation part of a project, dealing with the operational issues they incur is often an afterthought. When reliability, scalability and maintainability are part of the initial design process, many bleeding-edge stacks just don’t meet the bar. DevOps tooling in particular can impact the stability of all your products, so simplicity and determinism are key.

 

Chris Jones
Senior Manager of DevOps

Jones said collaboration and enablement are the two principles needed to build a successful DevOps culture. From there, a “continuous improvement” mindset is necessary in order to implement the best methodology for your team and project. Don’t rely on automation tooling to lead the DevOps team; Jones said it’s humans that make the culture a success at Outcome Health

 

How did you go about implementing a DevOps methodology within your tech organization?

There is no one-size-fits-all methodology for implementing DevOps, but there is certainly a DevOps mindset. That mindset can be summarized as “continuous improvement.” As I work toward that mindset, I’ll identify sources of waste and try to reduce them. Common examples include unnecessary hand-offs of work, a lack of adequate planning and lots of invisible work. In our organization, we’ve been gradually finding ways to eliminate hand-offs, or at least to reduce their frequency. 

For example, we’ve implemented a lightweight change management process that enables our engineering teams to do their own deployments if they don’t require any kind of manual or non-standard work from the DevOps team. We’ve instituted better planning processes designed to make cross-team dependencies visible so that we reduce surprises and churn.  

Finally, we’ve started allocating general buckets of “operational” work each sprint to capture the fact that many of our teams have day-to-day responsibilities that don’t rise to the level of stories but which take time away from other planned work and cannot be ignored. This keeps from unknowingly overcommitting our time and reduces context switching between tasks. Above all, we keep metrics on the predictability of our work and the things that have the greatest impact on that predictability. Regular retrospectives are one of our greatest assets. 

 

What are the key characteristics of a strong DevOps culture?

We are early in our DevOps transformation, but I think that two philosophies characterize any successful DevOps culture: collaboration and enablement. Our DevOps team was originally acting as a pure operations team, but over time we’ve worked with our engineers to change how we deliver software. 

These changes weren’t earth-shaking. For example, one change we made was to enable our development teams to request their own infrastructure changes. Such changes had previously been reserved to the DevOps team. Now, any team can request an infrastructure change just by creating a pull request. DevOps is still involved and offers guidance, but it’s a much more collaborative model rather than a transactional one. 

These changes are reflected by the development teams as well. We have ongoing discussions about what kinds of changes will enable our engineers to be more productive. For example, QA can work with our Android development team to make an application more testable or DevOps can work with our back-end developers to automate manual processes. 

Craft your own goals for your DevOps transformation. 

 

What advice do you have for other leaders who want to adopt a DevOps model in their organization? 

Craft your own goals for your DevOps transformation based on your organization’s particular business needs. Not every organization needs to adopt continuous deployment as a matter of practice, but the changes that are introduced to enable it will have a profound influence on your approach to software engineering. While DevOps often has an engineering focus, it needs to be advised by the needs and goals of the business for it to be truly effective. For example, you will require time to address technical debt and to automate manual processes such as builds and testing. There is an opportunity cost to that time, so be prepared to justify your DevOps transformation in business terms. You should also devise and keep good metrics so that you know when you have reached your goals. Finally, avoid focusing solely on tooling and technologies. While good tools are useful, it is sound practices and an emphasis on the human aspects of engineering that play the greatest role in the success or failure of DevOps. 

 

Dan Lenar
Technical Team Lead

Lenar said he encourages his team to stay current on DevOps trends by participating in book clubs, attending DevOps meetups and participating in conferences. Engineers can take what they’ve learned and apply it to the DevOps culture at Vail Systems. There are no set-in-stone rules when it comes to DevOps; Lenar said it’s a philosophy and thus is open to evolving. 

 

How did you go about implementing a DevOps methodology within your tech organization? 

The evolution of DevOps at Vail started with containerization. With the power of containers, a new developer could join a team and have an entire stack running in a matter of minutes instead of spending an entire day following a lengthy README to install all the external dependencies needed to get an application running on their laptop. 

First, teams were trained to build and run Docker images. Once teams were comfortable using Docker, we slowly introduced Kubernetes as our container orchestration system, which made deployments take a matter of seconds compared to our traditional methods with configuration management systems.

After implementing a DevOps methodology, we have seen more collaboration between development and operations teams on things like infrastructure code, and many silos that had formed over the year began to break down. With Kubernetes, developers were given the freedom to deploy their applications in development environments without help from operations staff.

Our ultimate goal is to build a GitOps methodology that will allow us to declaratively describe every part of pipeline as code; we can commit code to Git, which would trigger a CI/CD pipeline that would build a Docker image, run integration tests and deploy an image to a Kubernetes cluster.

DevOps is a philosophical mindset used to build modern applications.

 

What are the key characteristics of a strong DevOps culture? 

A strong DevOps culture should encourage team experimentation and not shy away from failure. In order to encourage more experimentation, we try to have less than 50 percent of all our work tied to toil. Toil can be best described as work tied to running a production service that tends to be manual, repetitive, automatable, tactical, devoid of enduring value and that scales linearly as a service grows. 

With this mindset, we have more time to think about how we can improve our processes and find the rights tools for the job. We encouraged our teams to become part of the Cloud Native Foundation community, attend and host various local DevOps meetups, participate in book clubs and attend KubeCon and DockerCon every year to keep up with current trends.

 

What advice do you have for other leaders who want to adopt a DevOps model in their organization?

One of the first things to understand about DevOps is that it is more than just technology or a toolset; it is a philosophical mindset used to build modern applications that take full advantage of cloud computing.

To effectively implement DevOps within an organization, buy-in from the top is essential. Once you have buy-in, it is best to start with small incremental changes while continuously soliciting feedback from development and operations teams on what’s working and what’s not. Communication between cross-functional teams and knocking down silos to empower every team member to make fast and reliable deployments is also essential for success.

 

Jobs from companies in this blog57 open jobs
All Jobs
Finance
Data + Analytics
Dev + Engineer
HR
Internships
Legal
Operations
Product
Project Mgmt
Developer
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Data + Analytics
new
IMC Trading
Chicago
Data + Analytics
new
IMC Trading
Chicago
Finance
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Finance
new
IMC Trading
Chicago
Data + Analytics
new
IMC Trading
Chicago
Finance
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Operations
new
IMC Trading
Chicago
Developer
new
IMC Trading
Chicago
Data + Analytics
new
IMC Trading
Chicago
Developer
new
Cat Digital
Chicago
Data + Analytics
new
Cat Digital
Chicago
Developer
new
Cat Digital
Chicago
Developer
new
Cat Digital
Chicago
Developer
new
Cat Digital
Chicago
Project Mgmt
new
Cat Digital
Chicago
Data + Analytics
new
Cat Digital
Chicago
Developer
new
Cat Digital
Chicago
Data + Analytics
new
Cat Digital
Chicago
Developer
new
Outcome Health
Chicago
Developer
new
Outcome Health
Chicago
Data + Analytics
new
IMC Trading
Chicago
Internships
new
IMC Trading
Chicago
Internships
new
IMC Trading
Chicago
Finance
new
IMC Trading
Chicago
Internships
new
IMC Trading
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Developer
new
Cat Digital
Chicago
Developer
new
Optiver
Chicago
Developer
new
Optiver
Chicago
Developer
new
Optiver
Chicago
Developer
new
Optiver
Chicago
Internships
new
Optiver
Chicago
Developer
new
Optiver
Chicago
Data + Analytics
new
Optiver
Chicago
Developer
new
Optiver
Chicago
Finance
new
Optiver
Chicago
HR
new
Optiver
Chicago
Operations
new
Outcome Health
Chicago
Product
new
Outcome Health
Chicago
Operations
new
Outcome Health
Chicago
Developer
new
Outcome Health
Chicago
Product
new
Outcome Health
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Developer
new
Vail Systems, Inc.
Chicago
Finance
new
Cat Digital
Chicago

Chicago startup guides

LOCAL GUIDE
Best Companies to Work for in Chicago
LOCAL GUIDE
Best Software Engineer Jobs in Chicago
LOCAL GUIDE
Coolest Offices in Chicago Tech
LOCAL GUIDE
Best Sales Jobs in Chicago