Before DevOps, there were developers and IT personnel — a.k.a. operations people. Those two groups worked in separate departments, a setup that lent itself to misunderstandings and rollout problems like rampant error messages, confused tech support and the burning house in this dev-gone-wrong meme.
Starting in the late aughts, companies began preventing glitches by blending developers into operations teams. The resulting agile teams — “DevOps” — helped resolve glitches, patch vulnerabilities and roll out new features. Quickly. And products grew increasingly responsive to user feedback. This was the future.
At least in theory.
High-functioning DevOps often requires a culture shift as well as reorganization. In old-school tech cultures, developers and operations personnel had conflicting perspectives on change. The job of developers was to instigate change with new software, whereas those on the operations side strove to create stable, functional environments — and were therefore prone to view change as a roadblock.
In a DevOps culture, though, change is constant. Small software tweaks roll out steadily within a “continuous deployment” workflow. At Microsoft’s search engine Bing, for instance, engineers deploy new code at least 10 times a week. Because divisional siloing is a thing of the past, that ceaseless change is no longer one department’s job and another’s problem. Instead, the guiding ethos is, “If you build it, you run it.”
That’s certainly true at Chicago’s Jellyvision, where, according to vice president of engineering Badri Rajagopalan, “Anybody and everybody in the engineering team is able to make changes and take code to production in a safe way. “You just want to get it done.”
In the future, Rajagopalan says, changes will get implemented even faster than they do now. “I think we’re going to continue to be able to reduce the time to market until it’s in the minutes, and we’re going to do thousands of deploys at any given point,” he says. “This will allow businesses to target features to individuals.”
Rajagopalan was one of three Chicago-based tech professionals who spoke with Built In about their experience as DevOps engineers, and offered tips for getting into the field.
What are the core responsibilities of a DevOps engineer?
The engineering team typically automates testing. That means creating ways to test new code and ensure that the quality is high. And then every time a developer makes a change, we want to keep merging those changes into the master branch. Often, we have 40 developers working on the same project. So you’re trying to integrate their changes simultaneously in a synchronous way. We also want to take that master branch through the QA environment, the staging environment, and if all that works, we want to move the change to production. That’s the pipeline that DevOps engineering teams are focused on automating. It’s testing and validating the code, taking it all the way to production and making sure that time is really fast.
We also try to increase the frequency of changes. The more changes we make, the lower the risk of a particular change causing an incident. Think about it from the perspective of editing: when you only give somebody two minutes to edit, the probability that they make the article really bad is very low.
A single team of five to seven DevOps engineers takes ownership of a single component or application as much as possible. The more capabilities you give them, the less likely they’ll need to wait on some other team for two weeks.
When application developers want to spin up a new application, we want that to be as easy as possible for them without perhaps compromising the security or reliability of production. So there’s always the challenge of giving developers flexibility while minimizing the risk of an accidental production outage or production impairment. For existing applications, we want to make deployments as easy as possible. So we want to minimize the timeframe from inception of an idea to the releasing of it to production, so it’s available for general public. The better we work on our infrastructure and our deployment pipeline, the quicker that happens.
My job is basically to let developers be developers, making sure that they’re not blocked by anything and removing as much noise as possible from the operational side.
When I was a DevOps engineer, a lot of what I focused on was observability, so building out a stable and scalable platform for the company, where they could effectively monitor applications. I also partnered with the development teams to help them build effective dashboards and alertings, so that they could support their applications in a seamless fashion. Before Signal, I worked for a company that got acquired by Expedia, and there was a change of platform there. In that role, a big part of my job was helping teams migrate to the new Expedia platform.
How did you get your first DevOps engineering role?
Rajagopalan: Back in the late 2000s, I was at Illinois State University and we set up the automation team. I was an assistant director in IT at the time, and that was part of my role day-to-day. I was managing that team, figuring out how to get the most code out the door. That wasn’t a formal DevOps role, but that’s where I started.
Four or five years ago, when I started working with the Cloud, that’s when I really started doing modern DevOps. The Cloud radically accelerated everything we do. In late 2010, if somebody came in and said, “Hey, I want to apply a new application in your data center,” we would go order a server, and Dell or HP would ship it over. It would take a few months. The Cloud really made this a lot simpler. You don’t have to worry about any of the hardware stuff.
Pasciak: I started out as an application developer. I spent 13 years in application development, front end, back end, database, optimization, things like that. And naturally, I got my feet wet with operations. You can’t just close your eyes and pretend it doesn’t exist. Projects had gaps that needed to be filled. The one I worked on right before I left for Vivid Seats had a gap in operations. I filled that gap, I was very interested in it. It was a new place for me. Then Vivid hired me for a DevOps role.
Richard: I got my first official DevOps title in 2016, at Expedia. Expedia had acquired the company I was working for, and at first, I worked on Expedia’s applications engineering team. It was an interesting mixed bag of stuff. We supported a lot of third-party products — so, we would license stuff and maintain relationships with those vendors. We also supported and developed most of the internal tools that the company used at the time. There was a codebase of some size where we would do development work and deploy things that the operations people and developers used in their daily lives. I was interacting with developers and operations as a developer, and then I was interacting with third-party vendors and some of our other teams from an operational standpoint. Learning to wear both of those hats helped me transfer into DevOps engineering pretty easily.
For someone who wants to get a DevOps engineering role, what are key first steps?
Rajagopalan: If you’re starting from scratch, start with a bootcamp. But the second part of DevOps, that they don’t teach you at developer bootcamp, is how to take the code out the door as quickly as possible. So, how to leverage everything from Jenkins to Amazon Web Services (AWS) or Azure or Google Cloud.
Imagine you’re a business and taking this code to production. If that is an exercise, we’ll eventually come to the conclusion that you have to have the right motivation script and use Jenkins to check your code and push it out the door to AWS. Then you learn the AWS support to it — how to make it run as quickly as possible there. So, start conquering through those types of challenges on the tech side. Jenkins and Cloud deployment are probably the places I would start.
Pasciak: I would say just go out and do it. If you’re looking to get into DevOps, you probably already have an idea of what it is. So find one thing that you don’t necessarily understand through to the end, and understand it. There’s a plethora of information out there today on the internet, from free online courses to YouTube channels to blogs. There is no shortage of information. Find something, understand it, and ultimately do it. Create a free Cloud account, and find a way to deploy an application. Get comfortable with automation. Automation and DevOps go hand in hand today.
Are graduate degrees important?
Pasciak: I have a masters in software engineering from DePaul, but I worked with many, many engineers, in both application and DevOps, that were just terrific and hadn’t gone to grad school. As a matter of fact, they hadn’t even done engineering-related courses or computer engineering courses. What ultimately makes or breaks a developer is their just genuine interest in the topic. If you’re genuinely interested, you’ll find the resources.
Richard: Often, DevOps engineers have BAs in computer science, computer engineering, security engineering kind of stuff, but it’s all over the place. I think one of the guys on my team right now has a political science degree. Grad school will get you more technical skill, though, and in a pool of DevOps candidates, the ones who have had grad school stand out.
But throughout my career, it’s become pretty apparent that there’s no replacement for hands-on experience. A majority of DevOps and operations engineers that I’ve worked with came out of internships. It’s essential to get your hands on problems in the real world and learn that way.
What resources do you recommend for learning more about DevOps?
Rajagopalan: There is a very good DevOps podcast out there, Arrested DevOps. I also recommend Thoughtworks’ Radar and podcast. The Google book on site reliability engineering is a good primer before DevOps. Microsoft has a whole DevOps publication that they also have on their website that is also very good. Reading up on any modern continuous integration and continuous deployment, or CICB, practices — all of those things sort of tie in with each other.
Pasciak: I would say Amazon Web Services, Google, a bunch of Cloud providers, they’ll give you resources on stuff related to DevOps. Google provides tutorials on how to use their infrastructure, how to use all the tools that they give you. Pretty much everybody that operates the Cloud, you’ll get some resources from them in their free membership tier.
Richard: I would have probably listed books years ago or a few years back, but these days the better ways to stay abreast of trends in DevOps and engineering is social media. The people who might speak at a conference also have Twitter accounts and some great discussion comes out of it. I especially like following John Arundel.
What’s the most difficult part of the job?
Pasciak: I would say prioritizing the things that need to be done. There’s certainly no shortage of work. The biggest challenge is deciding, from that giant bucket of things that we could do, which things are the lowest effort, highest impact. “This’ll be nice to have, but really doesn’t give us all that much,” versus, “This is very nice to have and it will increase productivity by so-and-so” — finding things in the second category.
Richard: When I started in DevOps, my interaction with the people that were marketing our products and interacting with our customers — business people — went up dramatically. I’d never really had to have a lot of engagement with people in those positions. As an engineer, trying to bridge all these gaps, to glue all this stuff together, it was difficult, but it was also a really great learning experience.
What’s the most rewarding part of the job?
Rajagopalan: At Jellyvision, when I started, it normally took us four days of like coordinating the deploy and trying to get the latest and greatest version of Alex out the door. Over seven or eight months, we got the deploy time way down. It now takes about 40 minutes to deploy finished code. It is a marked difference.
Pasciak: For me, the most rewarding part is when I see that developers are able to do the things they need to do without having to come to us. When somebody says, "Hey, we just sped up the deployment pipeline by 50%," that’s rewarding. When I don’t see any notifications on my Slack, that means things are humming along. That’s what we want.
Richard: As a DevOps engineer, as opposed to a traditional operations guy, I had more interaction with the development teams. It was super rewarding to be able to interact with the people who had been building our applications and platforms. I was able to just level myself up as an engineer and gain so much knowledge about how things are built in the real world.