8 Things IT Ops Hates—And What Devs Can Do to Help [Part 1]

[ibimage==48370==Original==none==self==ibimage_align-center]

By Al Sargent

This is part 1 of a 2-part series on the things that make your IT ops team members cry into their pillows at night—and how we can all help relieve their pain.

The concept—and even the very name—DevOps clearly suggests that dev and ops teams need to work together. But in the real world, the two sides often have very different considerations and perspectives. And all too often, the ops side gets left holding the short end of the stick. So in the spirit of helping devs better understand ops, I present eight things that can drive even the sunniest IT ops folks to question their career choices—and offer suggestions on how devs can help relieve some of the pressure. The topics were gleaned from our own experience as well as online resources like DevOps Reactions and Twitter accounts such as Honest Status Page. Both are worth reading since there’s a level of truth to what they humorously present.

1. Too many new developer technologies

[ibimage==48371==Original==none==self==ibimage_align-left]

Developers are measured on productivity, while sysadmins and other ops team members are measured on stability. Often these two goals are in opposition around new technologies. Frequently, developers will discover a new technology that lets them build new functionality more quickly.

Perhaps it’s Scala, which is surprisingly compact and expressive. Or MongoDB, which is schemaless and provides developers with high levels of flexibility in terms of updating their code.

But whatever the benefits might be for dev, these new technologies can cause unintended hardship for the ops team. One reason is that a typical company has dozens (or even hundreds or thousands) of apps that could be built on a wide variety of different technology stacks, each with their own set of dependencies that need to be acquired, patched, configured, tested, and security-hardened whenever ANY piece of that stack is upgraded.

Developers typically specialize in one app—or a handful of apps—making it possible to devote the time to becoming productive with a key new tool. But while there might be multiple developers working on a single app, each ops person is likely supporting many apps at the same time—in some cases, dozens or even hundreds of apps.

Tools that work great in a single use case might not be scalable or secure across the entire enterprise. Ops needs standardization to avoid being stuck with a long tail of stuff they need to learn and maintain, especially if the original developer is long gone and not available for consultation. It gets worse with open source solutions that might not always be vendor supported.

What devs can do to help:

Devs should ask ops what stacks are most common and well understood in their production environments—both today and roughly a year from now. (And ops teams: you need to be ready to answer this question.) Then do your best to stick to standardized components as much as possible. Sure, developers’ opinions differ as to whether it’s better to use MySQL or Postgres for their new app, but one factor to consider is that if a hundred other apps in your organization use a particular database, any incremental developer benefit for switching might not be worth the added pain to ops.

2. Having to deal with legacy code ... and lots of it

Developers are lucky. Because they’re always moving on to the next new thing, devs can often avoid dealing with legacy apps built and deployed by teams now long gone, and who likely left no documentation behind.

Ops, not so much. Many corporate data centers resemble a tech museum, made up of technologies that span decades. A lot of companies still rely on some funky app that looks like it was built in 2002 and that no one has a clue about. This creaky app does the job and upgrading it isn’t a priority, but the ops people still have to keep it running. Notable problems can crop up even when the code hasn’t been touched in years. This is especially problematic around security issues: no ops team wants to answer to the CEO why, as The Guardian noted, customer data was stolen due to software that wasn’t upgraded and had known security holes.

What devs can do to help:

This problem is essentially problem #1, but metastasized over time as the organization lost the institutional knowledge needed to maintain a particular stack. The solution is the same: keep stacks simple, and think of future team members who’ll need to maintain them. Given how certain technologies such as Cobol and mainframes just won’t go away, perhaps devs should think like the Iroquois and consider the impact of their decisions over the next seven generations.

3. Developers who sling over apps that don't work in production

Devs who say things like “It worked fine in dev” and then consider it ops’ problem understandably drive ops crazy. Too often, devs don’t test as much as they should before sending apps to production. Sometimes it seems that devs live by their own version of a certain beer ad credo: “I don’t always test my code, but when I do, I do it in production!” That attitude dumps all the work on the ops team, and the move toward more frequent deployment just makes things even more stressful.

What devs can do to help:

This is perhaps obvious, but it bears saying: devs need to remember that just because it works on their laptop doesn’t necessarily mean it works in production. To address this, Dockerize your apps in all your environments—dev, test, staging, production, and so on. (If you’re looking for a Docker book, here’s a free download of the first three chapters of Docker Up & Running, which will be available next month.)

Also, invite ops into your planning sessions and standups to ensure that they have input into test plans. Ensure that things like logging and monitoring are properly addressed, and ideally implemented in dev and test environments to maximize the similarity between those environments and production.

4. Taking the heat for other people's problems

There are many ways that things can break unexpectedly and it’s usually ops’ problem to make sure it’s fixed no matter whose fault it actually was. Given the responsibility to keep things running without the authority to fix the architectural issues that may underlie a systemic problem, it’s no wonder ops teams can get cranky. Inheriting other people’s problems and taking the heat for decisions you didn’t make is a clear recipe for unhappiness—especially when we’re talking about increasingly mission-critical apps where any failure is highly visible.

What devs can do to help:

Work with management to institute a “pause” mode, whereby dev can get a reprieve from code delivery schedules so they can lend a hand to ops when things go wrong. Often developers will have enough specialized experience with a technology to fix something, which a more generalized ops team member won’t have. At the end of the day, if a problem doesn’t get fixed fast, customers won’t care if the problem lay with dev or ops—they’ll just stop using your product.

So far I’ve covered the hardships of new technologies, old legacy code, untested apps, and other people’s problem. In part 2 I’ll go into the other four things that gives your IT ops team a major headache.

About the Author

Al Sargent is senior director of product marketing at New Relic. He's proud to lead an outstanding team of product marketing professionals who are responsible for go-to-market activities for New Relic's software analytics suite. View posts by Al Sargent.