[ibimage==50314==Original==none==self==ibimage_align-center]
By Ralph Bodenner
Every engineering team has processes they follow to get things done in the way that works best for them. Whether they’re in someone’s head or formally documented, they have to be communicated and shared because they are an implementation of the team’s culture at that point in time. But for a company and its engineering team to sustain its culture as it grows, even more important than what those processes are or how they’re recorded is how they undergo change.
At New Relic, we prize our culture of openness and inclusivity, so we found a way to leverage the GitHub pull-request process we use to collaborate when building software as a way to also manage how our engineering teams work. That includes debating and changing everything from the titles for various engineering roles, to technical architecture guidelines, tips for successful conference attendance, and practices for hiring the right people for our teams. The idea is to treat “culture as code” and try to “open source our culture” so that we use the same workflow for the team as we do for code.
At the Velocity conference in Santa Clara, Calif., a few months ago, I gave a talk on this concept called “Changing the Laws of Engineering with Pull Requests.” This blog post is adapted from that presentation.
Learning from, and leaving, the wiki
Like many other organizations, New Relic had relied on a number of different tools to manage change and best practices in our engineering team, including meetings, email, and the almighty wiki. We realized when we were a much smaller company that we needed to write down what processes we used, but we also realized that we needed a way to get rid of our previous approaches and introduce new ones. Processes can rapidly lose the connection to their original purposes, and sometimes end up just wasting time and creating overhead instead of adding value. Worse, having documents (the wiki) describing those now-outdated processes eroded trust in the system and dissuaded people from investing effort in the process.
Our wiki was treated like an official “manager’s tool,” in large part because changes to it took place immediately—anyone editing it was speaking for the whole team. There was no review or commenting unless you did that in meetings or over email, not conducive to engineers successfully participating.
Despite many attempts over the years, we were never quite able to meet our aspirations for defining and capturing the processes and policies of our engineering organization. Perhaps that’s because, for us, just as important as documenting how we deal with such things as site outages, software development processes, security, travel, training, and personnel policies and decisions, was the process of how we created those things.
To try and solve that, we came up with a short list of our broad goals:
- Processes embody culture
- Processes are repeatable, efficient, and fair
- Transparency
- Inclusivity
- Discoverability
- A single source of truth, written down
- Update/delete easily
- Bias for action
- Autonomy
- Culture > Process
And some specific requirements:
- A workflow that is a natural fit for engineers
- Less discussion happening in exclusive settings like meetings
- The “why” for every change recorded in one place
Going all in on GitHub
So one year ago in July of 2014, we ditched the wiki and began using GitHub pull requests to manage change in our engineering team. We store our cultural documentation in a git repo as Markdown and HTML, and use Jekyll to render it into a simple website. Since then, our entire engineering team, as well as others in the company with whom we collaborate, have become contributors to shaping our process in real time. We are closing the gap between our culture and our processes as engineers and managers alike propose and debate changes.
Of course, our business is not a democracy, and we can’t have just anyone changing processes, especially those that affect money or private information, like how salaries are determined, for instance. But it’s important to understand that open source isn’t a democracy either. While everyone is a contributor, we have only a few committers, who are managers. Everyone has a voice but not necessarily a vote.
The results
By the time of Velocity, we had 1,054 comments from 64 commenters—about a third of our Engineering staff. We merged 187 pull requests (and closed 30 more) with more than 25 engineers having changes merged. This is a much more inclusive method of collaboration, where before only the managers were directly participating.
But perhaps the biggest benefit is that we now capture the processes involved in our decision making. Why did we think doing something was a good idea? What did we know at the time? What was important? What assumptions went unchallenged?
It used to be that when we hired a new manager, we would manually dig up much of that “why” from old emails and discuss it with the new hire during onboarding. Now, the pull request knows why (not to mention who and when), and everyone in engineering can read it anytime.
What we’ve learned
Though we’re meeting our original goals with our new change process, there are still issues, of course. It’s not working perfectly for nontechnical people who might be less familiar with GitHub, and the UI for making changes to our documents could be easier as well.
We’re working to improve the inclusivity of our process with training, much like Aaron Suggs described in his Velocity talk on “GitHub for Poets” at Kickstarter. And we’re using pair programming techniques for process documents. To give early ideas a safe place to form, we’re turning focused conversations of a few people, in forks of the main culture repo, into proposals worth sharing and turning into pull requests.
We’ve also learned to actively discourage email discussions, closing threads with BCCs and pasting them into pull requests. We try to structure our repos and orgs so people can use GitHub notifications to stay informed. It’s also helpful to keep comments open after the merge, as we’ve seen cases where half of the discussion happens after that point. Finally, using GitHub’s quick pull requests really helps to lower the contribution barrier.
Hopefully, this post has inspired you to think about your own organization’s engineering culture and process, how to keep them in step, and, most of all, why it’s important to do so.
See the slides from my original presentation here.
About the Author
For more than 15 years, Ralph Bodenner has shipped customer-focused software, starting at Amazon.com, and then building developer tools at two reconfigurable hardware startups. Since his arrival at New Relic in 2009, he’s been dedicated to building effective software teams. As director of engineering, his experience crosses all aspects of software development, enabling his team to provide continuous delivery in a complex, software-as-a-service environment for a customer base that has grown to 90,000 in six years. Ralph holds an AB in Computer Science from Duke University. View posts by Ralph Bodenner.
