Groupon has tweaked and tested every corner of its e-commerce platform to find out precisely what makes customers click. Its platform is one of the world’s most optimized online destinations, but Groupon is still running daily experiments to add new features that increase business — and get rid of features that don’t.
Comprised of both data scientists and engineers, the Optimize team has built a tech platform for running those experiments with scientific rigor. We spoke with three team members about their efforts to reshape how Groupon thinks about data.
OPTIMIZE AT A GLANCE
TEAM DISCIPLINES: Engineers and data scientists.
WHAT THEY DO: Build tools that help Groupon iterate and improve on its customer experience; promote data literacy across the organization.
WHAT THEY DID BEFORE: The team includes statisticians, an astrophysicist, a music major, an economist and a theological studies major.
DID I READ THAT RIGHT?: Yep.
DEEPLY ROOTED: Optimize was one of Groupon’s founding teams.
INSPIRED BY: Pharma research, which pioneered a scientific way to “peek” at data and end experiments early.
THE STACK: Ruby on Rails, Node.js, Ember.js, Python, R, Hadoop/Hive.
What does Groupon’s Optimize team do?
Kristi Angel, data scientist: Experimentation is essential to product development at Groupon, and Optimize owns the platform for experimentation. Our software is used by product managers in developing new features and improvements on web and mobile platforms. We automate A/B testing, ensuring statistical rigor and provide reporting on experiment results.
I research statistical methodology to implement as features in our software, partner with product managers and leadership to develop a culture of statistical literacy, assist with the interpretation of experiment results and work with product analysts to design sound strategies for experimentation that does not fit within our framework. I also work on special projects related to the analysis of anomalous results and validation of data quality in our pipeline, for example.
Robb Broome, senior software engineer: The driving force is finding out whether the changes we make to our website are benefiting the business or hurting the business. As time goes by, we get better at finding solutions and ways of measuring impact that are more reliable and consistent.
What technologies power your experimentation platform?
David Oliver, engineering and product manager: Groupon’s website is built on a series of Node.js microservices, and our code is embedded as modules in those apps.
So, one of those Node modules that we own is Finch.js, which does two essential things: it buckets users into control or treatment groups, and it sends a message that the user saw the experiment. That message flows through a REST API, into Kafka and finally into Hadoop. We also have two Ruby on Rails applications. One is focused on consuming that data from Hadoop and piecing out the data in such a way that we can easily read it, and the other doubles as experiment configuration and the stats engine that powers our platform. Finally, we have an Ember.js app that our stakeholders use to create and manage experiments, and display experiment results.
If an experiment goes poorly, that can have a real impact on your bottom line. How do you balance those concerns against the need for statistical rigor?
Oliver: We use a technique called group sequential analysis, which was pioneered doing heart valve studies in the 1970s.
Angel: Imagine a clinical trial in which an experimental treatment is actually harming people. Ethically, one would want to stop the trial immediately, but statistically it is a bad thing to watch your data as it accumulates. All measurements have a natural variation and this randomness can easily be mistaken for a signal. We really only want to analyze the data once we know we have enough observations to be confident that what we are seeing is signal and not noise.
Group sequential analysis is a way to “peek” at the data in a controlled way, periodically checking in on the experiment in a statistically rigorous manner that limits false positive results. It turns out, a life-saving mechanism in pharma research is a revenue-saving mechanism at Groupon. We can more quickly capitalize on features that increase revenue and limit exposure to features that lose revenue.
What are the most interesting technological problems your team is solving?
Angel: We are working on arriving at the optimal attribution models across different areas of our business. For example, the attribution of a purchase to a specific experience — a home page feature, an email, a push notification — likely has a different window of time where we can reasonably say a specific purchase is a result of a specific experience.
Features that reduce friction on a site are likely to have a more immediate effect, whereas an email campaign might have an impact over a longer time frame. Mathematically, how do we find that optimal window? Technically, how do we implement and support a number of attribution methods for our framework?
Groupon has a long history of experimentation, and a lot of data to work with. Does that present unique challenges?
Broome: Groupon is already highly optimized, which means you need better science and better math. In the olden days, it was easy to make huge changes that you didn’t need very sophisticated systems to see. We also have to pay very close attention to whether the data is getting in on time, because there are billions of transactions coming in, and the paths are pretty complicated.
What is one important change to how Groupon works that has emerged from the Optimize team’s work?
Angel: Finch Express, which is what we call our platform, has changed the way Groupon does A/B experimentation. Before its existence, experimentation was a pretty loose concept. Experimenters would monitor results day over day and run things until they looked “good” — usually meaning that the data was susceptible to the experimenter’s biases.
Today, we do not display data about the metric of interest until the experiment’s conclusion. That prevents experimenters from checking the results before enough data has accumulated and limits false conclusions.
How does that impact Groupon’s approach to data?
Angel: Our platform further moves the needle by shifting our organizational culture. It isn’t hard nor is it enough to be data driven. We must also be data literate. By building out a sophisticated platform and committing to comprehensive support for our consumers, we grow data literacy in the organization and as a result we’re able to make better decisions.
What drew you to Groupon?
Oliver: The Optimize team and getting to work both with programming languages I was already familiar with, like Ruby and JavaScript, as well as getting to touch other languages like Scala and Clojure. Our team doesn't use Scala or Clojure anymore, but I love that Groupon as a whole isn't afraid to try different languages and technologies — and just as easily move on if they're not working. I was also attracted to having lots of moving parts and getting to work with all the different teams in the company.
What’s your favorite thing about Groupon’s culture?
Angel: I always felt supported in my personal development. Leadership here has a strong interest in making sure that you're doing the kind of work you want. And although people are really smart, they’re also really fun. There’s a pervasive sense of humor across the company that I really appreciate, because I’m kind of a goofy person.
What does your team look for in developers?
Angel: We obviously look for technical skills; coding ability matters. Our team works in Ruby and Javascript and is full stack. But a great candidate might actually be a Java developer or have limited experience with front end development. We’re also looking for someone who is curious and humble, with good analytical intuition, and who has a strong sense of ownership, good communication skills and a commitment to excellence.
Oliver: It’s important to us that people are good communicators and who can explain complex concepts in a clear and straightforward way. We also look for professionalism, which is one of those qualities that’s hard to describe, but you know it when you see it. You have to be able to represent the team to the rest of the company.
