3 frameworks for making better decisions around product releases

Written by Rachel Wolan, VP Product at LiveRamp, for our ebook, The Path to Product Excellence: Stories and Advice From the Field.

A release gone wrong

I used to run product at a call center software company. One Thursday in February, we rolled out a small but exciting change to fix a major data input problem that impacted agents. Before releasing it, we ran user tests and a month-long beta, thoroughly QA’d the agent experience, trained our support team, and notified our customers about the change. I was so comfortable that we did everything right that the day we rolled out the change to tens of thousands of call center agents, I left for a long ski weekend in Colorado.

We ended up completely breaking the existing process.

Agents were not getting credit for calls, which is how their pay is calculated. Supervisors accused agents of not answering enough calls. We received 10x the typical number of support tickets that day. Suffice to say, our customers were really unhappy, and I spent the majority of my vacation holed up in my hotel room rather than on the slopes!

Learning from mistakes

In the aftermath, we learned that several bugs logged by beta users were not escalated. We also discovered that our assumptions around the agent user persona were wrong and that our beta users had a higher tolerance for change than the overall user population.

I took that opportunity to re-examine many things around our decision-making process and my personal decision-making capacity. Too many roads passed through me. My team was not able to make bigger calls without me. Bugs were not getting escalated at the right time. Feature requests were not evaluated systematically. Basically, I was the bottleneck, and I needed to figure out better ways to decentralize decision-making power. Here are some simple frameworks I came up with to aid in release planning and real-time decision-making.

1. Release an internal alpha first whenever possible

We simply skipped the alpha stage in this case. We were lucky that we had both Support and Inside Sales teams since that is who we sold to. Involving our teams made them invested in the success of the product, and they gave feedback in real-time to our product managers, designers, and engineers. We created an internal product council based on our most vocal alpha testers.

I also instructed my product managers to review their beta plan with those teams. Each member of the internal product council had to sign off before we released agent-facing features. We realized that we had a greater margin of error for the supervisor and administrator functionality, but as soon as we touched the agent, we needed to run a much more careful release process.

2. Write the release notes as soon as the engineers begin working

I had my Associate Product Managers start working with the organization the moment we had locked in a release, which happened every month. That gave Support time to build detailed documentation, User Experience time to create user guides and onboarding guides, Sales Engineers time to adjust their demos, and Product time to produce a proper FAQ.

3. Make bug escalation a team sport

After the release incident, I realized that information was deposited throughout Support, Sales, Sales Engineering, Success, other executives, and more. The bugs we discovered after the rollout had been reported to Support, but not escalated to us. I hadn’t given those teams a proper outlet both to escalate concerns, without crying wolf, and to voice their opinions about the product.

I developed a framework to get cross-team buy-in so that anyone (i.e., Support, Sales, Success) could make real-time decisions about bugs on the spot based on our shared set of rules. Once we implemented it, we made tremendous strides in improving our bug resolution process.

Below, you will see the bug escalation framework we developed and refined over several months. Every time a bug was submitted by anyone in the organization, they filled out the following scorecard. Product and Engineering also filled out components of the scorecard when new tickets were submitted. This gave everyone a voice in the escalation process and distributed responsibility, ultimately making it much easier to make faster complex decisions.

By distributing responsibility for the success of the release process, we significantly de-risked future releases to agents, got earlier buy-in across the organization, and sourced better ideas from the highly engaged alpha product council.

. . .

This post is an excerpt from our ebook, The Path to Product Excellence: Stories and Advice From the Field. Get your copy now for more valuable insights from product management thought leaders.