This is a post in 3 parts, but the publication order got somewhat messy :). This article should be part 3 (How to prioritize with Product Discovery and rapid experiments), and then I will go back to parts 1 and 2. The reason is that I think Part 3 is so valuable that I could not wait to share it!
How relevant is prioritization for product development?
One of the main challenges when developing products is how to prioritize the backlog or roadmap of features that we want to implement. How to prioritize is the most frequently asked question I receive and probably the topic most Product Management blog posts are about (in fact I wrote last year another post on this topic).
As Product Manager, although I was hired not to prioritize but to achieve business results, the main task I must do to achieve that goal is to correctly determine what the team has to do next. Constantly define what is the most important task we have to start when we finish the one we are currently building.
In fact, in an analysis of Alpha UX, 86% of the Product Managers surveyed indicated prioritization and deciding what goes into the products at what time, as one of the key areas of responsibility.
Virtually any prioritization takes into account 2 aspects of the functionality (or thing to be done): the cost (how much effort or money will cost to make it and release it) and the business value (quantify somehow that I will achieve once it is released) .
While both aspects are extremely complex, in this post I will focus on the second: understanding what value will contribute to what I am considering doing.
Doing your homework
Prioritizing without a clear product strategy is like deciding what route to take without knowing where you want to go. Without clear objectives and product strategy, prioritizing is not only going to be more difficult, but it will also be completely inefficient.
Ergo, the first thing to do to prioritize correctly is to work on having clear objectives for the company and then establishing with the main stakeholders the objectives that the product will have aligned with those general goals.
Finally we must translate these objectives into performance indicators (known as KPIs) that will have a concrete value to achieve over a period of time. It is important to keep in mind that a same company target can be obtained with many different product KPIs, so we generally choose quarterly which ones we are going to focus in that particular period.
To illustrate this with a (very simplified) example, let’s consider a subscription product like Spotify or Netflix:
- The company wants to increase its sales
- The goal of the product will be to increase the number of buyers
- KPIs might be:
- Increase number of trial users (or free users) by 100.000 in 3 months
- Increase conversion rate to paid users by 10% in 3 months
- KPIs might be:
An example of prioritization without objectives
I have already told stories of my first steps in product, with a game of Poker for social networks. At that time the company had as a general rule 3 macro objectives: new users, engagement and monetization (this last one achieved by having more users buying virtual goods).
Following that concept, I systematically put into my roadmap functionalities that covered these 3 areas. The first monetization functionality we did was a success compared to other products: we managed to get 2% of users to pay (which in this industry was a pretty good value). The problem? We had about 10,000 users, these payments were called microtransactions of an average value of about $ 2 to $ 5, and for a company that was billing in the 7 figures this was truly insignificant.
Whenever I showed these monetization values, the stakeholder’s natural question was the same: That’s lovely. Now, how do we keep that monetization but with 1 million users?
In the early phases of the product it was much more important to achieve new users and engagement, and I should have focused the product KPIs only in those 2.
The options to prioritize
Now that we have clarified what objectives we are pursuing, we will focus on the tools that help you determine which of your ideas should have the highest priority.
I’ll present the options that I consider most relevant (not because I like them all – but because they are the most used ones), in a small matrix:
In general, prioritization methods have two characteristics: their complexity (how difficult is it to get a result) and their accuracy (in other words, how quantifiable their outcome is).
In part 2 of this article I will describe each method in detail, but now I want to skip all explanations and recommend you the best option.
The Inconvenient Truth About Financial Assessment
As organizations and their products evolve, most tend to seek ways to make their value estimation more precise, because they usually have more potential projects and want to prioritize them properly. The most traditional way to achieve this result is to do a financial evaluation, probably the method best explained at business schools (basically what you should find in a business plan).
You can not see my face red with embarrassment as I write this, but not long ago I was a strong believer in this method. Both the products in which I worked and the company itself had (or I thought they had) enough history of similar projects to use as inputs to make projections of the business results of new ideas we wanted to implement.
Why am I embarrassed?
These projections do not take into account two uncomfortable truths of product development and innovation:
- More than 50% of ideas will not have any business impact: no mather how nice your plan looks, the best companies in the world report this amount of failures. Booking.com for instance, that mentions that 70% of Its new developments do not end up seeing the light (and if we go to the Startups world, 9 out of 10 new products does not thrive).
- Although you have one of the ideas that is within the 50% of “successes”, the result is highly dependent on many aspects of execution: how well we communicate it, how user friendly we make it, and how much value the user really perceives. Trying to assess the value based on past executions does not have a solid ground.
In short, this method is great for industrial products or bridge constructions, and any other discipline that has a predictable result based on the behavior previously seen. But in general it is not good for innovative software products.
NOTE 1: Startups make the same mistake. For example, when you make a financial plan to get investments, you are doing this same exercise with even more uncertainty.
NOTE 2: There are software projects where there is less innovation and this method can work. For example, those that are for internal use and the user has no choice on whether to use the product or not 🙂
Using Product Discovery to prioritize
In the options matrix, I highlighted the financial assessment as one of the high accuracy methods, and in the previous section I discussed how most of the time its accuracy fails :S. The problem is that for lack of a better alternative, it really was the most accurate method. The estimate could be orders of magnitude different from the result, and yet it was better than other methods as expert judgment. And as an added bonus you have a precise number to show teams and stackeholders why this was more important.
Of course when what you deliver misses the estimate over and over again, your word loses credibility and even your job can be in danger!
The intuitive approach is trying to make the estimate more accurate: perfect your method adding more data points, variables you have not contemplated, etc. But this misses the point: you cannot get more accurate because the information you are looking for does not exist in your archive of past user behaviour.
Wouldn’t it be great if I told you that there is a more accurate method? And even better, if I told you that this method is less complex than financial evaluation?
Well the title of the section already anticipated the answer, but the methodology of Product Discovery and its techniques of rapid experimentation helps you to:
- Have a precise value to prioritize and communicate
- That will be very accurate because it is given by concrete feedback from real users
- You can run it with a prototype that takes a day to build
Okay, how does this “prioritize by experiments” works?
When you run an experiment you propose a goal.
Lets give an example using again our objective of Spotify increasing its paying users ratio by 10%. Suppose I want to experiment with adding a paid only feature that allows users to look for a song by singing to the microphone of their cell phone or computer.
Of course I can do a financial analysis, but as you can imagine, predicting how many users will pay for this feature would merely be an informed guess. The feature on the other hand is quite hard to build, so I can spend months building it only to see it fail after launch.
On the other hand, to run an experiment, I only have to write an email: Dear user, we are adding this new functionality that will cost $ X per month. Please, to subscribe, press the button below.
I think running this test would take around 20 minutes 🙂
Of course this is an oversimplification, but the information you will get is invaluable. But the information you will get is a very accurate number of people who would be willing to subscribe to the service.
You can refine the test with some quick options:
- Segment the user base, we do not want to spam all our users and maybe we have a segment more willing to opt for this
- We can achieve further accuracy taking the user to a landing with the possibility of entering the credit card, to measure how interested they really were (even if you do not charge them later and show them an apology message). With solutions like Wix, Leadpages or others, this would be very easy to do even without writing a single line of code.
After only hours I may have a precise conversion rate, which is a much more accurate description of future behaviour than an analysis based on the past.
What if the conversion rate is 0%? Definitely may require you to iterate the message or how you present this new feature. But also will give you a very certain feedback on not to prioritize this feature yet.
Being these tests so fast, I will be able to execute many in parallel (or over a few days), and then I will have candidate functionalities with their associated results:
- Search by voice – 0.8% conversion
- Embeded Video Clips – 1.3% Conversion
- Play songs reversing sound – 0.01% conversion
After weighing the effort of each one (which may be the subject of another article), prioritization is practically automatic. And not only that, the chances that once built will reach the estimated business value are higher.
I do not know if you can see the power of this but believe me that after having struggled with complex methods of prioritization and with the frustration that the estimates are not met, this really feels like a salvation.
I would love if you can comment what prioritization techniques you use, what challenges you are facing, and any feedback or opinion on prioritization with experiments.