For more than 10 years I have been obsessed with Product Development.
And since I have a result oriented demeanor, I’ve always tried to measure it and set goals for improvement.
The problem is that the things that are easy to measure most of the time do not mater. Quality of code, number of deploys, lines of code, response times, etc etc.
It is useless to measure any metric without understanding how it finally impacts the business. If you have 0 defects in a product that nobody uses, you wasted your time.
Good Product Development is the ability to efficiently generate great products that customer love and are profitable for the company.
How can we divide that definition in “things to measure”?
Part I — Business Results
If we take the first part — “that customer love and are profitable for the company” — we are basically talking about having a good business results. The problem is that it is often hard to isolate. Many parts of the company are moving towards the same goal. So how do you know if that conversion rate increase was due to good product development or the commercial team managing to provide better prices to customers?
Luckily we have A/B tests that can help you isolate the effect of new releases on a particular product. This is not applicable to all cases (like “measuring increase in SEO traffic”), but it can be used most of the time.
Measure good product development as the incremental impact of new product releases in the business metrics.
Let’s put an example. Lets say conversion in an E-commerce increased from 2% to 2,5% this quarter. You had the marketing team, the commercial team and the product team all wanting to take the credit. Marketing used new channels to acquire traffic. The commercial team managed to drop 5% the price in top selling products. And Product Development did several changes to the product details page.
If you did A/B testing in each release of the product release page, you would be able to say that through the quarter you released changes that affected the conversion +10% combined. So in that way you would say with more confidence that out of the 0,5% increase, Product Development should be credited with 0,2%.
There is one caveat: at the end of the day we want positive results. If conversion dropped, we do not want to “award” the product team for making good releases that made it drop less. Product teams should be empowered to care for increasing conversion even on things that are outside their direct sphere of influence. You do want that natural tension when the product team discusses with Marketing because they are buying low conversion traffic.
Keep the target as “increase X% business metric”, use the same goal for the multiple areas impacting the business result, and empower the product team to work with other areas to improve synergies.
In this way you provide the right incentive for the team, but keep measuring through A/B testing how effective your product initiatives were.
Part I I— Efficiency
This is the really hard part. And I will share my thoughts in the second part 🙂