3 tools to measure and pivot before its too late
A few weeks ago, I wrote about how to create a successful product strategy and succeed in a process that most companies fail.
However, we still face a challenge: this well-crafted strategy is supposed to provide a direction for execution for approximately 9 to 24 months. To know if a Strategy has been successful, you would need to wait years to see its results, which is unacceptably long.
For that reason, we will explore the best tools and options to measure the strategy’s success in its early days and how to take action with the results.
Strategy Feedback Loops
There are three tools free to get leading indicators for the success of your direction. They are sequential, starting with early, low effort, but less reliable inputs, and increasing in complexity and confidence as time and cost increase.
1. Qualitative Assessment
The earliest learnings of the defined strategy are the observations you get from your self-analysis and the communication sessions with other organization members. Getting actionable insights is challenging because you may get very subjective and biased opinions, especially if you are going with a “tell me what you think” kind of conversation.
To achieve more specific and applicable feedback, I use an 18-questions template (with a Likert scale and follow-up comments) to cover four aspects of strategy evaluation proposed by professor Rumelt (author of Good Strategy, Bad Strategy):
- Consistency: is it aligned with the company’s vision, needs, and resources?
- Consonance: does it consider customer and market trends?
- Feasibility: is it achievable and takes into account the team’s capabilities?
- Advantage: is it aligned and strengthens the desired positioning?
Using this survey, I have classified the types of feedback I usually find according to who is giving it.
1. Self Assessment — The obvious
Answering these questions by the same team that crafted the analysis should bring to life simple things that you took for granted or considered but were not fully articulated in the resulting communication document.
Some time ago, we were creating a strategy based on an opportunity for the automation of post-sale service activities. We had many manual processes that impacted operation costs and hurt our ability to scale.
By trying to answer the question, “The strategy helps increase the value we provide for the needs of our customers”, we realized that we had not articulated the impact on resolution speed and customer satisfaction.
2. Extended Product Team — The facts and feasibility
After running the same exercise with the rest of the product organization, we identified some skepticism around the data we were presenting and how reasonable it was to tackle the chosen problems.
Following our previous example, the feedback helped us improve the indicators involved in the presented process automation, like the number of requests and manual processing times to convey a more robust vision of the opportunity. When dealing with data-driven and deeply committed product teams, in-depth diagnosis information helps clarify the “why”.
Regarding the feasibility of the selected path, we included a few examples, not as requirements, but to provide visibility of what options could be pursued.
3. Stakeholders — The business compatibility
Depending on your level of communication with stakeholders, presenting your strategy can be one of those moments where you validate how aligned you are.
In our example, we did not receive any comment on the long term plan of automation (good!), but they did mention the need to have in shorter-term better support for the current manual processes. It was aligned with our strategy, but we had to consider how to roll out the path.
In summary, stakeholders’ feedback may be more aligned with consistency with business needs, the first item proposed by Rumelt.
2. Experiments
As we start to execute the strategy, the first step would be to discover the right solution through experiments. If you have a well-oiled discovery process, it should take little time to start running this type of validation. As soon as one month after starting executing the strategy, experimentation would provide useful learnings to adapt it if it is failing.
It may be tricky to find what type of experiment would be beneficial to get this type of validation. Which may not be good at this stage?
- On one extreme, you have early-research tools like exploratory interviews, that should have probably occurred before getting the insight for a strategy. (If you’ve decided on a path before doing interviews, it probably is a good idea to do them now).
- On the other extreme, you have specific solution validation, like usability tests or A/B tests. This type may invalidate your implementation but not necessarily the strategy.
The suit spot is with the type of tests that validate the problem and value of the opportunity. Problem interviews or prototype tests are good examples of such experiments.
Keep in mind that these tests become more important as uncertainty grows. For example, if considering a strategy with initiatives for the three-time horizons, experimenting with the more “moonshot” ideas for future growth would become even more critical. It’s quite different to validate an “automation of current activities” opportunity than to “lead X new trend in customer behavior”.
3. OKR results
If experiments have positive results, the next validation should be at the OKR level. What’s different? That you should see the results scale. After validating a suitable solution, you move on to progressive implementation and A/B tests that would confirm (or not) that the selected path holds valuable with more customers.
If the OKRs were correctly linked to the strategy, as initiatives outcomes are reflected on the key indicators, you would begin to see an impact on your strategic goals. OKRs can be monitored at any moment, probably starting after the 1st month of having implemented the strategy and its corresponding key results. By the end of the quarter, it should be clear if we are on the right track to achieve success.
Actions and Pivots
If we consider pivoting as a significant change in direction, while it may be frequent for startups looking for product-market fit, it should be a lower recurrence event for any company passed that stage.
We can follow this table to decide the actions based on the tool’s feedback:
The assessment feedback will mostly help you adjust your strategy. Since you don’t have any validation yet, it would be strange that you decide to pivot at this point. But the pieces you modify on your strategy should increase your chances of success.
In the case of experiments, there are two potential pivot opportunities:
- You discover a new insight that you believe is worth pursuing now. The first step would be to invest a bit more in experimentation to confirm the opportunity’s size and, afterward, go back to the strategy board and pivot to the new one with more confidence.
- In the case of strong invalidating feedback, a pivot is also required. If you have done discovery work previous to the strategy definition, this should be a rare case.
Finally considering OKR results, there are other scenarios to consider:
- A small offset in the results, like achieving a 35–50% score instead of the expected ~70%, should probably not be something to worry at the strategy level. Unless there are other indicators, you should revisit the execution plan rather than the selected path.
- Scores lower than 30% may happen for multiple reasons, which may be the hardest type of result to evaluate.
Without ignoring bad results, we need to assess if the result is due to the chosen strategy or due to our selected solution or incorrect execution. Did we build the wrong feature? Have we used an erroneous marketing channel?
My best advice at this point is to go back to experimentation. Considering what you have learned so far, formulate and hypothesis and put it to the test. It will require more effort, but it is probably worth it before giving up on a strategy that was the correct one.
The good news is that after the experiment, you will have much more confidence to decide whether you should:
- Stick to your strategy and make changes in execution
- Update your strategy with new insights
- Completely pivot
Conclusion
With excellent communication, appropriate alineation through OKRs, and effective early learning loops, you give your strategy real chances to succeed!
Scary news :) — I’m writing a book to describe the process to successfully create and link the Product Strategy, Roadmaps, and OKRs.
If you want access to the pre-launch free section, you can have it here.