Skip to Main Content
icon

Product experimentation and the idea overload challenge 

Product managers are struggling with idea overload. As per ProductPlan's 2023 State of Product Management Report, 52 percent of product managers say feature requests drive their product strategy. 

Whether a feature or a successful product, your task is to ship everything fast. Yes, you’re measured on usage or adoption metrics, but for most companies, the former trumps the latter. It’s easy to understand the need for speed. Customers are telling you to build a feature. The CMO is telling you to build a feature. The CEO is telling you to build a feature. Sales team is telling you to build a feature. The list goes on. Everyone seems to promise features that don't exist yet to close deals.

Yes, speed is important. However: 

How do you know you’re prioritizing the right features for the target audience?  

You need to remove the guesswork and validate your ideas by bringing experimentation earlier into your product development lifecycle.

Charles Schwab's Product Management Director Michael Vanderhoof took the stage at Opticon to show how the company combined product development and experimentation into one process.

How product experimentation works across each step of the product development lifecycle.

  • Idea generation: You can validate concepts before resource-intensive development. Plus, focus on those with proven user value and market viability. You can validate concepts before resource-intensive development. 
  • Feature prioritization: You can test and communicate your rationale behind every decision, and align it with the product vision and strategy. 
  • Development lifecycle: You can run painted door tests, A/B tests, and more to make adjustments, minimize risks, and optimize the product based on continuous insights. 
  • Reduce risks: A product delivery process needs smooth rollouts and no bugs. Feature flags can help you get started here. 
  • Performance improvements: Experimentation can help you evolve your product to meet changing market dynamics and real user expectations. 
Benefits of product experimentation across the development lifecycle

It helps prioritize features, prove before developing, and reduce risks when shipping. By testing before full-scale development, you can identify potential issues and ensure resources are invested wisely in features that resonate with users.  

Keep reading to learn more about how introducing experimentation at the start of your product development process can help you deliver better products the way your customers want. All of it without the delays and bugs.

Experimentation value includes achieved gains and avoided losses. It should lead to improvement and catch potential mistakes.

David Carlile
David Carlile
Senior Director of Product Strategy

product experimentation

What is it?

Product experimentation is a systematic and data-driven approach to testing and validating different elements of a feature or product before you even start building them out. It involves using controlled experiments, A/B testing, and more to gather insights and make informed decisions about features, products, and user experiences.  

Here's Optimizely's David Carlile showing how product experimentation can help you reduce risks across the development cycle.

Why do you need to bring experimentation earlier in the development cycle? 

So, a typical product development process without experimentation ends up looking something like this:

However, here’s the product development process you need:

You need to bring experimentation earlier into your product development lifecycle. It'll help you avoid assumptions about the product, such as who the target users are, what their needs and pain points are, how they will use the product, and what value they will get from it.  

You need experimentation from ideation to prioritizing features to using feature flags during feature delivery for easy, quick, and safe deployment. Run experiments to de-risk your product launches and roll back your bugs in seconds. 

Product experimentation process

If you want to build better, prove before developing 

Product ideation is the foundation of the product development cycle. Without it, you have nothing on the product roadmap.

First, let's understand what a product ideation process really is. It's about generating, developing, and refining ideas for new products or improvements to existing ones.

Validating ideas before developing ensures that the selected features align with business goals, user needs, and overall product strategy. It helps you avoid building the wrong product.

See how to host a virtual ideation session. 

Three pillars of an effective product ideation process

Dealing with HIPPOs   

By leveraging experimentation, you can present evidence to support your decisions, making it less about personal opinions and more about what the data suggests.  

Communication 

Regular updates, clear documentation, and collaborative discussions keep everyone on the same page regarding feature prioritization.  

Testing

Your product experimentation process becomes a strategic tool for not only prioritizing features but also for justifying those choices in the broader context of business objectives.  

Remember though... experiment results and analytics are not always a replacement for deep thinking about customer needs, a product vision, or a qualitative evaluation of product concepts before you build them.   

You need to speak to customers and stakeholders to understand what they actually want and how they intend to use a feature or a product. So don't guess. Test during the ideation to bring the best ideas to life and that needs a collaborative environment. Here’s how to collaborate during ideation.

Start by testing smaller changes first to measure user engagement and focus on building what's gathering interest.

Using a painted door test, you can create a minimal user experience that mimics a full feature set. It'll fetch you feedback from users and ensure you're building the right thing. 

Feature prioritization

Having ideas is overrated, picking the right one is underrated 

To ship features quickly, the next step is strategic prioritization. It's crucial to understand which features should be released first and in which version. Rather than diving into development blindly, proving out features through testing and experimentation is essential.

This process validates assumptions, identifies potential challenges, and fosters a culture of continuous improvement.  

Striking a balance between speed and strategic prioritization ensures that the product improvements are effective, meeting user needs and aligning with business goals.

Check out feature prioritization methodologies 

Types of experiments in your development lifecycle

There are many types and the list starts with A/B testing. Here, two versions of a webpage or app are compared against each other to see which one performs better. 

What is A/B testing:

Other types of experiments in product development: 

  1. In A/B/n testing, multiple versions are compared against each other to determine the one with the highest conversion rate.
  2. In server-side testing, variations of a test are rendered directly on the web server.
  3. Multivariate testing modifies multiple variables at the same time and finds out the variant with the most uplift.
  4. In A/A testing, two identical versions of an experiment baseline are put against each other. 
  5. Usability testing helps you test a feature with real users and evaluate its readiness for release. 
  6. If you want to increase the speed of development, build a minimal viable product (MVP) first through lean hypothesis testing to find a product-market fit.
  7. To reduce risks during product delivery, you can validate by releasing to a small percentage of users. Canary testing can help you deliver to a certain number of users at a time. 

Overall... 

A product experimentation framework brings a lot to the table. But there's a benefit that is almost entirely overlooked. 

Testing ideas around a given sample size can significantly reduce operational costs associated with web and mobile app development.

For example, making changes to your entire website requires time, resources, and capital. However, consider that most changes won't actually improve results or impact conversion metrics that matter So, it's cheaper to build them as experiments than to build all your ideas in production.

Building a test rather than a full-blown product requires less process and man-hours. Once you identify the highest-value product opportunities, only focus on building those features/products.

You'll always get asked to build every idea. It doesn't mean you go out and do it. Ask yourself: What's the smallest and simplest thing we can test to validate this concept? Only develop assumptions that have been validated. More than ROI, doing this will save you a lot of cash.  

If you achieve positive results, push the idea live and initiate the ideation process to build on the insights gained. If the result is negative, do not ship that feature. However, learn from what went wrong and avoid it in your remaining ideas.  

If the result is inconclusive, test alternative sources of evidence and variations. Lastly, no matter the kind of tests you run, you can’t scale it without key metrics.

Here’s how to supercharge your experimentation metrics: 

Feature management

Prevent negative experiences for your users

Feature management is about developing, delivering, and optimizing new features for a product. More importantly, it's about doing the best you can with the resources at your disposal. When you're out there building a new feature, you want to avoid negative user experiences. That'll only happen if you ship the best version of every feature without putting too much pressure on your resources. 

Here's how you can do the feature management part better. 

  • Reserve your developer resources for the most important development: Remember how you prioritized features based on their impact on user experience, business goals, and overall product strategy. Now, allocate your best resources to develop and test these features first.  
  • Validate code performance: Even if you choose to use a product experimentation tool as just a code editor, you can still test the performance and optimize accordingly before it goes out to your users. You can even gather feedback from beta testers before rolling out new features to everyone. 
  • Use data to understand your customers: Continuously analyze user behavior data to refine your product strategy and optimize feature management based on real-time insights. Combine quantitative data (metrics, analytics) with qualitative data (user feedback, surveys) to make informed decisions. 

Product delivery

Deploy quicker. Feel safer with feature flags 

Rolling out new features to all users can be a recipe for disaster. Instead, roll out more frequently to a subset of users and with fewer risks, validating your performance and impact on the customer experience before launching. 

Feature Flags, also known as switches or feature toggles, allow you to test functionality without deploying new code. This increases control, allowing you to release more frequently and test and learn without impacting the user experience. 

You can conduct controlled experiments to deploy changes faster, test variations, and tailor interactions with users. 

Here's an example of how feature flagging works in real time and ways to deploy:

When deploying a new feature you can control who sees the new feature, so you can grant access to a specific feature to a certain set of users.    

Typical rollout process with product experimentation

If you want to target specific audiences through multiple stages of release while maintaining collaboration among team members, you can conduct staged rollouts with progressive delivery.   

Every feature or product you build should meet customer needs. Learn more about improving product delivery. 

Product experimentation Tips

Optimize user journey at every step 

Setting up a product experimentation culture can work if you bring a concrete structure to make it more measurable. Here are more product experimentation tips: 

  • Assign a leader who's consistently thinking about how to improve the culture of product experimentation at your company. 
  • Set SMART (Specific, Measurable, Achievable, Relevant, and Time-bound) goals. For example, I want to run 2 product page layout-related experiments this month. 
  • Provide the ability for an experiment to iteratively move across your development environments from local to staging to production to roll out.  

Even practitioners can have doubts. So, have a space for questions, feedback, and iteration. 

As you grow with each iteration, it is an opportunity to gather valuable insights. By rigorously testing different features, layouts, and functionalities, you uncover what resonates most with your users and what drives conversions effectively. Through personalization, you can tailor user experiences to individual preferences and behaviors, leading to a more engaging and relevant interaction.  

This way, every digital experience you deliver resonates with each user on a deeper level, and retention will improve.

Experimentation across the digital lifecycle

Product experimentation tools

By now, you must be thinking about the best product experimentation tool.

The general advice here is to choose between the well-known brands but, within that, go with the one offering full-scale experimentation capabilities. Compare features like-for-like as much as possible and check if the platforms offering a lower price have a lot of functionality switched off.

Like any tool, focus on your requirements. For example, the team at Charles Schwab has been using Optimizely as a code editor right from the start. Calendly is crafting personalized experiences for its 20 million users.

If you have more questions on how to select the right product experimentation tool, check out this experimentation and feature flagging RFP template.

Conclusion

Product experimentation is a mindset. It brings agility, encourages data-driven decision-making, and cultivates a culture of experimentation.  

As a product manager, you have to cope with ambiguity and risk and make decisions with incomplete or conflicting information. You can navigate the complexities by optimizing user experience through data points, and clearly defined experimentation goals. You can explore and evaluate different possibilities and outcomes.  

Remember each test is a lesson, and every iteration takes your product closer to what the customers want.

Ready to learn more? Check out our no-nonsense report on the Evolution of Experimentation. It includes lessons and learnings from 127k experiments.