Experimentation · Reuters · 2025
The feature I stopped
AI article summaries shipped and engagement stayed flat. I stopped them and moved the investment to recaps for returning readers.
- +15%
- session frequency from returning-user recaps
- 20+
- A/B tests run with kill criteria set upfront
Context
AI features were under pressure to ship, and summaries were the obvious first one. The risk was keeping a feature alive because it demoed well.
The problem
AI summaries shipped behind an experiment with an explicit stopping rule. Engagement stayed flat.
What the evidence said
Summaries answered 'what is this article' for readers who were already on it. The unmet need was 'what did I miss' for readers coming back.
Options on the table
What shipped
- Experiment design with hypothesis, primary metric and stopping rule written before launch
- Summaries switched off at the pre-agreed threshold
- Returning-user recaps: what changed on stories you read since your last visit
Outcome
Recaps lifted session frequency 15%. The kill-criteria habit became standard across 20+ experiments.
My role, and the team's
- Me
- Set the hypothesis and kill criterion, made the call to stop, and reframed the opportunity as recaps.
- The team
- Data science ran the analysis; editorial set the rules for what a recap may say; engineering reused the summary pipeline.
From the working file
- Hypothesis
- Summaries at the top of long articles increase scroll depth and return visits.
- Result
- No significant change on the primary metric at the planned sample size.
- Decision
- Stop per the pre-registered rule. Redirect to returning-user recaps.
- What we learned
- Readers already on the page don't need a summary; returning readers need a catch-up.
What I'd do differently
I'd share the kill criterion with stakeholders before launch, not just with the team. It turns the stop from a debate into a formality.