Placement Experiments and Holdouts
Built-in A/B tests and standing holdouts per placement so every layout, experience, or parameter change is read as lift, not as a trend.
Every placement keeps a standing holdout, a matched group of viewers who don't see it. Experiments split viewers between versions of one change. Genuin reads the difference as lift, so you see whether a change helped before revenue moves.
This capability arrives as a proposal when your network needs it, filled in from your brand context, and goes live when you confirm it. It's part of Onsite, and Measure engagement explains how lift is counted against a holdout.
When to use it
Run an experiment before you change a placement's position, experience, or feed settings. Keep the holdout on permanently, for lift reporting.
| Variable | What you can compare |
|---|---|
| Position | Places on the page, such as after paragraph three or at the end of an article |
| Experience | Shorts, Shoppable, Community, or Live & Q&A |
| Layout | Layouts such as feed, carousel, or inline |
| Feed parameter | Exploration rate, freshness weight, repeat window, pinned slots, cohort use, or paid pacing |
How it works
- The holdout takes a share of each placement's viewers, 5 percent by default.
- Lift is the difference between viewers who saw the placement and the matched holdout. It's measured in watch time, pages per session, or return rate.
- Each experiment tests one variable at the split you choose, with a set duration and a minimum sample.
- Results show lift with confidence intervals, so a change reads as lift, not as a trend.
- The same holdout isolates Maya's contribution, so Relevance OS reads lift directly rather than inferring it.
- Holdouts also support incrementality, the holdout-based model in Attribution.
Set it up
Octo proposes the experiment with these steps already done. You review the result and confirm it.
- Open the placement under Network > Grid and choose Experiment. Learn how the grid works in The Hierarchical Grid.
- Pick the variable and the split. The variable is a position, an experience, a layout, or a feed parameter. Learn about feed parameters in Personalization and relevance.
- Read the results in Relevance OS, and promote the winner with one click. Learn how in Relevance OS: reading the analytics dashboard.
Settings
| Setting | Default |
|---|---|
| Holdout share | 5 percent per placement |
| Experiment duration | Set for each experiment |
| Minimum sample | Set for each experiment |
Set an experiment's duration and minimum sample when you create it under Network > Grid. The holdout share is also part of the Relevance OS Dashboard configuration.
What it reports
Relevance OS shows each experiment's lift, with confidence intervals, on these measures. Measure engagement defines each one.
- Watch time: watch time per user per minute, the engagement metric Maya is measured on.
- Completion: the share of plays that reach 90 percent.
- Follows: follows per thousand views.
- Revenue: what the placement earned.
Build on it
The Web SDK renders the placement you test. Add a Genuin placement to your website covers the placement and style IDs, and Embed the Genuin Web SDK on your website covers loading the script. Customize the Genuin video player sets sizes, tile layouts, and the profile menu.
Receive engagement events in your own analytics sends what viewers watch and do to your own analytics tool and warehouse.
The experiment and report calls in the Business Account API are proposed, not live. Live API today and the Business Account API roadmap maps what you can call now to what's planned.
