Nothing about this is a smaller version of the enterprise product. It is a different trade.
- One question at a time
- Traffic is not split across a queue of concurrent tests. One question gets all of it, which is the difference between an answer and a shrug.
- It waits
- A winner is called on the strength of the evidence, not on a schedule. On a quiet site that takes longer, and the engine is content to wait rather than promote something it cannot support.
- A tie still counts
- Most tests on a small site come back with no difference. That gets filed against the idea rather than thrown away, so your traffic never buys the same answer twice.
- What you learn keeps
- A belief about what your audience responds to survives a redesign, a new season and a change of theme. That is why patience costs you less here than it would with a tool testing layouts.
- A bigger question needs less traffic
- Not proportionally less. Squared. The visitors a test needs fall with the square of the effect it is looking for, so an idea that moves conversions twice as much needs a quarter of the traffic, not half. On a page converting at ten percent, hunting a ten percent lift takes about 14,000 visitors per version; hunting a twenty percent lift takes about 3,500. Testing one word is the expensive way to test, because the effect is small and small effects are what traffic gets spent on.
- One idea, told across the whole path
- A hypothesis is not confined to one element. The same argument can be made in the headline, the section under it and the page the button leads to, which moves a larger number than any of them would alone. That is the lever a small site has: not more visitors, but a bigger question asked of the ones it gets.