Prioritisation and the experiment loop

Your idea list is ready. Two questions remain: which to try first and how to read the result.

For the first, the most widely used simple framework is ICE. You score each idea on three axes (say 1-10):

• Impact — if it works, how big is the result?
• Confidence — how sure are you that it will work? This score is raised by evidence, not enthusiasm
• Ease — how quickly and cheaply can it be tested?

Score = the average of the three. You start from the highest.

ICE's value is not its precision — all three numbers are guesses. Its value is that it forces you to judge every idea from three separate angles. The feeling "I like this idea" conflates Impact with Ease; once separated, you often see that your favourite idea is a month of work resting on no evidence at all.

The experiment loop. Every experiment goes through the same five steps:

1. Hypothesis — "If we do [X], [metric Y] will change by [Z], because [reason]" 2. Metric — write down in advance which single number you will look at 3. Duration — write down in advance how long you will wait 4. Run — change one variable only 5. Decide — adopt / discard / retest

Every step matters, but writing steps 2 and 3 in advance is the one most often skipped. If you decide which metric to look at after seeing the results, you will always find a number that looks good — that is not learning, it is convincing yourself.

The truth about statistics on a small app. If your app gets 30 installs a day, you cannot reliably measure a 5% difference between two variants — that difference could be chance. The practical consequence: do not test small changes on small traffic. Try bold, large changes — their effect rises above the noise.