During the last Scrum training that I gave, I put to the test Wideband Delphi, a technique for reaching a consensus on estimations.
One common exercise that I propose is to estimate my own weight. Another, not based on Wideband Delphi, is to give high and low estimates on various data, such as the temperature of the sun, or the amount of US$ cash in circulation.
In this instance, there were 18 participants, about 50% more than the recommended number. I was running out of time, so I decided to merge the two exercises into one. I simply asked to estimate one thing: the amount of water in the US great lakes.
When the exercise is done with my weight, even the worse early estimates are not off by more than 10kg. Usually, by the end of the exercise, the consensus is within 1 kg of my actual weight, an impressive feat.
I was concerned about the amount of water, though. It is harder to relate to the US : participants were all living in the French country. I was hoping that some would mention having seen them, or at least seen a large European lake, such as Lac Léman (French for Lake Geneva), which could have been used as a point of reference. No such luck, so I was a bit concerned about the outcome.
The technique we used requires working in rounds. In each round, small groups work on their own, making up numbers (one low, one realistic, one pessimistic) in their own way. Then, numbers are shared and put on the whiteboard. An average is computed for each group, using the following formula:
average = (low + 4 x realistic + pessimistic) / 6
Finally, the groups are encouraged to discuss the ways by which they came up with their values. Inevitably, rather heated arguments ensue (“no way you can be right!”), but eventually, they tend to reach some kind of partial consensus (“see their values? we should double-check ours”).
On the first round, figures were around 600km3 for some, and around 40 000km3 for others. One group suggested 25 000km3 as a realistic number.
On the second round, we had figures ranging from 10 000 to 30 000 km3 (notice how the lowest value is now very different from the first round). At this point, some people tried getting more details from me (“does the figure include the Canadian side?”), but I explained that I had none to offer. Just like in real life when it is impossible to know the exact scope of a system.
On the third round, it went from 12 000 to 30 000 km3. The average was 25 000 km3.
Well, believe it or not, the actual figure is 23 000 km3. Yep, less than 10% off. Let me tell you, they were really impressed (me too).
Things to try next time:
- I shouldn’t know the correct value before hand; I feel that I might have influenced the participants, despite all the care I took.
- I was probably wrong to suggest having 3 rounds. A more involving way would have been to ask the participants themselves if they felt they had reached a consensus.
- I wonder if it would work for any type of figure? What about the classic “how many piano repairmen are there in NYC?”
