Six people estimate how many chairs a hall can hold. The organizer takes their average and calls it the group’s answer. That result may be useful, but it does not reveal whether the six people brought six pieces of information. Perhaps five copied the first estimate; perhaps one knows the fire-safety limit while the others guessed from the room’s appearance. A numerical average combines answers. To measure information aggregation, we also need to ask what information entered, how it moved between people, and whether the combination improved the answer under a fair comparison.
This Methods & Measures entry concerns the mechanisms and tests of combining distributed information into a group estimate, forecast or choice. It is narrower than a general “wisdom of crowds” claim that groups can sometimes be accurate, and different from social learning strategies, which ask when an individual should copy or sample others. Aggregation asks what the group does with its members’ information. An average, a vote, a discussion network, a market or an algorithm can all aggregate, but each has different assumptions and failure modes.
Define the information before the rule
A study cannot measure successful aggregation without defining what was distributed. In a factual estimation task, each participant may receive a private clue about the correct number. In a forecasting task, participants may have different background knowledge or different interpretations of public evidence. In a group discussion, the relevant information may be a reason that only one member knows. The information structure determines what an aggregation rule can possibly recover.
The simplest arrangement is independent numerical estimates of the same quantity. A mean may reduce unsystematic errors when the errors are not all pushed in the same direction. A median may resist a few extreme values. A weighted average gives more influence to some participants, perhaps because their past estimates have been more accurate. These are not interchangeable rules. One can outperform another under one error distribution and fail under another.
Kameda, Toyokawa and Tindale review information aggregation across several collective-intelligence settings. Their broad account makes a key point for measurement: a group output can arise through statistical pooling, social interaction, division of labor or a structured decision process. “The crowd” is not one intervention. The researcher should name the rule, the network and the information available to each participant before attributing an outcome to collective intelligence.
Even a correct group answer may not be evidence of aggregation. If a single expert’s estimate happened to dominate, the group may have followed rather than combined. If every member used the same public report, the group may have repeated one source six times. A study needs individual pre-aggregation responses and a record of subsequent exposure to others to determine whether distinct information was combined.
Choose an outcome appropriate to the question
Accuracy can be defined as distance from a known correct number, correctness of a categorical answer, or quality of a probabilistic forecast over many resolved events. For a chair count, absolute error may be intuitive; for a prediction with a probability, a proper scoring rule can reward honest, calibrated forecasts. One headline accuracy score can hide which errors matter. A group that performs well on easy questions may still fail on the uncertain cases for which aggregation was needed.
A fair comparison includes several baselines. Compare the final group output with individual initial estimates, with a simple mean or median of those initial estimates, and with a plausible alternative rule. If a discussion improves the final answer relative to the average individual, but not relative to the mean of independent first answers, the discussion itself has not shown added value. It may still have other value, such as helping members understand a decision, but the accuracy claim should be narrower.
Distribution matters. A mean can be close to the truth because positive and negative errors cancel in one sample. Repeating across questions and groups shows whether the pattern persists. The researcher should report individual variance, group bias and the distribution of errors, not merely the best illustrative case. If one extreme estimate swings the mean, a median or trimmed mean may be a useful comparator. No one summary statistic is a universal aggregation benchmark.
Process outcomes can also matter. Did unique information appear in discussion? Did a participant revise toward a more accurate peer or toward a confident but mistaken majority? Did the final answer become more accurate because a better rule was used or because members learned from feedback? These questions require process data: initial and final judgments, timing of exposure, communication records or controlled information flows. A final number alone cannot identify the route by which it arose.
Independence, correlation and duplicated evidence
The common intuition that averaging “cancels errors” depends on error structure. If people make different mistakes, pooling can reduce noise. If they all share the same false premise, averaging preserves the bias. Suppose six people estimate hall capacity from the same outdated floor plan. Agreement may look reassuring, but it does not represent six independent confirmations. An analyst should trace sources, not just count voices.
Dependence can arise before or during the task. Participants may read the same website, belong to the same professional network, hear an early speaker or see an online ranking before reporting. Even without direct copying, common training or incentives can align errors. A dataset of final judgments rarely reveals all these dependencies. Recording first answers before interaction provides at least one baseline; source questionnaires or randomized information assignments can reveal more.
Lorenz and colleagues experimentally exposed participants to others’ estimates or to the group average in a numerical estimation task. Social influence narrowed the diversity of judgments and, in their setting, undermined the accuracy advantage that independent aggregation could provide. This is evidence that a particular form of exposure can create correlated errors. It is not proof that all interaction is harmful; groups can share valid evidence and improve when the structure allows correction.
Becker, Brackbill and Centola studied network structure in collective estimates and found that social influence can improve or impair group accuracy depending on how connections are arranged. A decentralized network need not have the same effect as one in which a central person drives everyone else. The relevant measure is not simply “how much social interaction occurred,” but who saw whom, when, and how much independent information survived the interaction.
Dependence changes the meaning of sample size. A panel with one hundred people who copy the same initial estimate can contain less independent information than a panel of ten people with distinct evidence. Statistical uncertainty computed as though all one hundred answers were independent will be too small. A valid aggregation report should say what is known about shared sources and avoid treating headcount as an automatic measure of epistemic diversity.
What an average does and does not reveal
An arithmetic mean is transparent and often useful. It treats each numerical estimate equally and can benefit from error cancellation. A median is more robust to extreme entries but may discard useful information from a knowledgeable outlier. A weighted mean can incorporate past accuracy, but weights estimated from a small, selective history may simply reward luck. Choosing a rule after seeing the correct answers risks overfitting the evaluation.
Mannes studied how people revise their own judgments after seeing a group judgment. In the studied tasks, many participants underweighted the group estimate relative to what would have improved their accuracy. This illustrates a different measurement question: whether an individual uses an aggregate well. It does not show that the aggregate was always superior, or that people should always move toward the group. The quality and independence of the group estimate have to be established first.
For an analyst, a useful test is to calculate the performance of several pre-specified aggregation rules on held-out questions. One can then ask whether any advantage persists when the topic, group or question difficulty changes. If a sophisticated weighting model barely outperforms a simple mean on the same data used to tune it, there is little evidence that the complexity improved aggregation. The relevant claim is out-of-sample value for a specified use, not mathematical elegance.
The mean can also mask minority information. If a unique expert knows a legal cap on hall occupancy, their number may be far below the others’ estimates. Treating it as an outlier could be dangerous. A measurement protocol should therefore record why estimates differ and identify whether a rule is intended for noisy quantitative guesses or for cases with asymmetrically distributed expertise. Aggregation is not synonymous with suppressing exceptions.
Majority voting and the single-question problem
When answers are categorical, majority vote is a familiar aggregation rule. It uses the number of supporters, not the distribution of confidence or the information behind their views. A majority can be wrong if its members share a misconception or if a minority has rare but decisive knowledge. With only one question, researchers cannot average performance over a long series to identify who was reliable. This is the single-question crowd-wisdom problem.
Prelec, Seung and McCoy proposed a “surprisingly popular” method. Participants answer a question and predict how common each answer will be. The rule favors an answer more popular than respondents collectively expected, rather than merely the most popular answer. Their experiments show circumstances in which this extra elicitation can uncover information held by a knowledgeable minority. The method relies on assumptions about participants’ predictions of others, and it is not a guarantee that a surprising minority is correct.
This design is useful for construct validity because it asks for two different kinds of response: one’s own answer and a forecast about the distribution of answers. The distinction can reveal whether a minority knows something that the majority does not know it knows. Yet asking for a second response adds burden and can produce misunderstanding. A researcher should test whether participants comprehend both questions and compare the resulting rule with a simple majority on relevant tasks.
The truth may also be contested or unavailable. A legal or ethical group choice is not always a fact question with one hidden answer. Voting can legitimately aggregate preferences or authority rather than factual information. Calling a vote “information aggregation” in that context would confuse different purposes. The outcome measure should match the claim: factual accuracy when facts can be checked, preference representation when the task is collective choice, and procedural fairness when legitimacy is at issue.
Incentives change what people contribute
A participant may know a useful fact and still have reasons not to share it. Speaking up can be costly, especially if a high-status member has already given a confident answer. A forecaster may have an incentive to look distinctive or to avoid reputational risk. A paid participant may optimize the scoring rule rather than the group goal. These factors do not make aggregation impossible; they mean that the elicitation procedure is part of the measurement.
Bazazi and colleagues manipulated whether participants in a group spatial-estimation task were rewarded for individual accuracy or collective performance, while varying access to a displayed group mean. In that study, individual payoff combined with social information increased conformity and reduced diversity and group accuracy; collective payoff preserved more diversity. The experiment demonstrates that incentive structure can change how social information is used under its conditions. It does not prove that paying every real-world team collectively will produce better decisions.
Notice that incentives and dependence interact. If an individual can improve their own estimate by copying a group mean, copying may be locally sensible even while making the final group mean less informative because the same answer is counted repeatedly. A measurement design that records only the final mean could miss this tradeoff. Recording individual first and second responses, payoff conditions and group outcomes makes the mechanism visible.
Truthful elicitation is also a concern. People may strategically shade estimates if rewards, rankings or attention depend on appearing certain. Proper scoring rules can help elicit probabilities in some settings, but they require comprehension and a well-defined resolved outcome. A small practice round can reveal confusion about the rule. The design should not assume that a numerical response is a direct readout of belief without checking how it was obtained.
Networks and timing matter
Aggregation may happen all at once, as when independent ballots are counted, or sequentially, as when people see earlier answers before forming their own. Sequential systems can spread useful information quickly. They can also give early errors undue influence. If the first three answers are visible, later respondents may treat them as evidence and make the distribution look more certain than it was.
Network structure can preserve or destroy distinct information. A decentralized arrangement may let several pockets of evidence develop before they are combined. A highly centralized arrangement may amplify one person’s mistake. But decentralization has costs too: useful corrections can take longer to reach everyone. The design choice depends on the task, the speed required and how errors are detected. Becker and colleagues’ results support testing network structure rather than treating “discussion” as one homogeneous condition.
Timing is an observable variable. Researchers can collect a first private estimate, assign different exposure conditions, collect a second estimate and then evaluate the final aggregate. This sequence distinguishes initial diversity from revisions caused by others. It can also reveal whether a later improvement came from learning a new piece of evidence or merely moving toward a majority. If no process data are collected, those mechanisms remain speculative.
In a workplace, an analogous design is to ask for independent written estimates before a meeting. The subsequent discussion can then focus on the reasons behind differences, rather than allowing the first speaker to define the plausible range. This is a proposed application of the experimental logic, not a universally validated meeting protocol. Its success should be evaluated on the organization’s real decisions, including costs, inclusion and downstream outcomes.
Distinguish aggregation from social learning
A social learning strategy concerns what an individual does with others’ behavior or reports: copy the majority, follow a successful person, sample when uncertain or ignore social cues. Information aggregation concerns the group-level function that turns many signals into one output. A person can use an individually beneficial strategy that damages the information value of the collective estimate. Conversely, a group may design a rule that benefits from individual independence even when no one learns socially during the task.
This distinction affects measurement. To study an individual’s strategy, researchers examine when that person changes an answer after seeing a particular social cue. To study aggregation, they compare group output with truth and with baselines while tracking the dependence among contributors. The same experiment can study both, but it should not substitute one outcome for the other. “Participants followed the group” says little about whether the group became more accurate.
The broad wisdom-of-crowds question asks whether a collective result can outperform individuals under some conditions. This entry asks how the result was assembled and how to verify that information was actually combined. The former can be answered descriptively with group and individual scores. The latter needs evidence about inputs, rules, incentives, exposure and sources of correlated error.
Common confounds and reporting requirements
First, the baseline may be too weak. Showing that a final group answer beats the average individual does not show it beats a simple mean of independent initial answers. Second, answers may not be independent. If several participants used the same source, the effective information pool is smaller than the headcount. Third, questions may be selected after results are known. A rule that looks strong on a favorable subset may fail on a broader set.
Fourth, expertise may be unevenly distributed. Equal weights can dilute the one person who knows a constraint, while authority-based weights may privilege a confident but uninformed speaker. Fifth, feedback can change the task across rounds. Later improvement may reflect learning the question type rather than the aggregation rule. Sixth, incentives may affect both truthful reporting and willingness to contribute unique evidence.
Seventh, accuracy may not be the only relevant outcome. An aggregator can be accurate on average while repeatedly missing rare, high-cost cases. The evaluation should disclose how errors are scored and whose losses count. Eighth, apparent consensus can be mistaken for confidence. A narrow spread of answers after people saw the same group mean may indicate conformity, not independent confirmation.
A strong report therefore provides the task and ground truth, recruitment and information assignment, initial individual answers, exposure sequence, aggregation rule, outcome metric, comparison baselines, and sensitivity to dependence and outliers. If the result rests on a proprietary algorithm, the report should still disclose enough about weights and evaluation to assess whether the method added value. The checklist is Institute synthesis from the cited methods and experiments, not a validated index.
What the evidence does not show
The cited research does not show that larger groups are always wiser, that averaging always cancels error, or that social interaction is always damaging. Lorenz and colleagues found harm from particular social exposure in estimation tasks, while Becker and colleagues showed that network structure can change the direction of social influence. Prelec and colleagues’ method improved certain single-question judgments, but it is not a universal replacement for majority vote. Mannes studied individual use of group advice, not a guarantee that every group estimate is superior.
The incentive experiment by Bazazi and colleagues concerns a specific spatial-estimation setting; it does not prescribe a pay scheme for every organization. The reviewed mechanisms do not settle ethical priorities or make one aggregate answer legitimate for a decision affecting people with unequal stakes. A true factual estimate, an inclusive decision and a fair process are related goals that require different measures.
The article also does not establish a single “collective intelligence score” that travels across topics, populations and institutions. An aggregation rule can work when errors are diverse and fail when sources are shared. Its value has to be tested against appropriate baselines and on new questions. The practical reporting checklist is a way to make the claim reviewable, not evidence that a particular organization has already improved.
Return to the hall
The organizer can still take the average of six estimates. Before calling it sixfold knowledge, they can ask each person for a private first answer and its basis. If one person has the current safety certificate, that evidence matters more than the numerical average. If all six relied on the same outdated plan, agreement adds little. If they used different observations and the group rule outperforms the independent baseline on repeated questions, the aggregation claim becomes stronger.
Information aggregation is a process with inputs, a rule and an outcome. Measuring it well means seeing where the inputs came from, whether they remained distinct, how incentives and networks changed them, and what the rule added beyond a simple comparator. The goal is not to romanticize crowds or to dismiss collaboration. It is to learn when a group genuinely combines information and when a polished collective answer merely repeats one assumption.
Research references
- Kameda, T., Toyokawa, W., & Tindale, R. S. (2022). Information aggregation and collective intelligence beyond the wisdom of crowds. Nature Reviews Psychology, 1, 345–357. View source
- Lorenz, J., Rauhut, H., Schweitzer, F., & Helbing, D. (2011). How social influence can undermine the wisdom of crowd effect. Proceedings of the National Academy of Sciences, 108(22), 9020–9025. View source
- Becker, J., Brackbill, D., & Centola, D. (2017). Network dynamics of social influence in the wisdom of crowds. Proceedings of the National Academy of Sciences, 114(26), E5070–E5076. View source
- Mannes, A. E. (2009). Are we wise about the wisdom of crowds? The use of group judgments in belief revision. Management Science, 55(8), 1267–1279. View source
- Prelec, D., Seung, H. S., & McCoy, J. (2017). A solution to the single-question crowd wisdom problem. Nature, 541, 532–535. View source
- Bazazi, S., von Zimmermann, J., Bahrami, B., & Richardson, D. (2019). Self-serving incentives impair collective decisions by increasing conformity. PLOS ONE, 14(11), e0224725. View source