Research
The GRID Lab Research Program
Our research focuses on the study of judgment and decision making. An overview of the field can be found here.
Psychological Theory of Judgment and Choice.
Much psychological theory focuses on averages and central tendency. The variability of experiences in the world is vast, and the variability between people plays a substantial role in predicting and understanding judgments and choices. We have several research programs focused on developing theories that integrate concepts of variability into cognitive theories of information processing, perceptions of validity, and group level judgment accuracy.
Global-local Incompatibility
Perceptions of global risks are shaped by our personal experiences, which potentially contain little information about the risk we face. Global-local incompatibility is a statistical framework showing when variability in local environments is statistically incompatible with variability in global environments. In establishing the theory, we demonstrate that global-local incompatibility is present for perceptions of climate change.
- Broomell, S. B. (2020). Global–local incompatibility: The misperception of reliability in judgment regarding global variables. Cognitive Science, 44(4), e12831.
Open Science Material.
Global-local incompatibility can potentially be overcome by a large enough crowd of individuals. If a crowd facing a global risk contains individuals whose experience represents the full variability of the global risk, this crowd is more likely to be wise than any subgroup.
We found evidence for global-local incompatibility in:
Perceptions of the COVID-19 pandemic. Personal experiences with COVID-19 in the year 2020 varied greatly across the US. We analyzed public data on COVID-19 infection rates at the start of the pandemic and linked variability of infection rates across the US to variability in perceptions of the pandemic across survey participants.
- Broomell, S. B., & Kane, P. B. (2021). Perceiving a pandemic: Global–local incompatibility and COVID-19 superspreading events. Decision, 8(4), 227–236.
Open Science Material. - Broomell, S. B., & Chapman, G. B. (2021). Looking beyond cognition for risky decision making: COVID-19, the environment, and behavior. Journal of Applied Research in Memory and Cognition, 10(4), 512–516.
Perceptions of Tornado Season. Personal experiences from thunderstorms can shape perceptions of when tornadoes are likely to happen. In the American Southest, thunderstorm season and tornado season are not the same seasons. We found that survey respondents in the Southeast indicated their tornado season was similar to their thunderstorm season, as thunderstorms are more readily experienced compared to the atmospheric conditions for tornadoes.
Decompositions of Judgment
The accuracy of judgments can be understood by taking a measure of accuracy and decomposing it into its contributing component parts.
For a collection of expert judgments, the correlations between experts are likely to be due to the information available in the environment. Under these circumstances, differences between experts in how they use information plays a much smaller role in determining their final judgment. As correlations increase, less is gained from aggregating additional expert judgments.
- Broomell, S. B., & Budescu, D. V. (2009). Why are experts correlated? Decomposing correlations between judges. Psychometrika, 74(3), 531-553.
- Davis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014). When is a crowd wise? Decision, 1(2), 79–101.
- Davis-Stober, C. P., Budescu, D. V., Broomell, S. B., & Dana, J. (2015). The composition of optimally wise crowds. Decision Analysis, 12(3), 130-143.
Decision makers learn how to predict something from observations in the environment. If the environment is noisy, judgments across a group of people can be based on different mappings learned from their own idiosyncratic experiences, increasing judgment error within the group.
- Kane, P. B., & Broomell, S. B. (2020). Applications of the bias–variance decomposition to human forecasting. Journal of Mathematical Psychology, 98, 102417.
- Broomell, S. B., & Kane, P. B. (2021). Perceiving a pandemic: Global–local incompatibility and COVID-19 superspreading events. Decision, 8(4), 227–236.
Open Science Material.
Ordered Judgment Processes
This line of research is currently under development. We propose a new theory of judgment based on fusing operationally efficient algorithms from computer science with psychological theories of judgment to form a new theory of judgment that can dynamically react to changes in the environment.
The Role of Scientific Evidence in Judgment and Choice
The fringe of scientific knowledge is highly uncertain. Despite this uncertainty, the public still needs to make decisions based on the best evidence we have. Uncertain scientific evidence is the most difficult to communicate to the public. We have several research programs that focus on understanding and improving this process.
Public Perceptions of Scientific Evidence
Our research finds that the public may not perceive scientific uncertainty using the same terms as scientists, but instead through their own experiences with the predictions and results generated by science.
- Broomell, S. B., & Kane, P. B. (2017). Public perception and communication of scientific uncertainty. Journal of Experimental Psychology: General, 146(2), 286–304.
- Broomell, S. B., Budescu, D. V., & Por, H. H. (2015). Personal experience with climate change predicts intentions to act. Global environmental change, 32, 67-73.
- Broomell, S. B., Winkles, J. F., & Kane, P. B. (2017). The perception of daily temperatures as evidence of global warming. Weather, Climate, and Society, 9(3), 563-574.
- Dryden, R., Morgan, M. G., & Broomell, S. (2020). Lay detection of unusual patterns in the frequency of hurricanes. Weather, Climate, and Society, 12(3), 597-609.
- Anglin, S. M., Drummond Otten, C., & Broomell, S. B. (2023). Hypothesis Testing Preferences in Research Decision Making. Collabra: Psychology, 9(1), 73029.
Scientific communications may also include models, graphs, and data plots. We find that the public may graphically view the accuracy of predictive models differently than scientists.
Communication of Scientific Uncertainty
Scientific uncertainty is often assessed and captured using concepts from statistics, which are difficult to communicate clearly to the public without being overly technical. Verbal expressions of probability (e.g., likely) are commonly used as a way to express probability when an exact probability is not known. We have empirically tested the effectiveness of verbal probability and found that verbal probabilities do little to improve communication of uncertainty, and they may lead to an “illusion of communication.”
- Budescu, D. V., Broomell, S., & Por, H. H. (2009). Improving communication of uncertainty in the reports of the Intergovernmental Panel on Climate Change. Psychological science, 20(3), 299-308.
- Budescu, D. V., Por, H. H., Broomell, S. B., & Smithson, M. (2014). The interpretation of IPCC probabilistic statements around the world. Nature Climate Change, 4(6), 508-512.
- Harris, A. J., Por, H. H., & Broomell, S. B. (2017). Anchoring climate change communications. Climatic change, 140, 387-398.
Expert Judgments of Uncertain Outcomes
In contexts where personal experiences are likely to provide little information about a global risk, we can leverage a group of experts with experiences and observations that can span the variability of the risk.
- Broomell, S. B., & Davis-Stober, C. P. (2023). The strengths and weaknesses of crowds to address global problems. Perspectives on Psychological Science, 17456916231179152.
- Kane, P. B., Moyer, H., MacPherson, A., Papenburg, J., Ward, B. J., Broomell, S. B., & Kimmelman, J. (2020). Expert forecasts of COVID-19 vaccine development timelines. Journal of General Internal Medicine, 35, 3753-3755.
- Kane, P. B., Moyer, H., MacPherson, A., Papenburg, J., Ward, B. J., Broomell, S. B., & Kimmleman, J. (2022). Relationship between lay and expert perceptions of COVID-19 vaccine development timelines in Canada and USA. Plos one, 17(2), e0262740.
- Broomell, S. B., Wong-Parodi, G., Morss, R. E., & Demuth, J. L. (2020). Do we know our own tornado season? A psychological investigation of perceived tornado likelihood in the southeast united states. Weather, Climate, and Society, 12(4), 771-788.
Open Science Material.
Methodology for Studying Judgment and Choice
The study of judgment and choice faces unique challenges that require special attention to perform robust and generalizable modeling, data analysis, and statistical inference. We have several research programs that focus on developing, demonstrating, and testing robust methodology for the study of judgment and choice.
Information Aggregation
The Wisdom of the Crowd is based on the statistical benefits of averaging. Our research has defined when crowds may be wise along with approaches for squeezing a little more wisdom out of an aggregate of the crowd’s judgments.
- Huang, S., Broomell, S. B., & Golman, R. (2024). A hypothesis test algorithm for determining when weighting individual judgments reliably improves collective accuracy or just adds noise. Decision, 11(1), 7–34.
- Davis-Stober, C. P., Budescu, D. V., Dana, J., & Broomell, S. B. (2014). When is a crowd wise? Decision, 1(2), 79–101.
- Davis-Stober, C. P., Budescu, D. V., Broomell, S. B., & Dana, J. (2015). The composition of optimally wise crowds. Decision Analysis, 12(3), 130-143.
Compositional Data Analysis
A composition is a set of numbers that are contrained to sum to a constant value (such as probability or proportions). There are many types of variables that are measured in the psychological sciences that have this property. For example, judgments of the chances of a tornado in different seasons of the year should sum to 100% across the seasons. A composition contains dependencies between the measured variables that can be analyzed using compositional data analysis.
Methods for Comparing Choice Models and Estimating Their Parameters
Choice models are unique in their reliance on discrete observations from research participants. This limits the information that can be learned by a choice experiment, and the choice problems contained in an experiment determine the effectiveness of the inferences we hope to draw from our data.
- Broomell, S. B., Budescu, D. V., & Por, H. H. (2011). Pair-wise comparisons of multiple models. Judgment and Decision Making, 6(8), 821-831.
- Broomell, S. B., & Bhatia, S. (2014). Parameter recovery for decision modeling using choice data. Decision, 1(4), 252–274.
- Broomell, S. B., Sloman, S. J., Blaha, L. M., & Chelen, J. (2019). Interpreting model comparison requires understanding model-stimulus relationships. Computational Brain & Behavior, 2, 233-238.
- Sloman, S., Broomell, S., & Kusuma, T. (2020). Diagnosing pervasive issues with parameter estimation. In CogSci.
- Sloman, S. J., Oppenheimer, D. M., Broomell, S. B., & Shalizi, C. R. (2022). Characterizing the robustness of Bayesian adaptive experimental designs to active learning bias. arXiv preprint arXiv:2205.13698.
Directly Eliciting and Analyzing Uncertainty
We are developing methods for allowing participants to provide sets of answers on a survey. This can enable researchers to quickly and effectively understand uncertainty and ambiguity inherent in survey questions.
The data collected using interval-valued survey responses can be analyzed using a compositional data analysis framework.