Using Imperfect Surrogates for Valid Downstream Analyses: Design-based Supervised Learning for Social Science Applications of Large Language Models
Dec 10, 2023
Speakers
Naoki Egami
Speaker · 0 followers
Musashi Hinck
Speaker · 0 followers
Brandon Stewart
Speaker · 0 followers
Hanying Wei
Speaker · 0 followers
About
In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. The recent advancements in large language models (LLMs) can lower costs for CSS research by annotating documents cheaply at scale, but such surrogate labels are often imperfect and biased. We present a new algorithm for using outputs from LLMs for downstream statistical analyses while guaranteeing statistical properties—like asymptotic unbiasedness and proper uncertainty quantification—which are fundamental to CSS research. We show that direct use of LLM-predicted surrogate labels in downstream statistical analyses leads to substantial bias and invalid confidence intervals, even with high surrogate accuracy of 80-90%. To address this, we build on debiased machine learning to propose the design-based semi-supervised learning (DSL) estimator. DSL employs a doubly-robust procedure to combine surrogate labels with a few gold-standard labels. Our approach guarantees valid inference for downstream statistical analyses, even when surrogates are arbitrarily biased, without requiring stringent assumptions, by controlling the probability of sampling documents for gold-standard labeling. Both our theoretical analysis and experimental results show that DSL provides valid statistical inference while achieving root mean squared errors comparable to existing alternatives that focus only on prediction without statistical guarantees.In computational social science (CSS), researchers analyze documents to explain social and political phenomena. In most scenarios, CSS researchers first obtain labels for documents and then explain labels using interpretable regression analyses in the second step. The recent advancements in large language models (LLMs) can lower costs for CSS research by annotating documents cheaply at scale, but such surrogate labels are often imperfect and biased. We present a new algorithm for using outputs f…
Organizer
NeurIPS 2023
Account · 624 followers
Like the format? Trust SlidesLive to capture your next event!
Professional recording and live streaming, delivered globally.
Sharing
Recommended Videos
Presentations on similar topic, category or speaker
PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Closing remarks from organizers
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
![[Re] Replication study of SubgraphX: On Explainability of Graph Neural Networks via Subgraph Explorations](https://ma.slideslive.com/library/presentations/39010576/thumbnail/re-replication-study-of-subgraphx-on-explainability-of-graph-neural-networks-via-subgraph-explorations_GJrX70_small.png)
[Re] Replication study of SubgraphX: On Explainability of Graph Neural Networks via Subgraph Explorations
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
DreamHuman: Animatable 3D Avatars from Text
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
The Surprising Effectiveness of Diffusion Models for Optical Flow and Monocular Depth Estimation
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%
Natural Language Systematicity from a Constraint on Excess Entropy
Watch later
Total of 0 viewers voted for saving the presentation to eternal vault which is 0.0%