Join us for a Brown Bag Talk with Jack Blumenau, Associate Professor at UCL. His research examines the role of evidence in policymaking and the functioning of democratic political systems.
Abstract from the speaker:
Quantitative impact evaluations – studies which measure the effects of policy – play a central role in government decision-making, with many billions in public funds resting on their results. Recent literature demonstrating that quantitative research findings can be highly sensitive to the specific analysis decisions made by researchers is therefore concerning, as it threatens to undermine policymakers‘ confidence in evaluations. However, work examining this type of „analysis-dependence“ in policy evaluation is rare. We describe a set of linked many-analyst studies, where we coordinate a large number of social researchers to reanalyse 6 existing impact evaluations from UK government. In addition to allowing us to quantify the degree of analysis-dependence in existing impact evaluations, we also investigate the degree to which analysis-dependence is present in the analyses designed by large language models (LLMs). We use multiple models to independently generate analysis code for the same 6 government impact evaluations, allowing us to compare LLM-based heterogeneity to the heterogeneity present in human-led analyses. Our project promises to generate new evidence on the robustness of quantitative research that directly informs government decisions.
About the speaker:
Jack Blumenau is an Associate Professor in the Department of Political Science at University College London and a Senior Evaluation Advisor at the Evaluation Task Force in the UK Cabinet Office. His research focuses on two main areas: the use of evidence in policymaking and political behaviour in democratic systems.
He studies how policymakers incorporate research evidence into government decision-making and how impact evaluations can be made more reliable and effective. He is currently leading a UKRI-funded Metascience project aimed at improving the quality and reliability of government impact evaluations in the United Kingdom.
Bring your own lunch bag! Light pastries and drinks will be available in case you forget to bring it.
—
The Data Science Brown Bag Series is an informal and interactive gathering where participants bring their own brown bag lunch and engage in discussions on research and insights the field of data and computational social science (light pastries and drinks will be available if you forget your lunch bag!).
The series provides a platform for data enthusiasts, researchers, and practitioners to share their experiences, best practices, and emerging methodologies and research in using data science to analyze and understand social and political phenomena. The brown bag talk series is for anyone interested in data science and social science to network, learn, and share ideas in a casual and friendly setting.
