Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
The Socioscope project establishes a comprehensive data pipeline for Large-Scale Qualitative Research (LSQR) focused on the food system, documenting interactions among various organizations and actors. By leveraging AI for analysis, the project has successfully collected and processed extensive multimedia data, resulting in a corpus of 686 documented cases across 31 countries, encompassing 1,430 hours of recordings and 12.6 million words of transcripts. This paper serves as a technical reference, detailing the methodologies, operational frameworks, and ethical considerations that underpin the Socioscope data pipeline, providing a foundation for future research in LSQR.
The Socioscope pipeline has generated a massive, analyzable dataset that redefines how qualitative research can scale in complex domains like the food system.
The Socioscope project is a pioneering effort in Large-Scale Qualitative Research (LSQR) collecting comparable, open-ended, multimedia field data on hundreds of cases and using AI to make the material analysable at scale. The domain studied is the food system. The entities documented are the organisations that act in it: farms, processors, distributors, retailers, restaurants; and, at meso level, the actors that shape their environment, such as municipalities, government programmes, banks, NGOs and universities. This paper provides the technical reference for how the resulting data Corpus was built and managed to enable AI-augmented analysis. It describes the data pipeline end to end: the systemic sampling frame; the transaction grid used to capture each initiative's relations within the food system; the social contract that rewards participating interviewees, aiming to sustain access; the operational chain from scouting to interviews, including their uploading, transcription, translation, quality control and curation; the provenance rules (originals are immutable, every transformation is logged); and the installation of equipment, personnel and processes, including ethics and GDPR compliance. In its first phase (2023-2026) the pipeline produced 686 documented cases from 31 countries: some 1,430 hours of recordings, about 450,000 speech turns, and 12.6 million words of transcript. We report costs, metrics, lessons learned and limitations, so that other teams can reuse, adapt, and improve the Socioscope methodology.