Search papers, labs, and topics across Lattice.
The paper introduces the Social Media Data Toolkit (SMDT), a Python framework for standardizing, anonymizing, and enriching heterogeneous social network datasets obtained from diverse platforms. SMDT unifies data into a generic schema (Communities, Accounts, Posts, Actions, Entities) and includes configurable anonymization and enrichment modules leveraging LLMs and network analysis tools. The toolkit's utility is demonstrated via four case studies, showcasing its ability to facilitate cross-platform analysis while addressing data heterogeneity and privacy concerns.
Stop wrestling with messy social media datasets: this toolkit streamlines standardization, anonymization, and enrichment, unlocking cross-platform insights with ease.
The rapid diversification of social media platforms and the increasing restrictions on official APIs have significantly complicated cross-platform analysis. Researchers are often forced to rely on heterogeneous datasets obtained through web scraping and historical archives; however they often lack structural consistency. Prior to conducting cross-platform social media analyses, one needs to answer three critical questions: (1) What makes platforms different and similar? (2) How were the datasets collected? (3) How can we align the datasets of different platforms to conduct fair analyses? To address these questions, we introduce the Social Media Data Toolkit (\projectname{}), a comprehensive Python framework designed for the standardization, anonymization, and enrichment of social network datasets. \projectname{} unifies diverse data structures into a generic schema comprising Communities, Accounts, Posts, Actions, and Entities to facilitate multi-platform research. The framework features a configurable anonymization module to secure Personally Identifiable Information (PII) and an extendable enrichment layer that integrates Large Language Models (LLMs) and network analysis tools for downstream tasks such as stance detection and toxicity scoring without creating codebase for different datasets. We demonstrate the versatility of \projectname{} through four case studies spanning from textual analysis of the content to network analysis across platforms. To offer reproducible social media research, \projectname{} is released as an open-source tool featuring detailed documentation and practical guides for researchers at any skill-level. It can be accessed at github.com/ViralLab/SMDT and varollab.com/SMDT.
