Search papers, labs, and topics across Lattice.
This study systematically evaluates the impact of different preprocessing techniques on sentiment analysis of Twitter datasets, revealing that the order of application significantly influences model performance. The researchers found that tokenization is the most impactful technique, while spelling correction has minimal effect, and that stemming and stop-word removal can be used interchangeably, with a preference for retaining negation during stop-word removal. By establishing an optimal sequence of preprocessing steps鈥攖okenization, text cleaning, stemming, and stop-word removal鈥攖his work offers a streamlined approach for practitioners to enhance model efficiency and output quality.
Tokenization outperforms spelling correction in sentiment analysis preprocessing, reshaping how we prioritize text processing techniques.
Sentiment analysis in Twitter datasets is important because it enables monitoring public opinion on products and analysis of political and social movements. One critical step is preprocessing: the automated processing of text for machine learning algorithms. Preprocessing plays a critical role in reducing noise and improving efficiency. However, little research has systematically examined the order in which preprocessing techniques are implemented. We find that, when accounting for order, spelling correction is the least impactful preprocessing technique, whereas tokenisation is the most impactful. Stemming and stop-word removal are interchangeable, and it is better to remove stop words without removing negation. The best order for applying the preprocessing techniques was tokenisation, text cleaning, stemming, and then stopword removal. Our results provide a systematic approach for practitioners to deploy preprocessing to improve model output without the costly preprocessing exploratory phase.
