Sentiment Analysis of Roman Urdu Social Media Text Using Multilingual Transformer Models

The intensive growth of the social media has changed the way individuals communicate and share their views. In Pakistan this term is mostly used in Roman Urdu, a nonstandard form of Urdu, it is written with the use of the Latin alphabet, and is mostly used in the media like twitter/X, facebook and YouTube. Although this is widely used, the automatic methods of sentiment analysis in Roman Urdu are at a dismal state of development. Current systems are mostly developed to deal with English or formal Urdu script, which has a major gap in NLP research on this script. The processing tasks of Roman Urdu are characterized by such peculiarities as nonstandardized spelling rules, common code-switching of Urdu with English and local languages, informal grammar, and lack of sufficiently large labeled data. Most previous studies have been based on traditional machine learning models and shallow word embeddings with minimal attempt made on transformer-based approaches. This study will be a transformer-based sentiment classifier of Roman Urdu social media text. A collection of Roman Urdu data is created by appending a Roman dataset which is available publicly. This baseline was enriched with newly annotated YouTube video comments to construct a balanced corpus. Inter-annotator agreement was verified using Cohen's Kappa on a stratified validation subsample of 150 instances. Three multilingual transformers mBERT, XLM-RoBERTa, and UrduBERT are fine-tuned and assessed based on Macro F1-Score. The paper presents the first transformer-based Roman Urdu sentiment analysis benchmark, an annotated dataset, and a comparative evaluation framework that will assist future studies and practical uses such as monitoring social media and analysing the opinion of people.