Abstract:
Automatic analyzing and extracting useful information from the noisy social media content are currently getting
attention from the research community. It is common to find people easily mixing their native language along with the
English language to express their thoughts in social media, using Unicode characters or the Unicode characters written
in Roman Scripts. Thus these types of noisy code-mixed text are characterized by a high percentage of spelling mistakes
with phonetic typing, wordplay, creative spelling, abbreviations, Meta tags, and so on. Identification of languages at
word level become a necessary part for analyzing the noisy content in social media. It would be used as an intimidate
language identifier for chatbot application by using the native languages. For this study we used Sinhala-English codemixed
text from social media. Natural Language Processing (NLP) and Machine Learning (ML) technologies are used
to identify the language tags at the word level. A novel approach proposed for this system implemented is machine
learning classifier based on features such as Sinhala Unicode characters written in Roman scripts, dictionaries, and
term frequency. Different machine learning classifiers such as Support Vector Machines (SVM), Naive Bayes, Logistic
Regression, Random Forest and Decision Trees were used in the evaluation process. Among them, the highest accuracy
of 90.5% was obtained when using Random Forest classifier
