Skip to content

State of the Art Language models and Classifier for Kannada, which is spoken predominantly by Kannada people in India, mainly in the state of Karnataka

License

Notifications You must be signed in to change notification settings

goru001/nlp-for-kannada

Repository files navigation

NLP for Kannada

This repository contains State of the Art Language models and Classifier for Kannada, which is spoken predominantly by Kannada people in India, mainly in the state of Karnataka.

The models trained here have been used in Natural Language Toolkit for Indic Languages (iNLTK)

Dataset

Created as part of this project

  1. Kannada Wikipedia Articles

  2. Kannada News Dataset

Open Source Datasets

  1. IndicNLP News Article Classification Dataset - Kannada

Results

Language Model Perplexity (on validation set)

Architecture/Dataset Kannada Wikipedia Articles
ULMFiT 70.10
TransformerXL 61.97

Classification Metrics

ULMFiT
Dataset Accuracy MCC Notebook to Reproduce results
IndicNLP News Article Classification Dataset - Kannada 98.87 98.30 Link

Visualizations

Word Embeddings
Architecture Visualization
ULMFiT Embeddings projection
TransformerXL Embeddings projection

Pretrained Models

Language Models

Download pretrained Language Model from here

Tokenizer

Trained tokenizer using Google's sentencepiece

Download the trained model and vocabulary from here

About

State of the Art Language models and Classifier for Kannada, which is spoken predominantly by Kannada people in India, mainly in the state of Karnataka

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published