John Snow Labs' NLU is a Python library for applying state-of-the-art text mining, directly on any dataframe, with a single line of code. As a facade of the award-winning Spark NLP library, it comes with hundreds of pretrained models in tens of languages - all production-grade, scalable, and trainable.

Take a look at our official Spark NLU page: https://nlu.johnsnowlabs.com/ for user documentation and examples

• Tokenization
• Trainable Word Segmentation
• Stop Words Removal
• Token Normalizer
• Document Normalizer
• Stemmer
• Lemmatizer
• NGrams
• Regex Matching
• Text Matching
• Chunking
• Date Matcher
• Sentence Detector
• Deep Sentence Detector (Deep learning)
• Dependency parsing (Labeled/unlabeled)
• Part-of-speech tagging
• Sentiment Detection (ML models)
• Spell Checker (ML and DL models)
• Word Embeddings (GloVe and Word2Vec)
• BERT Embeddings (TF Hub models)
• ELMO Embeddings (TF Hub models)
• ALBERT Embeddings (TF Hub models)
• XLNet Embeddings
• Universal Sentence Encoder (TF Hub models)
• BERT Sentence Embeddings (42 TF Hub models)
• Sentence Embeddings
• Chunk Embeddings
• Unsupervised keywords extraction
• Language Detection & Identification (up to 375 languages)
• Multi-class Sentiment analysis (Deep learning)
• Multi-label Sentiment analysis (Deep learning)
• Multi-class Text Classification (Deep learning)
• Neural Machine Translation
• Text-To-Text Transfer Transformer (Google T5)
• Named entity recognition (Deep learning)
• Easy TensorFlow integration
• GPU Support
• Full integration with Spark ML functions
• +710 pre-trained models in +192 languages!
• +450 pre-trained pipelines in +192 languages!
• Multi-lingual NER models: Arabic, Chinese, Danish, Dutch, English, Finnish, French, German, Hewbrew, Italian, Japanese, Korean, Norwegian, Persian, Polish, Portuguese, Russian, Spanish, Swedish, and Urdu.

To get your hands on the power of NLU, you just need to install it via pip and ensure Java 8 is installed and properly configured. Checkout Quickstart for more infos

pip install nlu pyspark==2.4.7

import nlu 
nlu.load('sentiment').predict('I love NLU! <3') 

Get 6 different embeddings in 1 line and use them for downstream data science tasks!

nlu.load('bert elmo albert xlnet glove use').predict('I love NLU! <3') 

NLU provides everything a data scientist might want to wish for in one line of code!

• NLU provides everything a data scientist might want to wish for in one line of code!
• 1000 + pre-trained models
• 100+ of the latest NLP word embeddings ( BERT, ELMO, ALBERT, XLNET, GLOVE, BIOBERT, ELECTRA, COVIDBERT) and different variations of them
• 50+ of the latest NLP sentence embeddings ( BERT, ELECTRA, USE) and different variations of them
• 100+ Classifiers (NER, POS, Emotion, Sarcasm, Questions, Spam)
• 300+ Supported Languages
• Summarize Text and Answer Questions with T5
• Labeled and Unlabeled Dependency parsing
• Various Text Cleaning and Pre-Processing methods like Stemming, Lemmatizing, Normalizing, Filtering, Cleaning pipelines and more

Choose the right tool for the right task! Whether you analyze movies or twitter, NLU has the right model for you!

• trec6 classifier
• trec10 classifier
• spam classifier
• fake news classifier
• emotion classifier
• cyberbullying classifier
• sarcasm classifier
• sentiment classifier for movies
• IMDB Movie Sentiment classifier
• Twitter sentiment classifier
• NER pretrained on ONTO notes
• NER trainer on CONLL
• Language classifier for 20 languages on the wiki 20 lang dataset.

Working with text data can sometimes be quite a dirty Job. NLU helps you keep your hands clean by providing lots of components that take away data engineering intensive tasks.

• Datetime Matcher
• Pattern Matcher
• Chunk Matcher
• Phrases Matcher
• Stopword Cleaners
• Pattern Cleaners
• Slang Cleaner

Checkout the NLU Namespace for everything that NLU has to offer!

• Pandas DataFrame and Series
• Spark DataFrames
• Modin with Ray backend
• Modin with Dask backend
• Numpy arrays
• Strings and lists of strings

Checkout the following notebooks for examples on how to work with NLU.

• Kaggle Twitter Airline Sentiment Analysis NLU demo
• Kaggle Twitter Airline Emotion Analysis NLU demo
• Kaggle Twitter COVID Sentiment Analysis NLU demo
• Kaggle Twitter COVID Emotion Analysis nlu demo

Checkout the following notebooks for examples on how to work with NLU.

• 2 class Finance News sentiment classifier training
• 2 class Reddit comment sentiment classifier training
• 2 class Apple Tweets sentiment classifier training
• 2 class IMDB Movie sentiment classifier training
• 2 class twitter classifier training

• 5 class WineEnthusiast Wine review classifier training
• 3 class Amazon Phone review classifier training
• 5 class Amazon Musical Instruments review classifier training
• 5 class Tripadvisor Hotel review classifier training
• 5 class Phone review classifier training

• Train Multi Label Classifier on E2E dataset Demo
• Train Multi Label Classifier on Stack Overflow Question Tags dataset Demo

• NER Training example

• POS Training example

• Sentence Similarity with Multiple Sentence Embeddings
• 6 Wordembeddings in 1 line with T-SNE plotting

• Kaggle Twitter Airline Sentiment Analysis NLU demo
• Kaggle Twitter Airline Emotion Analysis NLU demo
• Kaggle Twitter COVID Sentiment Analysis NLU demo
• Kaggle Twitter COVID Emotion Analysis nlu demo

The following are Collab examples which showcase each NLU component and some applications.

• NER pretrained on ONTO Notes
• NER pretrained on CONLL
• Tokenize, extract POS and NER in Chinese
• Tokenize, extract POS and NER in Korean
• Tokenize, extract POS and NER in Japanese
• Aspect based sentiment NER sentiment for restaurants

• POS pretrained on ANC dataset
• Tokenize, extract POS and NER in Chinese
• Tokenize, extract POS and NER in Korean
• Tokenize, extract POS and NER in Japanese

• Translate between 192+ languages with marian
• Try out the 18 Tasks like Summarization Question Answering and more on T5
• T5 Open and Closed Book question answering tutorial

• Unsupervised Keyword Extraction with YAKE
• Toxic Text Classifier
• Twitter Sentiment Classifier
• Movie Review Sentiment Classifier
• Sarcasm Classifier
• 50 Class Questions Classifier
• 300 Class Languages Classifier
• Fake News Classifier
• E2E Classifier
• Cyberbullying Classifier
• Spam Classifier
• Emotion Classifier

• BERT, ALBERT, ELMO, ELECTRA, XLNET, GLOVE at once with t-SNE plotting
• BERT Word Embeddings and t-SNE plotting
• ALBERT Word Embeddings and t-SNE plotting
• ELMO Word Embeddings and t-SNE plotting
• XLNET Word Embeddings and t-SNE plotting
• ELECTRA Word Embeddings and t-SNE plotting
• COVIDBERT Word Embeddings and t-SNE plotting
• BIOBERT Word Embeddings and t-SNE plotting
• GLOVE Word Embeddings and t-SNE plotting

• BERT Sentence Embeddings and t-SNE plotting
• ELECTRA Sentence Embeddings and t-SNE plotting
• USE Sentence Embeddings and t-SNE plotting

• BERT Sentence Embeddings and t-SNE plotting
• ELECTRA Sentence Embeddings and t-SNE plotting
• USE Sentence Embeddings and t-SNE plotting

• Untyped Dependency Parsing
• Typed Dependency Parsing

• Tokenization
• Stopwords removal
• Stemming
• Lemmatization
• Normalizing
• Spell checking
• Sentence Detecting
• Normalize documents

• N Gram
• Entity Chunking

• Date Matcher

• Ping us on Slack
• Post an issue on Github

• NLU different output levels Demo