Get Started with NLP (Natural Language Processing)
Think about a scenario where we would like to predict the price of a house. We can frame it as a standard regression problem.
The features can be the house’s
total square footage
number of bedrooms and bathrooms
distance from the subway
etc.
All of them are numerical, so we can directly use them as independent variables. However, some landlords use vague descriptions like “large house”, “multiple rooms” and “convenient transportation”, making it hard to retrieve values for variables. Naturally, we would wish, that we could just use the text itself as input. And that is exactly what NLP is about.
In regression, if x = 3, it means there are three bedrooms. The value is a direct representation of the underlying variable. We aim to do the same for text, developing a unified way to convert unstructured text to numbers that are representative of the text content. Textual data can often be lengthy. To get a high-quality representation, we only want to focus on the most important parts, paying little attention to the things that appear in every sentence, etc. Techniques like this are called text preprocessing.
We will cover some commonly used approaches including Tokenization, Stemming, Lemmatization, Stopwords Removal, Part-of-Speech (POS) Tagging and Named Entity Recognition (NER).
After the ingredients are well chopped, what steps should we take to transform them into numbers? We will start with the basic count-based word embeddings, including One-hot Encoding, Bag of Words Model and TF-IDF, and more advanced prediction-based word embeddings that we actually use in practice like BERT.
Text Preprocessing
Tokenization
As said before, we want the wording embeddings to capture the meaning of the sentence. How to do that? Think about how the human brain understands a sentence: by reading through each word in order. We can do the same in NLP. Tokenization is the task of cutting a string into identifiable linguistic units that constitute a piece of language data (tokens), i.e. splitting the sentence into smaller chunks.
Three common token types are: words, characters, and subwords.
Word tokenization is simply splitting a sentence into individual words, typically using spaces as separators.
text = "Natural Language Processing is fun to learn."
tokens = text.split()
print(tokens)
# ['Natural', 'Language', 'Processing', 'is', 'fun', 'to', 'learn.']Notice in the result, that the period is together with the last word since they are not separated by blank space. To avoid this, we can further add a line to remove punctuations making use of regular expressions.
import re
import string
# Find punctuations and replace them with ""
tokens = [re.sub(f"[{string.punctuation}]", "", token) for token in tokens]For languages with no clear word boundaries like Chinese, it’s better to split by characters.
text = "NLP"
tokens = list(text)
print(tokens)
# ['N', 'L', 'P']Subword tokenization is something in between. It splits sentences into words and further splits each word into different parts that frequently occur in other words as well. For example, the word “unhappiness” can be tokenized into “un”, “happy” and “ness”. The prefix “un” means the opposite meaning, and can be found in many words like “unable”, “unknown”. The suffix “ness” indicates nouns, like “sadness”.
Let’s understand how this helps with an example. Suppose using word tokenization, we get three words: unhappy, unknown, sadness, and we store them as vocabulary. When a new out-of-vocabulary word appears, like happiness, we need to add one word to the vocabulary. However, if we do subword tokenization instead, we have:
word | subwords |
unhappy | “un”, “happy” |
unknown | “un”, “known” |
sadness | “sad”, “ness” |
The vocabulary is [“un”, “ness”, “happy”, “sad”, “known”].
When the new word “happiness” appears, instead of adding it to the vocabulary, it can be created by “happy” and “ness” from the vocabulary, so we don't need to store it. Now imagine a larger dataset, the number of unique words can be exceedingly large. Using subword tokenization will greatly reduce the vocabulary size and storage of course.
This is just one way of doing the splitting. Depending on the task, we can split into even smaller pieces, like a subword of two characters that frequently occur together (Byte-Pair Encoding, used by transformer models).
For subword tokenizer, building it from scratch can be complex. The good news is that there are many NLP packages available. Let's explore how to utilize them for various tokenization tasks.
NLTK for word tokenization.
import nltk
from nltk.tokenize import word_tokenize
nltk.download('punkt') # Download the Punkt tokenizer model
text = "Natural Language Processing is fun to learn."
tokens = word_tokenize(text)
print(tokens)
# ['Natural', 'Language', 'Processing', 'is', 'fun', 'to', 'learn', "."]SentencePiece for subword tokenization.
import sentencepiece as spm
# Train model using a large corpus
spm.SentencePieceTrainer.Train('--input=sample_data.txt --model_prefix=spm --vocab_size=50000')
# Load the pretrained model
sp = spm.SentencePieceProcessor(model_file='spm.model')
# Encode: Sentence -> Subword
text = "Tokenization is fascinating."
encoded_text = sp.EncodeAsPieces(text)
print(encoded_text)
# ['▁Token', 'ization', '▁is', '▁fascinating', '.']Stemming and Lemmatization
After splitting a sentence into a list of words, the next step is usually to normalize the text. The most common practice is actually to convert everything to lowercase, ignoring the distinction between “The” and “the” by using w.lower().
We always want to go further than this and strip off any other affixes, which is called stemming. For example, if we have the word “running”, after stemming it becomes “run”. The root of a word doesn’t have to be a real word, like “tries” becomes “tri” which is totally fine. By reducing a word to its root form, stemming saves a lot of memory. It also speeds up string search and comparison among the corpus.
Below is an example of using NLTK for stemming.
import nltk
from nltk.stem import PorterStemmer
from nltk.tokenize import word_tokenize
nltk.download('punkt')
ps = PorterStemmer()
sentence = "flies flying running runs runner ran"
words = word_tokenize(sentence)
stemmed_words = [ps.stem(word) for word in words]
print(stemmed_words)
# ['fli', 'fli', 'run', 'run', 'runner', 'ran']If we want to make sure the resulting root form is a valid word, instead of stemming, we should use lemmatization. It is particularly useful when we want to construct a vocabulary. Understandably, the additional checking and enforcing process makes the lemmatizer slower than the stemmers.
from nltk.stem import WordNetLemmatizer
nltk.download('wordnet')
lemmatizer = WordNetLemmatizer()
sentence = "flies flying running runs runner ran better best"
words = word_tokenize(sentence)
lemmatized_words = [lemmatizer.lemmatize(word) for word in words]
print(lemmatized_words)
# ['fly', 'flying', 'running', 'run', 'runner', 'ran', 'better', 'best']Stopwords Removal
When we read the sentence “The car is black.”, what really matters is just two words “car” and “black”. “The” and “is” are just grammar enforcements, they do not mean anything. We want the embeddings to focus more on words that actually convey something.
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
nltk.download('stopwords') # Downloading stopwords
nltk.download('punkt')
# Loading English stopwords
stop_words = set(stopwords.words('english'))
text = "Natural Language Processing is fascinating and has many applications."
words = word_tokenize(text)
filtered_words = [word for word in words if word.lower() not in stop_words]
print(filtered_words)
# ['Natural', 'Language', 'Processing', 'fascinating', 'many', 'applications', '.']Part-of-Speech (POS) Tagging
Part-of-speech tagging is the task of labeling each word in a sentence given the context, like if it is a noun, verb, adjective, adverb, etc. Why is it important? Think about the previous example, “The car is black.” which is reduced to “car” and “black”. If we don’t know their lexical categories, we would be wondering, is the car black? or is the black car? If we already know car is a noun and black is an adjective, the relationship will become clear.
Another use case of part-of-speech tagging is when a word possesses multiple meanings in one sentence. For example, “They refuse to permit us to obtain the refuse permit.”
The first refuse means reject, but the second refuse means trash. We would want to be able to distinguish them. A basic tokenizer won’t be able to distinguish them. That’s where POS comes in.
import nltk
from nltk.tokenize import word_tokenize
from nltk import pos_tag
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
sentence = "They refuse to permit us to obtain the refuse permit"
tokens = word_tokenize(sentence)
# Perform POS tagging
tags = pos_tag(tokens)
for word, tag in tags:
print(f"{word} ({tag})")
# They (PRP)
# refuse (VBP)
# to (TO)
# permit (VB)
# us (PRP)
# to (TO)
# obtain (VB)
# the (DT)
# refuse (NN)
# permit (NN)We can see the first refuse is tagged with VBP, meaning it’s a verb in non-3rd person singular present form. The second refuse is tagged with NN, meaning noun.
Apart from generating embeddings, POS is also important for text-to-speech tasks, since a word may be pronounced differently depending on the lexical category (refUSE or REfuse).
Named Entity Recognition (NER)
Named entity recognition is also a tagging task, but instead of tagging grammatical properties, it’s tagging the type of named entities (noun phrases that refer to specific types of individuals), including organization, person, location, date, time, etc.
The first motivation for using NER is similar to POS. In large texts, not all words are equally important. We usually are more interested in entities in a sentence, like who or what places this sentence is talking about. And NER allows us to retrieve this information.
NER also improves contextual understanding and thus leads to better word embeddings. Let’s use the first sentence from a news article as an example.
import nltk
from nltk.tokenize import word_tokenize
from nltk import pos_tag, ne_chunk
nltk.download('punkt')
nltk.download('averaged_perceptron_tagger')
nltk.download('maxent_ne_chunker')
nltk.download('words')
sentence = "WASHINGTON -- In the wake of a string of abuses by New York police officers in the 1990s, Loretta E. Lynch, the top federal prosecutor in Brooklyn, spoke forcefully about the pain of a broken trust that African-Americans felt and said the responsibility for repairing generations of miscommunication and mistrust fell to law enforcement."
tokens = word_tokenize(sentence)
tags = pos_tag(tokens)
# Perform NER
tree = ne_chunk(tags)
# Print the named entities
for subtree in tree.subtrees():
if subtree.label() != "S": # The top-level tree has label "S"
entity = " ".join([word for word, tag in subtree.leaves()])
print(f"{entity} ({subtree.label()})")
# WASHINGTON (GPE)
# New York (GPE)
# Loretta E. Lynch (PERSON)
# Brooklyn (GPE)From the result, we can see Washington, New York and Brooklyn are correctly identified as geo-political entity (GPE) and Loretta E. Lynch is identified as a person. Once named entities have been identified, we can then work on more advanced tasks like relation extraction.
Word Embeddings
Now we finished the preprocessing part, we can finally start to convert them to numbers.
One-hot Encoding
One-hot Encoding is the simplest way of converting sentences to vectors. It only considers if a word of the sentence exists in the vocabulary. If exists, then assign 1 as the value, otherwise, assign 0.
Let’s see an example. We have an ordered vocabulary [the, a, an, in, for, penny, pound]. The size of this vocabulary is 7. We want to encode the sentence “in for a penny, in for a pound”.
Vocabulary | the | a | an | in | for | penny | pound |
Exist or not | 0 | 1 | 0 | 1 | 1 | 1 | 1 |
The size of the vector should be the same as the vocabulary. The 1st element corresponds to the first word in the vocabulary, the word “the”. Since “the” is not contained in our sentence, we assign 0. Moving to the next word, “a”, we see it’s inside the sentence, so we assign 1. So on so forth.
The resulting vector should be x = (0, 1, 0, 1, 1, 1, 1).
Bag of Words (BoW) Model
The idea of BoW is very similar to one-hot encoding. The difference is that one-hot encoding only checks if the word exists or not, BoW also counts the number of occurrences of that word.
Vocabulary | the | a | an | in | for | penny | pound |
Count | 0 | 2 | 0 | 2 | 2 | 1 | 1 |
If we use BoW, the vector should be x = (0, 2, 0, 2, 2, 1, 1)
This is pretty simple. However, one fatal problem with this method is that we only remember the number of occurrences of words, but we don’t know in what order they occur in the sentence. To solve this problem, one solution is to engineer the vocabulary a bit. Instead of using single words as vocabulary (unigram), we can change each element in the vocabulary as a combination of words. For example, [“in for”, “for a”, “a penny”, …] is called bigram. We can further increase the number of words in each element, like trigram, and in general, n-grams where n is the number of words per element. Although n-grams partially solve the problem, we almost never use them in practice. Because increasing the size of the gram also increases the vocabulary size, the memory and sparsity of the vector will increase as well.
TF-IDF (Term Frequency-Inverse Document Frequency)
Consider the case where an entire article is treated as an observation. If we use BoW, the words being represented the most will very likely be “the”, “a”, “an”, etc. To prevent this from happening, a penalization must be added to reduce their final score, and that’s the overall idea of TF-IDF. Now let’s take a closer look at its components.
Term frequency measures how frequently a word appears in a document.
TF(t,d) = Number of times term t appears in document dTotal number of terms in document
Inverse document frequency measures the level of uniqueness of the word across all documents. If all documents contain it, the score will be low (e.g. “the”).
IDF(t) = log(Total number of documentsNumber of documents containing term t )
TFIDF(t,d)=TF(t,d) × IDF(t)
Here’s an example of calculating TF-IDF vectors using scikit-learn.
from sklearn.feature_extraction.text import TfidfVectorizer
documents = [
"""Apple Inc. is an American multinational technology company headquartered in Cupertino, California. It designs, manufactures, and markets mobile communication and media devices.""",
"""Unprecedented. Uncharted territory. A first in American history. Kevin McCarthy’s removal as speaker of the House of Representatives on Tuesday was a startling moment.""",
"""A party once known for its ruthless discipline is obviously unmanageable; Donald Trump piously bemoaned the infighting of a party he has done more than anyone to break."""
]
# Create the transform
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(documents)
# Retrieve TF-IDF values and show the results
tfidf_data = tfidf_matrix.toarray()
for doc, tfidf_values in zip(documents, tfidf_data):
print(f"Document: '{doc[:100]}...' has TF-IDF values: {tfidf_values}\n")BERT (Bidirectional Encoder Representations from Transformers)
Though count-based embeddings can focus on words with higher importance, they don’t really study the context. BERT, which you may heard of very often, standing for Bidirectional Encoder Representations from Transformers (a deep learning architecture), is a prediction-based embedding. Bidirectional means when processing a token, it considers its surrounding tokens from both sides whereas traditional models can only handle one direction, either from left to right or from right to left. In transformers architecture, there are both an encoder and a decoder. BERT only takes the encoder to process the input data and predicts a rich representation of it. We can understand it as a layer of mapping (or a function).
BERT is trained on a massive unlabeled dataset of 3.3 billion words. It is proven to be very useful for tasks including sentiment analysis, question-answering bot, text prediction, text generation and text summarization, etc. Below is an example of how to use BERT with Huggingface.
from transformers import BertTokenizer, BertModel
import torch
# Load BERT tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertModel.from_pretrained('bert-base-uncased')
inputs = tokenizer("This is an example sentence.", return_tensors="pt")
# Get BERT embeddings
with torch.no_grad():
outputs = model(**inputs)
# Extract embeddings
last_hidden_states = outputs.last_hidden_state
cls_embedding = last_hidden_states[0][0]
print(cls_embedding)While BERT embeddings are effective in general, for domain-specific tasks, it’s better to use fine-tuned domain-specific models. For example, we have:
Model Name | Finetune Dataset |
financial news and texts | |
social media posts and messages related to crypto | |
legislation, court cases and contracts | |
clinical notes |
To use a fine-tuned BERT model, we can load the model and its tokenizer by calling "username/model_name". If you are still unhappy about the performance, you can consider finetuning it with your own labeled dataset. Check out our post (link) to learn more about it!
Get Your Hands Dirty
If you need help with your first NLP project or use case, feel free to get in touch with us and let's take you through the first steps into the fascinating world of NLP!
View certificate