Text Augmentation in NLP
What is text augmentation?
To improve the performance of an NLP task, the emphasis is often overly placed on the algorithm design. While it is indeed important, everything starts with a good dataset. In reality, obtaining a well-labeled corpus that is large enough for training is extremely hard. The most common solution is to perform reasonable transformations on samples from the existing dataset, without changing the label, to artificially create a larger dataset. This process is called Text Augmentation.
Why should we use it?
Other than increasing dataset size, text augmentation has the following advantages.
By transforming text in various ways, the model can get access to a wider range of linguistic expressions, leading to an increase in the model’s ability to generalize on unseen real-world data.
When the training sample is limited, the model may learn noises from the dataset, instead of the underlying language pattern. By introducing a larger corpus, the risk of overfitting is reduced.
Real-world data is far from perfect. For example, when we input prompts to ChatGPT, sometimes we mean to say “translate this to English”, but what we actually type might be “translat eto engish”. By transforming text to include typos and grammatical errors, the model can better cope with applied situations.
Sometimes, we need to train the model to perform better in specific fields, like medical or legal. By transforming texts using domain-specific vocabulary and expression, the performance can be optimized toward the intended direction.
Lastly, for some classification tasks, samples associated with certain labels may be scarce. By augmenting them, we can improve the balance of the dataset.
Simple Text Augmentation Techniques
Synonym Replacement
Lexicon-based
This method is to randomly select a word from a sentence, replace it with its synonym found in a lexicon. The ‘wordnet’ lexicon from nltk is commonly used as a lexicon.
import nltk
from nltk.corpus import wordnet
import random
nltk.download('wordnet')
nltk.download('averaged_perceptron_tagger')
def synonym_replacement_lexicon(sentence, n):
words = sentence.split()
new_words = words.copy()
random_word_list = list(set([word for word in words if wordnet.synsets(word)]))
random.shuffle(random_word_list)
num_replaced = 0
for random_word in random_word_list:
synonyms = set()
for syn in wordnet.synsets(random_word):
for lemma in syn.lemmas():
synonyms.add(lemma.name())
if len(synonyms) > 0:
synonym = random.choice(list(synonyms))
new_words = [synonym if word == random_word else word for word in new_words]
num_replaced += 1
if num_replaced >= n:
break
sentence = ' '.join(new_words)
return sentence
# Example
original_sentence = "The quick brown fox jumps over the lazy dog"
augmented_sentence = synonym_replacement_lexicon(original_sentence, 2) # Replace 2 words
print(augmented_sentence)
'''
The quick chocolate-brown fox jumps over the indolent dog
The immediate brown bedevil jumps over the lazy dog
The quick brown bedevil leap_out over the lazy dog
'''Embedding-based
Instead of searching for synonyms from a lexicon, this approach searches for similar embeddings, which may result in words that are not synonyms but semantically similar.
import gensim.downloader as api
from gensim.models import Word2Vec
import random
# Load pre-trained word vectors
model = api.load('glove-wiki-gigaword-100')
def synonym_replacement_embedding(sentence, n):
words = sentence.split()
new_words = words.copy()
random_word_list = list(set([word for word in words if word in model]))
random.shuffle(random_word_list)
num_replaced = 0
for random_word in random_word_list:
synonyms = model.most_similar(positive=[random_word], topn=5)
synonym = random.choice(synonyms)[0]
new_words = [synonym if word == random_word else word for word in new_words]
num_replaced += 1
if num_replaced >= n:
break
sentence = ' '.join(new_words)
return sentence
# Example
original_sentence = "The quick brown fox jumps over the lazy dog"
augmented_sentence = synonym_replacement_embedding(original_sentence, 2) # Replace 2 words
print(augmented_sentence)
'''
The fast brown nbc jumps over the lazy dog
The give brown fox jumps up the lazy dog
The quick brown television jumps last the lazy dog
The quick brown fox jumps up the lazy puppy
The quick brown fox climb over the lazy dogs
The fast brown fox jumps out the lazy dog
'''Out of the six randomly generated sentences, only half of them make sense. That is why sometimes text augmentation without curation will decrease the model performance.
Random Insertion
This technique randomly inserts words into the sentence while trying to maintain the original meaning.
def random_insertion(sentence, n):
words = sentence.split()
for _ in range(n):
synonyms = []
word_to_replace = random.choice(words)
for syn in wordnet.synsets(word_to_replace):
for lemma in syn.lemmas():
synonyms.append(lemma.name())
if len(synonyms) > 0:
synonym = random.choice(synonyms)
index = random.randint(0, len(words))
words.insert(index, synonym)
return ' '.join(words)
# Example
augmented_sentence = random_insertion(original_sentence, 2) # Insert 2 words
print(augmented_sentence)
'''
The quick Brown brown fox jumps over the lazy dog
The quick quick brown fox jumps over the flying lazy dog
The quick brown fox jumps fast over brown the lazy dog
'''Random Deletion
This technique randomly removes words from the sentence with a given probability while trying to maintain the original meaning.
def random_deletion(sentence, p):
words = sentence.split()
if len(words) == 1:
return sentence
new_words = []
for word in words:
r = random.uniform(0, 1)
if r > p:
new_words.append(word)
if len(new_words) == 0:
return random.choice(words)
return ' '.join(new_words)
# Example usage
augmented_sentence = random_deletion(original_sentence, 0.25) # 25% probability of deletion per word
print(augmented_sentence)
'''
quick brown fox over the lazy dog
The quick brown jumps over the lazy
The quick fox jumps lazy dog
'''Introducing Typos
Fuzzy Matching
There are many ways to introduce typos. The most common one is based on a QWERTY keyboard. For example, when typing ‘a’, we may accidentally press ‘q/w/s/z’. So we can identify words containing ‘a’ and randomly replace the character with any character from ‘q/w/s/z’.
import random
def introduce_typo(word):
"""Introduces a typo based on neighboring keys on a QWERTY keyboard."""
qwerty_keyboard = {
'a': 'qwsz',
'b': 'vghn',
'c': 'xdfv',
'd': 'swerfxc',
'e': 'wrsdf',
'f': 'ertdgcv',
'g': 'rtyfhvb',
'h': 'tyugjbn',
'i': 'uojk',
'j': 'yuighknm',
'k': 'iujolm',
'l': 'iopk',
'm': 'jkn',
'n': 'bhjm',
'o': 'ipkl',
'p': 'oil',
'q': 'wa',
'r': 'edtf',
's': 'awedxz',
't': 'rfgy',
'u': 'yihj',
'v': 'cfgb',
'w': 'qase',
'x': 'zsdc',
'y': 'tghu',
'z': 'asx',
}
# Pick a random character from the word
char_idx = random.randint(0, len(word)-1)
char = word[char_idx].lower() # Convert to lowercase for the dictionary
# Find neighboring keys for the chosen character
neighbors = qwerty_keyboard.get(char, char)
# Replace the character with one of its neighbors
typo_char = random.choice(neighbors)
if word[char_idx].isupper():
typo_char = typo_char.upper() # Convert back to uppercase if needed
typo_word = word[:char_idx] + typo_char + word[char_idx+1:]
return typo_word
def typo_augmentation(sentence, n):
words = sentence.split()
new_words = words.copy()
for _ in range(n):
random_word_idx = random.randint(0, len(words) - 1)
new_words[random_word_idx] = introduce_typo(new_words[random_word_idx])
return ' '.join(new_words)
# Example
original_sentence = "The quick brown fox jumps over the lazy dog"
augmented_sentence = typo_augmentation(original_sentence, 2) # Introduce typos in 2 words
print(augmented_sentence)
'''
The quick brown eox jumps over the lazy dog
Ghe quick bdown fox jumps over the lazy dog
The quick broen fox jumps over the lazy eog
The quock brown fox jumps over tte lazy dog
'''Random Swap
Another common typo is to write alphabets in the wrong order, like “teh” instead of “the”.
import random
def introduce_order_swap_typo(word):
"""Introduces a typo in a word by swapping two adjacent characters."""
if len(word) < 2:
return word
# Choose a random position to swap, ensuring it's not the last character
swap_pos = random.randint(0, len(word) - 2)
# Swap the characters
swapped_word = (
word[:swap_pos] +
word[swap_pos + 1] +
word[swap_pos] +
word[swap_pos + 2:]
)
return swapped_word
def typo_augmentation(sentence, n):
words = sentence.split()
new_words = words.copy()
for _ in range(n):
random_word_idx = random.randint(0, len(words) - 1)
new_words[random_word_idx] = introduce_order_swap_typo(new_words[random_word_idx])
return ' '.join(new_words)
# Example
original_sentence = "The quick brown fox jumps over the lazy dog"
augmented_sentence = typo_augmentation(original_sentence, 2) # Introduce order swap typos in 2 words
print(augmented_sentence)
'''
The quick rbown fox jumps over the layz dog
hTe quick brown fox jumps ovre the lazy dog
The quick brown fox jumps voer the layz dog
'''Generative Models
Unlike the above-mentioned rule-based techniques, generative models utilize machine learning or deep learning architectures to generate new texts that are semantically similar to the original text. Results from this approach will also be more robust than the simple ones.
The simplest way as of today is to use ChatGPT to do it.
Alternatively, there are available Python libraries to perform similar tasks like nlpaug and TextAttack.
# !pip install nlpaug transformers
import nlpaug.augmenter.word as naw
# Initialize the BERT augmenter
aug = naw.ContextualWordEmbsAug(
model_path='bert-base-uncased', action="substitute")
original_sentence = "The quick brown fox jumps over the lazy dog"
# Generate augmented sentences
augmented_sentences = [aug.augment(original_sentence) for _ in range(3)]
for i, augmented_sentence in enumerate(augmented_sentences, 1):
print(f"Augmented Sentence {i}: {augmented_sentence}")
'''
Augmented Sentence 1: ['the white brown fox talked over this lazy dog']
Augmented Sentence 2: ['his old brown fox jumps over her lazy dog']
Augmented Sentence 3: ['the quick black fox jumps to the lazy boy']
'''Summary
In this diagram, we summarized commonly used text augmentation approaches and available libraries. It is important to choose appropriate methods based on the nature of the task. For example, when training a model to interact with humans, it is more reasonable to focus on introducing typos instead of synonym substitution.
We are only scratching the surface...
If you're interested in learning more about how to implement your NLP use case, feel free to reach out to us by leaving your contact information!
View certificate