History and Evolution of NLP: from the first Grammar Rules to Deep Neural Networks
You hit the office, open up VSCode, and yep, you’re tangled up with those super nested data structures again. So, you ask ChatGPT to figure out a function to navigate through this mess, perhaps also give the whole schema a revamp. And, you haven't even finished your first sip of coffee when—bam—it gives you the answer!
People have always dreamt of coding without writing the code, and now it’s actually happening. Looking at the mind-blowing capability of ChatGPT, it’s surprising to realize the field of Natural Language Processing (NLP) isn’t even a century old. It’s a young field, but definitely not a simple one. Over the years, scientists have experimented with numerous approaches to solve text-related tasks. Let’s dive in and explore these together now!
Early Rule-Based Systems: The Georgetown-IBM Experiment
Back in 1954, the Georgetown-IBM Experiment was held in New York to translate Russian to English.
This program has a vocabulary of 250 words and is based on six grammar rules. For example, one rule is, if the first code is 110, is the third code associated with the preceding complete word equal to 21? If so, reverse the order of appearance of words in output, otherwise, retain order.
This small demo brought a lot of optimism about the possibilities in the field of NLP. However, the limitations are also obvious. The hardware at that time was in its infancy, with limited speed and memory. The algorithms were unmanageable in realistic settings due to a phenomenon known as combinatorial explosion. And let’s not forget the complexity of human language, with its ambiguities and the need for common sense.
Rise of Statistical Modeling: Feature Engineering
The early rule-based systems heavily relied on linguists to design grammatical rules. However, after encountering limitations and facing realistic failures, scientists moved forward in a new direction. Frederick Jelinek, a scientist well known for ASR (Automatic Speech Recognition), once said "Every time I fire a linguist, the performance of the speech recognizer goes up". In the 80s, the paradigm shifted from “expert knowledge + rules” to “data + features”.
For example, the Hidden Markov Model (HMM) is a statistical model that is extensively used in speech recognition. Feature engineering is a crucial step before being able to use HMM at all. The raw speech signal is converted into a sequence of feature vectors that compactly represent the power spectrum of the signal. The quality of this feature extraction step directly affects model performance.
Even with the emergence of new advancements, statistical learning stands strong, continuing to be a significant driving force in the realm of NLP today.
The Deep Learning Wave: It’s all about Self-Learning!
Before the appearance of Deep Learning in 2015, NLP was mostly about structured prediction and feature engineering. The breakthroughs primarily come from how people curated smart and representative features. However, neural networks revolutionize the whole thing. Suddenly, it wasn’t about human-crafted features anymore. These network architectures could self-learn the embeddings, with no human intervention required! Let’s walk through a simple example of how neural networks work.
In the initial phase of training, embeddings are just randomly initialized.
During training, the neural network processes input data through multiple linear and/or non-linear layers. It produces an outcome, compares the network’s prediction to the true target values and calculates an error (loss function). This process is what’s called a forward pass through the network. Yet, without fine-tuning, the outcome at this stage doesn’t’ hold any practical value.
Backpropagation is then used to calculate the gradient of the loss with respect to each weight in the network, i.e. it computes how much each weight contributed to the error by moving backward from the last output layer to the first input layer. With the gradient calculated, we update the weight using this simple formula:
weight = weight - (learning rate × gradient)
This process is iterated numerous times across multiple epochs (the number of times you go through the entire dataset) until the error of the model reaches an acceptable level. With this well-trained model, the output can serve as embeddings for words!
It's this self-supervised nature that accounts for the prevalence of neural networks in almost all NLP models today. And what we’ve discussed is just scratching the surface. In reality, depending on the task, the architecture of neural networks can vary a lot. Let's walk through some of the commonly used architectures.
1. Fully Connected Networks (FCNs) / Dense Networks:
Dense networks consist of layers where each neuron is connected to every neuron in the nearby layers. It’s designed for the most basic classification and regression tasks.
2. Convolutional Neural Networks (CNNs):
Compared to dense networks, each neuron in a layer of convolutional neutral network is only connected to a local region of the input layer, typically via a small sliding window. It’s designed for image classification, object detection, and other computer vision tasks.
3. Recurrent Neural Networks (RNNs):
Recurrent neural networks use the same layer across different time steps—hence, the term ‘recurrent.’ It’s designed for sequential data like time series.
4. Long Short-Term Memory Networks (LSTMs):
LSTM is a special kind of RNNs capable of remembering and forgetting different things across time using “gates”, you can understand it as doors. If it opens, the old memories pass through and it remembers. If it closes, the old memories are blocked out so it forgets. It is also designed for sequential modeling and it mitigates the long-term computational problem in standard RNNs.
Large Datasets and Even Larger Models
The Transformer model was developed to overcome the limitations of previously mentioned models like RNN and LSTM. These models are sequential so it’s hard to parallelize across multiple processing units, limiting their training efficiency. Transformers allow for parallel processing of the entire sequence, making them highly efficient.
And there’s more; Transformers also employ a self-attention mechanism, allowing the model to assign different attention scores (or weights) to different parts of the input sequence.
Since around 2018, Transformer-based pre-trained models have become the standard. People focus less on architectural design. The focus has shifted less from architecture design and more to transfer learning—fine-tuning these pre-trained models using custom datasets. On the Huggingface website, BERT, RoBERTa, and DistilBERT are all Transformer models pre-trained with a large corpus of text ready to be called and finetuned.
More recently, it’s the era of LLM. Our beloved assistant, ChatGPT, which employs a Transformer architecture and is trained to predict the next word in a sequence of words (that’s how it answers your questions).
While the architecture remains unchanged, the focus has subtly shifted again—from transfer learning to, let’s say, prompt engineering. With an already intelligent large language model, the exploration is now more about how to craft prompts precisely to extract the desired outcome. The game has changed, and it’s all about asking the right questions.
Get Your Hands Dirty
To get started with NLP, you’ll often be pointed to hefty courses full of statistical models, neural network theories, and a mountain of math and computation. If you need help with NLP, feel free to get in touch with us and let's take you through the first steps into the fascinating world of NLP!
View certificate