Text Lemmatization Example with Spacy

      Lemmatization is a text normalization technique used in Natural Language Processing (NLP) and computational linguistics. Its primary purpose is to reduce words to their base or dictionary form, known as the "lemma." Unlike stemming, which focuses on heuristically removing common prefixes or suffixes, lemmatization employs linguistic analysis to ensure that the resulting word is a valid word found in a language's dictionary.

    In this blog post, we will explore lemmatization concept its application with Spacy library in Python. The tutorial covers:

1. The concept of lemmatization
2. Lemmatization in Python
   
3. Conclusion
   

     Let's get started.

 

The concept of lemmatization  

    Lemmatization is a text normalization technique used in Natural Language Processing (NLP) to reduce words to their base or dictionary form, known as the "lemma." Unlike stemming, which relies on heuristics to remove prefixes or suffixes, lemmatization considers the meaning and context of words, ensuring that the resulting word is a valid word found in a dictionary. Lemmatization aims to transform words into their most basic and meaningful form, the lemma. It involves linguistic analysis to consider factors like word inflections, tenses, and context.

Why We Need Lemmatization:

1. Meaning Preservation: Lemmatization retains the semantic meaning of words. It ensures that words are transformed to their dictionary form, which is crucial for understanding the text's meaning.
2. Data Consistency: In many NLP applications, consistency in words is essential. Lemmatization groups together related words, reducing variations and improving data consistency.
3. Language Understanding: Lemmatization assists in language understanding tasks, such as machine translation, sentiment analysis, and information retrieval. It ensures that the base form of words is used for analysis.
4. Higher Accuracy: Lemmatization is more accurate than stemming. It doesn't lead to the creation of non-standard words, which can occur in stemming.
5. Complex Languages: In languages with complex inflections, lemmatization is crucial for correctly reducing words to their base forms.
6. Text Generation: When generating text, lemmatization ensures that the words produced are meaningful and grammatically correct, making it useful in text generation tasks.

Here is the concept of lemmatization explained with example:
    - Original: "Running", "Runs", "Runner", "Ran", "Run"
    - Lemmatized: "Run"

    In this example, various forms of the word "run" are lemmatized to the common base form "run".

Lemmatization in Python

   In Python, we can use various libraries for lemmatization. In this tutorial, we use the Spacy library to perform lemmatization. Before we dive into the code, make sure you have installed Spacy library. You can use pip command to install it.

 

pip install spacy

python -m spacy download en_core_web_sm  

    In below example, we import the spacy and load its dataset. We provide a list of words to be lemmatized and apply lemmatization to each word in the list. Finally, we print original and stemmed words. 

 

import spacy

# Load the English language model

nlp = spacy.load("en_core_web_sm")

# Sample words

words = ["Agrees", "Agreed", "Agreeing", "Agree"]

# Lemmatize each token in the document

lemmatized_words = [token.lemma_ for word in words for token in nlp(word)]

# Display the original and lemmatized words

print("Original words:", words)

print("Lemmatized words:", lemmatized_words)

The output shows lemmatized words.

 

Original words: ['Agrees', 'Agreed', 'Agreeing', 'Agree'] Lemmatized words: ['agree', 'agreed', 'agree', 'agree'] 

 

To perform lemmatization for a given text you can use below example.

 

import spacy

# Load the English language model

nlp = spacy.load("en_core_web_sm")

# Sample text

text = "The cats are running and the dogs are barking loudly."

# Process the text with Spacy

doc = nlp(text)

# Lemmatize each token in the document

lemmatized_words = [token.lemma_ for token in doc]

# Display the original and lemmatized words

print("Original words:", [token.text for token in doc])

print("Lemmatized words:", lemmatized_words)

 

 

Original words: ['The', 'cats', 'are', 'running', 'and', 'the', 'dogs', 'are', 'barking', 

'loudly', '.'] Lemmatized words: ['the', 'cat', 'be', 'run', 'and', 'the', 'dog', 'be', 'bark', 'loudly', '.'] 

 

 

 

 Conclusion

 
    In summary, lemmatization is important in various NLP applications that require a deeper understanding of language, better text analysis, and the preservation of the semantic meaning of words. It ensures that the semantic meaning of words is preserved while reducing them to their most basic and meaningful forms.

 

 

References: