Techniques for text preprocessing in NLP

#remove words and digits
data['text'] = data['text'].apply(lambda x: re.sub('W*dw*','',x))

now observe the changes in the mail.

5) Remove Stopwords

Stopwords are the most commonly occurring words in a text which do not provide any valuable information. stopwords like they, there, this, where, etc are some of the stopwords. NLTK library is a common library that is used to remove stopwords and include approximately 180 stopwords which it removes. If we want to add any new word to a set of words then it is easy using the add method.

In our example, we want to remove the subject words from every mail so we will add them to stopwords and HTTP to remove web links.

#remove stopwords
from nltk.corpus import stopwords
stop_words = set(stopwords.words('english'))
stop_words.add('subject')
stop_words.add('http')
def remove_stopwords(text):
    return " ".join([word for word in str(text).split() if word not in stop_words])
data['text'] = data['text'].apply(lambda x: remove_stopwords(x))

Now the email text will be smaller because all stopwords will be removed.

here we have implemented a custom function that will split each word from the text and check whether it is a stopword or not. If not then pass as it is in string and if stopword then removes it.

6) Rephrase text

we may need to modify some text or change the pattern to a particular string which makes it easy to identify like we can match the pattern of email ids and change it to string like email address. In our example, we will modify the URL address and email address.

#email-id
df['text'] = df['text'].apply(lambda x: re.sub('b[w-.]+?@w+?.w{2,4}b', 'emailadd',x))
#url
df['text'] = df['text'].apply(lambda x:re.sub('(http[s]?S+)|(w+.[A-Za-z]{2,4}S*)', 'urladd', x))

you may not see any change in mails because we have already removed HTTP and punctuations before, But if you use the technique before applying the above preprocessing techniques, we can see the changes.

If you want to remove the URL then you can simply change the text to an empty string and it will replace fine. And you may also have different patterns which you need to look at and take care of. I think now you are capable to recognize the power of the regular expression library. isn’t it?

7) Stemming and Lemmatization

Stemming is a process to reduce the word to its root stem for example run, running, runs, runed derived from the same word as run. basically stemming do is remove the prefix or suffix from word like ing, s, es, etc. NLTK library is used to stem the words. The stemming technique is not used for production purposes because it is not so efficient technique and most of the time it stems the unwanted words. So, to solve the problem another technique came into the market as Lemmatization. there are various types of stemming algorithms like porter stemmer, snowball stemmer. Porter stemmer is widely used present in the NLTK library.

#stemming
from nltk.stem import PorterStemmer
stemmer = PorterStemmer()
def stem_words(text):
    return " ".join([stemmer.stem(word) for word in text.split()])
df["text"] = df["text"].apply(lambda x: stem_words(x))

we can see the words in the text are stemmed and some of the words it has stemmed which is not required, that’s only the disadvantage of this.

Lemmatization

Lemmatization is similar to stemming, used to stem the words into root word but differs in working. Actually, Lemmatization is a systematic way to reduce the words into their lemma by matching them with a language dictionary.

from nltk.stem import WordNetLemmatizer
lemmatizer = WordNetLemmatizer()
def lemmatize_words(text):
    return " ".join([lemmatizer.lemmatize(word) for word in text.split()])
df["text"] = df["text"].apply(lambda text: lemmatize_words(text))