3. Term Frequency Inverse Document Frequency a.k.a TF-IDF

"TF-IDF is a commonly used weighting technique that assigns weights reflecting the importance of a word to a document. The basis of this technique is the idea that if a word appears frequently across all documents, it is less likely to hold significant information about any specific document. On the other hand, words that appear frequently in one or a few documents and rarely across all documents are considered to have specific importance and should be assigned higher weights.

The mathematical expression of tf-idf (in one of the many forms) is:

$$ tfidf = frequency_{t,d} * log ( \frac{total \ documents}{total \ documents \ containing \ the \ term} ) $$

It is simply the multiplication of the number of times a word appears in a document by the logarithm of the total number of documents divided by the number of documents that contain the word

Intuitively, high-frequency words that appear in nearly all documents are weighted by the logarithm of 1 (log1), resulting in a weight of zero. Conversely, words with high frequency within a specific document and low frequency across the corpus will have a higher weight. Let's see an example using our small corpus above.

from sklearn.feature_extraction.text import TfidfVectorizer

tfidf_vectorizer = TfidfVectorizer( analyzer='word',          # Word level vectorizer
                                    lowercase=True,           # Lowercase the text
                                    tokenizer= word_tokenize) # Use this tokenizer)

tfidf_vectorizer.fit(corpus)
tfidf_features = tfidf_vectorizer.transform(corpus).toarray()

tfidf_df = pd.DataFrame(tfidf_features, columns=tfidf_vectorizer.get_feature_names_out())
tfidf_df

Notice that we now have weights computed. Because we are using a small corpus, the disparity of the weights is not highly visible but it does show. We will implement this for our review data.

4. CountVectorizer and TfidfVectorizer

We can use a vectorizer for text outside of the training data. It will create a vector corresponding to the column names and adds a tfidf value if the word is present in the column and zero otherwise.

bows_counter.transform(['python programming is great']).toarray()

array([[0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 1, 0, 0, 0]])

tfidf_vectorizer.transform(['python programming is great']).toarray()

array([[0.        , 0.        , 0.        , 0.        , 0.        ,
        0.61418897, 0.        , 0.        , 0.        , 0.        ,
        0.        , 0.        , 0.        , 0.61418897, 0.49552379,
        0.        , 0.        , 0.        , 0.        , 0.        ]]) 

4.1. Features to Matrix

The final step is to apply the tfidf vectorize to the dataset to obtain the feature and convert features into a matrix that can be ingested into a model for training. The example below demonstrates this implementation using the Amazon product reviews dataset.

from nltk.tokenize import word_tokenize
from sklearn.feature_extraction.text import TfidfVectorizer

review_tfidf_vectorizer = TfidfVectorizer( #max_feautures = 1000,        # Return the top 1000 features
                                        analyzer='word',              # Word level vectorizer
                                        lowercase=True,               # Lower case the text
                                        min_df = 5,                   # Use tokens that appear at least 5 times
                                        ngram_range=(1, 2),           # Create 1 n-grams
                                        tokenizer= word_tokenize,     # Use this tokenizer
                                        stop_words = 'english',       # remove english stopwords 
                                        sublinear_tf=1, smooth_idf=1, use_idf=1) # Additional Features

review_tfidf_vectorizer.fit(review_data.clean_review)
features_df = pd.DataFrame( review_tfidf_vectorizer.transform(review_data.clean_review).toarray(), 
                            columns=review_tfidf_vectorizer.get_feature_names_out())
features_df.head()

5. Pickling Vectorizer and Features Dataframe

Working with NLP often means working with sparse datasets and vectorizer classes which take a long time to run. Pickling these objects and datasets can help implement checkpoints to save time and compute. To complete this note, we implement pickling for the features dataframe and tfidt_vectorizer we developed.

import pickle

# vectorizer pickling
vectorizer_object = open( 'review_tfidf_vectorizer.pk' , 'wb')
pickle.dump( review_tfidf_vectorizer , vectorizer_object    )
vectorizer_object.close()

# features_data pickling
feature_object = open( 'features.pk' , 'wb')
pickle.dump( features_df , feature_object )
feature_object.close()

# sentiments_pickling
sentiment_object = open( 'sentiment.pk' , 'wb')
pickle.dump( review_data['sentiment'] , sentiment_object )
sentiment_object.close()

Now that features are defined and set, we can move to the sentiment Analysis Model. In part 3, we build on part two to generate two models for sentiment classification