1. CBOW - Continuous Bag of Words

Suppose we have some text: "generalized linear models have link functions that enable flexibility beyond that of ordinary least squares". Also, suppose that we want to predict the word vector for the word "models", the continuous bag of words uses the context words: "generalized linear ___ have link" to predict the target word. Given the corpus may contain other texts with similar context, then we can train a cbow model to provide the most probable word for the text above.

If you are interested in understanding the mechanics of training word2vec, it may be useful to visit the Stanford link above for more technical details of the training algorithms available for this estimation.

In Python, I use the gensim package to compute the words vectors for the review corpus using cbow model

Before we implement cbow word2vec, we need to determine the following:

• 1. size of vector: The size of the vectors for every word in the corpus
• 2. window size: The size of context words to use in computing the vector

In this example, let's use a window of 5 and a vector of size 10

from gensim.models import Word2Vec

vector_size = 10
window_size = 5

cbow_model = Word2Vec(  sentences= corpus_tokens,
                        vector_size = vector_size,  # Setting the vector size
                        window =window_size,  # Setting the Window size
                        sg=0,                                 # Initialize CBOW
                        min_count = 2,               # Minimum word count
                        sample=.0000001 )    # Lower Weighting/Downsampling of frequent words

The model has been trained. We can now extract a vector of size 10 using the 5 nearest words

cbow_model.wv['good']

array([ 0.07817571, -0.09510187, -0.00205531,  0.03469197, -0.00938972,
        0.08381772,  0.09010784,  0.06536506, -0.00711621,  0.07710405],
        dtype=float32)

1.2. CBOW Similarity

One of the advantages of using a CBOW model is we can then compute word similarity within the corpus. The cosine similarity method is used for this calculation.

Using the gensim package, we can call the most_similar method for any word as shown in the example below:

cbow_model.wv.most_similar('amazing')

[('few', 0.8727703094482422),
    ('seated', 0.8448812961578369),
    ('seen', 0.7667219638824463),
    ('full', 0.7428421974182129),
    ('said', 0.7405533790588379),
    ('brought', 0.7369356155395508),
    ('enjoy', 0.7215623259544373),
    ('heat', 0.718512773513794),
    ('stuffed', 0.6971375346183777),
    ('potatoes', 0.6860308647155762)]

Based on our dataset and tuning parameters of our model, the word amazing is similar to words 'seated', 'few', 'enjoy' and 'stuff' are generally positive words in a review.

2. Skip-Gram Model

The Skip-Gram model is trained to perform the reverse function of the CBOW model. That is, while the cbow model predicts the target word given context words, the Skip-Gram model predicts context words based on the presence of the target word.

In terms of the general implementation of skip-gram in gensim, we will only activate the skip-gram argument. Let's see the implementation.

skipgram_model = Word2Vec(  sentences= corpus_tokens,
        vector_size = 20,    # Setting the vector size
        window = 5,          # Setting the Window size
        min_count = 2,       # Minimum word count
        sg = 1,              # Initialize Skip Gram
        sample=.0000001 )    # Lower Weighting/Downsampling of frequent words

Just like with CBOW, we can generate a word vector for each word in the corpus

skipgram_model.wv['good']

array([-0.04121573,  0.04649187, -0.00098335, -0.00983143,  0.02302161,
        -0.02047365,  0.01371725,  0.03470667,  0.03032966, -0.03756103,
         0.04690514,  0.0233656 ,  0.01983496, -0.03122055,  0.0423056 ,
        -0.01075502,  0.04413366, -0.02680901, -0.04064848,  0.03411919],
       dtype=float32)

2.2. Skip-Gram Word Similarity

We can also retrieve similar words from the skip-gram model.

skipgram_model.wv.similar_by_word('amazing')

[('serve', 0.6504205465316772),
        ('incredible', 0.6315867900848389),
        ('your', 0.6247786283493042),
        ('flavorful', 0.6199944019317627),
        ('few', 0.5851094126701355),
        ("friend's", 0.5779340267181396),
        ('full', 0.5542045831680298),
        ('potatoes', 0.5514717698097229),
        ('said', 0.5444715023040771),
        ('overall', 0.5155755877494812)]

Unsurprisingly, words like 'flavorful', 'incredible' appear very similar to 'amazing'

3. Gensim Vocabulary

Suppose we have a new text input that may have new words that our models have not seen yet, it is therefore impossible to return the vectors for those words. It is always important to know how to access the vocabulary in the model so that you can provide an alternative to new words are you are trying to leverage the models for feature extraction.

The gensim vocabulary is a dictionary of all words with the vector object.

vocab = skipgram_model.wv.index_to_key
vocab[20:30]

['with', 'had', 'great', 'that', 'be', 'so', 'were', 'are', 'but', 'have']

len(vocab)

897

Saving this vocabulary list will be important in initializing vectors for new text, particularly for words not contained in the word2vec model.

4. Convert Word Vectors to Features

We have seen earlier how to implement bag of words and tfidf feature extraction. With those techniques, every word has a single value as a feature. With word vectors, every word has a vector of features so we will need to find a way to summarize the features such that every document has one vector representing all the tokens in the document. One simple approach is to sum the vectors and average them by the count of the words/tokens.

Let's see this in implementation

import numpy as np

def avg_word_vectors(words, model, vocabulary, feature_size):
    
    feature_vector = np.zeros((feature_size,), dtype='float64')
    word_count = 0.
    
    for word in words:
        if word in vocabulary:
            word_count += 1
            feature_vector = np.add(feature_vector, model.wv[word])
            
        if word_count:
            feature_vector = np.divide(feature_vector, word_count)
            
    return feature_vector

Testing the function on a sample of text

test = ["This", 'is', 'delicious', 'food']
avg_word_vectors(test, skipgram_model, vocab, 20 )

array([-0.12273251,  0.05990233,  0.20953931,  0.14712462, -0.02474864,
    -0.04072365,  0.12475592,  0.41853103, -0.23380686,  0.15436821,
     0.1927391 , -0.16070034,  0.25334843, -0.07903712,  0.13786161,
     0.13087943,  0.2813911 , -0.03589961, -0.17655849, -0.32781931])

Averaging across the full dataset

Now let's convert the averaged vectors to a full dataframe with words as the predictors/features

def avg_word_vectorizer(corpus, model, feature_size):
    vocabulary = set(model.wv.index_to_key)
    features = [ avg_word_vectors(text, model, vocabulary, feature_size) for text in corpus]
    
    return np.array(features)


skipgram_feaures = avg_word_vectorizer(corpus_tokens, skipgram_model, 20)

Converting Vectors into a Feature Dataframe

skipgram_df = pd.DataFrame(skipgram_feaures)
skipgram_df.head()

Conclusion

What we have done is reduce all of our text to a dataframe that has 20 features representing the average of all of the word vectors for the word in the text. We can use this matrix for some of the classification tasks that we did before. In the last section, we will cover how to use deep learning for the classification of the sentiments.