Introduction

Imagine you’ve been tasked with crafting the perfect subject line for a critical email campaign, but standing out in a cluttered inbox seems daunting. This article offers a solution with a step-by-step guide to generating smart email subject lines with Word2Vec. Learn how to harness the power of Word2Vec embeddings to craft compelling and contextually relevant subject lines that captivate and engage your audience. Follow along to transform your approach and improve your email marketing strategy.

Learning objectives

• Discover what vector embeddings are and how they represent complex data as numeric vectors.
• Learn how to calculate the semantic similarity between different pieces of text using cosine similarity.
• Build a system that can generate contextually relevant email subject lines using Word2Vec and NLTK.

This article was published as part of the Data Science Blogathon.

Embedding models: converting words into numerical vectors

Word embeddings are a method used to efficiently represent words in a dense numerical format, where similar words have similar encodings. Unlike manually setting these encodings, embeddings are trainable parameters: floating-point values ​​that are learned by the model during training, similar to how weights are learned in a dense layer. Embeddings range from 8 for smaller datasets to larger dimensions such as 1024 for large datasets, allowing them to capture relationships between words. This higher dimensionality allows embeddings to encode detailed semantic relationships.

In a word embedding diagram, a 4-dimensional vector of floating-point values ​​represents each word. Think of embeddings as a “lookup table” that stores the dense vector of each word after training, allowing you to quickly encode and retrieve words based on their vector representations.

Defining semantic similarity and its meaning

Semantic similarity is the measure of how closely two pieces of text convey the same meaning. It allows systems to understand the different ways ideas can be expressed in language without having to explicitly define each variation.

Introduction to Word2Vec and its functionalities

Word2Vec is a popular natural language processing technique for converting words into numerical vector representations.

Word2Vec generates word embeddings, which are continuous vector representations of words. Unlike traditional one hot encoding, which represents words as sparse vectors, Word2Vec maps each word to a dense vector of fixed size. These vectors capture semantic relationships between words, allowing similar words to have similar vectors.

Word2Vec Training Methods

Word2Vec uses two main training approaches:

Continuous bag of words

This method predicts a target word based on the surrounding context words. For example, if a word is missing from a sentence, CBOW attempts to infer the missing word using the context provided by the other words in the sentence.

Skip-Gram

During training, Word2Vec refines the word vectors by analyzing how often words co-occur within a defined context window. Words with more similar vectors are words that occur in similar contexts. Relations such as synonyms and analogies are well captured by this method (for example, the relation between “king” and “queen” can be derived from the analogy “king” – “man” + “queen” – “woman”).

Mechanism of action of Word2Vec

• Initialization: Start with random vectors for each word in the vocabulary.
• Course: For each word in a given context, update the vectors to minimize the prediction error between the actual and predicted words. This involves backpropagation and optimization techniques such as stochastic gradient descent.
• Vector representation: After training, each word is represented by a vector that encodes its semantic meaning. Words with similar meanings or contexts have vectors that are close to each other in the vector space.

Read more about Word2Vec here

Step-by-step guide to generating smart email subject lines

Discover the secrets to crafting compelling email subject lines with our step-by-step guide. Leverage Word2Vec embeddings for smarter, more relevant results.

Step 1: Set up the environment and preprocess data

Import essential libraries for data manipulation, natural language processing, word embeddings, and similarity calculations.

import pandas as pd
import nltk
from nltk.tokenize import word_tokenize
from gensim.models import Word2Vec
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

Step 2: Download NLTK data

Download the NLTK tokenizer data needed to tokenize text.

# Download NLTK data (only needed once)
nltk.download('punkt')

Step 3: Read the CSV file

Load the email dataset from a CSV file and handle any parsing errors.

# Read the CSV file
try:
    df = pd.read_csv('emails.csv', quotechar=""", escapechar="\\", engine="python", on_bad_lines="skip")
except pd.errors.ParserError as e:
    print(f"Error reading the CSV file: {e}")

Step 4: Tokenize email texts

Tokenize the email content into words and convert them to lowercase for uniformity.

# Preprocess: Tokenize email bodies
tokenized_bodies = (word_tokenize(body.lower()) for body in df('email_body'))

Step 5: Train the Word2Vec model

Train a Word2Vec model on the tokenized email content to create word embeddings.

# Train Word2Vec model on the email bodies
word2vec_model = Word2Vec(sentences=tokenized_bodies, vector_size=100, window=5, min_count=1, workers=4)

Step 6: Define a function to calculate document embeddings

Create a function that calculates the embedding of the body of an email message by averaging the embedding of the words.

# Function to compute document embedding by averaging word embeddings
def get_document_embedding(doc, model):
    words = word_tokenize(doc.lower())
    word_embeddings = (model.wv(word) for word in words if word in model.wv)
    if word_embeddings:
        return np.mean(word_embeddings, axis=0)
    else:
        return np.zeros(model.vector_size)

Step 7: Calculate inclusions for all email texts

Calculate the document embeddings for all email texts in the dataset.

# Compute embeddings for all email bodies
body_embeddings = np.array((get_document_embedding(body, word2vec_model) for body in df('email_body')))

Step 8: Define a semantic search function

Create a function that uses cosine similarity to find the most similar email text in the dataset for a given query.

# Function to perform semantic search based on the email body
def semantic_search(query, model, body_embeddings, texts):
    query_embedding = get_document_embedding(query, model)
    similarities = cosine_similarity((query_embedding), body_embeddings)
    best_match_idx = np.argmax(similarities)
    return texts(best_match_idx), similarities(0, best_match_idx)

Step 9: Sample Email Text for Generating Subject Lines

Define a new email text for which you want to generate a subject line.

# Example email body for which to generate a subject line
new_email_body = "Please review the attached documents and provide feedback by end of day"

Step 10: Perform a semantic search on the new email text

Use the semantic search function to find the email text in the dataset that most closely resembles the new email text.

# Perform semantic search for the new email body to find the most similar existing email
matched_text, similarity_score = semantic_search(new_email_body, word2vec_model, body_embeddings, df('email_body'))

Step 11: Get the corresponding subject line

Retrieve the subject line that matches the matched email body and print it, along with the matched email body and the similarity score.

# Find the corresponding subject line for the matched email body
matched_subject = df.loc(df('email_body') == matched_text, 'subject_line').values(0)

print("Generated Subject Line:", matched_subject)
print("Matched Email Body:", matched_text)
print("Similarity Score:", similarity_score)

Step 12: Evaluating Accuracy (Example)

Evaluating the accuracy of a model is crucial to understand its performance on unseen data. In this step, we define the function evaluate_accuracyuse a test dataset (test_df), and pre-computed embeddings (train_body_embeddings) to measure the accuracy of the model.

# Evaluate accuracy on the test set
accuracy = evaluate_accuracy(test_df, word2vec_model, train_body_embeddings, train_df('email_body'))
print("Mean Cosine Similarity for Test Set:", accuracy)

For the code implementation I used the Document dataset, which you can find here.

Output

A look at the dataset:

Real example

Let’s look at a real example to illustrate this step.

Suppose we have a test set (test_df) with the following email texts and subject lines:

import pandas as pd
import nltk
from nltk.tokenize import word_tokenize
from gensim.models import Word2Vec
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Download NLTK data (only needed once)
nltk.download('punkt')

# Example training dataset
train_data = {
    'email_body': (
        "Please send me the latest sales report.",
        "Can you provide feedback on the attached document?",
        "Let's schedule a meeting to discuss the new project.",
        "Review the quarterly financials and get back to me."
    ),
    'subject_line': (
        "Request for Sales Report",
        "Feedback on Document",
        "Meeting for New Project",
        "Quarterly Financial Review"
    )
}
train_df = pd.DataFrame(train_data)

# Example test dataset
test_data = {
    'email_body': (
        "Can you provide the latest sales figures?",
        "Please review the attached documents and provide feedback.",
        "Schedule a meeting to discuss the new project proposal."
    ),
    'subject_line': (
        "Request for Latest Sales Figures",
        "Feedback on Attached Documents",
        "Meeting for Project Proposal"
    )
}
test_df = pd.DataFrame(test_data)

# Preprocess: Tokenize email bodies
tokenized_bodies = (word_tokenize(body.lower()) for body in train_df('email_body'))

# Train Word2Vec model on the email bodies
word2vec_model = Word2Vec(sentences=tokenized_bodies, vector_size=100, window=5, min_count=1, workers=4)

# Function to compute document embedding by averaging word embeddings
def get_document_embedding(doc, model):
    words = word_tokenize(doc.lower())
    word_embeddings = (model.wv(word) for word in words if word in model.wv)
    if word_embeddings:
        return np.mean(word_embeddings, axis=0)
    else:
        return np.zeros(model.vector_size)

# Compute embeddings for all email bodies in the training set
train_body_embeddings = np.array((get_document_embedding(body, word2vec_model) for body in train_df('email_body')))

# Function to evaluate the accuracy of the model on the test set
def evaluate_accuracy(test_df, model, train_body_embeddings, train_texts):
    similarities = ()

    for index, row in test_df.iterrows():
        # Compute the embedding for the current email body in the test set
        test_embedding = get_document_embedding(row('email_body'), model)

        # Compute cosine similarities between the test embedding and all training email body embeddings
        cos_sim = cosine_similarity((test_embedding), train_body_embeddings)

        # Get the highest similarity score
        best_match_idx = np.argmax(cos_sim)
        highest_similarity = cos_sim(0, best_match_idx)

        similarities.append(highest_similarity)

    # Return the mean cosine similarity
    return np.mean(similarities)

# Evaluate accuracy on the test set
accuracy = evaluate_accuracy(test_df, word2vec_model, train_body_embeddings, train_df('email_body'))
print("Mean Cosine Similarity for Test Set:", accuracy)

Output:

Mean Cosine Similarity for Test Set: 0.86

Challenges

• Cleaning and preparing the email dataset for training can introduce issues such as malformed rows or inconsistent formats.
• The model may struggle to generate relevant subject lines for completely new or unique email content that differs significantly from the training data.

Conclusion

The project demonstrates how to generate smart email subject lines more easily using Word2Vec embeddings. To produce vector embeddings of email bodies, the procedure consists of preprocessing the email data and training a Word2Vec model. Further improvements include incorporating more advanced models and optimizing the methodology for improved effectiveness. Applications for this concept could be for a company that wants to improve the open rates of their email marketing campaigns by using more engaging and relevant subject lines. A news website wants to send personalized newsletters to its subscribers based on their reading preferences.

Key Points

• Discover how Word2Vec converts words into numeric vectors to represent semantic relationships.
• Discover how the quality of word embeddings directly impacts the relevance of generated subject lines.
• Recognize how to match new email text to current text using cosine similarity.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the author’s sole discretion.