Build Recommender Systems From Scratch 🤩
Imagine shopping in brick-and-mortar shops, we pick products by walking through each shelf and checking if there’s something we are interested in. However, this doesn’t work for digital items. Unlike physical stores with limited shelf space, displaying links, ebooks, movies, music, etc. online has zero cost. With this overwhelming amount of data, the old way of browsing through each of them no longer works. Even with filtering options like categories, price and rating, the remaining items are still a lot.
Now Recommender Systems are the primary way that most people (often unknowingly) interact with large collections.
Types of recommender systems
Popularity-based systems
The popularity-based recommender simply recommends items that are most popular to all users without any personalization, like recommending the top 10 rated movies, or top 10 best-selling products, depending on how you define popularity. Popularity-based model is commonly used as a benchmark. Any functional personalized recommender system is supposed to have better performance.
Another advantage of popularity-based systems is that it solves the problem of cold start. When users first come to the platform, we don’t have data like search history to perform more personalized recommendations. Recommending the most popular items to users can warm them up, start to browse more and generate histories that can be used for advanced recommenders.
Building a popularity-based recommender is fairly straightforward: Load the data, define and sort by popularity, and take top k. Below is the code for implementation.
import pandas as pd
# Load data
df = pd.DataFrame(data)
# Recommend movies based on popularity score
def recommend_movies(dataframe, top_k=5):
    # Sort the movies based on popularity score in descending order
    recommended_movies = dataframe.sort_values(by='popularity_score', ascending=False)
    # Return the top k most popular movies
    return recommended_movies.head(top_k)
# Recommend top 3 movies
top_movies = recommend_movies(df, top_k=3)
print("Top 3 recommended movies:")
print(top_movies)Content-based systemsÂ
The general idea of content-based systems is to recommend items to users based on their tastes and preferences. For example, if a user has read a lot of news on Cryptocurrencies, the system will recommend other news about Bitcoin to that user because they have similar topics.
The steps for creating a content-based system are as follows:
Create item profile
Each item is represented by a n x 1 vector where n is the number of features of this item. Different items should have vectors of the same dimension but different values for each dimension based on their specific content.Â
Features of movies can be the set of actors, the director, the year the movie was released, and the genre of the movie.
Features of images can be tags associated with them.Â
Using categories as features for documents would be too general. Alternatively, we identify words that characterize the topic of the document as features.Â
Tokenize and remove stopwords.
Calculate the TF-IDF score for each word in the document.Â
Pick the highest n words or a fixed n percentage of words, or define a threshold and take all words with a score higher than the threshold.
Create user profile
To create a user profile, we select all items the user once interacted with. We can take the simple average or a weighted average that takes into account how much the user likes this item (e.g. rating, viewing time, etc.).
Recommend items to the user
Now for each user, we have a vector representing their taste. We also have an ocean of vectors representing each item. To recommend an item to the user, calculate the cosine similarity between user and item vectors. The higher similarity means the item is closer to the user’s taste and can be the final recommendation.
Below is an implementation of the content-based recommender.Â
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
# Load data
df = pd.DataFrame(data)
# Create item profiles
tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf.fit_transform(df['content'])
# Create user profiles
user_likes_indices = [0, 2]Â # User likes articles with index 0 and 2|
user_profile = tfidf_matrix[user_likes_indices].mean(axis=0)
# Recommend items to users using cosine similarity
cosine_sim = cosine_similarity(user_profile, tfidf_matrix)
recommended_article_indices = cosine_sim.argsort().flatten()[::-1]
recommended_article_indices = [i for i in recommended_article_indices if i not in user_likes_indices]
top_recommendations = df.iloc[recommended_article_indices[:3]]
print("Recommended articles for the user:")
print(top_recommendations['title'])Collaborative FilteringÂ
Collaborative filtering works with a utility matrix where rows are users and columns are items. The goal is to fill in all empty ratings in the matrix, i.e. predict ratings, and then make recommendations based on predicted results.Â
There are three types of collaborative filtering algorithms: user-user, item-item, and latent factor model.
User-User and Item-Item
The overall idea of user-user collaborative filtering is, for each user, to find a group of similar users, for items that this user does not have a rating, use similar users’ weighted averages as predicted results. The idea of item-item collaborative filtering is largely the same. For each item, find similar items and use a weighted average to fill in the blanks.Â
The similarity score between user and user or item and item can be used as weights. While there are many ways to calculate similarity, like Jaccard similarity and cosine similarity, they both have biases like ignoring rating scores and treating missing ratings as disliking. Here we use Pearson Correlation to better capture the dynamics.Â
In practice, item-item collaborative filtering is more often used because items are by nature simpler than users, leading to better predictions.
Latent Factor Model
The goal of the latent factor model is still to predict missing ratings. However, it considers the user and item together by factoring the utility matrix into two smaller matrices. Using alternating least squares (ALS), we iteratively minimize the difference between available ratings and predicted ratings (the product of two matrices). After optimization, we will be able to roughly mimic the utility matrix with the two latent factor matrices, from which the predicted ratings can be used to fill in missing values.Â
# !pip install scikit-surprise
from surprise import SVD
from surprise import Dataset
from surprise import Reader
from surprise.model_selection import cross_validate
# Prepare sample data
# Each rating is in the form (user_id, item_id, rating)
data = [
    ('user1', 'item1', 4),
    ('user1', 'item2', 3.5),
    ('user2', 'item1', 2),
    ('user2', 'item3', 5)
]
reader = Reader(rating_scale=(1, 5))
# Load the dataÂ
data = Dataset.load_from_df(pd.DataFrame(data, columns=['user', 'item', 'rating']), reader)
# Modeling
algo = SVD()
cross_validate(algo, data, measures=['RMSE', 'MAE'], cv=5, verbose=True)
trainset = data.build_full_trainset()
algo.fit(trainset)
# Prediction and evaluation
user_id = 'user1'
item_id = 'item2'
actual_rating = 4Â Â
prediction = algo.predict(user_id, item_id, r_ui=actual_rating, verbose=True)Comparison
With a popularity-based model, we do not need any user data. Though the recommendations are not customized, they effectively solve the problem of cold-start compared to the other two methods.Â
For a content-based model, we only need data from the target user, compared to collaborative filtering where we need all users’ data to find similar groups. Also, it works particularly better for users with unique tastes (we cannot find similar users in this case). Another competitive edge against the popularity baseline is that it can recommend unpopular items that fit the user’s tastes. Â
The latent factor model based on collaborative filtering tends to perform better than all other models because it captures the underlying factors driving the interactions with unique latent factors learned for each user and item. It is also easily scalable to larger datasets.
Evaluation Metrics
Root Mean Square Error (RSME)
Like in regression, Root Mean Square Error (RSME) is a commonly used metric to quantify the difference between predicted and actual ratings given by users. A good latent factor model should have a low RMSE, but one drawback of this metric is that it does not consider the correctness of ranking.Â
AUC (Area Under the ROC Curve)
AUC (Area Under the ROC Curve) is a metric used in standard binary classification. In the context of recommender systems, it measures how often a positive interaction (user shows preference toward this item) is ranked ahead of a negative interaction.Â
For example, given a list of recommendations: - + - + + - -, there is no + before the first -, so the value is 0. Similarly, there is one + before the second -, so the value is 1. There are three +’s before the third -, so the value is 3, and the same for the fourth -. The denominator is the total number of possible combinations. With 4 negative interactions and 3 positive interactions, the value of the denominator is 3 x 4. Therefore, the AUC is calculated as (0+1+3+3)/(3x4) = 7/12. The higher this metric is, the better the performance of the recommender system.
Average Precision (AP)
Average precision (AP) is a weighted average of precisions at each step. For each + interaction, we calculate what fraction of higher-ranked items are also positive. Given the same list - + - + + - -, at the first positive interaction, the fraction of positive is 1/2. At the second positive interaction, the fraction of positive is 2/4. At the third positive interaction, the fraction is 3/5. The final AP as a weighted average is calculated as (½ + ½ +â…—)/3.Â
Reciprocal Rank (MRR)
Reciprocal Rank (MRR) is the reverse rank of the first positive interaction. Compared to other metrics, it focuses only on the first correctly recommended item, which is also intuitive as we should care about letting users see items they like within the top 3 or 5 recommendations.
Given the same list - + - + + - -, the MRR is ½. In general, the higher the MRR is, the better the performance is. However, if the list is + - - - - - -, the MRR is 1! That’s also something we want to avoid. Using a combination of all approaches can provide a holistic view of the performance.
A/B Testing
In practice, recommender systems exist in a feedback loop. It is extremely difficult to measure offline from observational data like above. So an alternative to all metrics is to use A/B testing and monitor the change of dependent variables.Â
For example, we are building a recommender for an online shopping website, and the purpose is to increase sales. We can randomly split users into two groups, one group using a popularity baseline and another using a customized recommender. With a month’s experiment time, we check if the click-through rate, conversion rate, and eventually the sales actually increase with statistical significance. With a proven increase, we can adopt the new method.Â
Infrastructure
In a practical setting, when the magnitude of data reaches gigabytes to terabytes, local computation resources become insufficient to even load the data. The commonly adopted infrastructures for recommender systems are distributed file systems like Hadoop File System (HDFS) and distributed computing systems like Apache Spark.Â
HDFS
HDFS is designed to split large blocks of data and distribute them across nodes, enabling parallel processing of data. HDFS enhances data durability by replicating data across multiple nodes. One node's failure will not affect data on other nodes. The replication also speeds up data processing by allowing the system to retrieve data from multiple nodes simultaneously. The replication factor means the number of copies of data we store in the entire cluster.
# Uploading a file from the local filesystem to HDFS
hadoop fs -put /local/path/to/file /hdfs/destination/path
# Downloading a file from HDFS to the local filesystem
hadoop fs -get /hdfs/path/to/file /local/destination/path
# Specify replication factor when uploading
hadoop fs -D dfs.replication=2 -put /local/path/to/file /hdfs/destination/path
# Check replication factor
hadoop fs -stat %r /hdfs/path/to/file
# Modify replication factor
hadoop fs -setrep 4 /hdfs/path/to/existing/fileSpark
Spark is an analytics engine to work with distributed data (e.g. data managed by HDFS). It supports data manipulation with dataframes using pandas and executing SQLÂ queries.
from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("data_wrangling").getOrCreate()
# Create a Spark DataFrame and show
df = spark.createDataFrame([
    (1, "foo"), (2, "bar"), (3, "baz")
], ["id", "value"])
df.show()
# Convert to a Pandas DataFrame
pandas_df = df.toPandas()
pandas_df['value'] = pandas_df['value'].str.upper()
# Create a temporary view to run SQL queries
df.createOrReplaceTempView("my_table")
# Run a SQL query
result_df = spark.sql("SELECT * FROM my_table WHERE id > 1")
result_df.show()It provides advanced analytical packages like MLlib for machine learning, GraphX for graph processing, and Structured Streaming for incremental computation and stream processing. Specifically for building recommenders, we use MLlib with well-written collaborative filtering models and evaluating metrics.Â
from pyspark.mllib.recommendation import ALS, Rating
from pyspark.mllib.evaluation import RegressionMetrics
from pyspark.sql import Row
# Read in the ratings data, parse and split to train and test
lines = spark.read.text("data/mllib/als/sample_movielens_ratings.txt").rdd
parts = lines.map(lambda row: row.value.split("::"))
ratingsRDD = parts.map(lambda p: Row(userId=int(p[0]), movieId=int(p[1]),
                                     rating=float(p[2]), timestamp=long(p[3])))
ratings = spark.createDataFrame(ratingsRDD)
(training, test) = ratings.randomSplit([0.8, 0.2])
More evaluation metrics are available at Evaluation Metrics.Â
We are only scratching the surface...
If you're interested in learning more about how to implement recommender systems to your use case, feel free to reach out to us by leaving your contact information!
View certificate