A recommender system is a type of information filtering system that provides personalized recommendations to users based on their preferences, interests, and past behaviors. Recommender systems come in a variety of forms, such as content-based, collaborative filtering, and hybrid systems. Content-based systems make recommendations for products based on how closely their characteristics match those of products the user has previously expressed interest in. Collaborative filtering systems recommend items based on the preferences of users who have similar interests to the user being recommended. Hybrid systems combine both content-based and collaborative filtering approaches to make recommendations.
We will implement this with the help of Collaborative Filtering. Collaborative filtering involves making predictions (filtering) about a user’s interests by compiling preferences or taste data from numerous users (collaborating). The essential premise is that, if two users A and B share the same opinion on a subject, A is more likely to share B’s opinion on a related but unrelated subject, x, than the opinion of a randomly selected user.
Recommender System using Pyspark
Collaborative filtering is implemented by the machine learning library Spark MLlib using Alternating Least Squares. These parameters apply to the MLlib implementation:
- The number of blocks used to parallelize computation is numBlocks (set to -1 to auto-configure).
- The number of latent factors in the model is its rank.
- The number of iterations to execute is known as an iteration.
- The regularisation parameter in ALS is specified by lambda.
- Whether to utilize the ALS variation tailored for implicit feedback data or the explicit feedback variant is determined by implicitPrefs.
- The implicit feedback variant of ALS has a parameter called alpha that controls the initial level of confidence in preference observations.
In this, we will use the dataset of the book review.
Step 1: Import the necessary libraries and functions and Setup Spark Session
#importing the required pyspark library from pyspark.sql import SparkSession
from pyspark.ml.evaluation import RegressionEvaluator
from pyspark.ml.recommendation import ALS
#Setup Spark Session spark = SparkSession.builder.appName('Recommender').getOrCreate()
spark |
Output:
SparkSession - in-memory SparkContext Spark UI Version v3.3.1 Master local[*] AppName Recommender
Step 2: Reading the data from the data set
#CSV file can be downloaded from the link mentioned above. data = spark.read.csv('book_ratings.csv',
inferSchema=True,header=True)
data.show(5)
|
Output:
+-------+-------+------+ |book_id|user_id|rating| +-------+-------+------+ | 1| 314| 5| | 1| 439| 3| | 1| 588| 5| | 1| 1169| 4| | 1| 1185| 4| +-------+-------+------+ only showing top 5 rows
Describe the dataset
data.describe().show() |
Output:
+-------+-----------------+------------------+------------------+ |summary| book_id| user_id| rating| +-------+-----------------+------------------+------------------+ | count| 981756| 981756| 981756| | mean|4943.275635697668|25616.759933221696|3.8565335989797873| | stddev|2873.207414896143|15228.338825882149|0.9839408559619973| | min| 1| 1| 1| | max| 10000| 53424| 5| +-------+-----------------+------------------+------------------+
Step 3: Splitting the data into training and testing
# Dividing the data using random split into train_data and test_data # in 80% and 20% respectively train_data, test_data = data.randomSplit([0.8, 0.2])
|
Step 4: Import the Alternating Least Squares(ALS) Method and apply it.
# Build the recommendation model using ALS on the training data als = ALS(maxIter=5,
regParam=0.01,
userCol="user_id",
itemCol="book_id",
ratingCol="rating")
#Fitting the model on the train_data model = als.fit(train_data)
|
Step 5: Predictions
# Evaluate the model by computing the RMSE on the test data predictions = model.transform(test_data)
#Displaying predictions calculated by the model predictions.show() |
Output:
+-------+-------+------+----------+ |book_id|user_id|rating|prediction| +-------+-------+------+----------+ | 2| 6342| 3| 4.8064413| | 1| 17984| 5| 4.9681554| | 1| 38475| 4| 4.4078903| | 2| 6630| 5| 4.344222| | 1| 32055| 4| 3.990228| | 1| 33697| 4| 3.7945805| | 1| 18313| 5| 4.533183| | 1| 5461| 3| 3.8614116| | 1| 47800| 5| 4.914357| | 2| 10751| 3| 4.160536| | 1| 16377| 4| 5.304298| | 1| 45493| 5| 3.998557| | 2| 10509| 2| 1.8626969| | 1| 33890| 3| 3.6022692| | 1| 37284| 5| 4.8147345| | 1| 1185| 4| 3.7463336| | 1| 44397| 5| 5.0251017| | 1| 46977| 4| 4.0746284| | 1| 10944| 5| 4.343548| | 2| 8167| 2| 3.705464| +-------+-------+------+----------+ only showing top 20 rows
Evaluations
#Printing and calculating RMSE evaluator = RegressionEvaluator(metricName="rmse", labelCol="rating",predictionCol="prediction")
rmse = evaluator.evaluate(predictions)
print("Root-mean-square error = " + str(rmse))
|
Output:
Root-mean-square error = nan
Step 6: Recommendations
Now, we will predict/recommend the book to a single user – user1 (let’s say, userId:5461) with the help of our trained model.
#Filtering user with user id "5461" with book id on which it has given the reviews user1 = test_data.filter(test_data['user_id']==5461).select(['book_id','user_id'])
#Displaying user1 data user1.show() |
Output:
+-------+-------+ |book_id|user_id| +-------+-------+ | 1| 5461| | 11| 5461| | 19| 5461| | 46| 5461| | 60| 5461| | 66| 5461| | 93| 5461| | 111| 5461| | 121| 5461| | 172| 5461| | 194| 5461| | 212| 5461| | 222| 5461| | 245| 5461| | 264| 5461| | 281| 5461| | 301| 5461| | 354| 5461| | 388| 5461| | 454| 5461| +-------+-------+ only showing top 20 rows
#Traning and evaluating for user1 with our model trained with the help of training data recommendations = model.transform(user1)
#Displaying the predictions of books for user1 recommendations.orderBy('prediction',ascending=False).show()
|
Output:
+-------+-------+----------+ |book_id|user_id|prediction| +-------+-------+----------+ | 19| 5461| 5.3429904| | 11| 5461| 4.830688| | 66| 5461| 4.804107| | 245| 5461| 4.705879| | 388| 5461| 4.6276107| | 1161| 5461| 4.612251| | 60| 5461| 4.5895457| | 1402| 5461| 4.5184| | 1088| 5461| 4.454755| | 5152| 5461| 4.415825| | 121| 5461| 4.3423634| | 93| 5461| 4.3357944| | 1796| 5461| 4.30891| | 172| 5461| 4.2679276| | 454| 5461| 4.245925| | 1211| 5461| 4.2431927| | 731| 5461| 4.1873074| | 1094| 5461| 4.1829815| | 222| 5461| 4.182873| | 264| 5461| 4.1469045| +-------+-------+----------+ only showing top 20 rows
In the above output, there are predictions for the book IDs for the user with userId “5461”.
Step 7: Stop the spark
spark.stop() |