Benchmarking recommender system and matrix completion algorithms could be greatly simplified if the entire matrix was known. We built a <sweetrs.org> platform with 77 candies and sweets to rank. Over 2000 users submitted over 44000 grades resulting in a matrix with 28% coverage. In this report, we give the full description of the environment and we benchmark the Soft-Impute algorithm on the dataset.READ FULL TEXT VIEW PDF
Matrix completion has a long-time history of usage as the core technique...
Completing a data matrix X has become an ubiquitous problem in modern da...
Top-N recommender systems have been investigated widely both in industry...
We consider a discrete-valued matrix completion problem for recommender
Recommender systems (RS), which have been an essential part in a wide ra...
The task of reconstructing a matrix given a sample of observedentries is...
Indian regional movie dataset is the first database of regional Indian
One of the problems in building any machine learning system is the limited access to ground truth. This problem is particularly prevalent in matrix completion when the matrices are very sparse, such as in the context of product recommendation. In many situations, a large dataset can be trimmed to a dense matrix by choosing specific users and items, yet it may introduce additional bias. In this project, we attempted to collect a dense matrix, by rating commercial products internationally known, such as candy bars and sweets.
We built a basic website sweetrs.org, where users can both rate and add new products. Participants were tasked to rate sweets on the scale from to or click "Never tried" in case they do not know or have not tasted the product. To the date, we collected over ratings from over users on items, giving the coverage of over matrix coefficients. Moreover, we identified a subset of users and products with the coverage of over matrix coefficients.
The project has been developed as a part of a Master Thesis at the University of Warsaw [Kid11].
Let be a matrix representing users rating items on the integer scale. Let be a set of all observed indices
. We attempt to approximate unobserved ratings. We estimate prediction error using cross-validation.
As our benchmark method we choose Soft-Impute [MHT10] due to its speed, efficiency and simplicity. Soft-Impute performs thresholded SVD in the presence of missing values.
We investigate how the prediction depends on the size of the training set and on the regularization parameter in Soft-Impute. We test settings for the size of the training set, with where is the ratio of the observed set used for testing.
We use cross-validation for estimating the Normalized Mean Squared Error:
where is the set on which we trained the algorithm and denotes the set difference.
In our preliminary experiments, we identified that analyzing the set is sufficient for finding best integer . Thus, we train the Soft-Impute algorithm on constellations . We center and scale each item before fitting. Next, we estimate NMSE for a given is performed as follows:
randomly choose training elements ,
fit the Soft-Impute model for on given the parameter ,
predict elements for ,
We repeat the procedure times for every . We present mean in Figure 1.
In this report we aimed at providing a benchmark and description of the dataset convenient for testing new recommender system techniques. We achieved over
of variance explained in the case whenof matrix coefficients are observed. We published the dataset and the sample
Rcode as a github repository111https://github.com/kidzik/sweetrs-analysis/.