Available via license: CC BY-NC-SA 4.0
Content may be subject to copyright.
Using Stable Matching to Optimize the Balance between
Accuracy and Diversity in Recommendation
Farzad Eskandanian
feskanda@depaul.edu
DePaul University
Chicago, Illinois, USA
Bamshad Mobasher
mobasher@cs.depaul.edu
DePaul University
Chicago, Illinois, USA
ABSTRACT
Increasing aggregate diversity (or catalog coverage) is an important
system-level objective in many recommendation domains where
it may be desirable to mitigate the popularity bias and to improve
the coverage of long-tail items in recommendations given to users.
This is especially important in multistakeholder recommendation
scenarios where it may be important to optimize utilities not just
for the end user, but also for other stakeholders such as item sellers
or producers who desire a fair representation of their items across
recommendation lists produced by the system. Unfortunately, at-
tempts to increase aggregate diversity often result in lower recom-
mendation accuracy for end users. Thus, addressing this problem
requires an approach that can eectively manage the trade-os be-
tween accuracy and aggregate diversity. In this work, we propose
a two-sided post-processing approach in which both user and item
utilities are considered. Our goal is to maximize aggregate diversity
while minimizing loss in recommendation accuracy. Our solution
is a generalization of the Deferred Acceptance algorithm which
was proposed as an ecient algorithm to solve the well-known
stable matching problem. We prove that our algorithm results in a
unique user-optimal stable match between items and users. Using
three recommendation datasets, we empirically demonstrate the
eectiveness of our approach in comparison to several baselines. In
particular, our results show that the proposed solution is quite ef-
fective in increasing aggregate diversity and item-side utility while
optimizing recommendation accuracy for end users.
ACM Reference Format:
Farzad Eskandanian and Bamshad Mobasher. 2020. Using Stable Matching to
Optimize the Balance between Accuracy and Diversity in Recommendation.
In 28th ACM Conference on User Modeling, Adaptation and Personaliza-
tion(UMAP’20), July 14–17, 2020, Genoa, Italy. ACM, New York, NY, USA,
9 pages. https://doi.org/10.1145/3340631.3394858
1 INTRODUCTION
Much of the early research in recommender systems has been fo-
cused on improving the accuracy of personalized recommendations.
These eorts have led to a number of eective recommendation
algorithms and strategies. In recent years, however, more attention
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than the
author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specic permission
and/or a fee. Request permissions from permissions@acm.org.
UMAP ’20, July 14–17, 2020, Genoa, Italy
©2020 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-6861-2/20/07.. .$15.00
https://doi.org/10.1145/3340631.3394858
has been focused on key objectives other than accuracy that may
also have a bearing on the overall satisfaction of users as well as
other possible stakeholders of the underlying system. One of these
key objectives is recommendation “diversity”.
Broadly speaking, two notions of diversity have been explored
in the context of recommendation: individual diversity and aggre-
gate diversity. Individual diversity reects the ability of the recom-
mender systems to recommend a broad range of items to individual
users (for example, items ranging over multiple categories, or span-
ning both popular items as well as niche items). This ability is
often measured as a function of the dissimilarity of items in each
recommendation list provided to users. Improving individual rec-
ommendation diversity is an important objective since it can lead
to higher user satisfaction and helps reduce the “lter bubble” eect
[21] aecting many personalized systems.
Aggregate diversity, on the other hand, measures the ability of
the system as a whole to recommend a wide selection of items. This
is an important system-level objective that ensures a fair oppor-
tunity for most items to be recommended. Increasing aggregate
diversity helps mitigate the popularity bias by increasing the cover-
age of niche or less popular items. It also promotes fairness from the
perspective of other stake-holders such as sellers or suppliers who
need to have a guarantee that the system provides a fair coverage
of their items across recommendation lists. Aggregate diversity is
often measured as a function of the number or the ratio of items that
are recommended to users relative to the set of all items available
across all suppliers.
In this paper we focus primarily on aggregate diversity especially
as an objective related to item side of the recommendation scenario.
In particular we are interested in a two-sided framework that can
produce a balanced match between items and users taking into
account both user utilities (reected in recommendation accuracy)
as well as item utilities (reected in aggregate diversity).
As with other beyond-accuracy objectives, there is often a trade-
o between aggregate diversity and personalization or recommen-
dation accuracy: approaches that try to increase aggregate diversity
generally do so at the cost of accuracy degradation. A variety of
approaches have been proposed to manage this trade-o including
optimization approaches using network ow models on the bipar-
tite user-item graph [
2
,
3
,
5
] as well as re-ranking methods involving
utilizing the dual recommendation perspective of recommending
(matching) items to appropriate users [
28
]. These approaches gener-
ally try to balance the diversity-accuracy trade-os either by trying
to optimize for recommendation accuracy while imposing certain
constraints on the items, or by using multi-criteria optimization
where both accuracy and diversity are optimized individually and
the trade-o is controlled through appropriate parameter settings.
arXiv:2006.03715v1 [cs.IR] 5 Jun 2020
We introduce a more principled two-sided approach to managing
the diversity-accuracy trade-o that takes into account both user
and item utilities. User utilities are maximized when the ranking
accuracy of the recommendation lists are maximized. Item utili-
ties, on the other hand, are maximized when the item is matched
with the most suitable list of users. We view the problem as an
assignment problem where the goal is to distribute items to each
user’s recommendation list of size
k
, such that both user and item
utilities are maximized and where each item is subject to a capacity
constraint
C
(i.e. can be recommended at most
C
times). We propose
a generalization of a well-known and ecient algorithm for solving
the two-sided stable matching problem adapted to recommenda-
tion context. Specically, we extend the Deferred Acceptance (also
known as Gale-Shapely) algorithm [
15
] used to solve the college
admission problem. In this problem, given a set of students and set
of colleges with quotas the goal is to nd the best (stable) matching
possible based on the ranking preferences of the two groups. In our
case, the ranking preferences for the user-side and the item-side are
specied based on user and item utility functions, respectively. The
algorithm then automatically nds the best stable match possible
subject to item cap constraints.
We show that the proposed algorithm is user-optimal (maximizes
user utilities) subject to the cap constraint. But the algorithm also at-
tempts to maximize item utilities while maintaining a stable match
(when there is a symmetry between user and item preferences),
and this, in turn, results in higher overall aggregate diversity. From
a practical perspective, our proposed algorithm provides added
exibility as a post-processing approach because it can be used
in conjunction with any top-
N
recommendation algorithm that
can generate preference rankings for users and items. In our ex-
periments, we empirically demonstrate the eectiveness of our
approach in comparison to several baselines and using dierent
underlying recommendation algorithms. Our results show that our
approach is eective in increasing aggregate diversity and item-side
utility while maintaining or improving recommendation accuracy
for end users.
2 RELATED WORK
As noted earlier, improving individual recommendation diversity
has been shown to lead to better user satisfaction [
18
,
23
] and
to reduce the “lter bubble” eect [
21
] aecting many personal-
ized systems. It is also an important consideration for reducing
popularity bias [
22
,
29
] that aects many collaborative recom-
mender systems. There has been an extensive research in develop-
ing methods for increasing individual diversity and in understand-
ing the trade-os between diversity and recommendation accuracy
[
11
,
12
,
24
,
27
,
30
,
31
]. Other related notions aecting the quality
of recommendations to end users, such as novelty and serendipity,
have also been extensively explored [1, 19].
Increasing aggregate diversity, on the other hand helps mitigate
the popularity bias by increasing the coverage of items recom-
mended from the long-tail [
4
,
9
,
22
,
29
]. In prior work, aggregate
diversity has been referred to using dierent terms including as
“sales diversity” [
14
,
17
,
20
,
28
], “catalog coverage”, or simply “cov-
erage” [
19
]. In this paper we use the term “aggregate diversity” to
broadly encompass these other representations of the concept.
In [
2
,
6
] a graph-theoretic approach has been used to increase
aggregate diversity and also maintaining accuracy. Specically, [
6
]
formulated the recommender system as a subgraph selection prob-
lem from a supergraph generated by any recommender system in a
bipartite graph of users and items. Then, they solved this problem
using a Minimum-cost network ow approach. In their approach a
similar constraint has been used to limit the number of times each
item can be recommended. Their goal is not to strictly satisfy this
constraint but to penalize the recommendations that try to deviate
from this constraint. Our approach and methodology is completely
dierent from theirs. First, the mentioned constraint will be always
satised completely in the resulting re-ranked recommendations.
Secondly, the stable matching proposed in this paper is a simple
and ecient (polynomial time complexity) method both in terms
of implementation and ease of understanding. Besides that, our
problem is a two-sided recommendation where utilities of both
users and items are incorporated into the model. However, in their
approach only user-side utilities are considered.
Another line of works which has been proposed recently by
[
8
] is to consider the objectives of dierent multistakeholders in
the system. For example, in any recommender system three stake-
holders can be dened: consumers, providers and the platform for
these two to be connected. Good examples of such platforms are
job recommendation and item recommendation in Amazon. In each
of these platforms the consumers and providers are separated from
the platform and the utility of both parties are important [
7
]. Very
recently [
26
] has formulated this problem using an integer linear
programming approach. Their goal is to maximize utility of both
consumers and providers using an optimization method. Our ap-
proach is similar in terms of overall goals. However, in our problem
we incorporate the utility of items in to the optimization while they
only consider the item utility as a constraint on the optimization
that needs to be satised. In other words, they do not take into
account the ranking preferences of items over users as we do it in
our method. In addition to that, our algorithm is completely dier-
ent and draws on well-established Gale-Shapely algorithm [
15
] for
the stable matching problem. The proposed method in this paper
is computationally more eective than the integer programming
approach.
3 BACKGROUND AND PROBLEM
DEFINITION
In a typical recommender system there are two types of entities:
Users and Items. Personalization is the main feature of recom-
mender systems allowing the system to match users with items
that are consistent with those users’ preferences. A recommender
system can be viewed as a bipartite graph
G=(U,I,µ)
, where
U
is the set of all users,
I
is the set of all items and
µ
is the set of edges
represented as user-items pairs
(u,i)
. The goal of a recommender
system is to rank the items such that preferred items by each user
appear higher in the ranking. The top
N
items in the ranking are
then presented to the user as recommendations. For our purposes,
we do not make any assumptions about which underlying algo-
rithm is used to rank the items, but assume that such a ranking can
be produced. We denote the preference ordering of user
u
on all
items in
I
based on the recommender system ranking by
≻u
. We
say user uprefers items i1to i2if i1≻ui2.
The preference ordering can be used to dene the notion of
user utility. As we shall see in the following, we use Normalized
Discounted Cumulative Gain (NDCG) to measure user utilities,
however, other ranking measures can also be used for this purpose.
In the typical recommendation setting, the goal is usually to max-
imize user utilities. However, a single-minded focus on user utilities
without considering the utility of items may result in an unfair or
uneven distribution of items among all the recommendation lists. In
reality, there are often few popular items that receive the lion share
of user attention in the system. Especially in collaborative recom-
mender systems, the abundance of user feedback on popular items
results in recommendations that are biased towards these items,
perpetuating the popularity bias. Thus, one-sided recommender
systems (that are focused solely on maximizing user utilities, tend
to produce a power law distribution of recommended items and an
unfair platform for most of the items to compete with a few popular
items. This, in turn, may reduce the incentive to participate on the
part of many sellers or producers.
Ideally, the distribution of recommended items should be uni-
form with maximum entropy. However, since the user preferences
are concentrated over a small set of items, any attempts to “atten”
the distribution of items in recommendations will result in signif-
icant loss in accuracy and personalization eectiveness. Ideally,
we would like to maximize the entropy of item distribution across
recommendations while minimizing accuracy loss.
One way to control the entropy of item distribution is to impose
a constraint
C
over the capacity of items (the number of times
each item can be recommended
ni
). This is the approach taken
in several prior methods that aim to increase aggregate diversity
while optimizing user utilities [
6
]. This approach,however, ignores
potentially relevant information available on the item side of the
recommendation platform. Just as user preference rankings of items
can be used as the basis for user-side utility function to provide the
most relevant items to users, a dual notion of item utility can be
used to represent the best match between each item and a ranked
list of users. Indeed, increasing aggregate diversity in a brute-force
fashion does not necessarily mean that each item is matched with
the most appropriate set of users.
Our goal in this work is to automatically achieve a balance be-
tween accuracy and aggregate diversity that takes into account
both user and item utilities. Specically, our approach re-ranks the
recommendations for users by maximizing both user and item util-
ities subject to a capacity constraint
ni≤C
in such a way to create
a stable match between users and items. We do so by extending the
well-known Gale-Shapely or Deferred Acceptance algorithm [
15
]
in order to handle the many-to-many matching problem which is
required in the context of recommender systems. In the following
we provide a more formal statement of the problem and in the next
section we present the details of our algorithm and discuss some of
its properties.
User Utility:
We use Normalized Discounted Cumulative Gain (NDCG) as
the basis for dening our notion of utility. NDCG is a well-known
measure of ranking quality in information retrieval which measures
the gain of each item based on the change in the ranking position of
the item in a ranked list of items. Given the ranking preference of a
user
≻u
and her recommendation list
L
, the Discounted Cumulative
Gain (DCG) is dened as:
DCG(L,≻u)=Õ
i∈L
rel(i,≻u)
loд2(i+1)(1)
Relevance scores can be dened in a small range of values such
as
rel =(
0
,rmax )
. The relevance score of each item for user
u
is
dened based on its ranking position in ≻u:
rel(i,≻u)=rank−1(i,≻u)
rmax
(2)
where the numerator is the reversed ranking position of item
i
in
user u’s ranking preference ≻u.
Given a user ranking preference order
≻u
and any other item
ranking L, the gain in user u’s utility relative to Lis dened as:
N DCGu(L)=DCG(L,≻u)
IDCG(u)(3)
where
IDCG(u)
is the ideal
DCG
of user
u
in which the relevance
scores for all items in equation 1 is equal to the maximum value
rel(i)=rmax .
In a one-sided recommendation setting the problem can be for-
mulated as a user utility optimization problem subject to certain
constraints. Specically, for each user
u
a recommendation list is a
set of user-item pairs
µu
. The user-side utility of the matching
µu
is computed using
N DCGu(µu)
. Therefore, given the item capacity
C
and set of all the matchings (recommendations)
µ={Ðu∈U µu}
the problem formulation would be as follows:
maximize
µU til ityU(µ)
subject to ni≤C,i∈ I.
(4)
where
U til ityU(µ)=N DC GU(µ)
and
ni
is the number of times
item iis recommended.
As noted earlier, however, our goal is to optimize both user and
item utilities.
Item Utility:
Similar to user-side utility, the item utility is dened by an item-
side version of
N DCG
. Given an item
i
ranking preference
≻i
over
all the users, the gain in item
i
’s utility given a matching
µi=
{(u,i),(u′,i), ...}is dened as follows:
U til ityI(µi)=N DC Gi(µi)=DCG(µi,≻i)
IDCG(i)(5)
Similar to equation 3, the ideal
DCG
for item
i
is denoted by
IDCG(i)
.
In a two-sided recommendation the problem can be dened as
simultaneously optimizing utilities of users and items:
maximize
µU til ityU(µ),U ti lit yI(µ)
subject to ni≤C,i∈ I.
(6)
It is important to note that there is an asymmetry between rank-
ing preferences of items and those of users. This means that utilities
of each side is derived independently and are not necessarily con-
sistent with each other. For example, if user
u
prefers item
i
the
most, it does not always hold that in item
i
’s preference ordering,
uis the highest ranked user, i.e.,
{∀i′∈I:(i≻ui′)} ̸⇐⇒ {∀u′∈U:(u≻iu′)} (7)
Therefore, nding a matching that maximizes the utility of one side
does not necessarily maximize the utility of the other side. Popular-
ity bias increases this asymmetry and this makes the simultaneous
optimization of both sides very dicult requiring an intelligent
way to manage the trade-o point between the two utilities.
4 STABLE MATCHING AND DEFERRED
ACCEPTANCE ALGORITHM
In the previous section we formulated the assignment problem such
that the utilities of both user and item sides are considered. In this
section, we introduce an ecient solution to this problem and show
the correctness of the algorithm.
The Deferred Acceptance (also known as Gale-Shapely) algo-
rithm [
15
] is a well-known method for solving the college admission
problem. In this problem, given a set of students and set of colleges
with quotas the goal is to nd the best matching possible based on
the ranking preferences of the two groups. Gale-Shapely’s method
is based on the key notion of a “stable match”.
In the recommendation context, a matching
µ
is said to be unsta-
ble if there exists another matching
µ′
where both utilities of users
and items have increased in
µ′
compared to
µ
. In other words, if
i≻ui′
and
u≻iu′
and also
{(u,i′),(u′,i)} ∈ µ
, then
µ
is unstable
because both
u
and
i
can be better matches for each other and
therefore both user and item utilities are higher in
µ′
. A matching
µ
is stable if no such unstable match exist for every pair of matches
{(u,i),(u′,i′)} ∈ µ
. Gale and Shapely showed that a stable match
always exists and the Deferred Acceptance algorithm always nds a
stable match. The output of this algorithm is a one-to-many match-
ing where each college will be matched to many students based on
their quotas. The ingenuity of the Gale-Shapely’s method is in this
intuitive strategy of solving a combinatorial optimization problem.
The input to the Deferred Acceptance Algorithm is a fully-
connected directed bipartite graph connecting users and items.
There are directed edges in each direction between user and item
nodes. There is a preference ranking for each node on each side of
this graph. The algorithm begins to match user-item pairs starting
from one side of the bipartite graph (here we assume it starts from
the user side). A user
u
is matched with an item
i
in increasing
order of her rankings. At a given rank
k
, two situations are possible.
First, item
ik
(
k
th item in user’s ranking) may not be currently
matched with any user. In this case, a match
(u,ik)
is created. On
the other hand, item
ik
may already be part of a match with another
user
v
. In this case if
ik
prefers
v
over
u
, then user
u
’s next ranked
item is considered since the current match
(v,ik)
is more stable.
Otherwise, it is replaced with the more stable match
(u,ik)
and
user
v
is considered again for subsequent matches. This process is
repeated until all users are matched with items. This would be the
one-to-one matching of Deferred Acceptance algorithm.
We extend this algorithm by allowing the members of each side to
have more than one matching. This way each user can be matched
Algorithm 1: MMDA: Many-to-Many Deferred Acceptance
Input:
The ranking preference of users and items:
{≻u
and
≻i
:
∀u∈ U and i∈ I};k: recommendation list size for
each user; ci: item caps.
Output: User-Item Matching µ
1µ← ∅
2for u∈ U do
3Su← ∅
4end
5while {∃u||µu|<k}do
6i←Top (≻u<Su)
7µu←µu∪i
8Su←Su∪i
9if |µi|>cithen
10 Φi←sort−1(µi)subject to ≻i.
11 µi←µi\{∀u∈µi|rank(u∈Φi)>ci}
12 end
13 end
14 return µ
to many items and hence receive these items as recommendations.
We introduce the
cap
for items as a constraint to limit the number of
times an item can be matched (recommended) to users. Therefore,
the combinatorial challenge of this problem would be to nd the best
matching that cannot be improved further (no other matching exist
that would be better o for both sides). We call this algorithm the
Many-to-Many Deferred Acceptance (MMDA) algorithm (see Algo-
rithm 1). Next, we introduce the notation that is required for this
algorithm. Two functions are used to work with preference rank-
ings in the algorithm. The rst function is
Top (≻u<Su)
which
returns the top-ranked item
i
from
≻u
where
i
does not belong
to
Su
. The second function is
[sort−1(µi),subject to ≻i]
which
reverse sort (or rank) the items in
µi
based on ranking preferences
in ≻i.
The algorithm starts with
µ
set to empty set. Also, for each user
we initialize an empty set to
Su
for the set of selected items by
u
.
As we iterate over users’ preference rankings
≻u
in the main loop
of the algorithm (line 5), we add items to
Su
. The main while-loop
continues until we assign
k
items to each user’s matching set
µu
.
This makes sure that we have recommended at least
k
item to each
user. Next at line 6, we set item
i
as the top preferred item in user
u
’s ranking which has not yet been selected. At line 7 we assign
item
i
to the user
u
’s matching and also, adding
i
to the selected
items of this user
Su
. At line 9, we check to see whether any of
the item constraints are violated. If so, we should nd the ranked
list (subject to
≻i
) of assigned users to item
i
and, reject the least
preferred user for item
i
by removing it from the assigned users
µi
to item
i
. Next, we discuss the observations and theorems that are
followed by MMDA.
Observation 1. Once item
i
is assigned to a user
u
, in later itera-
tions of the algorithm either the assignment
(u,i)
remains the same
or it gets better
(u′,i)
where
u′≻iu
. In other words, the utility of
each item’s assignment is monotonically non-decreasing.
Observation 2. On the other hand, individual users’ utilities get
worse over later iterations of the algorithm. Users always start with
their best options and in case of rejections they look for less preferred
items in their ranking preference.
Theorem 3. MMDA always nds a stable matching.
Proof.
(by contradiction). Let’s assume there is an assignment
which makes
µ
from algorithm 1 unstable. Therefore, by denition
there should be an unstable assignment for users
u
and
u′
and
items
i
and
i′
, where all of the followings are true:
u≻iu′
,
u≻iu′
,
(u,i)<µ,(u,i′) ∈ µ, and (u′,i) ∈ µ.
Since
u
prefers
i
, according to the observation 2 the algorithm
should assign
i
before assigning
i′
to
u
. Now, there can be two
possible cases for u′:
Case 1: u′
is assigned to
i
in a previous iteration. At this point,
u
rst selects
i
. Now,
i
should decide between
u′
and
u
. Based on
observation 1 items always choose the better user according to their
ranking preferences. Therefore,
µ
rejects the previous assignment
(u′,i)
and accept the new assignment
(u,i)
. This contradicts the
assumption that {(u,i′),(u′,i)} ∈ µ.
Case 2: u′
is assigned to
i
after
u
selecting item
i
and matched with
it in
(u,i)
. Since, by denition
i
prefers
u
to
u′
, then
i
will remain
matched with ucontradicting the instability assumption. □
Theorem 4. MMDA is user-optimal.
Proof. (by contradiction). Let’s assume that the output µof al-
gorithm 1 is not user-optimal. Therefore, there should exist another
matching
µ′
where at least a pair of assignments is dierent so
that
U til ityU(µ′)>U t ili tyU(µ)
. We can represent the dierence
between µand µ′is as follows:
µ={(u,i),(u′,i′)} ∪ {µ∩µ′}
µ′={(u,i′),(u′,i)} ∪ {µ∩µ′}
In the following, we show that if
µ′
exists with a greater user utility
satisfying the constraint
ni≤C
, then there is a contradiction. To
show this we consider two cases:
Case 1:
(
i≻ui′
). If user
u
prefers item
i
more than
i′
, then the
user-side utility of
µ
is greater than
µ′
and therefore
µ
is more
user-optimal than µ′.
Case 2:
(
i′≻ui
). In this, case assuming that there is a competition
between
u
and
u′
, user
u′
should also be more interested in
i′
. There
can be two cases with respect to the preferences of item
i′
: either
u≻i′u′or u′≻i′u. Again we break down these two cases.
Case 2.1:
Our assumptions are:
i′≻ui
,
i′≻u′i
,
u≻i′u′
. Based
on these assumptions for this case
i′
prefers
u
over
u′
and also
u
prefers
i′
over
i
. Then by denition the existence of assignment
(u,i) ∈ µ, makes µunstable. Which contradicts with theorem 3.
Case 2.2: (u′≻i′u)
. Since,
i′≻u′i
is true as an assumption, then
both user and item utilities are greater for µthan µ′.□
Note that there may be many stable matchings but there is only
one user-optimal stable matching (given that we assume users
make the rst selection of items in their ranking preferences over
items). Also, in the situation where too many users (more than
the cap) are interested in one item users who are not preferred
by that item will be rejected in order to maintain the stability of
the match. In the experiments we show how aggregated user and
item utilities increase with each given iteration of the algorithm.
The monotonicity of the utilities provides empirical evidence of the
correctness of this algorithm.
The computational complexity of the MMDA is
O(n×m×k loдk)
where
n
and
m
are the number of users and items in the system.
The worst-case scenario is when every user should go over all of
his/her ranking preferences of
m
items at most and each time sort
the items in µiwhere |µi|=kis the recommendation list size.
Controlling utility of each side can be done using item caps.
MMDA strictly tries to satisfy the items’ capacity constraint which
is the only parameter of this algorithm. However, there is a limit to
the increase of item-side utility if the algorithm is user-optimal. To
go beyond this limitation the item-optimal version of this algorithm
can be used where items start selecting users rst based on their
preferences.
5 EXPERIMENTS
In the previous section we proved that the MMDA algorithm is both
stable and user-optimal. In this section we provide further empirical
evidence for the eectiveness of the algorithm using three dierent
data sets and under dierent conditions. In our experiments the
resulting recommendations are evaluated for each side in isolation.
On the user side, we use traditional evaluation metrics for accuracy
and individual diversity of recommendations, while on the item-
side we measure changes in item utilities. We also use standard
metrics such as the Gini-index to measure changes in aggregate
diversity (see Section 5.3). First, we show the impact of MMDA
re-ranking on each of the user and item sides by measuring various
metrics. Secondly, we show the eectiveness of MMDA in dealing
with user-item utility trade-os. Also, we illustrate the impact of
setting dierent capacity values for items.
5.1 Datasets
In our analysis we used three publicly available datasets. The rst
dataset is MovieLens 1M (ML)
1
, a specic dataset for movie recom-
mendation which is widely used in Recommender system’s domain.
ML contains 6,040 users and 3,702 movies and 1M ratings. Sparsity
of ratings in this dataset is about 96%.
Two other datasets used in our experiments are from Amazon
categories [
16
]: Amazon Books and Amazon Movies. The original
Amazon books contains 2.5M users and about 1M items (books). Due
to huge sparsity of this dataset we have randomly selected a subset
of dataset in which each user has rated at least 20 items and each
item has received at least 20 ratings. The resulting dataset reduced
to 10K users and 9K items. The same process has been applied to
the Amazon Movies dataset. Our sampled dataset contains 6.5K
users and 6K items. The sparsity of the ratings are 72.3% and 98.5%
for Amazon books and Amazon movies, respectively.
5.2 Experimental Settings
MMDA is a post processing method which re-ranks the previously
generated recommendations. We use two well-known collaborative
ltering methods: User-based kNN [
25
] and Non-negative Matrix
Factorization (NMF) [
10
,
13
] as the underlying algorithms to gener-
ate the initial rankings. Utility of users are dened based on item
1https://grouplens.org/datasets/movielens
rankings from these recommendations. On the other hand, utility
of items can either be explicitly dened by item sellers, or they can
be implicitly derived by recommending users to items [
28
]. Since,
the seller preferences were not available to us, we used the implicit
approach to dene the item utilities.
For each dataset and recommendation method we rst generated
recommendations and then using the MMDA re-ranking method
and predened constraints (dierent
cap
s can be used for each item)
we generated the nal recommendations. The results are shown
in total of 6 tables (Tables 1 through 6) corresponding to dierent
dataset and recommender combinations.
We split the data into training, validation and testing sets with
80/10/10 percent ratios. For each dataset, we have tuned the pa-
rameters of base recommendation methods so that the accuracy
(precision) is the maximized over the validation set. The parame-
ters of the recommender methods used in our experiments are as
follows. In MovieLens dataset we set the number of neighbors to
10 in User-based kNN. Also, in same dataset the number of latent
factors is set to 40 when using NMF. For the Amazon datasets (both
books and movies) we set the number of neighbors to 40 for kNN
and the number of latent factors is set to 400 in NMF.
We used two baseline re-ranking methods to compare to our
approach.
Bayesian Re-ranking (BR)
[
28
]: Another re-ranking method
specically designed to increase catalog coverage. Given the score
of each user-item pair s(u,i), the BR re-scoring is dened by:
sBR (u,i)=s(u,i)Õ
u′∈U
s(u′,i)−α(8)
We have tuned the only hyper-parameter of this method
α
using
grid search. Not all the values of
α
produce desired results. For
example, setting values larger than 0
.
01 does not change neither
precision, nor gini-index.
Bi-criteria optimization using network-ow (NetFlow)
[
6
]:
We introduced this paper in the related work section. This method
tries to optimize for two objectives: minimum discrepancy and max-
imum relevance and formulates the objective as
[λ×discrepancy(H)
−
µ×relevance (H)]
(where
H
stands for hyper-graph that is the orig-
inal recommendations). We have tuned the two parameters (
λ
and
µ
) such that to obtain maximum accuracy at the various values of
aggregate diversity (Gini-index in our experiments, see section 5.3).
This is exactly the same approach that we used for tuning the pa-
rameters of our method. The hyper-graph used in our experiments
include all the recommended items available in the catalog and
their corresponding scores obtained from the recommender system
(either NMF or user-based kNN).
5.3 Evaluation Metrics
Accuracy-based measures of utility:
User-side Utility (Precision
@
k)
: We use precision to measure the
accuracy of re-ranked results.
Pr ecision
@
k
is the average of fraction
of relevant items in each user’s recommendation list Lu.
Pr ecision@k=1
|U | Õ
u∈U
|Lu∩ T
u|(9)
Figure 1: Amazon Movies: Utilities of both sides at each iter-
ation of the MMDA, item cap=15, base recommender=NMF.
Figure 2: Amazon Movies: Utilities of both sides given vari-
ous item caps c in MMDA.
Figure 3: Amazon Movies: Precision@10 and Gini-index
given various item caps c in MMDA.
where
k=|Lu|
is the recommendation list size and
T
u
is the user
u’s test set.
Aggregate Diversity Measures:
Gini-Index
: This is the measure of fair distribution of items into
recommendation lists of all the users. The ideal (maximum fairness)
case is when this distribution is uniform. The Gini-index of uniform
distribution is equal to zero and so smaller values of Gini-index are
desired.
Gini =1
|I |
|I |
Õ
k=1
(2k− |I| − 1) × p(ik|L),(10)
where
L
is the set of all the recommendations for users and
p(ik|L)
is the probability of the
k
-th least recommended item being drawn
from the recommendation lists in L:
p(i|L)=|{u∈ U|i∈Lu}|
Íj∈I | {u∈ U|j∈Lu}| (11)
Catalog coverage is another widely used metric dened as the
fraction of all the items that has been recommended at least once in
all recommendation lists. Although, it used extensively as a measure
of aggregate diversity, it is not a reliable measure. The main reason
is that it is prone to noise. In a typical recommendation dataset, you
would nd thousands of users. Just by recommending an item once,
it is not actually considered as covered. In that sense, Gini-index
is much more reliable and more dicult metric to optimize. Note
that in our tables in order to make results easier to understand
the inverse of gini-index is shown. Therefore, the greater values of
gini-index are more desired.
Intra-List Distance (ILD)
: This is the measure of individual user
diversity [
11
,
31
], which is a pairwise distance between all the items
in each user’s recommendation list.
ILD(Lu)=1
|Lu|(|Lu| − 1)Õ
i∈Lu
Õ
j∈Lu
d(i,j)(12)
Where
d(i,j)
is the Cosine distance between two items based on
ratings data. Note that, while our direct objective is not increas-
ing individual diversity, we nevertheless measured ILD to explore
possible correlations between aggregate diversity and individual
diversity.
User and Item Utilities:
We dened these utilities based on the
notion of NDCG and the ranking distance from each side’s ranking
preferences. The precise denition was provided in Section 3.
5.4 Results and Discussion
Figure 1 shows the increase in user and item utilities over successive
iterations of MMDA. Interestingly, in the beginning both utilities
are increasing at the same pace but over time the slope of the lines
shows more gain in item utility versus user utility. This is perfectly
aligned with our observations 2 and 1 in the previous section.
Also, we ran experiments on the Amazon Movies data set to
observe the eect of specifying various constraints for item ca-
pacity values cap in MMDA. Note that since this is an assignment
problem (meaning that for each user there are
k
empty positions
to assign items into those positions as recommendations) the
cap
value cannot be smaller (lower bound) than
|U | × k/| I|
where
k
is the recommendation list size. Choosing values smaller than the
lower bound will result in empty recommendation lists for some
users. Figures 2 and 3 show these results in terms of utilities (item-
side versus user-side) and accuracy-diversity trade-o, respectively.
In gure 3 the range of Gini-index values in x-axis are much larger
than precision values in y-axis. This shows the eectiveness of
MMDA in distributing items such that the loss in accuracy is mini-
mized. To the best of our knowledge no other re-ranking method
is capable of competing with MMDA when very low Gini values
(close to uniform distribution of items) is demanded. This would
be the major advantage of the MMDA compared to the baseline
methods as shown in the tables.
Figure 2 shows the performance of MMDA given various
cap
s.
Note that the range of change in user utility is much smaller com-
pared to the range of changes in item utility. This reects the ef-
fectiveness of MMDA in managing the trade-o in utilities such
that with little loss in user-side utility, the gain in item-side utility
would be substantial.
Among all of the existing methods in the literature our method
is the only one that can approach uniform distribution of items
(close to zero on the Gini-index in gure 3). More interestingly,
this huge lift in aggregate diversity does not result in a huge loss
of accuracy. Therefore, our approach eectively deals with the
accuracy-diversity trade o. Also, the increase in aggregated diver-
sity increases individual recommendation lists’ diversity which is
benecial both to users and item providers.
Results on tables 1-6 show a comparative evaluation of dierent
re-ranking methods using metrics discussed in section 5.3. Note
that in order to make the tables more understandable we reversed
the values of gini-index such that larger values of any metric are
desired. Tables 1 and 2 show the results for MovieLens dataset. The
concentration bias of ratings over popular items in training set
of this dataset is much more than other two datasets. This makes
managing the trade-o between accuracy and diversity more chal-
lenging. We can see all of the baseline methods are bound from
below in terms of Gini values. The Gini values of MMDA on the
other hand are not restricted to a specic range. For example, the
Gini values for MMDA with
cap =
17 and
cap =
20 are much
smaller representing near uniform item distribution across recom-
mendations with the cost of losing about 12% in precision. (also
compare it to the baselines) On the other datasets, the concentration
bias is slightly smaller and therefore MMDA is able to maintain a
high level of precision while decreasing Gini values (thus increasing
aggregate diversity) signicantly.
Even though our goal was not to increase individual diversity,
ILD measures in almost all of our results show the eectiveness of
MMDA in increasing Intra-list diversity of recommendations. This
suggest that both user and item sides are beneting from MMDA
re-ranking results.
6 CONCLUSION AND FUTURE WORKS
We have proposed a two-sided re-ranking recommendation ap-
proach that takes into account both item and user utilities in order
to produce the best possible match between items and users. Based
on an extension of the Deferred Acceptance algorithm for solving
the well-known stable matching problem, our method is guaran-
teed to always produce a stable match in an ecient manner. We
have proved this property of the algorithm and we have also shown
that it is user-optimal subject to specied item cap constraints.
These properties allow our algorithm to automatically manage the
trade-o between aggregate diversity objective on the item side and
the accuracy objective on the user side. This is a post-processing
NMF BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.0
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=0.99
µ=0.01
MMDA
ca p=17
MMDA
ca p=20
MMDA
ca p=50
MMDA
ca p=80
Precision@10 22.4 6.5 20.6 25.2 2.2 16.7 21.7 8.7 9.7 14.7 17.9
Gini(reversed) 6.3 6.1 13.0 8.0 85.3 35.1 16.6 95.9 81.6 35.1 20.7
Avд(p r ec ,дini)14.3 6.3 16.8 16.6 43.8 25.9 19.2 52.3 45.6 24.9 19.3
U t il it yU100.0 97.9 99.8 99.9 74.3 98.7 99.7 85.0 89.4 99.2 99.3
U t il it yI16.9 14.5 28.3 23.4 65.1 56.0 38.0 84.4 73.8 47.9 24.0
Avд(U t il it ie s)58.5 56.2 64.0 61.7 69.7 77.4 68.8 84.7 81.6 73.5 61.6
ILD 0.487 0.636 0.570 0.487 0.755 0.741 0.487 0.898 0.846 0.594 0.618
Table 1: MovieLens 1M dataset, NMF recommendations.
kNN BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=0.8
µ=0.2
MMDA
ca p=17
MMDA
ca p=20
MMDA
ca p=50
MMDA
ca p=80
Precision@10 22.4 3.1 17.5 21.6 2.5 6.7 9.1 9.4 10.1 14.6 16.9
Gini(reversed) 8.1 14.6 20.7 11.5 86.2 86.3 71.2 96.0 81.8 35.1 23.2
Avд(p r ec ,дini)15.2 8.9 19.1 16.6 44.4 46.5 40.1 52.7 46.0 24.8 20.1
U t il it yU100.0 97.3 99.9 100.0 77.3 96.2 98.4 86.3 93.8 99.2 99.6
U t il it yI31.2 30.7 51.1 43.2 66.0 74.1 71.6 82.8 73.3 47.9 41.1
Avд(U t il it ie s)65.6 64.0 75.5 71.6 71.7 85.2 85.0 84.5 83.5 73.5 70.3
ILD 0.512 0.908 0.539 0.505 0.771 0.883 0.893 0.964 0.925 0.659 0.634
Table 2: MovieLens 1M dataset, User-kNN recommendations.
NMF BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=0.8
µ=0.2
NetFlow
λ=0.01
µ=0.99
MMDA
ca p=12
MMDA
ca p=15
MMDA
ca p=20
MMDA
ca p=50
Precision@10 7.2 1.9 3.9 6.8 1.0 1.4 2.7 4.3 4.7 5.4 6.5
Gini(reversed) 11.1 15.3 32.0 20.4 85.7 85.5 7.5 88.6 71.9 55.2 24.6
Avд(p r ec ,дini)9.1 8.6 17.9 13.6 43.3 43.5 5.1 46.4 38.3 30.3 15.5
U t il it yU100.0 99.0 99.6 99.9 85.7 87.3 98.2 98.6 99.1 99.5 99.9
U t il it yI41.6 43.5 74.2 67.6 77.4 79.5 21.4 93.7 87.4 78.3 55.4
Avд(U t il it ie s)70.8 71.2 86.9 83.8 81.6 83.4 59.8 96.2 93.2 88.9 77.7
ILD 0.530 0.783 0.666 0.562 0.946 0.962 0.965 0.819 0.818 0.767 0.707
Table 3: Amazon Movies dataset, NMF recommendations.
kNN BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.0
NetFlow
λ=0.99
µ=0.01
NetFlow
λ=0.01
µ=0.99
MMDA
ca p=12
MMDA
ca p=15
MMDA
ca p=20
MMDA
ca p=50
Precision@10 7.1 1.4 4.9 6.9 1.1 3.5 7.0 5.4 5.8 6.2 7.0
Gini(reversed) 13.9 21.4 51.5 28.7 85.4 85.8 14.1 88.9 73.6 58.5 30.1
Avд(p r ec ,дini)10.5 11.4 28.2 17.8 43.2 44.6 10.6 47.2 39.7 32.4 18.6
U t il it yU100.0 98.7 99.7 99.9 88.6 99.4 100.0 99.2 99.5 99.6 99.9
U t il it yI49.0 30.0 77.2 71.1 63.2 74.5 49.5 81.4 76.6 72.2 60.5
Avд(U t il it ie s)74.5 64.3 88.5 85.5 75.9 87.0 74.8 90.3 88.0 85.9 80.2
ILD 0.690 0.838 0.752 0.745 0.964 0.98 0.922 0.810 0.757 0.718 0.722
Table 4: Amazon Movies dataset, User-kNN recommendations.
NMF BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.0
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=0.8
µ=0.2
MMDA
ca p=12
MMDA
ca p=15
MMDA
ca p=20
MMDA
ca p=50
Precision@10 14.3 3.6 4.2 11.5 2.4 8.6 14.2 9.4 10.4 11.4 13.8
Gini(reversed) 18.2 16.1 20.3 33.7 87.5 85.8 19.9 96.2 77.5 59.0 26.6
Avд(p r ec ,дini)16.2 9.9 12.2 22.6 45.0 47.2 17.0 52.8 44.0 35.2 20.2
U t il it yU100.0 99.4 99.5 99.9 90.9 99.6 100.0 98.6 99.5 99.8 99.9
U t il it yI47.8 37.8 61.9 74.7 85.6 97.3 52.1 96.3 88.2 77.7 53.9
Avд(U t il it ie s)73.9 68.6 80.7 87.3 88.2 98.4 76.0 97.4 93.8 88.8 76.9
ILD 0.845 0.921 0.921 0.897 0.97 0.939 0.845 0.94 0.906 0.903 0.883
Table 5: Amazon Books dataset, NMF recommendations.
kNN BR
α=0.01
BR
α=1e−3
BR
α=4e−4
NetFlow
λ=2.0
µ=0.01
NetFlow
λ=0.99
µ=0.01
NetFlow
λ=0.8
µ=0.2
MMDA
ca p=12
MMDA
ca p=15
MMDA
ca p=20
MMDA
ca p=50
Precision@10 12.8 2.4 10.3 11.4 3.6 7.0 12.6 10.1 10.7 11.4 12.6
Gini(reversed) 16.0 87.1 85.5 72.2 18.0 36.7 39.7 96.3 78.2 61.5 33.1
Avд(p r ec ,дini)14.4 44.8 47.9 41.8 10.8 21.9 26.2 53.2 44.5 36.5 22.9
U t il it yU100 92.7 99.7 99.8 99.2 99.5 99.9 97.9 99.4 99.7 99.9
U t il it yI59.7 75.3 94.9 93.8 34.9 87.2 89.4 94.7 91.1 86.8 74.5
Avд(U t il it ie s)79.8 84.0 97.3 96.8 67.0 93.3 94.7 96.3 95.2 93.2 87.2
ILD 0.831 0.918 0.883 0.883 0.955 0.935 0.856 0.965 0.93 0.942 0.888
Table 6: Amazon Books dataset, User-kNN recommendations.
approach that can be used in conjunction with any underlying
collaborative recommendation algorithm, adding to its exibility.
Our experimental evaluation using three datasets has provided
additional evidence for the eectiveness of our approach in increas-
ing aggregate diversity (and more generally item utilities) while
improving or maintaining recommendation accuracy.
In future work, we intend to experiment with additional base
recommendation algorithms to determine the impact of those al-
gorithms on item and user utilities resulting from the matching
algorithm. In addition we will explore the possibility of using dier-
ent underlying recommendation algorithms on the item and user
sides to improve the accuracy of initial item and user preference
orderings, respectively.
REFERENCES
[1]
Panagiotis Adamopoulos and Alexander Tuzhilin. 2015. On unexpectedness in
recommender systems: Or how to better expect the unexpected. ACM Transactions
on Intelligent Systems and Technology (TIST) 5, 4 (2015), 54.
[2]
Gediminas Adomavicius and YoungOk Kwon. 2011. Maximizing aggregate recom-
mendation diversity: A graph-theoretic approach. In Proc. of the 1st International
Workshop on Novelty and Diversity in Recommender Systems (DiveRS 2011). Cite-
seer, 3–10.
[3]
Gediminas Adomavicius and YoungOk Kwon. 2012. Improving aggregate rec-
ommendation diversity using ranking-based techniques. IEEE Transactions on
Knowledge and Data Engineering 24, 5 (2012), 896–911.
[4]
Chris Anderson. 2006. The long tail: Why the future of business is selling less of
more. Hachette Books.
[5]
Arda Antikacioglu and R Ravi. 2017. Post processing recommender systems for
diversity. In Proceedings of the 23rd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining. ACM, 707–716.
[6]
Arda Antikacioglu and R Ravi. 2017. Post processing recommender systems for
diversity. In Proceedings of the 23rd ACM SIGKDD International Conference on
Knowledge Discovery and Data Mining. ACM, 707–716.
[7]
Robin Burke, Nasim Sonboli, and Aldo Ordonez-Gauger. 2018. Balanced neigh-
borhoods for multi-sided fairness in recommendation. In Conference on Fairness,
Accountability and Transparency. 202–214.
[8] Robin D Burke, Himan Abdollahpouri, Bamshad Mobasher, and Trinadh Gupta.
[n. d.]. Towards Multi-Stakeholder Utility Evaluation of Recommender Systems.
[9]
Òscar Celma and Pedro Cano. 2008. From hits to niches?: or how popular artists
can bias music recommendation and discovery. In Proceedings of the 2nd KDD
Workshop on Large-Scale Recommender Systems and the Netix Prize Competition.
ACM, 5.
[10]
Andrzej Cichocki and Anh-Huy Phan. 2009. Fast local algorithms for large scale
nonnegative matrix and tensor factorizations. IEICE transactions on fundamentals
of electronics, communications and computer sciences 92, 3 (2009), 708–721.
[11]
Farzad Eskandanian, Bamshad Mobasher, and Robin Burke. 2017. A Clustering
Approach for Personalizing Diversity in Collaborative Recommender Systems.
In Proceedings of the 25th Conference on User Modeling, Adaptation and Personal-
ization. ACM, 280–284.
[12]
Farzad Eskandanian, Bamshad Mobasher, and Robin D Burke. [n. d.]. User
Segmentation for Controlling Recommendation Diversity.
[13]
Cédric Févotte and Jérôme Idier. 2011. Algorithms for nonnegative matrix factor-
ization with the β-divergence. Neural computation 23, 9 (2011), 2421–2456.
[14]
Daniel Fleder and Kartik Hosanagar. 2009. Blockbuster culture’s next rise or fall:
The impact of recommender systems on sales diversity. Management science 55,
5 (2009), 697–712.
[15]
David Gale and Lloyd S Shapley. 1962. College admissions and the stability of
marriage. The American Mathematical Monthly 69, 1 (1962), 9–15.
[16]
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual
evolution of fashion trends with one-class collaborative ltering. In proceedings
of the 25th international conference on world wide web. International World Wide
Web Conferences Steering Committee, 507–517.
[17]
Dietmar Jannach, Lukas Lerche, Fatih Gedikli, and Georay Bonnin. 2013. What
recommenders recommend–an analysis of accuracy, popularity, and sales di-
versity eects. In International Conference on User Modeling, Adaptation, and
Personalization. Springer, 25–37.
[18]
Nicolas Jones and Pearl Pu. 2007. User technology adoption issues in recom-
mender systems. In Proceedings of the 2007 Networking and Electronic Commerce
Research Conference. 379–394.
[19]
Marius Kaminskas and Derek Bridge. 2016. Diversity, Serendipity, Novelty, and
Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in
Recommender Systems. ACM Trans. Interact. Intell. Syst. 7, 1, Article 2 (Dec. 2016),
42 pages. https://doi.org/10.1145/2926720
[20]
John Paul Kelly and Derek Bridge. 2006. Enhancing the diversity of conversational
collaborative recommendations: a comparison. Articial Intelligence Review 25,
1-2 (2006), 79–95.
[21]
Tien T Nguyen, Pik-Mai Hui, F Maxwell Harper, Loren Terveen, and Joseph A
Konstan. 2014. Exploring the lter bubble: the eect of using recommender
systems on content diversity. In Proceedings of the 23rd international conference
on World wide web. ACM, 677–686.
[22]
Yoon-Joo Park and Alexander Tuzhilin. 2008. The long tail of recommender
systems and how to leverage it. In Proceedings of the 2008 ACM conference on
Recommender systems. ACM, 11–18.
[23]
Pearl Pu, Li Chen, and Rong Hu. 2011. A user-centric evaluation framework for
recommender systems. In Proceedings of the fth ACM conference on Recommender
systems. ACM, 157–164.
[24]
Shameem A Puthiya Parambath, Nicolas Usunier, and Yves Grandvalet. 2016. A
coverage-based approach to recommendation diversity on similarity graph. In
Proceedings of the 10th ACM Conference on Recommender Systems. ACM, 15–22.
[25]
Badrul Munir Sarwar, George Karypis, Joseph A Konstan, John Riedl, et al
.
2001.
Item-based collaborative ltering recommendation algorithms. WWW 1 (2001),
285–295.
[26]
Özge Sürer, Robin Burke,and Edward C Malthouse. 2018. Multistakeholder recom-
mendation with provider constraints. In Proceedings of the 12th ACM Conference
on Recommender Systems. ACM, 54–62.
[27]
Saúl Vargas and Pablo Castells. 2011. Rank and relevance in novelty and diversity
metrics for recommender systems. In Proceedings of the fth ACM conference on
Recommender systems. ACM, 109–116.
[28]
Saúl Vargas and Pablo Castells. 2014. Improving sales diversity by recommending
users to items. In Proceedings of the 8th ACM Conference on Recommender systems.
ACM, 145–152.
[29]
Hongzhi Yin, Bin Cui, Jing Li, Junjie Yao, and Chen Chen. 2012. Challenging
the long tail recommendation. Proceedings of the VLDB Endowment 5, 9 (2012),
896–907.
[30]
Mi Zhang and Neil Hurley. 2008. Avoiding monotony: improving the diversity of
recommendation lists. In Proceedings of the 2008 ACM conference on Recommender
systems. ACM, 123–130.
[31]
Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 2005.
Improving recommendation lists through topic diversication. In Proceedings of
the 14th international conference on World Wide Web. ACM, 22–32.
