Fig 1 - uploaded by Arman Dehpanah
Content may be subject to copyright.
The results of evaluating Elo, Glicko, TrueSkill, and PreviousRank using accuracy, MAE, Kendall's tau, MRR, AP, and NDCG on four different experimental set-ups: all players, best players, frequent players, and binned ranks.
Contexts in source publication
Context 1
... results of these experiments are given in Figure 1. In this figure, the rows correspond to evaluation metrics and the columns are associated with experimental set-ups. ...
Similar publications
Citations
... Finally, we used Normalized Discounted Cumulative Gain (NDCG) [23][24][25] and accuracy to evaluate rank predictions for battle royale and head-to-head games, respectively. ...
Competitive online games use rating systems for matchmaking; progression-based algorithms that estimate the skill level of players with interpretable ratings in terms of the outcome of the games they played. However, the overall experience of players is shaped by factors beyond the sole outcome of their games. In this paper, we engineer several features from in-game statistics to model players and create ratings that accurately represent their behavior and true performance level. We then compare the estimating power of our behavioral ratings against ratings created with three mainstream rating systems by predicting rank of players in four popular game modes from the competitive shooter genre. Our results show that the behavioral ratings present more accurate performance estimations while maintaining the interpretability of the created representations. Considering different aspects of the playing behavior of players and using behavioral ratings for matchmaking can lead to match-ups that are more aligned with players' goals and interests, consequently resulting in a more enjoyable gaming experience.
... The outcome of all tested models is an ordered list of players showing their predicted ranks. Therefore, we used an extension of Normalized Discounted Cumulative Gain (NDCG) for evaluating rank predictions [25]. We performed our experiments in three different setups. ...
Competitive online games use rating systems to match players with similar skills to ensure a satisfying experience for players. In this paper, we focus on the importance of addressing different aspects of playing behavior when modeling players for creating match-ups. To this end, we engineer several behavioral features from a dataset of over 75,000 battle royale matches and create player models based on the retrieved features. We then use the created models to predict ranks for different groups of players in the data. The predicted ranks are compared to those of three popular rating systems. Our results show the superiority of simple behavioral models over mainstream rating systems. Some behavioral features provided accurate predictions for all groups of players while others proved useful for certain groups of players. The results of this study highlight the necessity of considering different aspects of the player's behavior such as goals, strategy, and expertise when making assignments.
... Battle royale games consist of many teams and players, and the result of a match is often an ordered list of ranks. Therefore, we used NDCG to evaluate predictions [3,4] for our battle royale dataset; PUBG. ...
One of the main goals of online competitive games is increasing player engagement by ensuring fair matches. These games use rating systems for creating balanced match-ups. Rating systems leverage statistical estimation to rate players' skills and use skill ratings to predict rank before matching players. Skill ratings of individual players can be aggregated to compute the skill level of a team. While research often aims to improve the accuracy of skill estimation and fairness of match-ups, less attention has been given to how the skill level of a team is calculated from the skill level of its members. In this paper, we propose two new aggregation methods and compare them with a standard approach extensively used in the research literature. We present an exhaustive analysis of the impact of these methods on the predictive performance of rating systems. We perform our experiments using three popular rating systems, Elo, Glicko, and TrueSkill, on three real-world datasets including over 100,000 battle royale and head-to-head matches. Our evaluations show the superiority of the MAX method over the other two methods in the majority of the tested cases, implying that the overall performance of a team is best determined by the performance of its most skilled member. The results of this study highlight the necessity of devising more elaborated methods for calculating a team's performance-- methods covering different aspects of players' behavior such as skills, strategy, or goals.
... Rank correlation coefficients and their derivatives [4,[17][18][19][20] are among the metrics commonly used for this purpose. A recent study compared several metrics based on how they explain the predictive power of rating systems [21]. This study focused on solo battle royale matches where there are no teams, and players compete against each other as singletons. ...
... Elo was originally designed for head-to-head matches between two players. It was later extended to solo battle royale games where many players compete against one another as singletons at the same time in the same match [21]. ...
... Both σ and µ values are updated after each match by comparing predicted ranks R pred with observed ranks R obs . Similar to Elo, Glicko was originally designed for head-tohead matches between two players and was later extended to solo battle royale matches [21]. We extend Glicko to teambased battle royale games as we did with Elo. ...
Online competitive games have become a mainstream entertainment platform. To create a fair and exciting experience, these games use rating systems to match players with similar skills. While there has been an increasing amount of research on improving the performance of these systems, less attention has been paid to how their performance is evaluated. In this paper, we explore the utility of several metrics for evaluating three popular rating systems on a real-world dataset of over 25,000 team battle royale matches. Our results suggest considerable differences in their evaluation patterns. Some metrics were highly impacted by the inclusion of new players. Many could not capture the real differences between certain groups of players. Among all metrics studied, normalized discounted cumulative gain (NDCG) demonstrated more reliable performance and more flexibility. It alleviated most of the challenges faced by the other metrics while adding the freedom to adjust the focus of the evaluations on different groups of players.
... Other history-based outcome prediction approaches include machine learning on hand-crafted features for the MOBA genre [28] and modeling with rating systems for MOBA and shooter genres [29]. Work by Dehpanah et al. [30] extends rating systems for free-for-all games (Battle Royale genre) using a large dataset of matches in PlayerUnknown's Battlegrounds game. Analytics in eSports in not limited to winner prediction but also extends to players' behavioral analysis. ...
Proper training and analytics in eSports require accurately collected and annotated data. Most eSports research focuses exclusively on in-game data analysis, and there is a lack of prior work involving eSports athletes' psychophysiological data. In this paper, we present a dataset collected from professional and amateur teams in 22 matches in League of Legends video game. Recorded data include the players' physiological activity, e.g. movements, pulse, saccades, obtained from various sensors, self-reported after-match survey, and in-game data. An important feature of the dataset is simultaneous data collection from five players, which facilitates the analysis of sensor data on a team level. Upon the collection of dataset we carried out its validation. In particular, we demonstrate that stress and concentration levels for professional players are less correlated, meaning more independent playstyle. Also, we show that the absence of team communication does not affect the professional players as much as amateur ones. To investigate other possible use cases of the dataset, we have trained classical machine learning algorithms for skill prediction and player re-identification using 3-minute sessions of sensor data. Best models achieved 0.856 and 0.521 (0.10 for a chance level) accuracy scores on a validation set for skill prediction and player re-id problems, respectively. The dataset is available at https://github.com/asmerdov/eSports_Sensors_Dataset.
The new wave of Artificial Intelligence brought about by deep learning and reinforcement learning provides an "end-to-end" solution for agents from perception input to action decision-making output. Multi-agent learning is a frontier subject in the field of intelligent game confrontation, and it faces many problems and challenges such as adversarial environments, non-stationary opponents, incomplete information and uncertain actions. This paper starts from the perspective of game theory, firstly gives the organization of multi-agent learning system, gives an overview of multi-agent learning, and briefly introduces the classification of various multi-agent learning research methods. Secondly, around the multi-agent game learning framework, it introduces the basic multi-agent game and meta-game models, game solution concepts and game dynamics, challenges such as diverse learning objectives, non-stationary environment (opponent), equilibrium with complex to compute and easy to transfer. Once again comprehensively sort out the multi-agent game strategy learning methods, offline game strategy learning methods such as Markov game strategy learning, extensive game strategy learning, meta-game population strategy learning, online game strategy learning methods such as online optimization and no-regret learning, opponent modeling and exploitation, role matching and ad-hoc coordination. Finally, some frontiers of multi-agent learning are discussed from three aspects of agent cognitive modelling and collaboration, general game strategy learning methods, and distributed game strategy learning framework.
