Content uploaded by Osman Ali Sadek Ibrahim
Author content
All content in this area was uploaded by Osman Ali Sadek Ibrahim on Jan 15, 2024
Content may be subject to copyright.
An Innovative Hybrid-Box tool for Ranking
Preferences using Multi-objective Evolutionary
Gradient Strategies
Osman Ali Sadek Ibrahim ( osman.ibrahim@mu.edu.eg )
Minia University https://orcid.org/0000-0001-9254-3093
Research Article
Keywords: Articial Intelligence, Information Retrieval, Optimization, Learning to Rank, Evolutionary
Gradient Strategy, Hybridizing, Hybrid Evolutionary Machine Learning
Posted Date: January 15th, 2024
DOI: https://doi.org/10.21203/rs.3.rs-3857905/v1
License: This work is licensed under a Creative Commons Attribution 4.0 International License.
Read Full License
Additional Declarations: The authors declare no competing interests.
An Innovative Hybrid-Box tool for Ranking
Preferences using Multi-objective Evolutionary
Gradient Strategies
Osman Ali Sadek Ibrahim1*
1*Computer Science Department, Minia University, Minia, Al-Minya,
61519, Upper Egypt, Egypt.
Corresponding author(s). E-mail(s): osman.ibrahim@mu.edu.eg;
Abstract
Optimizing data features plays a crucial role in simplifying the process of selecting
instances and analyzing datasets, especially when dealing with ranking problems.
In scenarios such as ranking instances in medical diagnosis, search engine opti-
mization, and information retrieval, there is a need for models that can rank
data instances based on the significance of their features within the datasets.
This paper provides a hybrid-box tool which is an efficient Multiobjective com-
putational intelligence technique that generates ranking models by utilizing
training and validation datasets. These models are then assessed using unseen test
data to produce the predictive results. This tool uses a Multi-objective (1+1)-
Evolutionary Gradient Strategy algorithm with a novel gradient methodology
for adapting the mutation step-size. Furthermore, it uses five objective functions
besides the mutation hybridizing by four probability distributions as random
number generators. This tool is a novel technique in the continuous optimization
research domain for ranking data instances. Regardless of the hidden details dur-
ing the evolving procedure, it provides the best-evolved ranking model at the end
of each run. This makes the tool is hybrid between Black and White Box tools.
Keywords: Learning to Rank, Evolutionary Gradient Strategy, Hybridizing, Hybrid
Evolutionary Machine Learning
1
1 Introduction
Evolution Gradient Strategy (EGS) is an optimization technique in computational
intelligence that merges methods from evolutionary algorithms with gradient-based
optimization approaches. This method is categorized as an optimization technique
employed to discover optimal solutions for a range of problems, with a specific focus
on areas like reinforcement learning and policy optimization. EGS is designed to
enhance the performance of a parametric model through a series of iterative parame-
ter updates tackling a predefined task.
The key idea behind the Evolution Gradient Strategy is to combine the explo-
ration capabilities of evolutionary algorithms (creating diverse solutions) with the
exploitation capabilities of gradient-based optimization (finding locally optimal solu-
tions). This combination allows the method to navigate complex and high-dimensional
search solution spaces effectively, making it useful for tasks such as in Search Engine
Optimization, Recommender Systems, and Ranking medical cases.
This paper presents an innovative hybrid-box tool designed for the ranking of data
instances, employing five distinct fitness evaluation metrics. The approach incorporates
a (1+1)-Evolutionary Gradient Strategy that integrates two novel gradient techniques
in addition to the conventional method, all utilized for the purpose of normalizing
mutation step-sizes. The main motivation for this research is as follows:
•The task of ranking webpage documents, recommended items, and medical cases
based on their ground truth labels holds significant importance in terms of enhancing
user efficiency.
•It is worth noting that there exists a paucity of extensive research on multiobjective
learning to rank across various domains.
•It is noteworthy that only one research study in the realm of multiobjective learning
to rank, as demonstrated in [1], has provided a tool to facilitate research repro-
ducibility. Nonetheless, it is essential to acknowledge that this problem domain
has garnered substantial interest among researchers over the past two decades, as
evidenced by the works of [2–4].
•The prevailing trend in the research community is to furnish the tools employed in
applied research to ensure the reproducibility of research outcomes.
The primary objective of this research is to provide a tool for investigating the
practical implications associated with the balance between exploration and exploita-
tion in the context of Learning to Rank (LTR). Of particular importance is the idea
that a better understanding of these principles can enhance the effectiveness of Evo-
lutionary Algorithms (EAs) designed for LTR applications. For instance, a deeper
comprehension can aid in determining the most suitable mutation and initialization
procedures within a specific context or selecting the optimal gradient or normalization
for mutation operations. Additionally, it can lead to insights into differences inherent
in various Single and Multi-objective approaches. Since the tool can be used for single-
objective and multi-objective evolutionary Gradient Strategies for LTR. In addition,
there is a capability for using this tool in a hybrid approach by accepting the initial
2
solutions from other Evolutionary and Machine Learning models. Consequently, the
primary contributions of this paper can be summarized as follows:
1. Providing a novel tool for LTR problem, denoted as the Multiobjective Evolutionary
Gradient Strategy.
2. Introducing a new methodology for gradient ascent mutation normalization in this
tool. The reason for that is exploring various options of exploration-exploitation
balancing through the mutation step-sizes.
3. This hybrid-box tool can be initialized by linear ranking models from other evo-
lutionary and machine-learning techniques. This capability he researchers to use
this tool for hybrid metaheuristics and computational techniques through the
initialization procedure during the run of this tool.
The structure of this paper is outlined as follows: Section 2provides the back-
ground content for the tool presented. In Section 3, we delve into previous studies and
related research. Section 4introduces our novel hybrid-box tool for the Multiobjec-
tive evolutionary gradient strategy. Lastly, Section 5offers conclusions and outlines
recommendations for future research endeavors.
2 Background Material
Evolutionary Strategy
An Evolutionary Strategy (ES) is an optimization algorithm inspired by biological evo-
lution [5]. It is used to find the optimal solution for optimization problems, especially
when the objective function is complex, high-dimensional, and might not be differen-
tiable. ES maintains a population of candidate solutions and iteratively evolves this
population to improve the fitness of its members. Here’s a detailed explanation of the
(µ/ρ +λ)-ES variant, along with mathematical equations:
1. Basic Idea: ES maintains two populations: the parent population with µindividu-
als and the offspring population with λindividuals. In each generation, it selects the
µbest individuals from the offspring to become the parents for the next generation.
2. Initialization: Initialize the parent population Pwith λindividuals, typically
randomly generated. Specify the number of offspring λ(λ > µ).
3. Iteration: Create λoffspring by perturbing the parent solutions. For each offspring
i= 1,2, ..., λ: Choose a parent solution from P, often by selecting a replacement.
Perturb the parent solution using a mutation operator to create a new solution:
xi=xp+σ∗N(0, I) (1)
where: xiis the offspring. xpis the parent. σis a scaling factor that controls the
step size of the perturbation. N(0, I ) is a normally distributed random vector with
mean 0 and identity covariance matrix. Evaluate the fitness of each offspring by
applying the objective function: f(xi) Select the top µoffspring with the highest
fitness values to form the new parent population P for the next generation.
3
4. Termination: Continue the iterations for a fixed number of generations or until a
termination criterion (e.g., a sufficiently good solution is found) is met.
5. Parameters to Tune: µ: The number of parents. λ: The number of offspring. σ:
The mutation step size. Mutation operator: Defines how parents are perturbed to
create offspring.
6. Objective: The objective in ES is to maximize or minimize the fitness function
f(x) by finding the best parameters x within a search space.
7. Key Characteristics: ES is often used for hybrid-box optimization problems
where the objective function is not known analytically or is expensive to compute.
The use of mutation and stochastic selection helps explore the solution space effi-
ciently. ES can handle high-dimensional optimization problems. It’s important to
note that there are different variants of ES, such as (1+1)-ES, (µ+λ)-ES, and
others, which vary in the number of parents and offspring used. The (µ/ρ +λ)-
ES described here is one of these variants. The choice of variant depends on the
problem at hand and the trade-offs between exploration and exploitation.
Gradient Ascent
Gradient Ascent is an optimization algorithm used in machine learning, particularly
in the context of learning to rank [6]. Learning to rank is a supervised learning task
where the goal is to train a model that can rank a list of items (e.g., search results
or recommendations) based on their relevance to a particular query or user. Gradient
Ascent is used to optimize the parameters of your ranking model in order to maximize
the fitness metric solutions. The gradient ascent update rule for a parameter θis as
follows:
θ(t+1) =θ(t)+α∆L(2)
Where: θ(t)is the value of the gene weight at iteration t. αis the mutation step-size.
∆Lis the gradient of the loss function with respect to the parameter θ.
To compute ∆L, it will need to compute the gradient of the fitness metric values
with respect to your current and offspring model’s solutions. This involves taking par-
tial derivatives of ranking evolved models with respect to the current and offspring
solutions and then propagating the gradients through the model to update the param-
eters. The goal of this iterative process is to find the parameters that maximize the
evolved model accuracy, which, in turn, should result in a ranking model that assigns
higher scores to relevant items, effectively improving the quality of the ranking.
Black Box vs. White Box Software Tools
Black-box optimization algorithms are characterized by their minimal reliance on
domain-specific information to derive solutions. They serve as general-purpose
problem-solving techniques that can be applied to a wide spectrum of problems with
limited need for customization [7]. Domain-specific knowledge pertains to established
associations between representations of solutions and the corresponding objective
cost function. In essence, the degree of domain-specific information integrated into a
method inversely affects its flexibility and efficiency concerning a specific problem. For
4
instance, ’random search’ represents the most generic black-box approach, offering
the highest degree of adaptability by solely necessitating the generation of random
solutions for a given problem. However, this method permits domain resampling,
resulting in a worst-case scenario that is less favorable than exhaustively exploring
the entire search domain. In practice, the availability of prior problem-related knowl-
edge determines the extent to which a technique can efficiently discover a solution,
be it through heuristic or alternative means. Consequently, black-box methods find
relevance in scenarios where the problem domain provides minimal information that
can be leveraged by a problem-solving approach.
The concept of a ’black box’ is emblematic of an opaque external layer envelop-
ing an application, thereby obstructing testers’ access to its internal mechanisms and
necessitating their focus solely on the assessment of end-user interactions. Conversely,
the notion of a ’white box’ denotes a high level of transparency within the application,
affording researchers the capability to penetrate the outer enclosure and scrutinize
the underlying source code. Black box optimization is centered on the evaluation
of researchers’ experience, necessitating minimal technical expertise on the part of
the researchers. This approach holds significant value in terms of affording compre-
hensive testing coverage, particularly in contrast to white box testing, which can,
at times, exhibit such precision that it inadvertently overlooks broader contextual
considerations.
3 Literature Study
This section introduces prior research that has employed multiobjective techniques
in the context of Learning-to-Rank (LTR) problems. The primary objective of a
learning-to-rank process is to construct a ranking model capable of accurately pre-
dicting the relevance of a given set of data instances. This, in turn, enhances user
satisfaction and engagement. Achieving a ranking function involves a structured
sequence of steps. Once the training data has been curated and feature extraction
completed, it is utilized to develop a ranking function. Finally, this ranking function
is applied to generate a ranked list of data instances associated, thereby facilitating
the ranking process.
The [8] study introduced a novel multiobjective Learning to Rank (LTR) approach
for commercial search engines, utilizing the LambdaMART ranking algorithm, which
was considered state-of-the-art at the time. Their work focused on modifying the λ
functions within LambdaMART to address two specific issues associated with the
existing LambdaMART λ-gradient. The primary objectives of their modifications
were twofold. First, they aimed to prevent the ranking model from attempting to
differentiate between documents that were already appropriately ordered and sepa-
rated in the ranking. Second, they sought to mitigate ranking errors that persisted
throughout the training process.
5
This approach yielded significant enhancements in terms of ranking accuracy
compared to the established state-of-the-art LambdaMART ranker. To validate their
findings, experiments were conducted on a substantial real-world dataset, where each
query-URL instance was characterized by 860 distinct features. It is important to
note, however, that the dataset itself and the code package developed by the authors
for this research are not accessible to other researchers, thus limiting the potential
for research reproducible in this particular study.
Previous research endeavors in the context of Learning to Rank (LTR) have
frequently explored the domain of multi-objective function learning. However, a
noteworthy observation is that these research investigations have generally omitted
the provision of accompanying code packages, limiting the reproducibility of their
findings. This observation holds true for studies such as those conducted by [4,9–
11]. Notably, within the relatively recent landscape of LTR research, there exists
a scarcity of research tools dedicated to the Multiobjective Evolutionary Strategy.
Osman Ibrahim notably addressed this gap by introducing a research tool in his
Cooperated research. This tool was used for the purpose of evaluating an innovative
approach pertaining to the Pareto Frontier methodology in [1].
For Single Objective research and according to the research conducted by [12],
Learning to Rank (LTR) techniques can be classified into three main categories: point-
wise, pairwise, and listwise methodologies. This classification is based on the criteria
used to measure the loss or fitness function. In the pointwise approach, each distinct
query-document instance is treated as an individual learning instance. This cate-
gory includes methods such as Linear Regression (LR) [13], Boosting [14], Gradient
Boosted Regression Trees (GBRT or MART) [15,16], and Random Forest (RF) [2].
The pairwise approach involves the utilization of a loss or objective function that
relies on pairs of query-document instances associated with the same query as the
fundamental learning units. Prominent examples of pairwise methodologies include
RankNET (Rank Neural Net) as presented by [6], RankBoost, and SVMRank, as
explained by [17].
In the realm of information retrieval methodologies, the listwise technique encom-
passes the practice of treating the entire retrieved list of items as a unified learning
instance. This comprehensive approach involves the aggregation of all query-document
instances that pertain to a specific query. Notable algorithms that fall under the
listwise paradigm include ListNET, which was introduced by [18]. Additionally, the
listwise approach encompasses Coordinate Ascent as outlined by [19], AdaRank, as
proposed by [20], and RankGPES, as described in the work of [21].
The EGS-Rank algorithm, which includes ES-Rank and Gradient Ascent compo-
nents, falls into the listwise category of ranking algorithms. Prior research, exemplified
6
by investigations conducted by [22] and [18], has demonstrated that listwise method-
ologies generally demonstrate enhanced performance when contrasted with pointwise
and pairwise approaches.
4 The Presented Tool
This study endeavors to introduce a robust tool for enhancing multiobjective learning-
to-rank approaches, with the primary objective of optimizing performance within this
specific problem domain. The proposed tool introduces a set of three gradient-based
techniques, notable for the inclusion of two innovative methods that constitute novel
contributions to the field of multiobjective optimization.
Our newly proposed optimization method, known as the Multiobjective (1 + 1)-
Evolutionary Gradient Strategy, represents an innovative approach for addressing
complex Multi-Objective Learning to Rank (LTR) problems. In this method, we tackle
scenarios where there are multiple objectives to optimize, and it’s important to note
that no single solution can be considered optimal across all of these objectives simulta-
neously. Within this approach, we employ a technique that combines decision variables
through the use of random number generators derived from four distinct probability
distributions.
These random numbers are applied to a population of ”individuals,” with each
individual representing a potential solution to the problem. The four probability
distributions used as random number generators encompass Gaussian, Cauchy, Levy,
and Uniform distributions. To determine the ranking of these individuals, our algo-
rithm relies on the Pareto Frontier principle, employing three distinct methods that
were initially introduced in prior research [1]. For using the tool, firstly, the code
exists in this link [https://tinyurl.com/2wn6fru8]. After downloading the code and
installing Java Virtual Machine, the researcher should open the Command Prompt
in Windows or Terminal in Linux. Then, they write in the Command Prompt: ’java
∇-jar ∇(the code path) ∇(PRESS-ENTER-BUTTON)’, where ∇denotes
the Space button on the keyboard. The result for that command is in the Figure 1.
From this figure, the framework of the proposed Multiobjective (1 + 1)-Evolutionary
Gradient Strategy tool has the following structured parameters.
7
Fig. 1 The initial screenshot of the Multiobjective Evolutionary Gradient Strategy application.
8
Initialization
•The default gene initialization entails the assignment of an initial value of 0.5 to
each gene within the initial solution.
•The second alternative entails utilizing a pre-trained linear ranking derived from
Computational Intelligence methods as an initial solution. The parameters associ-
ated with this approach are denoted as follows (where ∇denotes the Space button
on the keyboard):
1. -algorithm ∇2
2. -initweight ∇(File Path that has the previously trained linear model)
Termination
The termination criterion utilized by this tool relies on the count of evolutionary
iterations. By default, the tool is configured to perform 1300 iterations; however, this
value can be adjusted using the parameter denoted as ’-iterations ∇No.’
Mutation
The number of genes that undergo the mutation is a random number less than the
number of genes existing in the chromosome, but it can be repeated of the same number
in the following evolving iterations if the current offspring solution outperforms the
current parent solution.
Thoroughly managing the mutation process entails meticulous consideration of
specific mutation steps. To achieve this objective, we employ Equation 3, wherein
the term ’PRG(0,1)’ denotes a random number generator characterized by Gaussian,
Cauchy, Levy, or uniform distributions, each possessing a mean of 0 and a standard
deviation of 1, following the guidelines outlined in [23]. Furthermore, the fitness val-
ues associated with both parental and offspring chromosomes, computed utilizing the
training dataset, are denoted as ’F(PCh)’ and ’F(OffCh) by which the tool calculates
the gradient steps by three different methods.
M Gen i =Gen i +P RG(0,1) ∗∆(F(P C h), F (Off Ch)) (3)
The parameter in the tool that represents ’PRG(0,1)’ is ’-DP ∇No.(between 1 to
4)’, where 1 represents Gaussian random number generator, 2 represents Cauchy ran-
dom generator, 3 represents Levy random number generator and 4 represents Uniform
random number generator.
Gradient Methods
The evolutionary process relies on the continuous assessment of the evolved solution’s
quality at each iteration. This assessment is conducted using two parts of the datasets,
namely, the training dataset and the validation dataset. Within this framework, three
distinct gradient options are made available for the purpose of dynamically adjusting
the mutation step size. This adjustment is aimed at enhancing the exploitation of the
nearest optimal solution. These gradient options are delineated as follows:
9
•The first choice:
1
((F(P Ch, trainData)−F(of f Ch, trainData)) + (F(P Ch, V al iData)−F(of fC h, valiData)))
(4)
•The second choice:
1
(P RG(0,1) ∗(F(P C h, trainData)−F(of fCh, tr ainData)) + (5)
1
P RG(0,1) ∗(F(P C h, V ali Data)−F(off Ch, valiData)))
•The Third choice:
C1 = P RG(0,1); C2 = P RG(0,1); (6)
1
(C1∗(F(P Ch, trainData)−F(of f Ch, trainData)) + (7)
1
C2∗(F(P Ch, V aliData)−F(of fCh, v aliData)))
where C1 and C2 are constants generated by Random number generator PRG(0,1).
The parameter in the tool that represents the Gradient choices is :’-GD ∇No. (between
1 to 3)’.
Pareto Coefficient and Parameters
The detailed explanation of Pareto solutions is presented in the context of a multi-
objective Evolutionary Strategy, as documented in the study by [1]. Notably, this
strategy does not incorporate variations involving gradient ascent in mutation step-
sizes. However, an enhancement can be achieved by incorporating gradient values into
the mutation step sizes, leading to improved exploitation toward the nearest optimal
ranking solutions. This augmentation complements the exploratory aspects inherent
in the Pareto selection, effectively extending its utility in exploring extensive search
spaces for optimal solutions. The parameter in the tool that represents the Pareto
choices is: ’-Pareto ∇No. (between 1 to 3)’.
Other Parameters
•’-train ∇(train data File Path)’ is used for assigning the training dataset to the
tool for evolving procedure.
•’-vali ∇(validation data File Path)’ is used for assigning the validation dataset to
the tool for evolving procedure.
•’-test ∇(test data File Path)’ is used for assigning the test dataset to the tool for
evolving procedure.
•’-Pmap ∇real No.’ represents the Pareto Coefficient for Mean Average Precision.
10
•’-Pndcg10 ∇real No.’ represents the Pareto Coefficient for Normalized Discounted
Cumulative Gain of top-10 instances.
•’-P10 ∇real No.’ represents the Pareto Coefficient for Precision top-10 instances.
•’-Pb10 ∇real No.’ represents the Pareto Coefficient for Best top-10 instances.
•’Prr10 ∇real No.’ represents the Pareto Coefficient for reciprocal rank top-10
instances.
•’Perr10 ∇real No.’ represents the Pareto Coefficient for expected reciprocal rank
top-10 instances.
5 Conclusion and Future direction
In summary, this paper introduces a novel and efficient Multi-objective computational
intelligence tool designed for ranking model generation, which leverages training and
validation datasets. The proposed tool employs a Multi-objective (1+1)-Evolutionary
Gradient Strategy algorithm, notable for its unique gradient-based approach to adapt
the mutation step-size. Additionally, it incorporates the use of five objective functions
and hybridizes mutations through the integration of four probability distributions as
random number generators. This novel tool, situated within the continuous optimiza-
tion research domain, is tailored specifically for ranking data instances. Importantly, it
consistently produces the best-evolved ranking model at the culmination of each exe-
cution, showcasing its distinctive hybrid nature, and bridging the gap between Black
and White Box tools.
An envisioned prospective avenue involves the utilization of this tool for the pur-
pose of prioritizing data instances in the contexts of medical diagnosis cases and search
engine optimization. Furthermore, there is a call for the advancement of additional
multi-objective learning-to-rank tools, aimed at facilitating researchers in achieving
greater levels of reproducibility in their research endeavors.
Acknowledgments. The authors wish to express appreciation to STDF, Egypt, for
their support in covering the open access fees for their paper published in Springer
journals.
Declarations
Ethical Approval
There is no involvement of humans or animals in this research.
Authors’ contributions
The paper was wrote and reviewed by the authors
Availability of data and material
The data is publicly accessible on the Microsoft Research website, and it has been
provided by the Microsoft research team. You can find the source code in here.
11
Conflict of interest
The authors affirm that there are no conflicts of interest.
Funding and Support
The authors wish to express appreciation to STDF, Egypt, for their support in covering
the open access fees for their paper published in Springer journals.
References
[1] Ismail, W.N., Ibrahim, O.A.S., Alsalamah, H.A., Mohamed, E.: Multiobjective
learning to rank based on the (1 + 1) evolutionary strategy: An evaluation of
three novel pareto optimal methods. Electronics 12(17) (2023) https://doi.org/
10.3390/electronics12173724
[2] Breiman, L.: Random forests. Machine Learning 45(1), 5–32 (2001) https://
doi.org/10.1023/A:1010933404324
[3] Ibrahim, O.A.S., Younis, E.M.: Hybrid online–offline learning to rank using
simulated annealing strategy based on dependent click model. Knowledge and
Information Systems, 1–15 (2022)
[4] Mahapatra, D., Dong, C., Momma, M.: Querywise fair learning to rank through
multi-objective optimization. In: Proceedings of the 29th ACM SIGKDD Con-
ference on Knowledge Discovery and Data Mining. KDD ’23, pp. 1653–1664.
Association for Computing Machinery, New York, NY, USA (2023). https://
doi.org/10.1145/3580305.3599482 .https://doi.org/10.1145/3580305.3599482
[5] Beyer, H.-G., Schwefel, H.-P.: Evolution strategies - A comprehensive introduc-
tion. Natural Computing 1, 3–52 (2002)
[6] Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., Hul-
lender, G.: Learning to rank using gradient descent. In: Proceedings of the 22Nd
International Conference on Machine Learning. ICML ’05, pp. 89–96. ACM, New
York, NY, USA (2005). https://doi.org/10.1145/1102351.1102363
[7] Droste, S., Jansen, T., Wegener, I.: Upper and lower bounds for randomized
search heuristics in black-box optimization. Theory of Computing Systems 39(4),
525–544 (2006) https://doi.org/10.1007/s00224-004-1177-z
[8] Svore, K.M., Volkovs, M.N., Burges, C.J.C.: Learning to rank with multiple
objective functions. In: Proceedings of the 20th International Conference on
World Wide Web. WWW ’11, pp. 367–376. Association for Computing Machin-
ery, New York, NY, USA (2011). https://doi.org/10.1145/1963405.1963459 .
https://doi.org/10.1145/1963405.1963459
12
[9] Mahapatra, D., Dong, C., Chen, Y., Momma, M.: Multi-label learning to rank
through multi-objective optimization. In: Proceedings of the 29th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining. KDD ’23, pp. 4605–
4616. Association for Computing Machinery, New York, NY, USA (2023). https:
//doi.org/10.1145/3580305.3599870 .https://doi.org/10.1145/3580305.3599870
[10] Hu, J., Li, P.: Collaborative multi-objective ranking. In: Proceedings of the 27th
ACM International Conference on Information and Knowledge Management.
CIKM ’18, pp. 1363–1372. Association for Computing Machinery, New York,
NY, USA (2018). https://doi.org/10.1145/3269206.3271785 .https://doi.org/
10.1145/3269206.3271785
[11] Momma, M., Bagheri Garakani, A., Ma, N., Sun, Y.: Multi-objective ranking
via constrained optimization. In: Companion Proceedings of the Web Confer-
ence 2020. WWW ’20, pp. 111–112. Association for Computing Machinery,
New York, NY, USA (2020). https://doi.org/10.1145/3366424.3382723 .https:
//doi.org/10.1145/3366424.3382723
[12] Liu, T.-Y.: Learning to rank for information retrieval. Foundations and Trends
in Information Retrieval 3(3), 225–331 (2009)
[13] Yan, X., Su, X.G.: Linear Regression Analysis: Theory and Computing. World
Scientific Publishing Co., Inc., River Edge, NJ, USA (2009)
[14] Freund, Y., Iyer, R., Schapire, R.E., Singer, Y.: An efficient boosting algorithm
for combining preferences. J. Mach. Learn. Res. 4, 933–969 (2003)
[15] Mohan, A., Chen, Z., Weinberger, K.Q.: Web-search ranking with initialized
gradient boosted regression trees. In: Journal of Machine Learning Research,
Workshop and Conference Proceedings, vol. 14, pp. 77–89 (2011)
[16] Friedman, J.H.: Greedy function approximation: A gradient boosting machine.
The Annals of Statistics 29(5), 1189–1232 (2001)
[17] Li, H.: Learning to Rank for Information Retrieval and Natural Language
Processing, Second Edition. Morgan and Claypool Publishers, ??? (2014)
[18] Cao, Z., Qin, T., Liu, T.-Y., Tsai, M.-F., Li, H.: Learning to rank: from pairwise
approach to listwise approach. In: Proceedings of the 24th International Confer-
ence on Machine Learning. ICML ’07, pp. 129–136. ACM, New York, NY, USA
(2007). https://doi.org/10.1145/1273496.1273513 .http://doi.acm.org/10.1145/
1273496.1273513
[19] Metzler, D., Bruce Croft, W.: Linear feature-based models for information
retrieval. Information Retrieval 10(3), 257–274 (2007) https://doi.org/10.1007/
s10791-006-9019-z
13
[20] Xu, J., Li, H.: Adarank: A boosting algorithm for information retrieval. In: Pro-
ceedings of the 30th Annual International ACM SIGIR Conference on Research
and Development in Information Retrieval. SIGIR ’07, pp. 391–398. ACM,
New York, NY, USA (2007). https://doi.org/10.1145/1277741.1277809 .http:
//doi.acm.org/10.1145/1277741.1277809
[21] Islam, M.A.: RankGPES: Learning to Rank for Information Retrieval using a
Hybrid Genetic Programming with Evolutionary Strategies (2013)
[22] Ibrahim, O.A.S., Landa-Silva, D.: An evolutionary strategy with machine learning
for learning to rank in information retrieval. Soft Computing 22 (2018) https:
//doi.org/10.1007/s00500-017-2988-6
[23] Zheng, Y., Wu, J., Wang, B.: Clgbo: An algorithm for constructing highly robust
coding sets for dna storage. Frontiers in Genetics 12 (2021) https://doi.org/
10.3389/fgene.2021.644945
14
