CEC’17 Competition on Evolutionary Multitask Optimization 




OVERVIEW AND AIMThe human possesses the most remarkable ability to manage and execute multiple tasks simultaneously, e.g., talking while walking. This desirable multitasking capability has inspired computational methodologies and approaches to tackle multiple tasks at the same time by leveraging commonalities and differences across different tasks to improve the performance and efficiency of resolving component tasks compared to when dealing with them separately. As a wellknown example, multitask learning [1] is a very active subfield of machine learning whereby multiple learning tasks are performed together using a shared model representation such that the relevant information contained in related tasks can be exploited to improve the learning efficiency and generalization performance of taskspecific models. Multitask optimization (MTO) [2][5] is a newly emerging research area in the field of optimization, which investigates how to effectively and efficiently tackle multiple optimization problems at the same time. In the multitasking scenario, solving one optimization problem may assist in solving other optimization problems (i.e., synergetic problemsolving) if these problems bear commonality and/or complementarity in terms of optimal solutions and/or fitness landscapes. As a simple example, if some problems have the same globally optimal solution but distinct fitness landscapes, obtaining the global optimum to any problem makes the others also get solved. Recently, an evolutionary MTO paradigm, socalled evolutionary multitasking [3], [4], was proposed to make the best use of the potential of evolutionary algorithms (EAs) incorporated with a unified solution representation space for MTO. As a populationbased optimizer, EAs feature the Darwinian survivalofthefittest principle and natureinspired reproduction operations which inherently promote implicit knowledge transfer across tasks during problemsolving. The superiority of this novel evolutionary multitasking framework over traditional ways of solving each task independently has been demonstrated on synthetic and realworld MTO problems by using a multifactorial evolutionary algorithm (MFEA) [3],[5] developed under this framework. Evolutionary multitasking opens up new horizons for researchers in the field of evolutionary computation. It provides a promising means to deal with the everincreasing number, variety and complexity of optimization tasks. More importantly, rapid advances in cloud computing would eventually turn optimization into the ondemand service hosted on the cloud, as illustrated in Fig. 1. In such a scenario, a variety of optimization tasks would be simultaneously executed by the service engine where evolutionary multitasking may become the technical backbone that harnesses the underlying synergy between multiple tasks to provide service consumers with faster and better solutions while generating higher economic gains for the service provider.
The promising perspectives and encouraging preliminary success of evolutionary MTO call for intensive research efforts in this new area. However, there is a lack of unified MTO test suites to facilitate research pursuits from both algorithmic and theoretical aspects. This fact motivates us to have designed two suites of MTO benchmark problems [6], [7] for singleobjective and multiobjective continuous optimization tasks, respectively. This competition aims at promoting research advances in evolutionary MTO where the two developed test suites are required to be used as a common platform to enable fair and easy performance evaluation and comparison. TEST SUITESSingleobjective and multiobjective continuous optimization have been intensively studied in the community of evolutionary optimization where many wellknown test suites are available. As a preliminary attempt, we have designed two MTO test suites [6], [7] for singleobjective and multiobjective continuous optimization tasks, respectively. The test suite for multitask singleobjective optimization (MTSOO) [6] contains nine MTO benchmark problems where each MTO problem consists of two singleobjective continuous optimization tasks which bear certain commonality and complementarity in terms of the global optimum and the fitness landscape. These nine MTO problems possess different degrees of latent synergy between their involved two component tasks. The test suite for multitask multiobjective optimization (MTMOO) [7] includes nine MTO benchmark problems where each MTO problem consists of two multiobjective continuous optimization tasks which bear certain commonality and complementarity in terms of the Pareto optimal solutions and the fitness landscape. The nine MTO problems feature different degrees of latent synergy between their involved two component tasks. All benchmark problems included in these two test suites are elaborated in technical reports [6], [7], respectively. Their associated code in Matlab is downloadable at http://www.cil.ntu.edu.sg/mfo/download.html. COMPETITION PROTOCOLPotential participants in this competition may target at either or both of MTSOO and MTMOO while using all benchmark problems in the corresponding test suites as described above for performance evaluation. For MTSOO test suite:(1) Experimental settingsFor each of 9 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudorandom number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one. For all 9 benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 300,000. In the multitasking scenario, one function evaluation means calculation of the objective function value of any component task without distinguishing different tasks. To enable comparison of the utmost performances of algorithms, the parameter setting of an algorithm is allowed to be calibrated w.r.t. each benchmark problem in this test suite. Note: Participants are required to report the used parameter setting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details. (2) Intermediate results required to be recordedWhen an algorithm is executed to solve a specific benchmark problem in a run, the so far achieved best function error value (BFEV) w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/100, k =1, …, 100 in this competition. BFEV is calculated as the difference between the best objective function value achieved so far and the globally optimal objective function value known in advance. As a result, for any benchmark problem, 100 BFEVs would be recorded w.r.t. each component task in each run. Intermediate results for each benchmark problem are required to be saved separately into nine “.txt” files named as “MTSOO_P1.txt”, …, “MTSOO_P9.txt” where the data contained in each “.txt” file must conform to the following format:
where BFEV_{j,k}^i (i = 1, 2; j = 1, …, 30; k = 1, …, 100) stands for the BFEV w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying two consecutive columns w.r.t. two component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference. (3) Overall ranking criterionTo derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 18 individual tasks. For each algorithm to be ranked, the median BFEV over 30 runs will be calculated at each checkpoint which corresponds to different computational budgets for each of 18 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline. For MTMOO test suite:(1) Experimental settingsFor each of 9 benchmark problems in this test suite, an algorithm is required to be executed for 30 runs where each run should employ different random seeds for the pseudorandom number generator(s) used in the algorithm. Note: It is prohibited to execute multiple 30 runs and deliberately pick up the best one. For all 9 benchmark problems, the maximal number of function evaluations (maxFEs) used to terminate an algorithm in a run is set to 300,000. In the multitasking scenario, one function evaluation means calculation of the values of multiple objective functions of any component task without distinguishing different tasks. To enable comparison of the utmost performances of algorithms, the parameter setting of an algorithm is allowed to be calibrated w.r.t. each benchmark problem in this test suite. Note: Participants are required to report the used parameter setting for each problem in the final submission to the competition. Please refer to “SUBMISSION GUIDELINE” for more details. (2) Intermediate results required to be recordedWhen an algorithm is executed to solve a specific benchmark problem in a run, the obtained inverted generational distance (IGD) value w.r.t. each component task of this problem should be recorded when the current number of function evaluations reaches any of the predefined values which are set to k*maxFEs/100, k =1, …, 100 in this competition. IGD [8] is a commonly used performance metric in multiobjective optimization to evaluate the quality (convergence and diversity) of the currently obtained Pareto front by comparing it to the optimal Pareto front known in advance. As a result, for any benchmark problem, 100 IGD values would be recorded w.r.t. each component task in each run. Intermediate results for each benchmark problem are required to be saved separately into nine “.txt” files named as “MTMOO_P1.txt”, …, “MTMOO_P9.txt” where the data contained in each “.txt” file must conform to the following format:
where IGD_{j,k}^i (i = 1, 2; j = 1, …, 30; k = 1, …, 100) stands for the IGD value w.r.t. the ith component task obtained in the jth run at the kth predefined number of function evaluations. The first column stores the predefined numbers of function evaluations at which intermediate results are recorded. The subsequent columns store intermediate results for each of 30 runs with each run occupying two consecutive columns w.r.t. two component tasks, respectively. Note: The comma is used as a delimiter to separate any two numbers next to each other in a row. As an example, “.txt” files obtained by MFEA are provided as reference. (3) Overall ranking criterionTo derive the overall ranking for each algorithm participating in the competition, we will take into account of the performance of an algorithm on each component task in each benchmark problem under varying computational budgets from small to large. Specifically, we will treat each component task in each benchmark problem as one individual task, ending up with a total of 18 individual tasks. For each algorithm compared for ranking, the median IGD value over 30 runs will be calculated at each checkpoint corresponding to different computational budgets for each of 18 individual tasks. Based on these calculated data, the overall ranking criterion will be defined. To avoid deliberate calibration of the algorithm to cater for the overall ranking criterion, we will release the formulation of the overall ranking criterion after the competition submission deadline. SUBMISSION GUIDELINEFor those who are interested in the competition, please archive the following files into a single .zip file and then send it to mtocompetition@gmail.com before the competition submission deadline (May 15, 2017):
Here, “param_SO.txt” and “param_MO.txt” contain the parameter setting of the algorithm for each benchmark problem in MTSOO and MTMOO test suites, respectively. “code.zip” contains the source code of the algorithm which should allow the generation of reproducible results.
The submission samples for MTSOO and MTMOO are available below:
If you would like to participate in the competition, please kindly inform us about your interest via email (mtocompetition@gmail.com) so that we can update you about any bug fixings and/or the extension of the deadline. COMPETITION ORGNIZERS
Kai Qin
Liang Feng
Yuan Yuan
YewSoon Ong
Xu Chi REFERENCES[1] R. Caruana, “Multitask learning”, Machine Learning, 28(1): 4175, 1997. 
