IPOP-CMA-ES and the Influence of Different Deviation Measures for Agent-Based Model Calibration

Abstract

Calibration is a crucial task on building valid models before exploiting their results. This process consists of adjusting the model parameters in order to obtain the desired outputs. Automatic calibration can be performed by using an optimization algorithm and a fitness function, which involves a deviation measure to compare the time series coming from the model. In this paper, we apply a memetic IPOP-CMA-ES for the calibration of an agent-based model and we study the effect of different deviation measures in this calibration problem. Classical metrics calculate the mean point-to-point error, but we also propose using an extension of dynamic time warping, which considers trend series evolution. In order to determine if calibrating with an specific metric leads to better solutions, we carry out an exhaustive experimentation by including statistical tests, analysis on the values of the calibrated parameters, and qualitative results. Our results show IPOP-CMA-ES obtains better performance than a genetic algorithm. In addition, MAE, MAPE and Soft-DTW are the metrics which report best results, although we get a similar behavior for all of them.