LSTM new gate for computing the efficiency on inputdata

Abstract

The recognized learning ability of neural networks (NNs) is determined by their training process. The NN data-dependent nature makes that their success depends to a large extent on the quality of the training data sets. This paper addresses the selection of the best training sets for Long Short Time Models (LSTMs) in reallife contexts. If such selection from a theoretical standpoint is already complex, it becomes challenging when addressed for real scenarios, where effectiveness should be replaced by efficiency (i.e., effectiveness in the shortest possible time and without increasing the computational cost). Our proposal revolves around the introduction, definition and structuring of a new LSTM gate: the efficiency gate, that applies to those inputs that have passed the forget gate. The efficiency gate is responsible for ensuring that training advances only with the best inputs in the sense that the corresponding outputs match as closely as possible with a target value (best estimators). For this purpose, new mathematical results will be demonstrated here in order to prove that our efficiency gate is mathematically feasible. We also fully describe how the new gate joins the LSTM operation, blending in with it. The consequence of such natural integration is that the computational effort is practically unchanged. Our methodology also allows the user to set the admissible error thresholds (which are different in each real scenario) as part of the setting options.