Event-based Vision for Early Prediction of Manipulation Actions Déniz Cerpa, José Daniel Fermüller, Cornelia Ros Vidal, Eduardo Rodríguez Álvarez, Manuel Barranco Expósito, Francisco Event-based vision Online prediction Manipulation action prediction This work was supported by the Spanish National Grant PID2019-109434RA-I00/ SRA (State Research Agency /10.13039/501100011033). We acknowledge the Telluride Neuromorphic Cognition Engineering Workshop (http: //www.ine-web.org), supported by NSF grant OISE 2020624 for the fruitful discussions on neuromorphic cognition and their participants for helping with the recording of the dataset. Neuromorphic visual sensors are artificial retinas that output sequences of asynchronous events when brightness changes occur in the scene. These sensors offer many advantages including very high temporal resolution, no motion blur and smart data compression ideal for real-time processing. In this study, we introduce an event-based dataset on fine-grained manipulation actions and perform an experimental study on the use of transformers for action prediction with events. There is enormous interest in the fields of cognitive robotics and human-robot interaction on understanding and predicting human actions as early as possible. Early prediction allows anticipating complex stages for planning, enabling effective and real-time interaction. Our Transformer network uses events to predict manipulation actions as they occur, using online inference. The model succeeds at predicting actions early on, building up confidence over time and achieving state-of-the-art classification. Moreover, the attention-based transformer architecture allows us to study the role of the spatio-temporal patterns selected by the model. Our experiments show that the Transformer network captures action dynamic features outperforming video-based approaches and succeeding with scenarios where the differences between actions lie in very subtle cues. Finally, we release the new event dataset, which is the first in the literature for manipulation action recognition. 2026-03-04T07:34:42Z 2026-03-04T07:34:42Z 2023-07-26 conference output Daniel Deniz, Cornelia Fermuller, Eduardo Ros, Manuel Rodriguez-Alvarez, Francisco Barranco. Event-based Vision for Early Prediction of Manipulation Actions. DOI: https://doi.org/10.48550/arXiv.2307.14332 https://hdl.handle.net/10481/111859 10.48550/arXiv.2307.14332 eng http://creativecommons.org/licenses/by-nc-nd/4.0/ open access Attribution-NonCommercial-NoDerivatives 4.0 Internacional Cornell University