SASD: Self-Attention for Small Datasets- A case study in smart villages

Abstract

Understanding repeat visitation patterns in tourism is important for optimizing economic benefits, as loyal visitors significantly contribute to the stability and growth of destinations. However, this area remains underexplored, especially in smart villages where data limitations challenge traditional machine learning (ML) approaches. Although neural networks (NN) have proven effective in various research fields, they struggle with small datasets. We propose a ML application for tracing repeat visitors using NN suitable for small datasets. Specifically, we designed SASD (Self-Attention for Small Dataset), a deep learning architecture that incorporates self-attention layers to address data limitations. We applied SASD to predict tourists’ visit intentionality in the next 12 months in a smart village region, using as training data, information from License Plate Recognition sensors, and questionnaires. We evaluated its performance against various ML algorithms; Decision Trees, Random Forests, K-NN, Logistic Regression, Gradient Boosting, Naive Bayes, SVM, MLP, RNN, and LSTM, TabNet and TabTransformer. Our results demonstrate greater accuracy, recall, precision, and F1-score. Specifically, SASD outperforms other models by up to 3% on the weighted average F1 score. Our results also confirm that in NN, the incorporation of self-attention layers accelerates convergence and reduces processing time by 32%. The best results are achieved with two self-attention layers placed at the beginning and end of the NN. Our results provide insights for policymakers, business managers, local communities, and environmental organizations, enabling informed decisions and optimal resource allocation for tourism development.