On the disagreement problem in Human-in-the-Loop federated machine learning

Abstract

The popularity of Artificial Intelligence (AI) has risen sharply in recent years, revolutionizing applications in most sectors with unprecedented functionalities. Milestones and achievements like ChatGPT demonstrate not only the impressive capabilities of AI, but also how accessible such technologies have become in recent times. However, the success of AI applications depends heavily on the underlying information integration processes. Among the most important processes are the training of the AI model at the core of the application and the collection and pre-processing of training data. In particular, the task of collecting high-quality training data can be very costly and resource-intensive, as in many cases large amounts of data have to be annotated manually. Human annotators must have extensive expertise for certain tasks in order to provide high-quality training data. In this paper, we present a framework to maximize the efficiency of human experts in a Machine Learning (ML) scenario, with the aim of optimizing the use of human expertise in active learning. This is done by constantly measuring the quality of human experts’ input, as well as by involving human annotators only when needed. We showcase the benefits of our proposed framework by applying it to a problem in image classification, proving its usefulness to reduce the cost of annotating training data. The source code of the framework is publicly available at https://github.com/human-centered-ai-lab/app-HITL-annotator.