Publication Details

Download

Changkun Ou
The Intelligence in the Loop: Empirical Explorations and Reflections
PhD Dissertation. Faculty of Mathematics, Computer Science and Statistics, LMU Munich, April 03, 2023, Munich, Germany. 130 pages. https://doi.org/10.5282/edoc.32969 (bib)

For decades, engineering in computing systems has used a human-in-the-loop servo mechanism. A conscious human being is usually believed, in a rational manner, to operate, assist, and control the machine to achieve desired objectives. Over time, researchers have started to use human-in-the-loop schemes in more abstract tasks, such as iterative interface design problems. However, with the observations and developments in social science, the underlying rationality assumption is strongly challenged, and humans make mistakes. With the recent advances in computer science regarding artificial intelligence, data-driven algorithms could achieve human-level performance in certain aspects, such as audio recognition, image segmentation, and machine translation tasks. The human-in-the-loop mechanism is being reconsidered and reshaped towards an extended vision to assist human decision-making or creativity in the human-computer interaction (HCI) research field. This thesis explores the boundary for human-in-the-loop optimization systems to succeed and be beneficial. In the interaction loop, machine agents are designed rationally to interact with human beings that may behave using incomplete rational policies iteratively. The thesis first examines and deliberates common principles in mainstream HCI research regarding the advice for building human-in-the-loop systems using existing computation techniques concerning decision-making support, utility-based optimization, and human concepts regarding preferences, satisfaction, and expertise. To reflect real-world constraints in a human-in-the-loop optimization system, the thesis explores three design problems: text summarization, image color enhancement, and 3D polygon reduction. These design problems are selected to involve human perception and intelligence, aesthetic preference, and rational judgments. Specifically, to understand and analyze the interaction loop, the thesis conducted a series of experiments to study the impact of various building blocks in human-in-the-loop systems that observes exploration and exploitation of human users, including problem context, solution space, reliability of human inputs regarding preference and expertise, and relevant user interfaces for inputs. Combining the findings of the experiments, the thesis revisits vulnerable assumptions that may be largely ignored when designing a modern human-in-the-loop optimization system. The experiment on the impact of user interfaces narrows down the exploration space of this thesis and empirically demonstrates how different preferential user interfaces influence the overall interaction performance. Based on the findings, subsequent experiments further investigate how human judgments can be a flaw of a human-in-the-loop optimization system. The result shows that, due to cognitive limitations and unrealistic system assumptions, inconsistent and unstable preferences commonly exist in this human-in-the-loop optimization system, resulting in suboptimal machine outcomes and user dissatisfaction, which conflicts with the objective of using a human to gain the expected output. With a deeper look into human aspects, another experiment attempts to reveal the potential causes, such as involved level of human expertise. The system further tests the usage of individuals with different levels of expertise. Based on the observation and analysis, higher-level expertise leads to lower subjective satisfaction and more interactions, whereas novices terminate faster and also achieve expert-level performance, which not only reveals challenges to utilizing the obtained human insights but also be considered as an indicator to reveal how we can better involve a human in an optimization loop for exploring a solution space. All these contributions in human-in-the-loop optimization systems lead to a rethinking of the source of intelligence and engage philosophical discussions. These topics eventually approach more fundamental questions regarding the definition of intelligence and how we might succeed in keeping our intelligence in the loop.