Ognjen Stefanovic

Recent advancements in large language models have transformed human-Al interactions. Humans mostly leverage them as tools to address simple, well-defined everyday problems. However, we pose the question: how can a human domain expert use AI as a collaborative partner to solve difficult, vaguely defined problems? One common situation of particular interest is the following.

Consider a scenario where a deployed system malfunctions due to an unforeseen situation. Typically, a human expert is required to diagnose and resolve the issue. During their interaction with the system, several challenges may arise. Both the human and the system may lack clarity regarding what specifically needs to be fixed, resulting in a vague objective function. The set of possible actions to fix the issue might be only partially known. The system's state is only partially observable, making it difficult to assess the situation accurately. The human may attempt several fixes, some of which may fail, necessitating an iterative approach to problem-solving. Therefore, interaction between the human and the system to elicit information and define solutions is key to successfully solving the problem.

We propose that this interaction occur through natural language, as it is the most intuitive form of communication for humans. The system would utilise language models (LMs) to facilitate dialogue. This collaborative approach allows the human and the language model to rapidly design, build, and test multiple solutions-a process referred to as Experimental Design. We aim to explore this process through a Bayesian lens. Several additional challenges will be addressed in this project, such as whether the human and AI share a common understanding of the problem, whether the LM can reason consistently, whether the LM has a model of the problem from the user's perspective, and whether the limited information provided by the user during a few interactions is sufficient for the AI to grasp the requirements.

By tackling these questions, we hope to enhance the effectiveness of human-Al collaboration in solving complex problems.