Privacy Aware Multimodal Dialogue System
Aishik Mandal (Ph.D. Student)
Clinical Natural Language Processing (NLP) has the unique potential to support doctors in improving patient care, in some cases even saving lives, while at the same time lowering costs. In previous work, electronic health records (EHR) were used for tasks like patient outcome, in-hospital mortality, and length-of-stay prediction based on admission notes, thereby supporting doctors in patient care. Others have developed information extraction systems for EHRs to support clinical research. However, most of such works are done on English data. One reason for this is a scarcity of clinical data in languages other than English. This lack of available datasets comes down to the singular issue of privacy. As clinical data is extremely sensitive, it is extensively protected through GDPR and national laws. In this PhD project, we propose to create end-to-end workflows that can tackle this privacy issue. We propose to create psychiatry dialogue datasets from interviews and therapy sessions between doctors and patients. We choose psychiatry dialogues due to their high degree of privacy requirement. However, using only textual modality is not sufficient for psychiatric downstream tasks. Acoustic features like tone, pitch and visual features like gaze and gestures provide a large amount of information in such situations. Thus we propose a multimodal approach. In conclusion, the workflow will consist of two major steps: 1. Transcription and De-identification: This step includes transcription of interviews and therapy sessions between doctors and patients. However, to maintain privacy, the transcription must remove any protected health information and personal identifiers. The audio and video recordings will also be modified in a way to guarantee privacy with minimal loss in information. 2. Downstream Applications: To train multimodal models for downstream task applications, we will explore the trade-offs for different privacy measures. Downstream tasks could include understanding the mental states of patients during a session, predicting chance of suicide, understanding improvements in a patient over multiple sessions.
| Primary Host: | Iryna Gurevych (Technical University of Darmstadt) |
| Exchange Host: | Aurélien Bellet (INRIA) |
| PhD Duration: | 01 September 2023 - 30 September 2027 |
| Exchange Duration: | 01 September 2024 - 28 February 2025 - Ongoing |
