Challenges in natural language processing require coordination across a large scientific network

NLP is a subfield of AI focusing on the interaction between computers and human language. It enables computers to process large amounts of unstructured text data, and to understand and generate human language in a written or spoken form. It’s fundamental for applications in areas such as speech recognition, voice command, search engines, language translation and sentiment analysis, and contributes to the broader goal of creating intelligent systems that can interact with humans effectively.

The ELLIS Program ‘Natural Language Processing’ aims to promote excellence in NLP research and to shape further progress in this field by fostering cooperation among leading European researchers and labs working on a diverse set of NLP areas. Three top-notch NLP scientists lead the program: Iryna Gurevych, Professor at the Computer Science Department of the Technical University Darmstadt (Germany), André Martins, Associate Professor at Instituto Superior Técnico (Portugal), and Ivan Titov, Professor and Personal Chair in Natural Language Processing at the University of Edinburgh (UK).

Together, André Martins and Iryna Gurevych provided the answers to the following questions addressing their program’s research field and the cooperation among the program members.

The recent public debate about language models such as ChatGPT has brought the field of natural language processing into the spotlight. Looking at the broader picture: What are the major current breakthroughs in this field, and what are key challenges and limitations in current NLP models?

The field of NLP has seen a tremendous development in the last years with the advent of large language models, of which ChatGPT is an example. Neural network architectures, particularly transformers, became ubiquitous for addressing many tasks in a unified manner, first by pre-training on large datasets and finetuning for specific tasks, then learning multiple tasks together, and more recently via prompting and in-context learning. Not only this led to a significant progress in the field of NLP, it also inspired other fields, which extended attention mechanisms and transformers to several other domains, including computer vision, robotics, and computational biology, as well as multimodal applications which are a combination of these.

However, there is still a lot to be done. Language models are known to hallucinate too often and they do not provide reliable uncertainty estimates about the information they generate. This poses considerable risks since the high fluency of the generated text may mislead users. They are also not very good at tasks that require reasoning, common sense, grounding, and physical intuition. Existing systems are too English centric and their performance tends to degrade in other languages – this is particularly relevant in Europe which is highly multilingual (e.g., there are 24 official languages in the EU). Training and running these models is computationally very demanding and the current transformer-based architectures do not scale very well to long contexts. All these challenges need to be overcome and there are very active areas of research tackling them.

Your ELLIS Program fosters European research cooperation in NLP. Why is this important in your opinion? What do you hope to achieve with the program?

No single research lab, no matter how well-resourced, would be able to solve all the problems outlined above while working in isolation. These are really fundamental challenges that require coordination across a large scientific network. It is very important for European research institutions to work collaboratively to overcome these challenges and to assume leadership in the field. For example, most of the ongoing progress in large language models worldwide has been driven by large “big science” teams with very good coordination and access to significant computational resources. It is very important that Europe does not stay behind. The ELLIS program is a very good opportunity to create such synergies.

What projects are you currently working on in the program, and how’s the research cooperation among the program members organized?

Our workshops have been very inspiring and successful in fostering collaboration between the different program members. Several of our program members collaborate in European projects. Since NLP is a large field, we really benefit from hearing from everyone’s perspectives. Right now, large language models are of course in the middle of everyone’s attention and we have recently conducted a joint workshop with another ELLIS program on this topic. Our workshops always include breakout sessions for brainstorming various NLP topics and the groups which organize each breakout often follow up to write surveys, tutorials, position papers, or simply to collaborate in projects. For example, a group of researchers including ELLIS students who have all met in our Dagstuhl workshop for the first time have co-authored a survey on an emerging topic of uncertainty in natural language generation. The co-supervision of ELLIS students is also a very effective means to spark collaborations among program members.

What are your thoughts on the future of NLP? Which challenges and research directions will shape the field in the coming years?

We are very excited about the coming years. We expect significant progress in language grounding from combining multiple modalities (e.g. text and video). We expect new architectures that scale better to long context to become a reality. We also expect progress in more efficient NLP systems, and in fact we are already seeing smaller models to catch up with large language models. Controlling generation, preventing hallucinations, quantifying uncertainty, improving models’ interpretability are all directions which will certainly shape the field in the near future.

More information

View all members of the ELLIS Program ‘Natural Language Processing’ here.  

Learn more about all 16 ELLIS Programs here. ELLIS research programs focus on areas with the potential to move the needle in modern AI - from basic research in theory and algorithms to applications in health and climate sciences. They include outstanding researchers as ELLIS Fellows and Scholars. ELLIS program workshops are regularly supported by the EU-funded ELISE project which is building on ELLIS.

Follow ELLIS

Follow ELLIS on X, LinkedIn, Mastodon and Facebook, and subscribe to our monthly newsletter.

Contact ELLIS here.