AI Foundation Models: A Roadmap for Europe

The development and deployment of large-scale machine learning systems such as GPT-4 have ushered in a new era of computational requirements. The recently discovered scaling laws indicate that performance continues to increase as training sets grow, and the best performance can thus only be obtained using large-scale machine learning on vast computational resources.

The expenditure to operate a supercomputer that runs GPT-4 is well above one billion EUR. This financial barrier means that the AI landscape is increasingly dominated by the most affluent players, raising concerns about a concentration of power with a few overseas companies, and preventing Europe from playing a significant role in the current race for AI supremacy.

Call to establish an intergovernmental multi-centric AI research organization with high-performance computing facilities We argue that a viable solution and indeed an urgent need is the establishment of publicly funded research institutions and high-performance computing facilities. We recommend the creation of an intergovernmental, multi-centric AI research organization with strong institutions in all parts of Europe that will generate real innovation for Europe, best leverage Europe’s cultural diversity, and integrate European values in the development of future technology. It should be equipped with cutting-edge computational facilities providing the capacity required to train the next generation of foundation models. Europe would greatly benefit from at least one, if not several, such large data and compute centers.

Data and compute centers in locations where energy can be produced sustainably Choosing the locations for and designing these compute centers requires careful consideration. Ideally, they should be situated where energy can be produced sustainably and affordably, where cooling is readily achievable, and where waste heat can be potentially repurposed. These sites need not be linked to existing research institutions – indeed, decoupling them would avoid potential political disputes. In prominent private (OpenAI, USA) and public (Jean-Zay, France) examples, the compute centers are indeed not colocated with where the AI research is being done; often, the AI researchers do not even know where their algorithms are run. The primary focus should be on functionality, flexibility and sustainability, rather than symbolic strength by centralization.

While Europe is not home to any of the large AI companies, it plays a central role in the open source movement, which goes largely unnoticed in the general public. In the open source community, researchers and volunteers of diverse backgrounds and expertise voluntarily contribute their time to public projects, and everybody benefits: scientists get fast access to other methods, unnecessary duplication is avoided, and start-ups and companies are able to leverage state-of-the-art software technology. In particular, freely available AI models reduce the need for unnecessary data collection and compute (and thus CO2 generation). The open source ethos encourages scientific progress and knowledge sharing, both essential for innovation. Yet, the potential of these efforts is currently limited by the lack of access to high-performance computing systems. While Europe has an outstanding open source community, and it is conceivable that advancements in technology could enable independent developers to train large AI models and challenge tech giants in the future, the training of modern foundation models currently cannot be parallelized across multiple locations: certain computations strictly need localized infrastructure for high-bandwidth communication during training. We thus must address the present lack of computational resources.

The future of AI lies in broadening access to high-performance computing An open letter that the ELLIS Board recently co-signed emphasizes the importance of open source AI research for both competitiveness and security. Transparency allows independent researchers to quickly identify vulnerabilities and mitigate risks related to the misuse of AI technologies. While transparency also bears the risk of lowering the threshold for potential misuse of AI systems, the benefits of transparency, collaboration, and open exchange of ideas have historically been demonstrated to outweigh the risks. Openness in science has proven to be a successful model, encouraging the competition of ideas that drive innovation, and allowing newcomers with a fresh perspective to enter a field and challenge the received wisdom.

The future of AI lies in broadening access to high-performance computing, fostering open-source practices, investing in attracting and retaining the best minds and demanding transparency in AI research. This approach not only democratizes AI development but also contributes to a more secure and competitive AI landscape.

Initial concrete steps made in Europe are demonstrating benefits of this approach. An example is the large-scale data collection and training of BLOOM, a 176B-Parameter Open-Access Multilingual Language Model, which was a result of a collaboration of more than 1,000 researchers from 70+ countries and 250+ institutions on the French national supercomputer infrastructure Jean Zay. Another example is the large-scale open data collection effort LAION, which resulted in the publicly available OpenCLIP model trained on more than 5 billion image-language pairs.  

A multi-centric AI lighthouse for Europe, aligned with the EU’s strategic objectives ELLIS, the European Laboratory for Learning and Intelligent Systems, is a grassroots activity that has developed an ambitious plan to make Europe competitive in modern AI. This plan is highly synergistic with the creation of one or several high-performance computing facilities for large-scale machine learning, as proposed in related calls from other parts of the community.

As argued here, we advocate a multi-centric AI lighthouse across Europe aligning with the EU’s strategic objectives for AI, offering the following benefits:

• For communities and industries across Europe to benefit from AI advances, AI research and translation must be embedded in local ecosystems. Harnessing the potential of AI as a pervasive technology requires technology adoption across all industry sectors, the public sector, and different sizes of businesses – AI must be responsive to needs ‘on the ground’.
• Connected, regional networks of excellence offer the best opportunity for Europe to attract and retain high-quality AI talent. Europe is already home to several centres and networks of AI excellence, which can be leveraged for the benefit of the wider continent, coordinating across centres to attract talent and skills to all of the EU.
• Europe needs to act fast in strengthening Europe’s global position in AI in the face of fast innovation. Multi-centric networks can double down on existing strengths and offer an agile mechanism to promote world-leading European AI research, while responding to societal issues or concerns through closer engagement between AI researchers, policymakers and the public.
• “AI made in Europe” should reflect and leverage Europe’s culture and diversity. Europe’s strength comes from its diversity, a sense of purpose, and a tradition of excellence -- all three key catalysts for disruptive innovation. Europe needs excellent AI across the continent, so that its benefits can be shared with and distributed to every company (big and small), start-up, institution and citizen.

ELLIS has demonstrated how an excellence-driven approach to AI research and innovation can help secure European AI leadership. In its first 5 years, it has attracted 5,000 applications from students and researchers seeking to work in European AI labs, created a tightly-knit community of more than 1,000 excellent AI scientists, 171 ERC grant winners, obtained substantial funding from the European Commission and established collaborations with other European networks of excellence, and built a network of 39 ELLIS units, selected on the basis of scientific excellence, which have committed more than 300M Euro of their own funding for an initial period of five years. The strength of engagement with ELLIS from researchers across the EU illustrates the desire in the European AI community to collaborate in a distributed network that is built on bringing together outstanding scientists who want to make a difference.

Focus on excellence and functionality We emphasize the importance to keep the focus on excellence rather than symbolic strength through geographic concentration into a single site. While meeting no immediate need, a centralized research body would risk isolating AI research from the sectors, research communities and citizens that it should serve. Faced with a situation where a centralized site in a single member country absorbs large parts of future EU funding, but not the talent (which is the most likely outcome), AI talent currently working at the most dynamic European centres of AI excellence may even consider moving to top AI hotspots overseas.

A multi-centric laboratory with strong institutions in all parts of Europe will generate real innovation for Europe, best leverage Europe’s cultural diversity, and integrate European values in the development of future technology. This will ensure that Europe does not become a mere consumer of AI technology developed elsewhere, building on other values, but instead builds a genuine “AI made in Europe”.

ELLIS offers its expertise and mechanisms to create an intergovernmental AI research organization We have previously issued a call to action that argued for Europe to invest in computing infrastructure for AI. In the years that have elapsed since then, the urgency has dramatically increased. We now recommend that the European nations focus their efforts and resources on aiding the creation of an intergovernmental multi-centric European AI research organization. This organization would benefit from adopting the guiding principles of ELLIS: pan-European collaboration, an emphasis of scientific excellence as the driver of true innovation, and a strong focus on attracting the next generation of international talent. It should be endowed with cutting-edge computational facilities providing the capacity required to train the next generation of foundation models. AI will continue to transform our world, and the development may still be in its infancy. It is not too late for Europe to play a major role. The ELLIS community, which now includes most of the leading machine learning scientists in Europe, is fully prepared to rise to this challenge by offering its expertise and existing structures (including robust mechanisms for international quality control) to European governments, the EU and other stakeholders in order to create an organization that puts Europe at the forefront of modern AI research.

About ELLIS ELLIS is a pan-European AI network of excellence. It builds upon machine learning as the driver for modern AI and aims to secure lasting international leadership of AI made in Europe by connecting top researchers in this field and by creating a multi-centric AI research laboratory. Founded in 2018, ELLIS has grown into a network that counts 39 ELLIS units at world-class institutions in 14 countries, 16 ELLIS programs and a pan-European PhD program. The members of ELLIS are committed to shaping the future of AI in Europe by pushing the scientific and technological boundaries for human-centered, beneficial and safe AI. More about ELLIS: ellis.eu