ETH Zurich, INSAIT, and LatticeFlow AI announce the release of the first evaluation framework of the EU AI Act for Generative AI models.

The release, available at https://compl-ai.org, includes the first technical interpretation of the EU AI Act, mapping regulatory requirements to technical ones, together with a free and open-source framework to evaluate Large Language Models (LLMs) under this mapping. The launch also features the first compliance-centered evaluation of public foundation models from organizations such as OpenAI, Meta, Google, Anthropic, and Alibaba against the EU AI Act technical interpretation.

Thomas Regnier, the European Commission's spokesperson for digital economy, research, and innovation, commented on the release: ​“The European Commission welcomes this study and AI model evaluation platform as a first step in translating the EU AI Act into technical requirements, helping AI model providers implement the AI Act.”

First Technical Interpretation of the EU AI Act

The EU AI Act outlines high-level regulatory requirements without providing detailed technical guidelines. To address this, the European Commission has launched a consultation on the Code of Practice for providers of general-purpose Artificial Intelligence (GPAI) models, tasked to supervise the implementation and enforcement of the AI Act rules on GPAI. COMPL-AI can also benefit the GPAI working groups, which can use it as a starting point for their efforts.

COMPL-AI includes also a free and open-source framework built upon 27 state-of-the-art benchmarks that can be used to evaluate LLMs against these technical requirements.

“We invite AI researchers, developers, and regulators to join us in advancing this evolving project,” said Prof. Martin Vechev, Full Professor at ETH Zurich and Founder and Scientific Director of INSAIT in Sofia, Bulgaria. “We encourage other research groups and practitioners to contribute by refining the AI Act mapping, adding new benchmarks, and expanding this open-source framework. The methodology can also be extended to evaluate AI models against future regulatory acts beyond the EU AI Act, making it a valuable tool for organizations working across different jurisdictions.”

“With this framework, any company can now evaluate their AI systems against the EU AI Act technical interpretation. Our vision is to enable organizations to ensure that their AI systems are not only high-performing but also fully aligned with the regulatory requirements,” said Dr. Petar Tsankov, CEO and Co-Founder at LatticeFlow AI.

View source version on businesswire.com: https://www.businesswire.com/news/home/20241016758271/en/