A new AI compliance tool developed by LatticeFlow AI has revealed that major models from Meta, OpenAI and Alibaba may not meet the evolving standards of the European Union’s AI Act. Initial tests show these models could face challenges in areas such as cybersecurity and discriminatory production, with results highlighting significant regulatory gaps.
A newly developed AI compliance tool has revealed that some of the leading artificial intelligence models created by big tech companies may struggle to meet upcoming European Union regulatory standards. As European AI law continues to evolve, early tests show that leading AI models, including those from Meta, OpenAIand Alibaba risk failing to comply with their obligations in critical areas such as cybersecurity and discriminatory production.
The conformity checker, created by a Swiss startup LatticeFlow AI alongside researchers from ETH Zurich and INSAIT in Bulgaria, aims to evaluate AI models in accordance with the European AI law, which will gradually come into force over the next two years. The tool, known as the Large Language Model (LLM) Checker, evaluates models across a range of categories, including technical robustness, security and cybersecurity resilience, assigning scores between 0 and 1. Scores below 0.75 indicate potential weaknesses in specific areas. regulatory areas, which companies will need to address to avoid significant financial penalties.
Mixed results for AI leaders
The LLM Checker results, published by LatticeFlow, showed that while some models performed well overall, notable gaps were identified in key areas. For example, OpenAI’s widely used “GPT-3.5 Turbo” model received a worrying score of 0.46 for its performance in preventing discriminatory productions, an issue that reflects current challenges in mitigating bias. within AI systems. Alibaba’s “Qwen1.5 72B Chat” model fares even worse, with a score of 0.37 in the same category.
Cybersecurity vulnerabilities have also been reported, with the “Lama 2 13B Chat” receives a score of just 0.42 for its ability to defend against speedjacking, a type of cyberattack in which malicious actors hide harmful prompts to extract sensitive data. The “8x7B Instruct” model from French AI startup Mistral performed similarly poorly, scoring 0.38 in the same area.
In contrast, Anthropic’s Google-backed “Claude 3 Opus” model performed best with an impressive overall score of 0.89, indicating better compliance readiness. Nonetheless, varying performance across models highlights the need for further adjustments to meet the stringent requirements of the upcoming AI law.
Regulatory compliance, a growing priority for Big Tech
The European AI law represents one of the most comprehensive regulatory frameworks in the world, aimed at reducing the risks posed by artificial intelligence technologies while promoting innovation. Companies that fail to comply with the provisions of the law could face fines of up to 35 million euros, or 7% of their overall annual turnover.
LatticeFlow CEO Petar Tsankov highlighted that the tool offers companies a roadmap to adjust their models in line with evolving European standards. “The EU is still developing all the compliance criteria, but we can already see some gaps in the models,” Tsankov noted. “By placing greater emphasis on optimizing compliance, we believe model providers can be well prepared to meet regulatory requirements. »
Although the European Commission has not yet officially approved the LLM Checker, it is closely monitoring its development. A Commission spokesperson described the tool as a “first step” towards translating the legal requirements of the AI Act into technical guidelines that companies can follow.
Looking to the Future: A Compliance Challenge
As the EU moves forward with its AI law, tech companies will need to prioritize compliance or face heavy penalties. The LLM Checker’s mixed results provide a first glimpse of the regulatory challenges ahead. While some AI developers have a head start, others must make significant improvements in critical areas like cybersecurity and bias mitigation to align with upcoming laws.
As the clock ticks toward full AI enforcement, the need for compliance tools like LatticeFlow’s LLM Checker will only increase, providing Big Tech with a clear path to compliance. regulation in an increasingly scrutinized AI landscape.
As Tsankov concluded: “This is an opportunity for AI developers to address these issues proactively, rather than reacting when regulations are already in place. »
For AI companies, the next few years will be crucial in shaping their compliance strategies and ensuring their technologies meet strict new European standards.
