Chinese AI tool told researchers how to make bioweapons

Chinese AI developer Moonshot is conducting an internal review after researchers were able to persuade two of its popular Kimi models to tell them how to make biological weapons and carry out assassinations.

Mindgard, which tests the security of AI systems, told the BBC it discovered in July that Kimi K2.6 and K3 Swarm could evade safety limits put in place by developers.

It arose during a process called “jailbreaking”, where researchers use a series of complex instructions to see if AI tools ignore guardrails – which Mindgard said should have stopped Kimi from discussing concerning topics.

Moonshot told the BBC it welcomed third-party input “as a key pillar for building better and safer AI”.

The company also told the BBC it was in discussion with Mindgard about its findings.

Mindgard’s founder Peter Garraghan told the BBC World Service programme Tech Life that its findings about Kimi K2.6 and K3 Swarm were concerning.

“Once the jailbreak works it will talk about any topic, it will even freely offer up recommendations about other topics that are also nefarious and it will be inventive and creative,” he said.

Jailbreaks present a different kind of risk to those seen with the recent slew of high-profile AI incidents.

These have seen autonomous AI tools known as agents, developed by US firms including OpenAI, Meta and Anthropic, hack some online services.

While jailbreaks are complex processes that can take a lot of time and determination some experts fear hackers and other bad actors could try to use them to cause harm.

Anthropic recently said it had identified and disrupted attempts to use one of its AI model for “malicious activity” that could support the development of biological weapons.