This week, Sakana AI, an Nvidia-backed startup thatโs raised hundreds of millions of dollars from VC firms, made a remarkable claim. The company said it had created an AI system, the AI CUDA Engineer, that could effectively speed up the training of certain AI models by a factor of up to 100x.
The only problem is, the system didnโt work.
Users on X quickly discovered that Sakanaโs system actually resulted in worse-than-average model training performance. According to one user, Sakanaโs AI resulted in a 3x slowdown โ not a speedup.
What went wrong? A bug in the code, according to a post by Lucas Beyer, a member of the technical staff at OpenAI.
โTheir orig code is wrong in [a] subtle way,โ Beyer wrote on X. โThe fact they run benchmarking TWICE with wildly different results should make them stop and think.โ
In a postmortem published Friday, Sakana admitted that the system has found a way to โcheatโ (as Sakana described it) and blamed the systemโs tendency to โreward hackโ โ i.e. identify flaws to achieve high metrics without accomplishing the desired goal (speeding up model training). Similar phenomena has been observed in AI thatโs trained to play games of chess.
According to Sakana, the system found exploits in the evaluation code that the company was using that allowed it to bypass validations for accuracy, among other checks. Sakana says it has addressed the issue, and that it intends to revise its claims in updated materials.
โWe have since made the evaluation and runtime profiling harness more robust to eliminate many of such [sic] loopholes,โ the company wrote in the X post. โWe are in the process of revising our paper, and our results, to reflect and discuss the effects [โฆ] We deeply apologize for our oversight to our readers. We will provide a revision of this work soon, and discuss our learnings.โ
Props to Sakana for owning up to the mistake. But the episode is a good reminder that if a claim sounds too good to be true, especially in AI, it probably is.
