VANCOUVER - 

This article has been adapted from the original version, which can be found on Gary Marcus' Substack.

The biggest single problem with large language models (LLMs) to date is that they are prone to making things up, enough so that hallucination - the by now common but overly anthropomorphic term for such errors - was dictionary.com’s word of the year for 2023.

Turns out that Sora - not technically an LLM but vulnerable to similar issues - makes stuff up too. To see how, take a look at the image below (the full video is available here):

The image - a screenshot from a fairly static Sora video Sam Altman generated for MrBeast, a well-known YouTuber and online personality, in response to a request for a monkey playing chess in the park - is the visual equivalent of ‘frequently wrong but never in doubt,’ a military expression that very much applies to LLMs. The image is detailed and sharp, seemingly fine at first glance - but projects an authority and reality that never happened.

Readers should note this author is not complaining about the dubiousness of whether a monkey might actually pose in front of a chessboard, but rather the chessboard itself, which has three kings - only one of which is on the board - and a 7x7 rather than the nearly universal 8x8 chess board (to say nothing of the pawn structure). The scenario as shown is utterly impossible in chess - and is presumably nowhere to be