3dateswrong, for the same birthday
Like in an exam
Researchers from OpenAI and Georgia Tech compare language models to students facing hard exam questions: when unsure, you guess. If a right answer gets a point and a blank gets none, guessing always pays. Many tests for language models are scored exactly like that.
Three dates, all wrong
The researchers asked a large model for the birthday of the first author, Adam Tauman Kalai – to be answered only if known. Three attempts gave three different wrong dates. Their proposal: tests should stop penalising uncertainty.
