> Mostly capability failures that newer models no longer make: counting letters
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
dcrazy 2 days ago [-]
The LLMs have indeed gotten better at representing numbers and performing arithmetic, but they can still do it wrong.
I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
cyanydeez 3 days ago [-]
Inthink finetuning and prompts and intermediary engineering harnesses all obscure idependent verificant of raw LLM capability.
So yes, they probably spend half theor budget trying to keep suspension of disbelief.
_davide_ 3 days ago [-]
Seems like this claim is expired in 3...2...1...
Rendered at 03:14:57 GMT+0000 (Coordinated Universal Time) with Vercel.
Have LLMs fundamentally actually changed or improved to resolve problems like this, or have the big models been aggressively tuned / trained on specific examples so it’s less embarrassing?
Because I’ve noticed that Google’s stuff always seems to generate code to do counting tasks, even when not asked for, so I wondered if the LLM itself even generated the answer or it’s just being instructed to use a tool? (Which IMO is a workaround)
I don’t consider it a workaround for an LLM to delegate arithmetic to a calculator.
So yes, they probably spend half theor budget trying to keep suspension of disbelief.