AI fails

[briantrumpet]

I was just going to make a quip about how if my French house at 500m ASL were flooded by rising sea levels, that'd mean that not one UK town would be above sea level. So I asked.

The answer surprised me, so I looked at one of the sources, and it seems that AI can't understand the sources it quotes: the elevations they quote are of nearby mountains.

https://sothebysrealty.co.uk/the-journal/top-10-highest-towns-in-england/

Buxton: "The town claims to be the highest market town in England and, indeed, is situated near Shining Tor, a peak with an elevation of 559 metres (above sea level)."

Marsden "Located near the southern edge of the Pennines, Marsden is a market town set among rolling hills and breathtaking landscapes. The town provides access to the Pennine Way, a footpath that leads through the Pennine range and their villages. At 582 metres above sea level, Black Hill boasts sweeping views of the surrounding moorland and valleys."

Okehampton: "From the town of Okehampton, walkers can access Dartmoor National Park and climb its highest point, High Willhays – which rises 621 metres above sea level."

Even given that Sotheby's entitles the piece "Top 10 highest Towns in England" and includes Hay-on-Wye because of the Black Mountains it's fairly near, AI's inability to understand even one sentence doesn't give one much faith in its ability to 'understand' anything at all. "Available data", my ârse. It's like a 5-year-old using the word 'data'.

 

[briantrumpet]

An interesting article on AI hits & misses in specific professional fields:

https://inews.co.uk/inews-lifestyle/gp-tested-against-ai-results-scared-patients-3985378

 

[matticus]

The answer surprised me, so I looked at one of the sources, and it seems that AI can't understand the sources it quotes: the elevations they quote are of nearby mountains.

https://sothebysrealty.co.uk/the-journal/top-10-highest-towns-in-england/

There's two problems (at least):
- failure of AI to properly read the source
- the source being tosh:

That article certainly doesn't list the 10 highest towns in the England (Alston??). And it gets at least one simple fact wrong:
"Hay-on-Wye.. provides access to Black Mountain – with an elevation of 811 metres" no way are any of those hills 811 meters high!
bollox - back to map-reading school for me. I've even walked up that one!

It's no wonder that forums (like this) and social media are being flooded with utter gash, often in good faith (peppered with extreme ignorance!)

 

[briantrumpet]

There's two problems (at least):
- failure of AI to properly read the source
- the source being tosh:

That article certainly doesn't list the 10 highest towns in the England (Alston??). And it gets at least one simple fact wrong:
"Hay-on-Wye.. provides access to Black Mountain – with an elevation of 811 metres" no way are any of those hills 811 meters high!

It's no wonder that forums (like this) and social media are being flooded with utter gash, often in good faith (peppered with extreme ignorance!)

Quite so. As I've mentioned before, scepticism is an important part of assessing information/data - "Does this look right?", in assessing both source material and output - and AI does not have that capability, and I'm not convinced it ever can.

 

[briantrumpet]

That said...

 

[C R]

This could have gone to the reform thread, but there's an AI side to it.

https://www.worcesternews.co.uk/news/25557344.bid-overturn-election-result-dismissed-high-court/

In one of the wards of Worcestershire County Council there was a dead heat between a Green candidate and a Reform candidate, so the returning officer drew a name, and the Green candidate won.

The Reform candidate appealed, but the appeal was dismissed for being filed too late. Among the comments from the judge was that one of the cases cited in support of the appeal doesn't exist, and is likely an AI hallucination.

The Reform candidate now owes about £19000 in costs.

 

[briantrumpet]

This could have gone to the reform thread, but there's an AI side to it.

https://www.worcesternews.co.uk/news/25557344.bid-overturn-election-result-dismissed-high-court/

In one of the wards of Worcestershire County Council there was a dead heat between a Green candidate and a Reform candidate, so the returning officer drew a name, and the Green candidate won.

The Reform candidate appealed, but the appeal was dismissed for being filed too late. Among the comments from the judge was that one of the cases cited in support of the appeal doesn't exist, and is likely an AI hallucination.

The Reform candidate now owes about £19000 in costs.

These hallucinated legal cases seem to be all too common, as with the one that Copilot dreamt up for me about differential haircutting rates for girls/boys women/men, but that I then couldn't get it to find again. If the '10 highest UK towns' example is anything to go by, it's got a long way to go before its comprehension of text is trustable.

 

[Psamathe]

it's got a long way to go before its comprehension of text is trustable.

Except as I posted earlier, in part cannot be solver as the hallucinations are inherent in the math techniques used.

Why OpenAI’s solution to AI hallucinations would kill ChatGPT tomorrow
The paper provides the most rigorous mathematical explanation yet for why these models confidently state falsehoods. It demonstrates that these aren’t just an unfortunate side effect of the way that AIs are currently trained, but are mathematically inevitable.

Suggests a radical change of design.

 

[briantrumpet]

Except as I posted earlier, in part cannot be solver as the hallucinations are inherent in the math techniques used.

Suggests a radical change of design.

I might have been being optimistic.

 

[briantrumpet]

There's enthusiasm, then there's bordering on fraud in order to boost your product...

https://techcrunch.com/2025/10/19/openais-embarrassing-math/?mc_cid=b5f09ec8a4

The Decoder reports that in a since-deleted tweet, OpenAI VP Kevin Weil declared that “GPT-5 found solutions to 10 (!) previously unsolved Erdős problems and made progress on 11 others.” (“Erdős problems” are famous conjectures posed by mathematician Paul Erdős.)

However, mathematician Thomas Bloom, who maintains the Erdos Problems website, said Weil’s post was “a dramatic misrepresentation” — while these problems were indeed listed as “open” on Bloom’s website, he said that only means “I personally am unaware of a paper which solves it.”

In other words, it’s not accurate to claim GPT-5 was able to solve previously unsolved problems. Instead, Bloom wrote, “GPT-5 found references, which solved these problems, that I personally was unaware of.”

Sebastien Bubeck, an OpenAI researcher who’d also been touting GPT-5’s accomplishments, then acknowledged that “only solutions in the literature were found,” but he suggested this remains a real accomplishment: “I know how hard it is to search the literature.”