Dennis Schubert@denschub@mastodon.schub.social
A contact just told me that my old "LLMs generate nonsense code" blog post from 2 years ago is now very outdated with GPT5 because it's so awesome and so helpful. So I asked him to give it a test for me, and asked it my favorite test question based on a use-case I had myself recently:
Without adding third-party dependencies, how can I compress a Data stream with zstd in Swift on an iPhone?and here is the answer from ChatGPT 5: https://chatgpt.com/share/68968506-1834-8004-8390-d27f4a00f480
Very confident, very bold, even claims "Works on iOS 16+".
Problem with that: Just like any other LLM I've tested that provided similar responses, it is - excuse my language but I need to use it - absolute horseshit. No version of any Apple SDK ever supported or supports ZSTD (see https://developer.apple.com/documentation/compression/compression_algorithm for a real piece of knowledge). It was never there. Not even in private code. Not even as a mention of "things we might do in the future" on some developer event. It fundamentally does not exist. It's completely made up nonsense.
This concludes all the testing for GPT5 I have to do. If a tool is able to actively mislead me this easy, which potentially results in me wasting significant amounts of time in trying to make something work that is guaranteed to never work, it's a useless tool. I don't like collaborating with chronic liars who aren't able to openly point out knowledge gaps, so I'm also not interested in burning resources for a LLM that does the same.
Lazy B0y@lazyb0y@mastodon.social
@denschub@mastodon.schub.social
Maybe it's a "cultural" thing.
LLMs want to be as polite as possible, which includes not replying that a question can not be answered and preferring to answer completely wrong than refusing the request.
I have the impression many people like that, unfortunately.
I even got a case at work recently where i had to answer a very vague question and the manager asking seemed to be very uninterested in getting more specific, instead preferring any kind of incorrect answer fast.
