Brutkey

Taggart
@mttaggart@infosec.exchange

Brutal:

The findings across task, length, and format generalization experiments converge on a conclusion: [Chain-of-Thought reasoning] is not a mechanism for genuine logical inference but rather a sophisticated form of structured pattern matching, fundamentally bounded by the data distribution seen during training. When pushed even slightly beyond this distribution its performance degrades significantly, exposing the superficial nature of the "reasoning” it produces.

NosirrahSec 🏴‍☠️ guillotine enthusiast
@NosirrahSec@infosec.exchange

@mttaggart@infosec.exchange The thing is, this isn't even surprising to anyone that studies these.

It's just a simple fact. You can only force math to look like it's "thinking" on the surface, any deeper glance and it's just a mess.

Taggart
@mttaggart@infosec.exchange

@NosirrahSec@infosec.exchange You'd think so, but my mentions are filled with professionals in the field who love to tell me the opposite. I'm hoping the eventual preponderance of evidence brings some folks out of the cult.

NosirrahSec 🏴‍☠️ guillotine enthusiast
@NosirrahSec@infosec.exchange

@mttaggart@infosec.exchange It's very, very, complex. I don't really fault some of us for not knowing as much about how it all works.

Even my understanding is very meager, but I DO try to learn more about it. I link these videos every time I talk about it, and it's still very basic, but seems daunting at first.

https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi

He normally focuses on physics/mathematics, but he's EASILY the best learning animator out there and I'll die on that hill.