Very interesting read, thank you for sharing. A couple of questions if I may:
1) So, when AI companies talk about 'reasoning models' they flat out lie? Because AI doesn't reason. It only extrapolates and interpolates data. If this is the case how does Apple's paper fit in that says "models break down at a certain level of complexity'? If there is no reasoning, then reasoning cannot break down?
2) Could data quality and quantity become so huge in certain areas that the problem doesn't matter anymore and promises can be kept? As you said, any sufficiently advanced pattern matching is indistinguishable from intelligence. I am wondering whether this fundamental flaw will ultimately crash the party or whether we will just steamroll over it.
3) You said the model doesn't know the probability. But shouldn't it be possible to derive that based on the underlying statistical process? A confidence metric could be assigned to the output that indicates where the output lies within the curve, similar to the R-Squared of a simple regression?
1) Most are using a loose definition of reasoning which includes heuristics, pattern-matching, algorithms etc. I've started to use the term "understanding" sometimes in place of reasoning to clarify more distinctly. But due to this often conflation of terms, it is sometimes unclear if the authors perception is one in which they also think "understanding" is part of what they consider reasoning.
2) I think the key here is as you say "in certain areas". So, in certain domains we will likely have the error rate low enough that it is acceptable, but it will never be a zero error rate like a deterministic calculator. We probably don't ever want them hands-off managing financial records for example.
3) Yes, some of the papers I mention discuss using the internal probabilities. The details were a bit much to include here, but making practical use of those numbers is a lot more problematic than one might think in concept. The papers mention the issue that probability does not always align with being correct. An answer that has many possible choices, but each valid could have a low probability for its tokens. An incorrect answer could have high confidence simply because it is a common pattern.
Since this would also increase inference costs, probably nobody wants to do it unless it turns out it is really good and I suspect that just isn't the case so far.
You're making the assumption that the AIs are rather "neutral" -- that is, they don't have their own awareness or motivation. That might be true if the AI in question is acting alone, but they probably aren't. Sooner or later the AIs (all of them) are going to be invested by evil spirits, and they do have awareness and motivation. It might even be true now.
I don't even like using the word "hallucination."
It seems wrong to anthropomorphize computer errors.
I also like saying "regurgitative" rather than "generative."
I agree, but it is the term everyone understands now.
More than a Rube Goldberg machines, LLMs are "humans all the way down", meaning there are always humans in the loop somewhere
Very interesting read, thank you for sharing. A couple of questions if I may:
1) So, when AI companies talk about 'reasoning models' they flat out lie? Because AI doesn't reason. It only extrapolates and interpolates data. If this is the case how does Apple's paper fit in that says "models break down at a certain level of complexity'? If there is no reasoning, then reasoning cannot break down?
2) Could data quality and quantity become so huge in certain areas that the problem doesn't matter anymore and promises can be kept? As you said, any sufficiently advanced pattern matching is indistinguishable from intelligence. I am wondering whether this fundamental flaw will ultimately crash the party or whether we will just steamroll over it.
3) You said the model doesn't know the probability. But shouldn't it be possible to derive that based on the underlying statistical process? A confidence metric could be assigned to the output that indicates where the output lies within the curve, similar to the R-Squared of a simple regression?
Thank you!
1) Most are using a loose definition of reasoning which includes heuristics, pattern-matching, algorithms etc. I've started to use the term "understanding" sometimes in place of reasoning to clarify more distinctly. But due to this often conflation of terms, it is sometimes unclear if the authors perception is one in which they also think "understanding" is part of what they consider reasoning.
2) I think the key here is as you say "in certain areas". So, in certain domains we will likely have the error rate low enough that it is acceptable, but it will never be a zero error rate like a deterministic calculator. We probably don't ever want them hands-off managing financial records for example.
3) Yes, some of the papers I mention discuss using the internal probabilities. The details were a bit much to include here, but making practical use of those numbers is a lot more problematic than one might think in concept. The papers mention the issue that probability does not always align with being correct. An answer that has many possible choices, but each valid could have a low probability for its tokens. An incorrect answer could have high confidence simply because it is a common pattern.
Since this would also increase inference costs, probably nobody wants to do it unless it turns out it is really good and I suspect that just isn't the case so far.
You're making the assumption that the AIs are rather "neutral" -- that is, they don't have their own awareness or motivation. That might be true if the AI in question is acting alone, but they probably aren't. Sooner or later the AIs (all of them) are going to be invested by evil spirits, and they do have awareness and motivation. It might even be true now.