Thanks for the response! A few thoughts in rough order of reading
- it's not obvious to me that reasoning is 'the ability to synthesize rules', nor that this is different from pattern matching, nor that LLMs cannot do this. I think this paper shows LLMs struggle *on these kinds of tasks* but that's different from 'llms can't generalize rules at all ever'
- LLMs also can identify when they make mistakes. They have additional limitations right now due to things like effective context window limitations, but within that window it's been demonstrated that LLMs can identify errors. See the s1 paper.
- it's not obvious to me that an LLM wouldn't be able to solve these problems with enough time and a large enough context window. This is hard to do because it's expensive, but I have no reason to believe it's impossible
I think your response hinges on some causal reason for why LLMs are incapable for learning these things, what you call an "incapability of learning logical rules." The problem is, I disagree that LLMs can't learn logical rules. More generally, no one can point to a causal reason for why such things are *impossible*. Impossible is a big word! The halting problem is impossible. Going faster than the speed of light is impossible. Towers of Hanoi? IDK man, I just don't see it. This paper identifies weaknesses in the current generation of models on a specific subset of tasks, and imo that's really all it shows. It's pretty narrow! I think It's hard to reach for theoretical conclusions about why something doesn't work when we don't even have a particularly good idea of why some of this stuff*does* work
I get that it's not immediately obvious that that's the case, but ultimately it is the most sophisticated way of describing the act of reasoning I have come across, with nothing else really coming close. Brandom's essay Reasoning and Representing found here https://sites.pitt.edu/~rbrandom/Texts%20Mark%201%20p.html migt be a good introduction, or thus the work of Richard Evans on AI, specifically the material in his PhD. Also because it is ultimately quite simple, just requiring an agent to grasp and follow the rules of first order logic so as to be able to commit themselves to certain concepts and judgements. To me it seems that LLM's are in some sense able to grasp these rules, but they lack a proper ability to follow them, as it stays at a pattern-match level, instead of going to a truly logically necessitating level. You need pattern-matching to grasp such rules, but ultimately the rules are not reducible to pattern-matching, but lead a life of their own.
I might have worded it a bit too harshly, but I am convinced that LLM's can't follow their own rules right now, not that it is impossible for them at all ever. But looking at it from this perspective gives us a great way to explain the errors that we see LLM's making all the time, which is namely a weird lack of rigidity in its responses. For me it explains the weird fact that an LLM can quite handily summarize a large text, but once you ask it to hand you a specific quote from such a text it falters completely and just starts making shit up. It's in some weird way both extremely smart, and also less capable of grasping and following a rule than your average 10-year old. And I think the reason for this, which the Apple paper shows quite well, is that it is unable to synthesize and follow its own rules in a proper way, and thus also unable to follow the steps of an algorithm to solve for example the Towers of Hanoi, which shouldn't be a hard task if you're given the algorithm! It namely then changes from an NP task to a P task, which you should expect an agent able to write extremely complex code to be able to do, but weirdly doesn't. It can even write such an algorithm, which we see because it can write code to solve it, thus it can synthesize rules, but it can't follow them itself, and I think it would become a lot more capable, and actually reasoning, if it could follow the logical rules it devises for itself.
Also the error thing is a bit of a technical point concerning what it means to follow a rule, and importantly it can retroactively see it hasn't followed a rule correctly, but it can't then actually start following the rule in the new situation. They're weirdly stunted in that sense, primarily because they lack this sort of conceptual and logical rigidity needed for complex reasoning.
Also I imply no causal reason, I think there is a problem in the formal makeup of their architecture, and that maybe adding something like a logic-solver like the one described by Evans inside the architecture, to which it would be able to feed certain input and rules and get a logically sound response it could then use in its reasoning process would solve a lot of problems. Using such a logic solver would namely give it a deterministic response, thus if the LLM would give its internal logic engine something like a matrix and the rules for multiplication (or a symbolic representation of the Towers of Hanoi problem and the algorithm needed to solve it) it would *always* provide the same answer, which if the rules are correct would give the correct answer, instead of the nondeterministic behaviour we observe in LLM's now, which basically just guess at the solution to a quite simple matrix multiplication, and give different responses every time due to the nondeterministic nature of their responses. This kind of reflects the fact that it is quite able to solve NP problems, but solving relatively complex P problems makes it stumble way too soon. If it would have such a rule-engine in its architecture which it could access, to which it gives input by translating the complex data it gets to structured symbolic data and rules, then it could quite handily start solving Towers of Hanoi on its own.
Or better than a logic solver would maybe be to spend more time to hammer the rules of logic and the imperative to follow these into some specific part of their architecture? I'm reminded of Wittgenstein trying to teach Austrian kids logic through repetition.
While it's true that the response to this is overblown, I think you're missing an important aspect of reasoning that the LLM's indeed don't show right now, as reasoning in the proper sense is the ability to synthesize rules, which one then tries to follow in a strict sense. LLM's just aren't doing this right now, and therefore they're not really yet doing the whole reasoning business, just approximating it very well. What the paper shows is that the LLM's are unable to construct rules to logically deconstruct a problem, and are also unable to follow a rule-set solving such a problem, even when handed to them. You argue that humans also make mistakes when following such rules, but in some sense this is not true, for when a human is given the algorithm to solve a Rubik's Cube, or a 10 disk Tower of Hanoi, it will eventually solve it by following the algorithm, even when making some mistakes in the meantime, because it can realize that it has made a mistake in following the rules at a certain point, whereafter it can redo the operation to actually succeed in following the rules. Moreover, if the human is careful enough, it should be able to without fail follow the rules to its completion, which makes the whole point about the billion additions kind of redundant. It is perfectly conceivable that a human would be able to solve that, given enough time, yet the crux is that the LLM (as it stands now) is wholly unable to follow such rules strictly enough to ever succeed in such an operation, or solve a complex problem.
Allowing the LLM to use code can allow it to bypass this inherent problem, but as of yet it is still unable to give itself logical rules, and then to abide by these. Robert Brandom has done a lot of philosophical work to show that reasoning is not just pattern-matching, but is the ability to formulate and follow such logical inference-rules, and for example Richard Evans has shown in a PhD at Deepmind how such a system could be implemented in AI. This is also why I agree with you that the paper shouldn't be seen to discount the possibility of reasoning for LLM's, but it does show that the current implementations hit a principled wall, which should prompt us to revise our ideas about what it means to reason for an AI, not just to hope to pattern-match it away, as if you follow Brandom's arguments, mere pattern-matching will never get us to where we want to be.
The real problem is not that LLM's make mistakes, it's really that they aren't even able to make a mistake, as they aren't able to actually follow a logical rule in the first place! You can only make a mistake when you try to do something correctly, but it has no notion of this at all. Yet this seems wholly achievable if we attack the problem it in the right manner.
Thanks for the response! A few thoughts in rough order of reading
- it's not obvious to me that reasoning is 'the ability to synthesize rules', nor that this is different from pattern matching, nor that LLMs cannot do this. I think this paper shows LLMs struggle *on these kinds of tasks* but that's different from 'llms can't generalize rules at all ever'
- LLMs also can identify when they make mistakes. They have additional limitations right now due to things like effective context window limitations, but within that window it's been demonstrated that LLMs can identify errors. See the s1 paper.
- it's not obvious to me that an LLM wouldn't be able to solve these problems with enough time and a large enough context window. This is hard to do because it's expensive, but I have no reason to believe it's impossible
I think your response hinges on some causal reason for why LLMs are incapable for learning these things, what you call an "incapability of learning logical rules." The problem is, I disagree that LLMs can't learn logical rules. More generally, no one can point to a causal reason for why such things are *impossible*. Impossible is a big word! The halting problem is impossible. Going faster than the speed of light is impossible. Towers of Hanoi? IDK man, I just don't see it. This paper identifies weaknesses in the current generation of models on a specific subset of tasks, and imo that's really all it shows. It's pretty narrow! I think It's hard to reach for theoretical conclusions about why something doesn't work when we don't even have a particularly good idea of why some of this stuff*does* work
I get that it's not immediately obvious that that's the case, but ultimately it is the most sophisticated way of describing the act of reasoning I have come across, with nothing else really coming close. Brandom's essay Reasoning and Representing found here https://sites.pitt.edu/~rbrandom/Texts%20Mark%201%20p.html migt be a good introduction, or thus the work of Richard Evans on AI, specifically the material in his PhD. Also because it is ultimately quite simple, just requiring an agent to grasp and follow the rules of first order logic so as to be able to commit themselves to certain concepts and judgements. To me it seems that LLM's are in some sense able to grasp these rules, but they lack a proper ability to follow them, as it stays at a pattern-match level, instead of going to a truly logically necessitating level. You need pattern-matching to grasp such rules, but ultimately the rules are not reducible to pattern-matching, but lead a life of their own.
I might have worded it a bit too harshly, but I am convinced that LLM's can't follow their own rules right now, not that it is impossible for them at all ever. But looking at it from this perspective gives us a great way to explain the errors that we see LLM's making all the time, which is namely a weird lack of rigidity in its responses. For me it explains the weird fact that an LLM can quite handily summarize a large text, but once you ask it to hand you a specific quote from such a text it falters completely and just starts making shit up. It's in some weird way both extremely smart, and also less capable of grasping and following a rule than your average 10-year old. And I think the reason for this, which the Apple paper shows quite well, is that it is unable to synthesize and follow its own rules in a proper way, and thus also unable to follow the steps of an algorithm to solve for example the Towers of Hanoi, which shouldn't be a hard task if you're given the algorithm! It namely then changes from an NP task to a P task, which you should expect an agent able to write extremely complex code to be able to do, but weirdly doesn't. It can even write such an algorithm, which we see because it can write code to solve it, thus it can synthesize rules, but it can't follow them itself, and I think it would become a lot more capable, and actually reasoning, if it could follow the logical rules it devises for itself.
Also the error thing is a bit of a technical point concerning what it means to follow a rule, and importantly it can retroactively see it hasn't followed a rule correctly, but it can't then actually start following the rule in the new situation. They're weirdly stunted in that sense, primarily because they lack this sort of conceptual and logical rigidity needed for complex reasoning.
Also I imply no causal reason, I think there is a problem in the formal makeup of their architecture, and that maybe adding something like a logic-solver like the one described by Evans inside the architecture, to which it would be able to feed certain input and rules and get a logically sound response it could then use in its reasoning process would solve a lot of problems. Using such a logic solver would namely give it a deterministic response, thus if the LLM would give its internal logic engine something like a matrix and the rules for multiplication (or a symbolic representation of the Towers of Hanoi problem and the algorithm needed to solve it) it would *always* provide the same answer, which if the rules are correct would give the correct answer, instead of the nondeterministic behaviour we observe in LLM's now, which basically just guess at the solution to a quite simple matrix multiplication, and give different responses every time due to the nondeterministic nature of their responses. This kind of reflects the fact that it is quite able to solve NP problems, but solving relatively complex P problems makes it stumble way too soon. If it would have such a rule-engine in its architecture which it could access, to which it gives input by translating the complex data it gets to structured symbolic data and rules, then it could quite handily start solving Towers of Hanoi on its own.
Or better than a logic solver would maybe be to spend more time to hammer the rules of logic and the imperative to follow these into some specific part of their architecture? I'm reminded of Wittgenstein trying to teach Austrian kids logic through repetition.
While it's true that the response to this is overblown, I think you're missing an important aspect of reasoning that the LLM's indeed don't show right now, as reasoning in the proper sense is the ability to synthesize rules, which one then tries to follow in a strict sense. LLM's just aren't doing this right now, and therefore they're not really yet doing the whole reasoning business, just approximating it very well. What the paper shows is that the LLM's are unable to construct rules to logically deconstruct a problem, and are also unable to follow a rule-set solving such a problem, even when handed to them. You argue that humans also make mistakes when following such rules, but in some sense this is not true, for when a human is given the algorithm to solve a Rubik's Cube, or a 10 disk Tower of Hanoi, it will eventually solve it by following the algorithm, even when making some mistakes in the meantime, because it can realize that it has made a mistake in following the rules at a certain point, whereafter it can redo the operation to actually succeed in following the rules. Moreover, if the human is careful enough, it should be able to without fail follow the rules to its completion, which makes the whole point about the billion additions kind of redundant. It is perfectly conceivable that a human would be able to solve that, given enough time, yet the crux is that the LLM (as it stands now) is wholly unable to follow such rules strictly enough to ever succeed in such an operation, or solve a complex problem.
Allowing the LLM to use code can allow it to bypass this inherent problem, but as of yet it is still unable to give itself logical rules, and then to abide by these. Robert Brandom has done a lot of philosophical work to show that reasoning is not just pattern-matching, but is the ability to formulate and follow such logical inference-rules, and for example Richard Evans has shown in a PhD at Deepmind how such a system could be implemented in AI. This is also why I agree with you that the paper shouldn't be seen to discount the possibility of reasoning for LLM's, but it does show that the current implementations hit a principled wall, which should prompt us to revise our ideas about what it means to reason for an AI, not just to hope to pattern-match it away, as if you follow Brandom's arguments, mere pattern-matching will never get us to where we want to be.
The real problem is not that LLM's make mistakes, it's really that they aren't even able to make a mistake, as they aren't able to actually follow a logical rule in the first place! You can only make a mistake when you try to do something correctly, but it has no notion of this at all. Yet this seems wholly achievable if we attack the problem it in the right manner.
Ugh substacks app screwed up. My response became a top level comment instead of a reply. See here: https://theahura.substack.com/p/a-few-quick-thoughts-on-apples-illusion/comment/124506302?utm_source=share&utm_medium=android&r=5sutf