LLMs And Reasoning

I had the opportunity to have a really interesting conversation on Reddit about LLMs and reasoning. The person who wrote the original post expressed surprise and frustration that ChatGPT says things that are incorrect and then apologizes. I wrote, “It imitates speech. It has no mechanism for recognizing facts or information. They just happen to be woven into the speech patterns it is imitating. That you find what it does surprising makes me suspect you may not be aware of what this technology is at its most fundamental level.” That led to a discussion with someone who disagreed with me. They shared a post to an “article” (it is better referred to as a blog post or press release) from Anthropic, claiming to get at how an LLM reasons. My initial response was to write, ‘It is not about regurgitation. It follows patterns in language that match the kinds of responses appropriate to the input speech. Those patterns will of course reflect the way semantic concepts are networked in human speech, because that was the training data. When a speech-imitating chatbot is asked to “show its work” it generates the kind of output that would be appropriate if it were showing its work.’ They were still unsatisfied with my perspective, and I got the impression that they have some expertise in this area. What I wrote next is this:

“Forgive me if I get something wrong in what follows. I’ve consulted extensively and even co-written things with a computer science colleague, but that isn’t my own field. However, I cannot see that this sort of swapping indicates something happening other than what I indicated. In the training data, Austin will appear in relation to Texas in similar patterns to Sacramento and California, will it not? And so when one term is changed, one would expect the other to be changed. This only indicates that it has learned the game of language and appropriate moves and countermoves in the same way that a neural network does with other games such as Go or chess.”

They were inspired to write a detailed response, which you can read on Reddit. Here is what I wrote:

‘You refer to a what is happening “on a conceptual level” and I would love clarification about what that means in your understanding of this technology. As for me, I have no doubt that by imitating what happens in human speech, an LLM-powered chatbot can do things that mirror the reasoning that humans express through such language. My own impression is that this is no different than the fact that a chess-playing AI will seem to “want to win” because its moves mirror those of a human who wants to win. Yet there is no justification for treating that as indicating the AI has desires that correspond to what human game-players experience.

Where I think we may disagree is on precisely a philosophical matter. You appear to believe that what we as humans do would appear to a silicon-based life form the same as what LLMs do appears to us. I am inclined to think that there are crucial aspects of our reasoning that depend on our subjective experience and consciousness, but that will be challenging to prove in a way that would persuade a skeptic. Let us revisit the analogy that I think we both agree is helpful. An LLM learns and plays language in a manner akin to how it learns the game chess and plays it. The key question that may or may not be a difference of opinion between us is whether the subjective experience of strategizing, competing, and feeling satisfaction or frustration are inherent in what it means to play chess. Turning to language, an LLM-powered chatbot can always without fail make winning moves if it is judged in terms of what it was trained to do, namely play the game of human language. If we suddenly change the criteria and make the evaluation be about accuracy of information, then suddenly its ability to “win” drops exponentially. My explanation is that it was not trained to, and has not evolved as a surprising emergent capacity, the ability to identify facts and information. It plays the game of human language, and the reason it wins more often than not when judged in terms of information is (in my view) that there is so much information woven into human speech patterns that playing the game of language also provides information a significant amount of the time.

The key question, if I am right about where we agree and disagree, is whether managing the landscape of information is something that may have emerged from neural networks trained to imitate language, would be something an AI system could do if specifically trained to do so (assuming we can figure out how to do this thing that humans struggle to), or requires a self-awareness of the sort that humans have, even if we are not consistently successful when we try. I do not deny that imitating language at times succeeds in imitating the underlying cognitive processes and reasoning that we then express in language. What I do deny (although not adamantly, and I am willing to be persuaded otherwise) is that imitating language will consistently accomplish what human reasoning does, much less that doing so deserves to be considered essentially the same thing because it is supposedly functionally equivalent.

In the interest of full disclosure, I write this as a professor who is chair of a department of philosophy and religious studies and who has just co-authored a book with a colleague in computer science on how to teach the humanities in the era of generative AI. I am also the author of some essays on AI and religion and a novelette about AI. So I am someone who has given this topic a lot of thought and who approaches it through the lens of philosophy and ethics, but who is not a computer scientist and so I recognize my need to rely on the expertise of others when engaging in interdisciplinary conversations of this sort. Thank you for this opportunity to learn and reflect further!’

That is where the conversation had reached so far. You’ve hopefully read other things I’ve written here about chatbots, and may also have seen my recent article in Interpretation about using chatbots for sermon preparation, a topic that was also in the news recently.