You're right that understanding and prediction aren't identical. Prediction doesn't imply understanding but understanding does implies prediction. There are a lot of issues that are conflated and it will help to make some distinctions. We understand functions when we understand how the function maps inputs to outputs. We can do this through an exhaustive specification of input/output pairs or by specification of how generic inputs are transformed into outputs. But understanding from direct specification breaks down when the space of operations is too big to comprehend. If all we can do is run a program to see its output because its state space is too large and/or complicated to comprehend, then we lack understanding.
This is the state we find ourselves when it comes to LLMs. Their space of operations is too large to comprehend as a direct sequence of transformations. Any hope of understanding will come from the identification of relevant features of the system and how those features impact the behavior of the process. Understanding implies prediction because of the connection between causally/semantically relevant features and their influence on behavior. When we say we understand how something works, we are saying we comprehend the relevant features of the system and how they influence relevant behavior. Prediction is a consequence of this kind of understanding.
The danger in LLMs is that their sequence of operations is so large and opaque that we do not know at what level features of the trained network or the input will impact the output and what if any bounds there is on such impact. There is plausibly some semantically empty (to us) feature of the input that can have an outsized impact on its output (relative to its semantic relevance, e.g. adversarial perturbations). As long as we cannot conclusively rule out such features or put bounds on their impact on the expected behavior of the system, we cannot say we understand how they work. The potential existence of ghost features that can cause relevant deviations in expected output just points to structure within the network that we are blind to. Unknown relevant structure just means gaps in understanding.
As far as your RNG example goes, we say we understand the RNG because we understand how it captures entropy from various sources and how it manages its state buffers. We understand how it maps inputs to outputs. We can't predict its output without first knowing its input, but that's neither here nor there.
>> But understanding from direct specification breaks down when the space of operations is too big to comprehend. If all we can do is run a program to see its output because its state space is too large and/or complicated to comprehend, then we lack understanding.
I don't agree. That's [edit: part of] why maths are established as the language of science, because they're a formal language of abstractions by which we can avoid having to deal with large, or even infinite, spaces, by writing down a finite formula that fully defines the relevant concept.
For example, I don't have to count to infinity, to understand infinity. I can study the definition of infinity (there are more than one) and understand what it means. Having understood what infinity means, I can then reuse the concept in calculations, where again I don't have to count to infinity to get to a correct result. I can also reuse the concept to form new concepts, again without having to count to infinity.
With LLMs then, we have the maths that define their "space of operations". We can use those maths to train LLMs! Again- what else understanding remains to be had? I do think you're still talking about tracing the operations in an LLM to fully follow how inputs become outputs. But that's not how we commonly approach the problem of understanding, and even predicting the behaviour, of large and complex technological artifacts. Like, I don't reckon there's anyone alive that could draw you a diagram of every interaction between every system on an airliner. Yet, we "understand" those systems and in fact we can analyse them and predict their behaviour (with error).
That kind of analysis is missing from LLMs, but that's because nobody wants to do it, currently. People are too busy poking LLMs and oohing and aaahing at what comes out. I'm hoping that, at some point, the initial rush of hype will subside and some good analytical work will be produced. This was done for previous LLMs although rarely of course, and poorly, because of the generally poor methodologies in machine learning research.
>> The danger in LLMs is that their sequence of operations ...
That's relevant to what I say above. Yeah, that work hasn't been done and it should be done. But that's not about understanding how LLMs work, it's about analysing the function of specific systems.
The SEP article on understanding perhaps will be helpful to break the impasse. It cites an influential theorist[1]:
>Central to the notion of understanding are various coherence-like elements: to have understanding is to grasp explanatory and conceptual connections between various pieces of information involved in the subject matter in question.
Understanding individual operations in isolation is a far cry from understanding how the system works as a collective unit, i.e. "grasping explanatory and conceptual connections". If you accept weak emergence as a concept (and you should), then you recognize that the behavior of a complex system can be unpredictable from an analysis of the behavior of the components. The space of potential interactions grows exponentially as units and their effects are added. Features of the system that are relevant to its behavior are necessary to comprehend in order to be said to understand the system. Ghost features, i.e. unidentified structure, undermine the claim of understanding. We presumably understand the rules of particle physics that constrain the behavior of all matter in the universe. But it would be absurd to claim that we understand everything about the objects and phenomena that make up the universe. It just ignores the computational and interaction complexity as irrelevant to understanding. This is plainly a mistake.
Regarding your point about science, the difference is that the process of science is, at least in part, about increasing predictive accuracy independent of understanding. Mathematics obviously helps here. But this doesn't say anything about whether predictive accuracy increases understanding, which you already assert is an independent concern. Science results in understanding when the models we develop correspond to the mechanisms involved in the real-world phenomena. But it is not the model that is the understanding, it is our ability to engage with features of the model intelligently, in service to predictive and instrumental goals. If all we can do is run some mechanized version of the model and read its output, we don't understand anything about the model or what it tells us about the world.
This is obviously a verbal dispute and so nothing of substance turns on its resolution. But when you say that we definitely do understand them, you are just miscommunicating with your interlocutor. I think its clear that most people associate understanding with the ability to predict. You're free to disagree with this association, but you should be more concerned with accurate communication rather than asserting your idiosyncratic usage. Convincing people that we understand how LLMs work (and thus can predict their behavior) has the potential to cause real damage. Perhaps that is an overriding concern of yours rather than debating the meaning of a word or grinding your axe against the ML field?
This is the state we find ourselves when it comes to LLMs. Their space of operations is too large to comprehend as a direct sequence of transformations. Any hope of understanding will come from the identification of relevant features of the system and how those features impact the behavior of the process. Understanding implies prediction because of the connection between causally/semantically relevant features and their influence on behavior. When we say we understand how something works, we are saying we comprehend the relevant features of the system and how they influence relevant behavior. Prediction is a consequence of this kind of understanding.
The danger in LLMs is that their sequence of operations is so large and opaque that we do not know at what level features of the trained network or the input will impact the output and what if any bounds there is on such impact. There is plausibly some semantically empty (to us) feature of the input that can have an outsized impact on its output (relative to its semantic relevance, e.g. adversarial perturbations). As long as we cannot conclusively rule out such features or put bounds on their impact on the expected behavior of the system, we cannot say we understand how they work. The potential existence of ghost features that can cause relevant deviations in expected output just points to structure within the network that we are blind to. Unknown relevant structure just means gaps in understanding.
As far as your RNG example goes, we say we understand the RNG because we understand how it captures entropy from various sources and how it manages its state buffers. We understand how it maps inputs to outputs. We can't predict its output without first knowing its input, but that's neither here nor there.