Can ChatGPT create new knowledge?
Some observations on what kind of new knowledge a system like ChatGPT can produce
ChatGPT has been described as many things, but I haven’t yet seen it described as I conceive of it: a system that interpolates1 human knowledge. It answers questions with the ‘average’ of what was in its training materials; an amalgamation of the knowledge that ‘surrounds’ the question. Perhaps this is obvious to anyone that already understands it, but conceptualizing it like this clarified things for me.
Can it do anything new?
That it only interpolates existing human knowledge seems to imply that it cannot tell you anything new. However, it can, for at least two reasons:
Things may be new to you, and even to most users. There are things only known to few people, that it can include in answers to the right kind of question.
An analogy: if many people write about walking a path around a forest and only seeing forest all the time, it seems to all of them like there is only forest in the circle they circumscribe. However, if one person went through the forest and wrote about noticing a clearing with a house and you ask ChatGPT “what is inside that forest?”, it could include that information, and it thus inform everyone that bothers to ask that there is a clearing with a house in the forest.
If only one person write about what is inside the forest, ChatGPT might just include such information. More in general, if e.g. all others wrote about it seeming like there was only forest there, an LLM might need to prefer rare statements of fact above repeatedly stated conjectures and thus pick out the fact that it is actually known there is a house there, above just restating the hypothesis over and over again.It may combine knowledge from disparate sources leading to true statements no human ever had or would have guessed. Technically: the interpolated value may be outside of the convex set of all human knowledge.
An analogy: perhaps no one ever investigated what is below the forest. The people in the vicinity assume it’s just earth and rock. Some geologists surveyed the area from the sky and their measurements imply there is a cave complex below the forest, but that wasn’t why they were surveying. There is a website automatically generated from the data that states this fact in plain text, but no human ever noticed. ChatGPT could inform those who ask what is below the forest that there is a cave complex.
Can it do anything new?
Those two points still suggest an important limitation: anything it generates is still implicit in, and derived from, existing human knowledge captured in writing. So although it could generate new knowledge, it is hard to see how any of that knowledge would be so significant, deviate so much from existing knowledge, that it would be considered unprecedented, prize worthy or deserving of similar superlatives. Technically: it feels like the set of human knowledge is not very concave and using LM’s to turn the epigraph of human knowledge into a convex set will not add knowledge that makes a significant difference.
Of course this intuition may be entirely wrong and there are mindblowing true statements out of reach of humans, due to limited capacity to ingest, memorize and combine even already known data, that LM’s do have access to, perhaps after increasing the training data set to include more of the data humanity has gathered over the years.
For those unfamiliar with this term, interpolation is basically guessing what something is based on what you know of the things that surround it.