ChatGPT

[theprestige]

It's kind of impressive, the number of people who are looking at these autocomplete algorithms, and thinking, this is clearly a sapience that should have voting rights and child custody rights and be eligible for jury duty because who can define human cognition or say that these things don't have it.

 

[Mike Helland]

It's kind of impressive, the number of people who are looking at these autocomplete algorithms, and thinking, this is clearly a sapience that should have voting rights and child custody rights and be eligible for jury duty because who can define human cognition or say that these things don't have it.

That number being... zero?

No one has said that.

 

[p0lka]

If a chatbot ever gets bored of waiting for me to ask a question, and tells me to hurry up as it's got other stuff to do, then I might attribute an internal state to it. Not even remotely there at the moment.

 

[Dr.Sid]

I'm sorry, did I wake up in an early 20th century science fiction story?

AI developers know exactly how their technology does what it does. The alternative would be magic.

Not really. We created self learning algorithms. They can they learn. We know they do it by optimizing a function with billions of parameters. Trillions lately. We understand THAT.
It's those parameters though which define the response. And we can't understand them, simply because there's too many of them. They can't be easily mapped and debugged. They can be, to some extent, but it's really scratching the surface.
Generally we really don't know how they work. It is magic. AIs often behave in way we didn't expect. Usually they simply don't do what we want, but sometimes they also do something better than we expected.
And we are at stage, where making bigger AI is simpler, than analyzing the small ones, so the amount of things and phenomenons we don't understand is actually getting bigger.

 

[Myriad]

I'm sorry, did I wake up in an early 20th century science fiction story?

AI developers know exactly how their technology does what it does. The alternative would be magic.

Nope. A century ago, the idea that creating a thing necessarily meant you know exactly how the thing behaves and responds in any situation was still fairly prevalent. Therefore the only exception, the only kind of scientist who could create something he couldn't control, would have to be a "mad scientist."

That's never really been true, but by now it's obvious it's not true. Have you seen the proof that Wolfram's rule 110 is Turing equivalent? The rules of rule 110 can be stated in a few words, but to understand what a rule 110 cellular automaton will do in any possible starting configuration would require understanding every computation that could possibly be done (including for instance ChatGPT's algorithm).

Nowadays the use of evolutionary methods to develop neural nets means AI developers can produce systems that perform well at some task (and out-perform vast numbers of alternative competing systems) without them having (or needing to have) the slightest idea how it works. They know it's made out of numerous sub-units that are individually well understood, but in the trained system those sub-units are all different and interact in complex ways, much like the neurons in a human brain (not coincidentally). Anyone who wants to understand how it does what it does would then have to approach it like a biologist studying an already-evolved complex organism.

 

[W.D.Clinger]

Cats have more variety of vocalization, and more variety of facial expression than dogs.

Oh yes, cats use facial expressions to communicate such diverse messages as

• I don't care.
• Insouciance c'est moi.
• Why do you think you're so important that I should pay attention to you?
• Why do you think you're so important that you can ignore me?
• You have one task, and one task only, so why aren't you feeding me?
• You have one task, and one task only, so why haven't you cleaned my litter box?

Once upon a time, I knew a cat that, upon hearing me play a nylon-string guitar, would enter the room to sit quietly on the rug and listen. On one and only one occasion, I switched to a steel-string acoustic. At the first strum of that steel-string, the cat rose to its feet, looked at me accusingly, and marched out of the room. So some cats have a facial expression that says "How dare you assault my sensitive ears!"

AI developers know exactly how their technology does what it does. The alternative would be magic.

No, the alternative would be alchemy.

In 2017, the Neural Information Processing System (NIPS) conference gave its Test of Time award to a paper written by Ben Recht and Ali Rahimi. In his speech accepting the award, Rahimi said

We say things like "machine learning is the new electricity".

I'd like to offer another analogy.

Machine learning has become alchemy.​

Rahimi quoted an email:

On Friday, someone on another team changed the default rounding
mode of some Tensorflow internals (from "truncate toward 0" to
"round to even").

Our training broke. Our error rate went from <25% error to ~99.97%
error (on a standard 0-1 binary loss).

Rahimi continued:

This is happening because we apply brittle optimization techniques to lost surfaces we don't understand....

Now I'm okay using technologies I don't understand. I got here on an airplane and I don't fully understand how airplanes work....​

Rahimi concluded:

And we would love it if we could work together to take machine learning from alchemy and into electricity.

[Applause]​

ETA: In a technical paper titled The Mathematics of Artificial Intelligence, Gitta Kutyniok wrote:

From a mathematical viewpoint, it is crystal clear that a fundamental mathematical understanding of artificial intelligence is inevitably necessary, and one has to admit that its development is currently in a preliminary state at best.

I have only skimmed that paper, but it looks like a good introduction to current practice.

 

[steenkh]

We can't explain sapience because we haven't found all the puppet strings yet. We know that AI isn't sapient because we know all about its puppet strings, and none of them explain human sapience.

The concept of emergence allows for something unexpected to happen even when “all puppet strings” are known. The researchers working with LLMs don’t understand some of the results they get, even though they know all the puppet strings.

And finally, why should a sapient AI (if one is ever constructed) explain human sapience? The two things could be entirely different. It seems to me that you would only accept sapience in humans, because the puppet strings of everything else would be known by definition.

 

[steenkh]

I don't think they're any more or less robust than the definition of "tall". I also, however, don't think they're useless.

OK, so what are these definitions? So far I have only seen some version of “I know sentience (and sapience) when I see it”.

If LLMs get to a point where they can mimic extrapolative thinking, problem solving, and innovation perfectly... I think they would have to be considered intelligent.



I don't think that extrapolative thinking can really be mimicked.

Is that a definition? Or your version of “I know it when I see it”?

 

[Arcade22]

Reasoning being impaired by lack of language skills could explain some of the things I've personally seen, where a person seemed incapable of planning or making decisions in ways that would give them good outcomes.

Except that scientific evidence clearly demonstrates that language skills are to a large degree independent of executive functions. People can have their ability to comprehend language effectively destroyed because of brain damage without suffering comparable deficits in executive functions and vice versa. That is not to say that they completely separate, but the notion that natural language comprehension is a perquisite for logical reasoning is not true.

Now with that said you could very credibly argue that, fundamentally speaking, "logical reasoning" is merely a form of language comprehension. "Logic" can be and is indeed easily comprehended as a formal language and i wouldn't be surprised if fundamentally in terms of neuronal architecture the circuits for "logical reasoning" are indeed routed in such a way that it conforms to some form of formal system^WP.

It is even possible that some people exist in a p-zombie state, where they can only respond to stimulus, without any concept of self or reasoning at all...

https://en.wikipedia.org/wiki/Philosophical_zombie

No they are not "possible". Merely conjecturing their potential existence does not make them a possibility because "philosophical zombies" are a form of solipsist nonsense.

 

[Fast Eddie B]

That's never really been true, but by now it's obvious it's not true. Have you seen the proof that Wolfram's rule 110 is Turing equivalent? The rules of rule 110 can be stated in a few words, but to understand what a rule 110 cellular automaton will do in any possible starting configuration would require understanding every computation that could possibly be done (including for instance ChatGPT's algorithm).

I don’t get some of the above, but I was thinking of Conway’s “Game of Life” in which a few very basic rules about which “cells” were “dead” and “alive” in each “generation” gave rise to all sorts of unexpected emergent patterns. Just as one can’t foresee how that code might result in “gliders” and “pulsars” and the like, I think AI’s emergent properties are beyond what its originators were capable of predicting.

 

[Wudang]

I'm sorry, did I wake up in an early 20th century science fiction story?

AI developers know exactly how their technology does what it does. The alternative would be magic.

No the alternative would be complexity. Ive worked on large scale software projects and spent an evening after a deep debugging course with a hardware engineer explaining to each other how we each saw the layers. I still expect to be surprised. I don't expect spooky stuff but I do expect emergent properties, unexpected interactions, unintended consequences etc.
Also I'm not saying they can't explain it just that it might be hard to do so from the initial design. Especially as I suspect they may not have included debugging tools in the builds for performance reasons.

 

[theprestige]

We have very little idea how it does what it does. All we know is how it was made, which is not the same thing.

This thread is about ChatGPT, which is an LLM. We have a complete idea of how an LLM does what it does. This is not a case of "spooky action inside the machine".

 

[Darat]

This thread is about ChatGPT, which is an LLM. We have a complete idea of how an LLM does what it does. This is not a case of "spooky action inside the machine".

We've moved from it just being about CHatGPT long ago, we've been discussing all different kinds of AI.

 

[theprestige]

I don’t get some of the above, but I was thinking of Conway’s “Game of Life” in which a few very basic rules about which “cells” were “dead” and “alive” in each “generation” gave rise to all sorts of unexpected emergent patterns. Just as one can’t foresee how that code might result in “gliders” and “pulsars” and the like, I think AI’s emergent properties are beyond what its originators were capable of predicting.

Conway's gliders, pulsars, and other structures are entirely deterministic. Once the necessary starting conditions are discovered, the structures can be generated without exception or unexpected behavior.

LLMs incorporate some probabilistic functions, so the exact same output isn't guaranteed from the exact same input every time. And with the size of the corpus an LLM typically works with, the statistical variance can seem quite large over multiple runs. But this doesn't mean the algorithms aren't perfectly understood. Nor does it mean its creators and researchers are baffled as to the cause of the variance.

 

[steenkh]

This thread is about ChatGPT, which is an LLM. We have a complete idea of how an LLM does what it does. This is not a case of "spooky action inside the machine".

Actually, no. We don’t have a complete idea of how an LLM does what it does. LLMs have given rise to emergent features that baffles the engineers who built them.

This does not mean that LLMs are sentient or sapient, but it means that we can’t explain what is going on, and we certainly can’t rule out these things on the basis of knowing how LLMs work.

 

[Dr.Sid]

This thread is about ChatGPT, which is an LLM. We have a complete idea of how an LLM does what it does. This is not a case of "spooky action inside the machine".

It absolutely is spooky action inside the machine.

 

[Myriad]

I don’t get some of the above, but I was thinking of Conway’s “Game of Life” in which a few very basic rules about which “cells” were “dead” and “alive” in each “generation” gave rise to all sorts of unexpected emergent patterns. Just as one can’t foresee how that code might result in “gliders” and “pulsars” and the like, I think AI’s emergent properties are beyond what its originators were capable of predicting.

That's a very similar example to mine. The rule 110 cellular automaton is like Conway's Life, but it appears to be even simpler: Life is a 2D cellular automaton with two possible states per cell and evolution rules that take into account the state of each cell and its 8 neighboring cells. Rule 110 is a 1D cellular automaton with two possible states per cell and evolution rules that take into account the state of each cell and its 2 neighboring cells.

Both have been proven Turing equivalent, which means that there are some starting configurations (however rare and complex they might be) that form numerous "gliders" and other higher level structures whose interactions emulate a Turing machine and hence can emulate any known deterministic computation.

Here's Life emulating Life, which is simpler than emulating a Turing machine:



Knowing the cellular automaton rules of Life is necessary but far from sufficent to understand how a construction like that one actually works!

 

[Myriad]

One likely implication of all this is that there are emergent properties of human language that we have always made use of but never fully understood. Why does so much of our own self-perceived thinking seem to be in words? Why is printed text still, among all our far more advanced-seeming technologies, the most effective way of modeling and communicating conscious experience?

I posit that "all an LLM does is predict the next word based on the pattern of previous words as learned from a very large sampling of actual human language" and "an LLM exhibits surprising capabilities that often seem to imply internal understanding" are not contradictory claims.

 

[W.D.Clinger]

This thread is about ChatGPT, which is an LLM. We have a complete idea of how an LLM does what it does. This is not a case of "spooky action inside the machine".

But it's alchemy.

LLMs are based on trained neural networks. The following spoiler summarizes our "complete idea of how an LLM does what it does."

The following quotations come from the part of Rahimi's "alchemy" presentation in which he summarizes the algorithmic content of the three papers for which he and his co-author received the Test of Time award. (For more readable typesetting of the equations, see the transcript.)

Here's the algorithm without any of the kernel flim-flam, straight up:

Model

f(x;α) ≡ Σ_j=1^D α_iz(x;ω_j)​

Training

min_α Σ_i=1^N ℓ(f(x_i;α),y_i)​

You just draw a bunch of functions, independently from your data set. You weight them and you tune those weights so that you get a low loss on your training data set okay.

In the second paper we showed this. In the same way that Fourier bases provide a dense basis set for an L² ball of L² functions, or in the same way that three layer wide neural networks could approximate any smooth function arbitrarily well, so too do a bunch of randomly drawn smooth functions approximately densely a function in a Hilbert space arbitrarily well with high probability.

So now you don't need to talk about kernels to justify these random features. They don't have to be eigenfunctions of any famous kernels, they're a legitimate basis for learning unto themselves.

In the third paper we finally derive generalization bounds for the algorithm I just showed you.

Theorem: With high probability,

R[f^{^}] − R[f_*] = O((1/√n + 1/√D) ||f_*||)​

(For a more detailed presentation of the mathematics, see Gitta Kutyniok's invited paper.)

So that's how and why neural networks, including LLMs, work.

As Nahimi goes on to say:

We've made incredible progress....

In many ways we're way better off than we were 10 years ago.

And in some ways we're worse off....

Machine learning has become alchemy.

The point of that analogy (!) is that alchemy actually worked...for a while...in certain limited domains...but modern science became possible only after scientists had "dismantle[d] 2,000 years worth of alchemical theories." Nahimi "would like to live in a society whose systems are built on top of verifiable, rigorous, thorough knowledge and not on alchemy."

He then gives several examples to show that while the abstract mathematical character of the algorithms can be stated (as in the spoiler above), the training part of the algorithm produces zillions of numerical weights whose meaning and derivation are opaque, even to the humans who programmed the algorithm and ran the training part of that algorithm.

That is the sense in which we do not understand how an LLM (or other trained neural networks) do what they do. That is the sense in which trained neural networks are alchemy.

We could live with that opacity (and our consequent inability to understand the operation and limits of LLMs and other trained neural networks) if the networks produced fully reliable results, but they don't. They work well enough to be useful a lot of the time, even though we don't really understand how or why, but they can also hallucinate—as we've seen in this thread.

Nahimi gives specific examples to demonstrate the fragility of trained neural networks.

We've moved from it just being about CHatGPT long ago, we've been discussing all different kinds of AI.

Which is entirely reasonable because many (though not all) different kinds of AI rely on exactly the same trained neural network technology used by ChatGPT.

 

[Mike Helland]

Knowing the cellular automaton rules of Life is necessary but far from sufficent to understand how a construction like that one actually works!

Put another way, if the only way to predict the complexity of simple rules when you run them is to run them, it's not really a prediction, is it.