I mean, your argument is still basically that it’s thinking inside there; everything I’ve said is germane to that point, including what GPT4 itself has said.

My argument?

That doesn’t mean they’re having thoughts in there I mean. Not in the way we do, and not with any agency, but I hadn’t argued either way on thoughts because I don’t know the answer to that.

Are you assuming I’m saying that LLMs are sentient, conscious, have thoughts or similar? I’m not. Jury’s out on the thought thing, but I certainly don’t believe the other two things.

I’m not saying it’s thinking or has thoughts. I’m saying I don’t know the answer to that, but if it is it definitely isn’t anything like human thoughts.

Simply because its interior is a black box doesn’t mean we don’t understand how we built that black box, or how it operates and functions.

Wait a sec. I think we’re saying the same thing here. I guess depending on what you mean by how it operates and functions. I’ve said multiple times we understand the math and the code. We understand how values propagate through it because again, that’s all the math and code people wrote. What we don’t understand is how it uses that math and code to actually do thinks that seem intelligent (putting aside the point of whether it is or is not intelligent). If that’s what you’re arguing then great, we’re on the same page!

I also can’t inspect the electrons moving through my computer’s CPU. Does that mean we don’t understand how computers work? Is there intelligence in there?

Well, I don’t have the equipment to look at electrons either (I don’t think that tech exists), but I can take a logic probe and get some information that I could probably understand, or someone who designs CPUs could look at the gates and whatever and tell you what they did and how they relate to whatever higher level operations. You’re bringing up something completely different here. Computers are not a black box at all. LLMs are-- you just said that yourself.

No, that is not my main objection. It is your anthropomorphization of data and LLMs

I’m not anthropomorphisizing them. What are you talking about? I keep saying they don’t work like human brains. I just said I don’t think they’re sentient or conscious. I said they don’t have agency.

I think you’re getting caught up in trying to define what intelligence is; but I am simply stating what it is not.

How do you know what it’s not if we can’t define what it is?

It is not a complex statistical model with no self-awareness, no semantic understanding, no ability to learn, no emotional or ethical dimensionality, no qualia…

Jury’s still out on whether human brains are complex statistical models. I mean (from here)…

Our brains have learned, through evolution and experience, the statistical properties of our natural environments and exploit this knowledge when performing perceptual tasks.

I don’t make any claim to understanding neuroscience, and I don’t think that article is saying for sure we know that.

Anyway, in-context learning is a thing for LLMs. Maybe one day we’ll figure out how to have them adjust their weights after training, but that’s not happening now (well people are experimenting with it).

New research is showing they do have semantic understanding.

They don’t by themselves have self-awareness, but a software framework built up around them can generally do that to some extent.

They do understand emotions and ethics. Someone built a fun GPTrolley web site a while ago. I think it died pretty quickly because it was too expensive for them, but it had GPT 3(?) answering Trolley Problem questions. It did (in my memory of it) like to save any “AGI” on one track over humans, which was amusing. They don’t have emotions, no. Does something have to have emotions to be intelligent?

And no, I’ve said all along they aren’t conscious, so no qualia. Again, is that required for intelligence?

This is the crux of the problem: it is not a “square” to a computer because a “square” is a human classification. Your thoughts about squares are not just more robust than GPT’s, they are a different kind of thing altogether. For GPT, a square is a token that it has been trained to use in a context-appropriate manner with no idea of what it represents. It lacks semantic understanding of squares. As do all computers.

No. A square to GPTs is not just a token. It’s associated with some meaning. I’m not going to re-hash embedding and word vectors and whatever since I feel like I’ve explained that to death.

If you’re saying that intelligence and understanding is limited to the human mind, then please point to some non-religious literature that backs up your assertion.

I’m disappointed that you’re asking me to prove a negative.

I’m literally not. “Intelligence is limited to the human mind” is not a negative.

The burden of proof is on you to show that GPT4 is actually intelligent. I don’t believe intelligence and understanding are for humans only; animals clearly show it too. But GPT4 does not.

I feel like I’ve laid out my argument for that mostly through the Microsoft and Max Tegmark papers. Are you saying intelligence is only the domain of biological life?

Here’s a question-- are you conflating “intelligence” with “general intelligence” like AGI? I find a lot of people think “AI” means “AGI.” It doesn’t help that some people do say those things interchangeably. I was just reading a recent argument between Yann LeCun and Yoshua Bengio and they were both totally doing that. Anyway, I don’t at all believe GPT4 is AGI or that LLMs could even be AGI.

For an overview of how many different kinds of LLMs function, here’s a good paper: https://arxiv.org/pdf/2307.06435.pdf

Looks like a great paper-- I hadn’t seen it yet. I know how LLMs are constructed (generally-- while I could go and write some code for a multi-layer neural network with back propagation without looking anything up, I couldn’t do that for an LLM without looking at a diagram of the layers or whatnot).

I’m aware of that date.

The OpenAI GPT-4 video literally states that GPT-4 finished training in August 2022.

Either way, to clarify / reiterate, you’re refuting a different point than I’ve made. I said:

Its understanding of AI is from before it was trained, so it is wildly out of date at this point given how much has happened in the space since.

I’m not talking about whether it knows about its own training (I doubt that it does). I’m talking about it knowing about what’s happened in the broader AI landscape since.

Care to provide some proof of that? They did update their system prompt to include a few things like it is now GPT4 (it used to always say GPT3). Other than that, I don’t think it knows anything. But in general, I was more talking about developments in AI since it was trained which it certainly does not know.

Edit: hmm I just reviewed our discussion and I note you only provided one link which was to the psychological definition of intelligence. You otherwise are providing no sources to back up your claims while my responses are full of them. Please start backing up your assertions, or provide some evidence you are an expert in the field.

I’d just like to step in here and mention that asking an LLM is probably not a good proof (and this is directed at both of you). Its understanding of AI is from before it was trained, so it is wildly out of date at this point given how much has happened in the space since.

But we know exactly what they’re doing conceptually, and individually, and in aggregate.

Can you define and give examples of what you mean at each level here? Maybe we’re just not understanding each other and mean the same thing.

Read your own sources from your previous post, that’s what they’re telling you.

The Anthropic one is saying they think they have a way to figure it out, but it hasn’t been tested on large models. This is their last paragraph:

Our next challenge is to scale this approach up from the small model we demonstrate success on to frontier models which are many times larger and substantially more complicated. For the first time, we feel that the next primary obstacle to interpreting large language models is engineering rather than science.

They are literally only able to do this on a small one layer transformer model. GPT 3 has 96 layers and 175 billion parameters.

Also, in their linked paper:

A key challenge to our agenda of reverse engineering neural networks is the curse of dimensionality: as we study ever-larger models, the volume of the latent space representing the model’s internal state that we need to interpret grows exponentially. We do not currently see a way to understand, search or enumerate such a space unless it can be decomposed into independent components, each of which we can understand on its own.

Under the Future Work heading:

Scaling the application of sparse autoencoders to frontier models strikes us as one of the most important questions going forward. We’re quite hopeful that these or similar methods will work – Cunningham et al.'s work [17] seems to suggest this approach can work on somewhat larger models, and we have preliminary results that point in the same direction. However, there are significant computational challenges to be overcome.

How are you getting from that that this is a solved problem?

Concepts are indeed abstract but LLMs have no concepts in them, simply vectors. The vectors do not represent concepts in anything close to the same way that your thoughts do. They are not 1:1 with objects, they are not a “thought,” and anyway there is nothing to “think” them. They are literally only word weights, transformed to text at the end of the generation process.

Again, you aren’t making sense here. Word/sentence vectors are literally a way to represent the concept of those words/sentences. That’s what they were built for. That’s how they are described. Let’s take a step back to try to understand each other.

Are you trying to say that only human minds can understand concepts? I don’t buy the human brains are magic bit, and neither does our current understanding of physics. Are you assuming I’m saying that LLMs are sentient, conscious, have thoughts or similar? I’m not. Jury’s out on the thought thing, but I certainly don’t believe the other two things. There’s no magic with them, same with human brains. We just don’t fully understand what happens inside either. Anthropic in the work I quoted is making good progress at that, and I think they may be pretty close, but in terms of LLMs (and not Small LMs), they are still a black box. We know the math behind them, the software, etc. We have some theories. We still do not understand. If you can prove otherwise, please provide me with a source. Stuff is happening really fast in AI, and maybe I blinked and missed something.

I think you’re maybe having a hard time with using numbers to represent concepts. While a lot less abstract, we do this all the time in geometry. ((0, 0), (10, 0), (10, 10), (0, 10), (0, 0)) What’s that? It’s a square. Word vectors work differently but have the same outcome (albeit in a more abstract way).

the vectors do not represent single words, but tokens

I was talking word vectors where the vectors DO represent words. It’s in the name. LLMs don’t specifically use word vectors, but the embeddings they do use work similarly.

Tokens do not represent the meaning of a word/partial word/phrase, just the statistical use of that word given the data the word was found in.

You are correct tokens don’t represent the meaning of a word. However, tokens are scalars. You are conflating tokens and embeddings / word vectors here. Tokens are used to simplify converting a string into a format a neural network can understand (a vector). If we used each ascii character in the input/output string as a vector input to the network, we’d have to have a lot more parameters than if we combine the characters in some way (i.e. tokens). As you said, they can be a word or a part of a word. There’s no statistics embedded with the tokens (there are some methods of using statistics to choose what tokens to use, but that’s decided before even training the model and can not ever change [with our current approach]). You can read here for more information on tokens. Or you can play around with the gpt3 tokenizer.

Your concept of a chair is an abstract thought representation of a chair. An LLM has vectors that combine or decompose in some way to turn into the word “chair,” but are not a concept of a chair or an abstract representation of a chair. It is simply vectors and weights, unrelated to anything that actually exists.

If you know Python, you should grab nltk and experiment with gensim, their word vectors.

model.most_similar(positive=[‘woman’,‘king’], negative=[‘man’], topn = 1) [(‘queen’, 0.71181…)]

king + woman - man = queen

Seems like an abstract representation of those things as concepts using math. For the record, word vectors are actually pretty understandable/understood by people because you can visualize them easily. When you do, you find similar concepts clustered together (this is how vector search works except with text embeddings). Anyway, it just really seems like linking numbers to concepts is not clicking with you, or you somehow think it’s not possible. Reading up on computational linguistics might help.

That is obviously totally different in kind to human thought and abstract concepts. It is just not that, and not even remotely similar.

Yes, neural networks (although initially built thinking they were a computer version of a neuron), are a lot different from how actual brains work as we’ve learned in however many decades since they were invented. If you’re saying that intelligence and understanding is limited to the human mind, then please point to some non-religious literature that backs up your assertion.

You say you are familiar with neural networks and AI but these are really basic underpinnings of those concepts that you are misunderstanding. Maybe you need to do more research here before asserting your experience?

I’m pretty confident in my understanding, though I’m always open to new ideas that are backed with peer reviewed research. I’m not going to get into a dick waving contest here, so I guess we’ll have to agree to disagree.

As a side note, going back to your definition of intelligence. That was for psychology. I’ll note that the Wikipedia page for Intelligence has this to say:

The definition of intelligence is controversial, varying in what its abilities are and whether or not it is quantifiable.

And so I’ll reiterate that we don’t have a good definition of intelligence.

They are saying the internal vector space that LLMs use is too complicated and too unrelated to the output to be understandable to humans.

Yes, that’s exactly what I’m saying.

That doesn’t mean they’re having thoughts in there

I mean. Not in the way we do, and not with any agency, but I hadn’t argued either way on thoughts because I don’t know the answer to that.

we know exactly what they’re doing inside that vector space – performing very difficult math that seems totally meaningless to us.

Huh? We know what they are doing but we don’t? Yes, we know the math, people wrote it. I coded my first neural network 35 years ago. I understand the math. We don’t understand how the math is able to do what LLMs do. If that’s what you’re saying then we agree on this.

The vectors do not represent concepts. The vectors are math. When the vectors are sent through language decomposition they become words, but they were never concepts at any point.

“The neurons are cells. When neurotransmitters are sent through the synapses, they become words, but they were never concepts at any point.”

What do you mean by “they were never concepts”? Concepts of things are abstract. Nothing physical can “be” an abstract concept. If you think about a chair, there isn’t suddenly a physical chair in your head. There’s some sort of abstract representation. That’s what word vectors are. Different from how it works in a human brain, but performing a similar function.

A word vector is an attempt to mathematically represent the meaning of a word.

From this page. Or better still, this article explaining how they are used to represent concepts. Like this is the whole reason vector embeddings were invented.

I’m not really interested in papers that either don’t understand LLMs or play word games with intelligence

I mean, my first paper was from Max Tegmark. My second paper was from Microsoft. You are discounting a well known expert in the field and one of the leading companies working on AI as not understanding LLMs.

Human intelligence is a mental quality that consists of the abilities to learn from experience, adapt to new situations, understand and handle abstract concepts, and use knowledge to manipulate one’s environment.

I note that’s the definition for “human intelligence.” But either way, sure, LLMs alone can’t learn from experience (after training and between multiple separate contexts), and they can’t manipulate their environment. BabyAGI, AgentGPT, and similar things can certainly manipulate their environment using LLMs and learn from experience. LLMs by themselves can totally adapt to new situations. The paper from Microsoft discusses that. However, for sure, they don’t learn the way people do, and we aren’t currently able to modify their weights after they’ve been trained (well without a lot of hardware). They can certainly do in-context learning.

Yes. LLMs are not magic, they are math, and we understand how they work. Deep under the hood, they are manipulating mathematical vectors that in no way are connected representationally to words. In the end, the result of that math is reapplied to a linguistic model and the result is speech. It is an algorithm, not an intelligence.

We understand how they work? From the Wikipedia page on LLMs:

Large language models by themselves are “black boxes”, and it is not clear how they can perform linguistic tasks. There are several methods for understanding how LLM work.

It goes on to mention a couple things people are trying to do, but only with small LLMs so far.

Here’s a quote from Anthropic, another leader in AI:

We understand the math of the trained network exactly – each neuron in a neural network performs simple arithmetic – but we don’t understand why those mathematical operations result in the behaviors we see.

They’re working on trying to understand LLMs, but aren’t there yet. So, if you understand how they do what they do, then please let us know! It’d be really helpful to make sure we can better align them.

they are manipulating mathematical vectors that in no way are connected representationally to words

Is this not what word/sentence vectors are? Mathematical vectors that represent concepts that can then be linked to words/sentences?

Anyway, I think time will tell here. Let’s see where we are in a couple years. :)

I’m not really interested in papers that either don’t understand LLMs or play word games with intelligence

In the end of the bit I quoted you say: “basically no world at all.” But also, can you define what intelligence is? Are you sure it isn’t whatever LLMs are doing under the hood, deep in hidden layers? I guess having a world model is more akin to understanding than intelligence, but I don’t think we have a great definition of either.

GPT-4 cannot alter its weights once it has been trained so this is just factually wrong.

The bit you quoted is referring to training.

They are not intelligent. They create text based on inputs. That is not what intelligence is, unless you have an extremely dismal view of intelligence that humans are text creation machines with no thoughts, no feelings, no desires, no ability to plan… basically, no internal world at all.

Recent papers say otherwise.

The conclusion the author of that article comes to (LLMs can understand animal language) is… problematic at the very least. I don’t know how they expect that to happen.

The armor is way too symmetrical. Can’t have the same thing on both arms according to Bungie! Hah.

Check out this recent paper that finds some evidence that LLMs aren’t just stochastic parrots. They actually develop internal models of things.

Wikipedia: In copyright law, a derivative work is an expressive creation that includes major copyrightable elements of a first, previously created original work.

I think you may be off a bit on what a derivative work is. I don’t see LLMs spouting out major copyrightable elements of books. They can give a summary sure, but Cliff Notes would like to have a word if you think that’s copyright infringement.

From what I’ve seen, here’s what happened. GPT 4 came out, and it can pass the bar exam and medical boards. Then more recently some studies came out. Some of them from before GPT 4 was released that just finally got out or picked up by the press, others that were poorly done or used GPT 3 (probably because of gpt 4 being expensive) and the press doesn’t pick up on the difference. Gpt 4 is really good and has lots of uses. Gpt 3 has many uses as well but is definitely way more prone to hallucinating.

What’s with all the hit jobs on ChatGPT?

Prompts were input to the GPT-3.5-turbo-0301 model via the ChatGPT (OpenAI) interface.

This is the second paper I’ve seen recently to complain ChatGPT is crap and be using GPT3.5. There is a world of difference between 3.5 and 4. Unfortunately news sites aren’t savvy enough to pick up on that and just run with “ChatGPT sucks!” Also it’s not even ChatGPT if they’re using that model. The paper is wrong (or it’s old) because there’s no way to use that model in the ChatGPT interface. I don’t think there ever was either. It was probably ChatGPT 0301 or something which is (afaik) slightly different.

Anyway, tldr, paper is similar to “I tried running Diablo 4 on my Windows 95 computer and it didn’t work. Surprised Pikachu!”

Yeah, I generally agree there. And you’re right. Nobody knows if they’ll really be the starting point for AGI because nobody knows how to make AGI.

In terms of usefulness, I do use it for knowledge retrieval and have a very good success rate with that. Yes, I have to double check certain things to make sure it didn’t make them up, but on the whole, GPT4 is right a large percentage of the times. Just yesterday I’d been Googling to find a specific law or regulation on whether airlines were required to refund passengers. I spent half an hour with no luck. ChatGPT with GPT4 pointed me to the exact document down to the right subsection on the first try. If you try that with GPT3.5 or really anything else out there, there’s a much higher rate of failure, and I suspect a lot of people who use the “it gets stuff wrong” argument probably haven’t spent much time with GPT4. Not saying it’s perfect-- it still confidently says incorrect things and will even double down if you press it, but 4 is really impressive.

Edit: Also agree, anyone saying LLMs are AGI or sentient or whatever doesn’t understand how they work.

As I see it, anybody who is not skeptical towards “yet another ‘world changing’ claim from the usual types” is either dumb as a doorknob, young and naive or a greedy fucker invested in it trying to make money out of any “suckers” that jump into that hype train.

I’ve been working on AI projects on and off for about 30 years now. Honestly, for most of that time I didn’t think neural nets were the way to go, so when LLMs and transformers got popular, I was super skeptical. After learning the architecture and using them myself, I’m convinced they’re part of but not the whole solution to AGI. As they are now, yes, they are world changing. They’re capable of improving productivity in a wide range of industries. That seems pretty world changing to me. There are already products out there proving this (GitHub Copilot, jasper, even ChatGPT). You’re welcome to downplay it and be skeptical, but I’d highly recommend giving it an honest try. If you’re right then you’ll have more to back up your opinion, and if you’re wrong, you’ll have learned to use the tech and won’t be left behind.

extraordinary claims without extraordinary proof

What are you looking for here? Do you want it to be self aware and anything less than that is hot garbage? That latest advances in AI have many uses. Sure Bitcoin was over hyped and so is AI, but Bitcoin was always a solution with no problem. AI (as in AGI) offers literally a solution to all problems (or maybe the end of humans but hopefully not hah). The current tech though is widely useful. With GPT4 and GitHub Copilot, I can write good working code at multiple times my normal speed. It’s not going to replace me as an engineer yet, but it can enhance my productivity by a huge amount. I’ve heard similar from many others in different jobs.

Yeah I think you’re right on about the students not being able to afford GPT4 (I don’t blame them. The API version gets expensive quick). I agree though that it doesn’t seem super well put together.

Oh ok! Got it. I read it as you saying ChatGPT doesn’t use GPT 4. It’s still unclear what they used for part of it because of the bit before the part you quoted:

For each of the 517 SO questions, the first two authors manually used the SO question’s title, body, and tags to form one question prompt3 and fed that to the Chat Interface [45] of ChatGPT.

It doesn’t say if it’s 4 or 3.5, but I’m going to assume 3.5. Anyway, in the end they got the same result for GPT 3.5 that it gets on HumanEval, which isn’t anything interesting. Also, GPT 4 is much better, so I’m not really sure what the point is. Their stuff on the analysis of the language used in the questions was pretty interesting though.

Also, thanks for finding their mention of 3.5. I missed that in my skim through obviously.