Bill Gates feels Generative AI has plateaued, says GPT-5 will not be any better::The billionaire philanthropist in an interview with German newspaper Handelsblatt, shared his thoughts on Artificial general intelligence, climate change, and the scope of AI in the future.
You got it the wrong way around. We already have a ton of compute and what this kind of AI can do is pretty cool.
But adding more compute power and parameters won’t solve the inherent problems.
No matter what you do, it’s still just a text generator guessing the next best word. It doesn’t do real math or logic, it gets basic things wrong and hallucinates new fake facts.
Sure, it will get slightly better still, but not much. You can throw a million times the power at it and it will still fuck up in just the same ways.
I mean, that’s more-or-less what I said. We don’t know the theoretical limits of how good that text generation is when throwing more compute at it and adding parameters for the context window. Can it generate a whole book that is fairly convincing, write legal briefs off of the sum of human legal knowledge, etc.? Ultimately, the algorithm is the same, so like you said, the same problems persist, and the definition of “better” is wishy-washy.
It will obviously get even better, but you’ll never be able to rely on it. Sure, 99.9% of that generated legal document will look perfect, till you overlook one sentence where the AI hallucinated. There is no fact checking in there, that’s the issue.
This is short-sighted.
The jump to GPT 3.5 was preceded by the same general misunderstanding (we’ve reached the limit of what generative pre-trained transformers can do, we’ve reached diminishing returns, ECT.) and then a relatively small change (AFAIK it was a couple additional layers of transforms and a refinement of the training protocol) and suddenly it was displaying behaviors none of the experts expected.
Small changes will compound when factored over billions of nodes, that’s just how it goes. It’s just that nobody knows which changes will have that scale of impact, and what emergent qualities happen as a result.
It’s ok to say “we don’t know why this works” and also “there’s no reason to expect anything more from this methodology”. But I wouldn’t dismiss further improvements as a forgone possibility.
Another way to think of this is feedback from humans will refine results. If enough people tell it that Toronto is not the capital of Canada it will start biasing toward Ottawa, for example. I have a feeling this is behind the search engine roll out.
ChatGPT doesn’t learn like that though, does it? I thought it was “static” with its training data.
I was speculating about how you can overcome hallucinations, etc., by supplying additional training data. Not specific to ChatGPT or even LLMs…
Toronto is Canadian New York. It wants to be the capital and probably should be but it doesn’t speak enough French.