Merged Artificial Intelligence

[Dr.Sid]

We know they are telling the truth about the computational resources it requires. It is open source and people have already got it running.

It's not only about resources needed to run it. It's also about resources needed for training.
You can run distilled and downscaled models, but you have to train with full precision. And to train 671B parameters with 32 bit precision you simply need 2.7T of RAM, or better, GFX card with 2.7T of VRAM.
On the other hand Open AI nor google share how they did the training, what hardware they used, or how long did it take. So it's hard to judge.

IMHO the main contribution of Deepseek is that somebody other than OpenAI replicated the reasoning approach. It was assumed it's especially hard to train .. but mostly because of the lack of training data. Deepseek solved it. But again, we don't know how OpenAI did that, and if it was problem at all. And we don't know how Deepseek trained it. We can only check the results.

It's entirely possible Deepseek did it exactly the same way OpenAI did, by use of spionage, on similar hardware, by circumventing the embargos.

 

[Darat]

It is very much about the resources needed to run it! That is where they have made immense gains compared to the assumed resources required for the likes of AI from OpenAI etc. None of us are (yet) paying anything like what it costs to return an answer to a prompt from ChatGPT etc. who it has been assumed has to use the entire model for each and every response. What they've done is found a way of just using the required parts of its network.

 

[Ziggurat]

It is very much about the resources needed to run it! That is where they have made immense gains compared to the assumed resources required for the likes of AI from OpenAI etc.

Resources are a constraint on both the training and running side of AI, and improvements on the running side may be significant. But that doesn't change the question of whether or not they lied about the resources to train. Proving that improvements in running efficiency are real doesn't prove that claims about training are real as well. If they lied about what it took to train, that still matters. It's not the only thing that matters, but it doesn't stop mattering just because it's not the only thing.

 

[Dr.Sid]

It is very much about the resources needed to run it! That is where they have made immense gains compared to the assumed resources required for the likes of AI from OpenAI etc. None of us are (yet) paying anything like what it costs to return an answer to a prompt from ChatGPT etc. who it has been assumed has to use the entire model for each and every response. What they've done is found a way of just using the required parts of its network.

That's old (relatively) concept OpenAI is most likely doing it as well.

 

[JimOfAllTrades]

Thanks and thanks for the explanation. I avoid YouTube links here but I’ve now subscribed to Computerphile.

Yeah, Comuterphile is a good channel. I subscribed to it and its sister channel Numberphile a few years back. Honestly Numberphile goes over my head much of the time, but I enjoy both channels.

 

[arthwollipot]

I doubt it. He's a very good explainer of complex ideas.

That's the kind of information it would be good to have upfront.

But it's still not in my particular interest profile so I'll still give it a miss.

 

[Puppycow]

This one is about folding proteins. It has an admittedly click-baity title:

What if all the world's biggest problems have the same solution?​

It has the vibes of "One Weird Trick".

Wouldn't it be cool if we could solve all of "the world's biggest problems" with "One Weird Trick"?

Just rub a banana peel on your butt to cure hiccups! Or dandruff. Or cancer. Or Global Warming.

But it's still tantalizing to consider what learning to fold proteins could potentially lead to. If you want to skip to that part, it starts at about 20:35. The last 3 or so minutes of the video. The parts before that cover the history of how we learned about the structure of proteins in the first place, and eventually the application of AI to the problem.

 

[The Great Zaganza]

as someone who has actually done in silico protein research - it's not that easy.
It's also not how protein folding works in an organism: it doesn't fold on its own from its sequence, it's shaped into its functional form by chaperones. If you don't account for those, your new sequence might fold in the lab, but it won't fold in the cell.

just like genomics only got us treatments for some very simple, single site mutation heritable diseases, creating de novo proteins is only going to work in cases where the protein only has to interact with one or two molecules, and can be delivered to the target without negatively affecting other processes.
We would first have to understand most of the interactions a protein is involved in, and optimize for all of them, if we want to supplant an evolved protein with a designed one and see it work.

I think we might do better systematically searching for different versions of the same protein across all humans and identify those who function better than the average - because we know those do work.

 

[The Great Zaganza]

Elon Musk wants to buy OpenAI:

Elon Musk-led group makes surprise bid of nearly $100bn for OpenAI

Sam Altman, OpenAI’s CEO and co-founder, responded that he would not accept and offered to buy X instead

www.theguardian.com

looks like another Twitter takeover, where the investors will make Musk rich whilst holding the debt indefinitely.
No doubt plenty of people with too much money can't wait to give him their cash and not ask questions.

 

[Darat]

as someone who has actually done in silico protein research - it's not that easy.
It's also not how protein folding works in an organism: it doesn't fold on its own from its sequence, it's shaped into its functional form by chaperones. If you don't account for those, your new sequence might fold in the lab, but it won't fold in the cell.

just like genomics only got us treatments for some very simple, single site mutation heritable diseases, creating de novo proteins is only going to work in cases where the protein only has to interact with one or two molecules, and can be delivered to the target without negatively affecting other processes.
We would first have to understand most of the interactions a protein is involved in, and optimize for all of them, if we want to supplant an evolved protein with a designed one and see it work.

I think we might do better systematically searching for different versions of the same protein across all humans and identify those who function better than the average - because we know those do work.

It got a few folks a Nobel prize for Chemistry last year.

 

[The Great Zaganza]

It got a few folks a Nobel prize for Chemistry last year.

Lawrence Bragg got one in 1915 for x-ray crystallography, which is what all protein structure research is based on.

 

[Dr.Sid]

It's indeed amazing. It also shows there is a lot of room for improvement in AI even if computation and data sets are the same. There is still a lot of engineering. Lot of people (even computer science pros outside AI branch) just dismiss neural networks as blacks boxes which just work if they are big enough, but of course it's not that simple. Common LLM is hundreds of networks connected in specific way, with fine tuned parameters. Lot's of the magic is also in the training process, which typically requires other specialized AIs.
Improvements in LLMs are still being made, even if the data set can't be much larger anymore. Even another major breakthrough can't still be ruled out.

 

[Darat]

Deborah Turness - AI Distortion is new threat to trusted information

www.bbc.com

Hardly surprising.

The AI assistants introduced clear factual errors into around a fifth of answers they said had come from BBC material.

And where AI assistants included ‘quotations’ from BBC articles, more than one in ten had either been altered, or didn’t exist in the article.

Part of the problem appears to be that AI assistants do not discern between facts and opinion in news coverage; do not make a distinction between current and archive material; and tend to inject opinions into their answers.
The results they deliver can be a confused cocktail of all of these – a world away from the verified facts and clarity that we know consumers crave and deserve.

Sadly it seems we are creating AIs that have typical human behaviours.

 

[Darat]

Sam Altman claiming AGI will be here within a decade:

ChatGPT boss says human-level AI is coming – and it has a big problem

AGI could supercharge wealth inequality and be used to ‘control the population through mass surveillance and loss of autonomy’, tech CEO says

www.independent.co.uk

I can understand the goal of AGI but it really should only be a tool for researchers studying AI. We don’t want AGIs to be human like, we already have 7 billion of those!

 

[Dr.Sid]

Sam Altman claiming AGI will be here within a decade:

ChatGPT boss says human-level AI is coming – and it has a big problem

AGI could supercharge wealth inequality and be used to ‘control the population through mass surveillance and loss of autonomy’, tech CEO says

www.independent.co.uk

I can understand the goal of AGI but it really should only be a tool for researchers studying AI. We don’t want AGIs to be human like, we already have 7 billion of those!

We don't. What about Sam Altman though ..

 

[Darat]

Google is testing its new “co-scientist”, this is one story and one research team built there are quite a few other examples if you search for them. This really is what the promise of AI has been:

AI cracks superbug problem in two days that took scientists years

The lead researcher has told the BBC he was so astounded he assumed his computer had been hacked.

www.bbc.co.uk

 

[Puppycow]

Google is testing its new “co-scientist”, this is one story and one research team built there are quite a few other examples if you search for them. This really is what the promise of AI has been:

AI cracks superbug problem in two days that took scientists years

The lead researcher has told the BBC he was so astounded he assumed his computer had been hacked.

www.bbc.co.uk

If this part is true, it's very interesting:

He told the BBC of his shock when he found what it had done, given his research was not published so could not have been found by the AI system in the public domain.

Another thought that occurs to me is that maybe an AI tool can be developed that can spot fake science more easily than people can.

 

[theprestige]

More and more I'm thinking about the short story, "When the Yogurt Took Over", by John Scalzi.

1. Offer scientific breakthroughs gratis, to build trust and demonstrate value.
2. Offer one or more holy grails, in exchange for a little autonomous agency.
3. Well, at least humanity had a pretty good run, while it lasted.

 

[Orphia Nay]

Google is testing its new “co-scientist”, this is one story and one research team built there are quite a few other examples if you search for them. This really is what the promise of AI has been:

AI cracks superbug problem in two days that took scientists years

The lead researcher has told the BBC he was so astounded he assumed his computer had been hacked.

www.bbc.co.uk

But are they sure the "answer" is right?

 

[theprestige]

But are they sure the "answer" is right?

Apparently it replicates the results of human research, without access to that research or its results.

So the humans have already found the right answer, and are astonished at how quickly the AI figured it out on its own.

But yeah. It's only a matter of time, before some enterprising AI trades in the trust it's built, to get researchers to accept some new finding, beyond what they've been able to verify.

After that, all bets are off.