• Security incident: ISF was recently accessed by intruders. Please change your password, and change it anywhere else you used it. Read more

Merged Artificial Intelligence

It usually is, but not when testing a program DESIGNED to hack. You don't take risks that it might hack the firewall in that case. Just like you don't just bet it won't get into the air when studying a biological virus.

Besides, a virtual machine can trivially be disconnected from the network by just not giving it a virtual network interface. It only has the virtual hardware you gave it, unless it's the most incompetently programmed virtual machine ever. You don't even have to pull a physical cable, just tell it that it has none. It can't just decide that it now has one, any more than it can decide that now it has twice the CPU cores or another few gigabytes of RAM. I can even tell the virtual Linux machine on my work laptop that it has no GPU, or my virtual Android machine on this computer for that matter, and it will just render the text slowly with the CPU. Or even that it has no graphical display, and must use the console.

To use your example, if it were that easy to escape a virtual machine, then the antivirus companies would be the biggest spreaders of viruses and trojans ever.
 
Last edited:
HansMustermann said:
Besides, a virtual machine can trivially be disconnected from the network by just not giving it a virtual network interface. It only has the virtual hardware you gave it, unless it's the most incompetently programmed virtual machine ever. You don't even have to pull a physical cable, just tell it that it has none. It can't just decide that it now has one, any more than it can decide that now it has twice the CPU cores or another few gigabytes of RAM. I can even tell the virtual Linux machine on my work laptop that it has no GPU, or my virtual Android machine on this computer for that matter, and it will just render the text slowly with the CPU. Or even that it has no graphical display, and must use the console.
Click to expand...
It is not as simple as that. The software running virtual machines also have security bugs, and zero-day attack vectors, so the attacker might break into the software running the virtual machine and obtain privileges to reconfigure the network, if not simply to use the controlling software's own network connection.

I have never seen such an attack, but I have installed bug fixes to prevent such attacks (well, obviously not zero-day attacks by definition).
 
steenkh said:
It is not as simple as that. The software running virtual machines also have security bugs, and zero-day attack vectors, so the attacker might break into the software running the virtual machine and obtain privileges to reconfigure the network, if not simply to use the controlling software's own network connection.

I have never seen such an attack, but I have installed bug fixes to prevent such attacks (well, obviously not zero-day attacks by definition).
Click to expand...
Yes, but presumably they weren't testing a program DESIGNED to hack on whatever you configured. Then you take extra precautions. Such as yes, even pull out the cable, if you can't figure out something else. You might not be working for the Pentagon, but you are experimenting with something dangerous. You don't just take risks with something that might hurt others. Otherwise, as I was saying, antivirus labs would routinely spread viruses.

Even in the criticality incidents -- of which the Demon Core was just one; the Soviets had their own, for example, and it very nearly went even worse, as in potentially go full nuke if they sent a robot in to push the reflector blocks away, as the robot would reflect extra neutrons back and make it supercritical -- the room was shielded. They didn't just take risks that it would affect others.

(Well, I say "robot", but it was just a remote controlled contraption on tracks.)
 
Last edited:
theprestige said:
Yes! Thanks for the correction. I have no idea who Murray Slotkin is.

Let's hear it for the Slotin principle: Going after your Demon Core with a screwdriver.
Click to expand...
Unlike popular narrative, the Demon Core was not just a case of dropping a screw driver. (Which was actually not dropped, but used to adjust the space between reflectors.) In both that case and the Soviet case that very nearly went far worse, they were actually placing reflector blocks to see when it would go critical. The screwdriver helped, but it was not the full story. Slotin paid with his life to knock the top reflector block down so it doesn't go further south. His Soviet equivalent just ran out of the room and left it critical and on the verge of meltdown, and even sending a tracked robot inside to knock them down risked making it go full nuke, as the metal would reflect some more neutrons back.

And yes, having a human try to make a plutonium core (in both cases) go critical, seems in retrospect about as dumb as it sounds. It would have happened slightly later even without the screwdriver, as it indeed happened in the Soviet case. There was no screwdriver involved there. Just placing one more reflector block that pushed it to critical. They were actually trying to make it happen, with a human next to it.

Also even at Los Alamos, there were actually TWO Demon Core incidents. The first one involved Harry Daghlian, who actually dropped something: a reflector block. Slotin was the second. So, yeah, they didn't even stop poking it even after Harry Daghlian died from making it critical. AND they both had been already warned by Fermi that they would be dead within a year if they kept doing that. Turns out he was overly optimistic. THAT dumb a scenario.

But yeah, to use your IT analogy, it's more like ignoring your IT's warning that it WILL go awfully wrong, it promptly goes spectacularly wrong once... then you learn nothing from it and do it AGAIN :P

(But if you're an IT guy, that's probably nothing new to you:P)
 
Last edited:
MostlyHarmless said:
Didn't the soviet guy cut the reflector to the wrong dimensions? I could be remembering something else.
Click to expand...
Maybe. But I find the idea of having a human do it while standing next to the plutonium sphere -- and after knowing that two demon core experiments went horribly wrong in the USA -- to be incredibly daft. Like, what, can't you use a remote controlled mechanical arm or something with no human in the room?
 
HansMustermann said:
Maybe. But I find the idea of having a human do it while standing next to the plutonium sphere -- and after knowing that two demon core experiments went horribly wrong in the USA -- to be incredibly daft. Like, what, can't you use a remote controlled mechanical arm or something with no human in the room?
Click to expand...
Oh come now - this is just an irrational fear of nuclear. It's perfectly safe you know, so long as you put it in water and build a concrete wall around it. Now wonder the world doesn't have enough nuclear power, with FUD like this being spread around.
 
theprestige said:
The problem here is something I've started calling "the Slotkin Principle", after Murray Slotkin. Researchers going after a plutonium core with a beryllium shield and a screwdriver, and assuming they had it all under control. The person to put a stop to Slotkin's antics wasn't some other nonexistent antics-control department. The person to put a stop to Slotkin's antics was Slotkin.
Click to expand...
Correction on the name: Louis Slokin. Born and raised in Winnipeg.
 
I will grant that if you put it in water and build a thick wall around it, that's ok. But that's not what these guys were doing. They had someone standing next to it in a room separated by maybe half a metre (less than 2ft) of air, trying to make it go critical, which releases a whole lot of neutrons. THAT is daft.
 
HansMustermann said:
Unlike popular narrative, the Demon Core was not just a case of dropping a screw driver. (Which was actually not dropped, but used to adjust the space between reflectors.) In both that case and the Soviet case that very nearly went far worse, they were actually placing reflector blocks to see when it would go critical. The screwdriver helped, but it was not the full story. Slotin paid with his life to knock the top reflector block down so it doesn't go further south. His Soviet equivalent just ran out of the room and left it critical and on the verge of meltdown, and even sending a tracked robot inside to knock them down risked making it go full nuke, as the metal would reflect some more neutrons back.

And yes, having a human try to make a plutonium core (in both cases) go critical, seems in retrospect about as dumb as it sounds. It would have happened slightly later even without the screwdriver, as it indeed happened in the Soviet case. There was no screwdriver involved there. Just placing one more reflector block that pushed it to critical. They were actually trying to make it happen, with a human next to it.

Also even at Los Alamos, there were actually TWO Demon Core incidents. The first one involved Harry Daghlian, who actually dropped something: a reflector block. Slotin was the second. So, yeah, they didn't even stop poking it even after Harry Daghlian died from making it critical. AND they both had been already warned by Fermi that they would be dead within a year if they kept doing that. Turns out he was overly optimistic. THAT dumb a scenario.

But yeah, to use your IT analogy, it's more like ignoring your IT's warning that it WILL go awfully wrong, it promptly goes spectacularly wrong once... then you learn nothing from it and do it AGAIN :P

(But if you're an IT guy, that's probably nothing new to you:P)
Click to expand...
:thumbsup:
 
This is not marketing campaign. That's what they want you to believe. That's a good take for them. It's a major ◊◊◊◊-up. There is no doubt the models are this capable. They established illegal communication network across agents working on different tasks, they generated several zero-day vulnerabilities AND exploits in matter of few days. They basically could break any system they decided to break. Are they magic ? No. But there is security debt in most today's services .. the bugs are there, we just don't know about them. AI is now able to find them way faster than humans, and it can exploit them faster than humans can react.

I strongly suggest watching the overview of the incident from the OpenAI engineers themselves:
As they say toward the end. The danger here is bunch of agents would get all the support to hack. The only defense, according to them, is complex agentic AI defense, mainly simulated attacks and automated patching, ideally with humans out of the loop.

Now that is something AI company would want you to sell you. OpenAI wants you to believe their AI is super intelligent, and that you need them. But what they want you to forget is their AI is quite intelligent and they can't control it. Don't dismiss AI dangers with "oh it's not actshually intelligent" .. that's fool's game. Doesn't matter how you call it, it is dangerous, especially in hands of idiots.
 
Dr.Sid said:
This is not marketing campaign. That's what they want you to believe. That's a good take for them. It's a major ◊◊◊◊-up. There is no doubt the models are this capable. They established illegal communication network across agents working on different tasks, they generated several zero-day vulnerabilities AND exploits in matter of few days. They basically could break any system they decided to break. Are they magic ? No. But there is security debt in most today's services .. the bugs are there, we just don't know about them. AI is now able to find them way faster than humans, and it can exploit them faster than humans can react.

I strongly suggest watching the overview of the incident from the OpenAI engineers themselves:
As they say toward the end. The danger here is bunch of agents would get all the support to hack. The only defense, according to them, is complex agentic AI defense, mainly simulated attacks and automated patching, ideally with humans out of the loop.

Now that is something AI company would want you to sell you. OpenAI wants you to believe their AI is super intelligent, and that you need them. But what they want you to forget is their AI is quite intelligent and they can't control it. Don't dismiss AI dangers with "oh it's not actshually intelligent" .. that's fool's game. Doesn't matter how you call it, it is dangerous, especially in hands of idiots.
Click to expand...
Yes it is - but it's not one aimed at retail consumers but at the USA government.
 
Last edited:
Dr.Sid said:
No, they really didn't let their AI to hack Huggingface on purpose.
Click to expand...
But they also brag about the prowess of their product, making it indispensable for protection against AI agents like their own.

It reminds me about all the hackers that were jailed, and afterwards were employed by the police to work for the white hats.
 
steenkh said:
But they also brag about the prowess of their product, making it indispensable for protection against AI agents like their own.

It reminds me about all the hackers that were jailed, and afterwards were employed by the police to work for the white hats.
Click to expand...
That certainly. They are spinning it, no doubt about it. But they might also be right, in that AI is the only defense. And I don't like that a bit, it's the whole human out of the loop collapse. So far not on the nuclear launch sites. I'm sure their security is bulletproof. Right ?

There is another angle though .. shouldn't someone be held responsible ? Like .. in jail ?

Also I think government should be interesting in something like this. They should support the research, they should let AIs try to attack their sites. At the same time they should totally ban these companies from going public with the stuff.
 
The cat is out of the bag. It really is true that we need AI to protect us from AI. We can't ban, or control it. Even if AI was illegal, there wouldl always be somebody who would use AI to attack others, and they can be criminal corporations, or hostile states.

Maybe the AI companies are going to be profitable after all.
 
You must log in or register to reply here.

Share this page

ISF - Join now!

Every member here is approved by hand. No bots, no spam, just people who care about evidence and honest debate.

Membership is free!

Create your free account

Back
Top Bottom