You're forgetting one of Watts' stipulations: Evolution is hostile. It happens when there's environmental pressure on an organism to evolve or go extinct. Once an organism evolves into a comfort zone, it stops evolving. Humans reached an evolutionary sweet spot in our current environment, without ever having to ditch consciousness or die out.
That's Watts, anyway. What we know, even without Watts, is that evolution does not automatically mean an organism has no maladaptive traits. We know that not every trait in a successful organism is a net advantage. Humans are getting by with sickle cell anemia and autism, not because these are beladaptive traits, but because our overall fitness for our current environment doesn't put pressure on us to evolve away from them. So why not consciousness?
Which is related to my first problem with his theory, i.e. that a species does not actually "win" evolution. There is no endgame to evolution; technically there is no "better" or "worse", and even if we set some arbitrary parameters like "stronger" or "smarter" or "reacts without awareness", environmental pressures won't necessarily result in those traits, because those pressures are also arbitrary, and it is also arbitrary whether the emerging environment will favour one or the other. And at the very end, of course, all life forms lose.
Consciousness is as much of an arbitrary trait as all the others, and it might work in some environments, and fail in others, but I think the idea that it will suddenly be a hindrance in interplanetary warfare (or whatever real threats the future might hold) is a bit of a leap.
I think one reason that the observer has been far from superfluous is that it's able to "cheat". And I suppose Watts' proposition is that his aliens can also cheat, just better, but I think this is also a leap.
And some people with maladaptive traits might be thriving because we cheat and transform the environment to suit them, but I don't see this as a problem; our artificial environment isn't any more or less arbitrary than the natural one. Maybe it will break down at some point, but this wouldn't be because we evolved stopped evolving or evolved "wrong" -- we just ran out of luck, like the dinosaurs who never adapted to deflect a gargantuan space rock (ironically, we probably could deflect a gargantuan space rock).
But I'm not even appealing to Blindsight here. I'm just pointing out that the developers of LLMs haven't included an observer in the architecture, and that, for LLMs, such an observer would be a waste of resources.
I don't think that's why. I mean, it might be a good reason at some point, but currently we don't actually know how to do that. Potential uses and/or horrors will have to wait for when it's actually possible. Currently, LLMs are either a toy, a tool to perform drudgery, or a dubious way to bypass copyright laws, which is only a bit useful.
But some people in this thread seem under the impression that LLMs totally change our understanding of consciousness, when they very clearly don't. They only change our understanding of how difficult or easy consciousness is to fake.
but you aren't saying why we can't.)
