Optical illusions are definitely brain failures (though in direct response, they're summated poorly from eye stimuli). When the reality of depth or color* is mistaken in your brain, then it fails to accurately depict reality. But the brain tries oh so hard...
It's a brain failure.
Let's talk color.
Here is an image:
For reference, this image comes from a
presentation by Beau Lotto, which I recommend you see, at TED.
In this image, there are tiles that I have labeled a1, a2, b1, and b2. The color of those tiles, on the screen, are all the same. However, in the context of the sample images, given your criteria, there should be a "correct" color that corresponds to the "actual" colors of the object. On the left hand image, which we can simply call "A", the tile a1 is in a shadow, and the tile a2 is outside of it. On the right hand image, which we can call "B", the tile b1 is under the same approximate lighting conditions as the tile b2.
Obviously, there is less light in a shadow than there is outside of one. So if two tiles, let's say a1 and a2, are emitting the same amount of light, but a1 is in the shadow, then a1 should be reflecting a greater percentage of the light that is available to it than a2 (since it reflects the same amount, but there's less light available), and thus, it should be whiter. But if those same two tiles are under equal lighting conditions--let's say b1 and b2--then if they are reflecting the same amount of light, they should be the same color.
There are similar sorts of adjustments that occur in lighting conditions where different color lights are present... for example, take this image:
Here, there are tiles labeled A and B on the left and on the right. Both of the tiles A and B are drawn using the same exact color of gray; however, if the squares on the left are illuminated by a bluish light, and the squares on the right by a yellowish light, then the tile A's "correct" color would probably be yellow, and the tile B's "correct" color would probably be blue.
So here's the point. These are the kinds of things our brain has to do to an image in order to get the "right" color of an object--simply because of the way light comes from objects in an image. There is no way to describe the workings of a hypothetical brain that gets the "correct" color of objects based on the presentation of an image, because the problem of getting the correct color of such objects is an impossible problem. It is for this reason that I object to your use of the term "brain failure".
What our brains do, instead, is make an assumption that objects present themselves in a natural world in "normal" ways, and working off of the assumption that objects have a definite color that does not change under lighting conditions, applies heuristics to try to perceive that color. The reason we have color optical illusions is that these heuristics can be exploited.
So when we look at this image:
...and our brains experience a "failure" whereby we perceive two different shades of orange, you should note that this "failure" is due to a heuristic that allows it to "succeed" in judging the color of oranges under more normal scenarios.
I think the model you're trying to apply is too simplistic--it ignores the fact that perception is different than the perceived, and that the former is merely a model of the world. It's too naively realistic for my tastes. I prefer a model of the world where objects exist entirely independent from our perceptions, where qualities of our perceptions are not part of the external world's properties, but instead are properties of how our brain builds a model of the world that is useful for our purposes.
*Example: Synesthetics "see" colors that correspond to letter shapes or even sound. But they aren't actually seeing these colors, because the medium they are viewing through their eyes isn't reflecting the electromagnetic "light" that we all generally see.
You're kind of begging the question here, and are mixing categories.
Colors are perceptual qualities, not inherent properties of light (though there is an alternate definition of color that we can say is a property of light; however, using that definition, color is not a perceptual quality). It appears you're using "to see" to describe perceiving something through the visual apparatus. In this case, I see nothing but your arbitrary declaration to support the notion that color grapheme synesthetes do not see shapes as colors. After all, they see shapes visually, and those shapes are imbued with colors.
Their brain is failing, it is somehow doing something wrong.
Please describe what's wrong with the way their brain is working, without appealing to my suggested alternative of comparing their brain to a reference brain.
But that doesn't mean what the synesthetic sees is "real" because it isn't because there is no corresponding light they're seeing; it's just their brain failing.
In the case of color grapheme synesthetes, there most certainly is corresponding light that they are seeing.