I don't see how "your yellow could look to me like my green" can not imply something separate in your brain that is "looking" at what the rest of your brain has processed, and that could be swapped for that same separate entity in someone else's brain. What else could it mean?
I am not talking a seperate entity looking at the visual image my brain creates, i am talking about the visual image itself.
Think about the color blue. You know the color when you see it, but could you ever describe it? No, the only way you could describe it would be to say "it is the color of the sky" or something similar.
Now think about the color red... same goes, you could never describe a color to someone who is blind and has never seen.
Now think about the situation; if the way that i see the sky, i know that the name for that visual image of color i am seeing is blue... but what if the way i see blue, is the way that you see red.
When asked, what is the color of the sky, we will both answer "blue".
The light is hitting my eye, going through its filters, the information is being passed to my brain for processing, and i am presented with the image into my concious mind. An image has been constructed that I identify as vision, which contains color. But what really is color? it is merely my brain presenting me with the seperated out information of the light frequencies it has recieved. Who is to say that my brain would create the same type of image as yours? Why should there be a static way that light frequencies are contructed into visual information accross humanity? It is likely there is... but yet it could vary, and we would never be able to tell.