That doesn't mean the consciousness is ephemeral (I won't use "observer" anymore; your only intent with it is to obfuscate).- Your brain may come from a limited pool of possible brains, but your brain does not determine "who" your observer is -- in that two exact matches in brains would not produce the same observer.
As has been repeatedly said, the consciousness is an emergent property of entirely physical things, including the brain itself but also including physical experiences.