Because I have a continuous memory of being consciously aware. Sure, during deep dreamless sleep I can't tell, but nor can anybody else about their life! The members of the cabal are mind controlled in their ordinary roles, so that's something that can be ruled out in my case.
Svali, and her husband, were only mind controlled during the night, but there were signs of being tired during the day etc. I don't find anything in Svali's story that match my own life. Svali was a programmer, meaning she programmed the other members in the cabal, so she had a special role. Most of the members are probably mind controlled during their ordinary daily lives.