Yeah, even if people outside of stoner circles get that the numbers "420" refer to recreational marijuana usage, it's pretty likely that they don't know "why" or it's origins. You're not out of the norm by any means.
I'm so ◊◊◊◊◊◊◊ sick of "May the 4th be with you" and "Revenge of the 5th", I'm about to lose my ◊◊◊◊. The 5th of May is Cinco de ◊◊◊◊◊◊◊ Mayo and everything else can eat an enormous sack of dicks. Now, have I ever watched Star Wars? No. Do I want to? No, but I would put this on the same level as 420 in that I know it's popular based on a movie, but I don't know the story of those ◊◊◊◊ ass movies.
I feel like to be "in the know" you would have to know more than numbers = weed. Perhaps I'm the odd one out, but I bet you could go up to 50 people outside of a bank and ask, "Do you recognize 420?"
They'd say, "Yes."
"Do you know how it originated and the relevance?" and I would bet a large amount of money the answer would be "no".