In the USA where the actual smoking ban for employment actually exists, yes you could. Of course, you could also have that same smokers only requirement. So it is even handed even if all of these concepts are over the top for most any industry.
BTW, as for the semiconductor industry the issue of smoke breathing/clothing particulates is well known. A few companies have tried a no smoking at all while at work policy and it just was not worth the effort. Too many irritable employees did not outweigh the very small benefit in less particulate in the breath/clothing. Its only an issue on the really heavy chain smokers. Basically, if you can smell something on the production floor that is coming from someone else it is damaging product. This applies to make up, perfume, flatulence, cigarettes, chewing gum, food, deodorant, sweat ect. The clean room suits and masks we wear generally accounts for most of this, but a few people can build up very powerful (highly particulate) smells that can make their presence consistently known on the work floor. Such people usually get banished to areas it might not matter, or get fired for small excuses. Or usually, it just gets ignored unless massive product damage gets noticed by someone high up. In a mature line a few people breaking the rule on odors should have a very small impact. It also helps that in some places you wear different clothes under the clean room suits than you wear on your breaks. I have seen a few heavy smokers pick up the habit of brushing their teeth after their breaks too.
Interesting stuff. How about walking into work after passing through a street heaving with diesel particulates? Could that contaminate your workplace? In my country Athens has banned diesel engines in the city centre apart from buses and taxis. And studies elsewhere have linked diesel emissions to asthma attacks. Should we, logically, deny employment to drivers of diesel cars?