I am from a conservative part of the country and I had a couple questions.
Although I understand you want questions answered I should point out that you shouldn't trust what you read on forums. In this case you have at least one person who is claiming to be a mason here who is most likely not (and if he is, he demonstrates 0 of the qualities of the real Freemason). Of course, he would because hes trolling say the same thing about me or Fitzgibbon, so the point is when your asking these sort of questions you MIGHT get the real answer but you are best off going to a masonic web site. I'd recommend something like
http://askafreemason.org or
http://www.thefreemason.com/community/.
Some people choose to not talk about their membership for a variety of reasons. Some lodge cultures put more value on secrecy than others. Some come from evangelical, Catholic, or otherwise anti-masonic parts of the community (active resistance to freemasonry is small but fierce).
Women can't join a mainstream lodge because its a fraternity. They can join other types of freemasonry like Le Droit Humain, International Co-Masonry, etc. These are not considered "mainstream" lodges by the lodges that get to arbitrate such things. However, lodges like the United Grand Lodge of England have stated that besides the fact that these lodges admit women, they are otherwise like mainstream freemasonry and regular in every other way. I consider it a bit like college fraternities and sororities.
Any mason who told you "people of color" couldn't join his lodge is either not a mason or in a lodge that doesn't practice the values they teach. Someones race (along with their social status, sexual orientation, religion, etc.) has no impact on them joining. My local lodge is actually minority Caucasian, and I'm in North Carolina. If its a real masonic lodge the answer is sad but simple: racism. Unfortunately, some lodges have a racist past - its a sad fact but a reality. I would never attend such a lodge, and would in fact do everything I can to stop such practices - as would 99% of real masons.
There are what are called historically African American lodges called Prince Hall. They are like other lodges in every way, and were in fact early on recognized as regular masonry by things like the UGLE. Today these lodges are only "historically black" sort of like HBCUs - anyone from any race is welcome.
Smaller lodges may not meet that often or during certain periods of the year (they call it "going dark" - usually happens in the summer). So that may be why you never see anyone there. I wouldn't go there just because its your grandfathers lodge. Find one that fits your personal culture. Local lodges vary widely in terms of demographics, dress code, and community involvement. I'd be surprised if there wasn't one within a 30 to 45 minute drive that fits you. To find other lodges, type in "Grand Lodge of <Your State" in google and make sure that the website for your state matches one in the following list:
http://www.ugle.org.uk/about-ugle/recognised-foreign-grand-lodges/grand-lodges-in-north-america/
Most of the state grand lodges now have search engines that list all the lodges by location, so you can find one that might be more active and to your interests.
Again, don't trust a word I've written just as you shouldn't trust a word by anyone else here due to the prevalence of fakemasons - before acting on any of this make sure you double check for yourself.