On the other hand, you can push the little gobs from the broken old-fashioned thermometer around on your counter and then into a trashbag without apparent harm, as long as you vigorously wash afterward. Just don't eat it, inhale it, or touch it in a form that is absorbable.
Mercury is everywhere on the planet, and you ingest small quantities in many ocean fish, some foods, and just by touching dirt and breathing. If you live in an area with recent (in geologic terms) volcanic activity, you will be exposed to more. Here in the Pacific Northwest, our "background" mercury level is much higher.
As ectoplasm mentioned above, it also depends what form it is in. Some moves readily back out of the human body, and some is retained for longer timespans. Mercury miners from the 60s developed horrible neurological problems, which helped us get a better idea on what kind of damage occurs versus what kinds of exposure levels.