Check out this video. It may not look like much, but it's pretty amazing.
What you're looking at is a model showing all of the asteroids discovered from 1980-2010 as they're discovered. The red ones are 'Earth-crossers'. They're the ones to be scared of. But take heart: We don't live on a 2D plane, so they can always go above or below us :)