a guy pulls up as I leave my house, watches me while on his phone, I leave for 20 minutes come back he's still there, I go inside and close my door I hear his car start up and drive away fast. Should I be worried?

I guess it would depend if you see him all the time or is it just a one-time thing?

If you are seeing the same guy doing every time you leave your house, then you might have cause for concern. If this is the case, if this were me, I would walk out of my house, holding my phone as if I were recording them - their car - the license plate. I don't have video on my phone, so I would be taking pictures of the person, license plate, car, etc.

Keep a log of every time you see this person and see if you can determine a pattern. Then you have evidence to call police on the non-emergency line, unless he does something that makes you feel like you are in immediate danger.

I would consider this to be "suspicious activity."

While such a phrase is primarily related to being vigilant about terrorism, the concept is more generic:  A suspicious activity is when a person's conduct or action does not fit the normal day-to-day activity of our neighborhood- an occurrence that is out of place and should not be happening in our neighborhood.

And that quote is from this site---and it's worth a look:  http://www.mlpa.org/content/recognizing-and-reporting-suspicious-activity

I occasionally do the same thing---namely coincidentally being in the same place when I choose to pull over and make a cell phone call---but I don't stare at anyone while there, and I fully expect to possibly be challenged by police on patrol who want to know what I am doing.

There is no need for you to wonder about what is going on---call 911 and ask them to check it out if it has happened more than once.

Yes Id be concerned especially if you don't recognize this guy as this constitutes stalking so this could be a bad thing if you notify Police and you should do so, so they can find out whats going on here. Personally, I suspect hes girl shy and afraid to meet you here  but he should have enough common sense to realize what hes doing  here but best to be on the side of caution and safety here.  Good luck