I can understand your point. If you were travelling, or knew someone who was travelling, I can imagine being worried, even fearful. If you live in a large city, one which might be considered a target for a terrorist act, a certain amount of fear is understandable. I will admit, I was concerned that there might be additional attacks, but I can't say that I was afraid for myself or my family. I just couldn't see us being in the bullseye.
There are probably a few people here who remember the 50's and early 60's, when the threat of nuclear war was extremely high. I remember having to practice diving under our school desks and covering our heads with our arms as air raid drills. And each Saturday at noon, all the air raid sirens in the area would blast for a minute's test. (I can remember thinking that if the Russian's attacked at noon no one would know it was a real air raid.) I suppose that growing up with that threat has made me less fearful of other threats. I also grew up not far from Newark, NJ during the height of the race riots in the 60's. Perhaps that also works to temper my fear reflex.
I take it from your last paragraph that you were in the military, or maybe in Newark. I have never been in positions such as you describe. I can well believe that one would develop a fine sense of caution and awareness, and even fear, as survival mechanisms. And I am concerned about the prospect of terrorists launching a nuclear or biological attack against this country or other countries. But I'm with you: panic is more often than not counterproductive. And blaming everyone with a Middle East background for the acts of a few fanatics is also counterproductive. Yet that is what panic will do.