Its most likely a faulty HOP somewhere. I have had no problem at all. And since it only affect some people, it's either clientside or somwhere along the path there.
I suggest trying to get some traceroutes from people that have problem, identify the faulty HOP and report it to your host.
My bet is a faulty switch somewhere just outside the host.