Depends on where you are. In the U.S. no one's really gotten burned yet, diCorcia probably spent a lot in legal fees winning his case, but he was getting $10k-$20k a print, he could afford it.
If you're not making huge money like diCorcia you're just not an attractive target for a lawsuit--no lawyer is going to put the time required into a case for a few hundred dollars.
You're especially unlikely to face litigation if you simply remove the offending piece from further displays upon complaint. This is a little tougher for a book project though...
Do beware of anything defaming, humiliating, or that could be considered an invasion of privacy. Those CAN lead to big damages. but they're also pretty easy to avoid.
France and Quebec have both had cases where photographers lost, they're definitely "no-street-sales" zones. My understanding is that Anglophone Canada is different than Quebec in this area, but, as an American my understanding of Canada is pretty suspect.