It's partly the size of the front element, and partly the nature of the camera. An SLR typically has a large 'eye', and who wants to be stared at by one of those? SLRs say serious in a way a point and shoot never will. If people think they're being captured on an amateur, 'fun' camera they react in a different way to being studied by a professional one.

I recall one photographer saying he covered his camera in silly stickers to make it, and him, appear less threatening.