In a RF typically the viewfinder is on one side of the camera. When you raise it it only covers half of your face, and leaves one eye free, and the nose visible. The subject sees the nose, one eye, and probably the mouth of the photographer, while with a SLR almost the entire face is covered.
It might be that for this reason the photographer with a RF is less conspicuous. For the gut, the instinct so to speak, he's raising "half a camera" to his eyes (or at least, it appears "half" because it covers only half of the face).
Rationally this shouldn't make any difference. It's obvious that both are taking a picture. But somewhere in the gut if the camera is bigger and it covers both eyes (and most of the face) it is obvious that the photographer "menaces" to take a picture. If the camera is smaller and the photographer goes on looking at the scene then probably "also the images is going to be smaller..." or it's a snap or it's not really being taken. The gut is not very rational.