Generally I would agree with you about most people not minding. I rarely get booted. Most people when they see me working with a view camera, are curious about what they see as a novelty.
But it would depend on where you are. Here in NJ you have to be careful. I wanted to photograph a section of a barn and the light was just perfect, accentuating the texture of a board. Had I taken the time to go find the owner to ask, the light would have been lost, so I took a chance and lost, got kicked off.
Another time I drove down a dirt road in a local park that was marked for authorized vehicles only. Had I taken the time and walked in, I would have lost the clouds. I took a chance and lost, a ranger came by and ticketed me for $70.00 and I still did not get the shot.
That image of the tricycle entangled in barbed wire was done in Jeresy City, not a very nice town. Fortunetly I was able to park right in front of the scene. While I was quickly and nerveously working, an unsavory guy came over, boom box on his shoulder, and asked me what I was taking a picture of, and was my camera worth anything. Fortunetly it was a sunny Sunday afternoon. At the time, I looked pretty unsavory myself with long greasy looking hair and beard, it was a hot afternoon, and I am bigger then he was.
The risks we take to get the picture are many. Those taken by a fine art photographer are nothing when compared to other's risks.