IME, the size and sound of a camera make little difference in whether or not people notice and/or get mad. If this is an issue, carrying yourself differently and approaching your subjects differently are the keys, not a new camera. The 1N is an outstanding street camera because of it's speed, it's AF, and it's durability. It would be hard to do a whole lot better IMO. If anything, go for a 1NRS, or the lower-end EOS camera with a pellicle mirror. (I forgot the model name.)
I also feel on a conceptual level that it is not a bad thing if people notice you. We are not invisible observers in society. We are participants in it. So, be a part of the picture, not a voyeur. Some of my favorite street photography of all time has direct eye contact, and sometimes visible annoyance/surprise.