you have to rotate around the entrance pupil, which is generally not identical to the nodal point:
http://en.wikipedia.org/wiki/Entrance_pupil