I owned both systems and now only have an OM-2Sp and OM-4Ti.
Both systems are good, but I prefer the OM bodies and the 40/2 seems, from my limited experience with it, much better than the Nikon 35mm Ai-S lenses.
Other than that...the 24mm I have is superb, as is the 85/2.

the main thing is that the fast Olympus lenses are mostly smaller than their slower Nikon cousins.