That 2/3rds of a stop (1/2 stop is f1.7 to f1.4) can be a big deal for dedicated low-light shooters. The 1.5 stops between f2 and f3.5 on the 35 is even bigger (getting you from sconsumer zoom speeds to a fast prime). The shallower D0F allowed by the faster lenses is also a major reason.
I shoot a lot of low-light high ISO stuff, and keeping the shutter speed at 1/30 is a big deal when shooting with the 50's or 35's. Last night I was shooting in a bar, with 3200TMZ. I had my 35/2.5 E and 55/3.5 Micro with me and was missing my 50/1.8 E badly, the extra stop over the 35 and almost 2 stops over the 55 are a big deal in low light, I eventually had to pull out the flash because I had enough light to shoot a 1.8 acceptably (A 1.4 or 1.2 would have got me into 'no problem' territory) but the slower primes were only netting me 1/15 or slower, which simply is too low for acceptable shooting.