So, my conclusion is that I really like it. I see virtually no drawbacks- The 24-105f/4 is darker indoors, but then I would never really shoot it in low light to begin with.
Focus is way clearer than before, I can happily manually focus on something not in the middle od the frame for instance so I don't need to focus&recompose.