I use a Durst 606 for 110, 35mm and 6X6. Later I added a Kindermann 6X9 for my Mamiya Press negs. I probably could get by with the Kindermann alone, but I do like the Durst. I also have a Minox enlarger, but that's pretty much a special case.
I would think a 4X5 would handle everything, although there might be some difficulty with small or subminiature film sizes.