I can understand why you feel that way about 4x5. I shot my first 4x5 slides recently and looked at the result and thought, "Cute, but small." Seems that I never think that way when doing B&W, even when contact printing. Part of it could depend on what you're going to do with it.
I used to shoot 5x7 and love the size. Had a monorail B&J and a car, so it travelled a bit. If you're going to carry the thing outside at all away from your vehicle, size issues will play a HUGE role.
If you haven't played with 4x5 much yet, then enjoy it first! It changed everything about the way I shoot. I worked on 4x5 for about 5-6 months before I dusted off the 5x7 and put it to use. Making FNG mistakes is a lot cheaper - like the time I went to a Shuttle launch with the film loaded in backwards. Remember, I said cheaper, not less of a heartbreak.