There are only two things I worry about with long exposure times:
1) Heat buckling the film.
2) Vibration.

Depending on the type of enlarger and whether your heat filters are in place, the longer the light is hitting the film, the greater the chance the film will buckle and move out of the zone of focus, causing blurry enlargements. Altering your ƒ-stop or using a glass negative carrier can counteract this tendency but you should still remember that more time might mean more heat. Certain types of cold light and/or diffusion enlargers won't have this problem so much. My Beseler does buckle the film, even with the heat filter in place. My Omega doesn't do it so much but I still have to think about it.

The other problem is that, the longer the exposure, the more chance there is for the enlarger to vibrate or get bumped, causing a blurry image.
Depending on the construction of your building, where the darkroom is located and how well your equipment is set up, even a truck driving by could cause vibration that could be noticed in a big enlargement. If you're a klutz like me, how many times have you accidentally whacked your head against the enlarger while working? Do that while you're making an exposure and it will cause a problem. I live right next to an airport runway and about a half-mile from some railroad tracks; both sources of vibration. Although my darkroom is in the basement, mostly immune from these sources of vibration, I still have noticed it when a big plane lands or when a truck drives by.

I'm not saying that you shouldn't use long exposure times. Sometimes you have to. I'm only mentioning the fact that, the longer the exposure, the greater the chance of problems. That's why I try to keep my exposure times under a minute. If I can get them down to 30 seconds, even better.