Actually, I'm loving combining the two so far. I shoot portraits, but in a very 'fine art' sort of way. I choose the location (don't currently use my studio) and the mood of the image is largely determined by the subject (mostly kids). People generally know what to expect from my website and from an initial consultation. During the session, I shoot certain 'safe' portrait images first, and then we play.
I have a gallery opening in February, and many of the images were made during portrait sessions (making sure, of course, that there's a signed model release in place.) I'm just now starting to explore this as a full-time venture (since my company made that decision for me ) and it will be interesting to see if I still feel the same way this time next year!