For many (most?) of us, this is the one area where we're not bound by the expectations of others. Whether it's clients, bosses, co-workers, family, etc., it often seems our lives (and our time) belong to others. When we're doing photography, our only obligation is to satisfy ourselves.
We should consider ourselves fortunate to lead these "double lives" (triple is a bit much, Thomas... ). I know far too many people without much in the way of outside interests. You know the feeling you get when you take a print off the drying rack and think, "Wow... I can't believe I made this"? 99% of the world never experiences that moment. Embrace your separate lives!