If it keeps you sane, maybe you should do it. But talk it over with your wife first. I'm not currently working, but my son (age 2, just) goes to daycare two days a week so I can have my only "me" time - usually photography related. If it weren't for that, we'd be in pretty bad shape (hubby is deployed and little dude doesn't sleep through the night most nights - I have gotten about 9 good nights of sleep in the last 2 years). I don't know how I'd do any photography if I was working. My problem is that I have sooo many ideas of things I want to try and projects I want to do and only those two days to do them. Chasing Nate and keeping things running in general (did I mention we also have four dogs?) means I'm totally exhausted by the time I get him into bed. I do end up sitting in front of the TV because I can't find the energy to do anything else most of the time.
Stealing away to the darkroom feels a bit selfish, maybe like stealing from time i should be with family.