I have been there a number of times without problem, like you on weekdays. Contact Brett Bertollio at the prison at email@example.com He will give you all the info you need for your get together.
Mornings is usually busiest as groups come through, by mid day and into the afternoon you usually only encounter families and similar.
The prison will offer you a special group behind the scenes guided tour for additional $$$. If this is your first time, there is more than enough in the various cell blocks to keep you occupied your first time around.
Great place, enjoy!