If the silver is removed, I don't believe it to be particularly dangerous. Main thing is that it's usually quite acidic. It's used as a chelating agent in pools, for example, and even given to humans to reduce heavy-metal poisoning. So it's not hugely toxic but as always, the dose/concentration matters.
I believe photo labs are permitted to release silver-free thiosulfate; at least Kodak's tech docs certainly give that impression by talking about permissible silver levels (a few ppm) in waste discharge streams resulting from a fixer replenishment & silver-reclamation cycles.