I'm not sure if color chemicals are more toxic than BW chemicals. But it is always recommend to use safely precautions. This is particularly important if you have contact with the chemicals on a regular basis (daily). The things you can do:

Avoid direct contact withe chemicals;
Wear gloves in mixing chemicals from powder or concentrates, some even wears goggles with liquid concentrate
Wear gloves and/or use tongs when developing in open tanks
Chemical area must be ventilated to the outside
Clean spill immediately and thoroughly.
Avoid hand contact with eyes, nose and mouth.
Wash hands immediately if get contact with chemicals
Wash hands thoroughly after each darkroom session with soap

For casual users, the darkroom chemicals are safe. But the safety procedures should be followed.