Studies have confirmed that silver and its compounds are not particularly toxic. I cite the following monograph by the EPA.
In actuality iron and it's compounds are more dangerous. As little as 20 mg per kg can cause serious poisoning. Iron poisoning is endemic in certain portions of Africa where the people subsist on a corn meal porridge cooked in iron pots.