I chimed in on this before, stating that I feel "toy" cameras are made for children, and that I do not consider a Polaroid a toy. My Great Grandmother's The 800 certainly isn't a toy.
The more I think about it, though, many today think of Polaroids only as the plastic One-Step style cameras. While still not toys, they are not as "serious" as some of the other (and older) offerings from Polaroid. Also, even though they were not made for children, often they were given to children as a first camera. They are fairly durable, easy to use, and give immediate gratification. Maybe this contributes to the perception.