I've lived in Hanover, Wingham, Lucknow, Kincardine, and (briefly) Point Clark. I've spent a significant amount of time in Port Elgin.
Having that base, I've come to the conclusion that none of them are safe for any sane person under 30. Boredom is the small town killer of teens. One of the few things to do for fun is drink out in the middle of nowhere (barn dances, stags, etc.), which is followed by driving home. Toss in snowmobiling, dirt biking, pool drownings, and small arms accidents if you like.
I know 3 people from those towns (separate incidents) who have some form of severe disability resulting from the above activities (drunk driving in a snow storm is just stupid -- trees get involved).
Yes, you can do that in a big city as well, but more importantly you also have other options to have a good time and far more transportation options for afterward.
You are more likely to see something un-safe in Toronto, but far more likely to be involved in it within the small town. Suburbs seem to have the worst of both worlds.