It's all about safety. I wouldn't call one safe at any sort of speed, it looks like you'd roll one at any sort of speed. Plus, I wouldn't want to be in one in an accident. I know motorbikes are just as unsafe there, but they have the speed and the maneuverability to hypothetically GTFO of the way.
Although, in a lot of places, especially more remote ones, they are road-legal. I know in Yellowknife you can get road permits for side-by-sides (As well as quads with road tires), and Whitehorse is considering the same thing.