It sucks to drive onthestreet without a front swaybar, and IMO it's actually dangerous. Been there, done that, don't want to do it again!
Definitely better offroad without it though. Like was said before, a little more body roll, but if you're going slow it's not bad. The extra articulation more than makes up for it. The sway doesn't damp any up and down motion (like shocks do), so it won't make the ride any bouncier.
I'd leave the rear bar in, it doesn't really hinder flex that much, but I feel it provides some lateral stability on sidehills. I don't have a rear sway, and with the front disco'd as well I've been on some sidehills where I've felt a LOT of roll to the downhill side. The rear bar helps to keep the body a little more horizontal in those situations. I actually think the rear sway helps more OFF road than on, as I don't notice the lack of it when I'm on the street.