It does fight me sometime, but yeah, grab one end and pull it toward the other end, and up.
I keep mine in, only for the fact that if you're in an accident, it will at least slow down whatever items may be unsecured in the cargo area. But I agree, it can be a pain in the arse when you're trying load certain things, and have to remove it. As far as people walking by and looking in, they pretty much have to take an obvious look through the glass. The tint makes it a bit hard to see just walking by.