Well, you can try and do it without pulling the dash, but it is going to be a beeotch. Pulling the e-brake assembly will give you a bit more room to work with on the driver's side. If you have big hands, forget about it.
The passenger side isn't bad for removal and install, it's the driver's side that is a Major PITA. I tried doing it without pulling the dash and almost got the driver's side installed but because the threads on the mounting studs weren't straight I couldn't get the speaker to align properly and set flush, so I ended up pulling the dash.