Shouldn't take too long, maybe half and hour if the first time you're doing it (just to figure out which bolts you need to loosen to pivot, what size they are, walking back and forth to the tool box, etc.)?
The belt is probably just loose, but if you bought a new one already, put the new one on and keep the old one for emergencies (I keep one in a plastic bag stuffed in the engine compartment under the battery tray).
I'm not sure where the 4.0 pivots, my 4.2 w/AC is tensioned by the alternator.
Black '88YJ 4.2L
2.5" Skyjacker, 32x11.5s, 4.11s, Weber, HEI, etc.