When the belt is removed, and the compressor clutch is not energized (as would be the case with the ignition "off"), the pulley should turn smoothly and not wobble in any manner.
Since yours does, it's the pulley bearing that needs replacing. There have been some write-ups on this forum on how to do this. If you cannot find those, PM me and I can send you the pages from the FSM.
1992 XJ 2" lift -- RIP
2000 XJ 3" lift -- Australian Ford Big Brakes up front, rock rails, ZJ rear discs (with WORKING parking brake), C4x4 rear bumper/tire carrier, Hella H4 headlights, IPF driving lights, OEM skid plates, OME springs/shocks, JKS LCA's, JKS Sway Bar links.