Sounds like not enough toe in. Caster could cause the same problem with that much lift but usually caster effects return to center and toe in corrects wondering. Besides pinion angles more important than caster.
Anyway sounds like you need to check the alignment.
[SIZE="2"][B][COLOR="Red"]Red 2005 X[/COLOR] - RE 4.5" w/ 3.5" Springs, 33"x12.50" Mud Dawgs, "Drive Shaft is a Woody", TeraFlex SYE, 4x KC Slimlite, 48" Hi-Lift Jack, EK Grab Handles, Custom Made Rear Bumper
[COLOR="Red"]Red Jeep Member #354[/COLOR][/B][/SIZE]