I daily drive my 96, and I trust it more than most other vehicles on the road around me. I'm also a mountain biker, and it'll get you where you need to go.
Vehicle-specific things to look for are floorpan rust, header crack, broken seal and/or rust around the windshield, and be very thorough and particular when looking over the cooling system. Other than rust, few problems are very expensive to fix (but he seller doesn't need to know that), but the more thorough you are, the better prepared you can be.
KBB is less than reliable when discussing older/niche vehicles, but I've always had a much lower than average regard for that "book." If you live anywhere that a reliable 4x4 is especially desired, you'll pay significantly more than you might in a warm, flat area. Around here, even beat-up XJs can go for over 4 or 5 thousand (though I think those people are idiots). I paid less than 3000 for mine with 108k miles in very good condition.
If you buy it, don't trust anything he PO says, and don't take chances--change all the fluids and give it a tune-up. It's all very easy and cheap to do, but is priceless.
My 96 XJ is for sale in the event you want something to restore or build into a wheeling rig. Sale ad is HERE