I drove a number of the 4.7L powered and 5.7L powered WK's before I bought my LTD with the 5.7L. 4.7L was ok, not bad overall for drive-ability, and the gearing seemed ok. I cannot imagine that the 3.7L in an XK would be much fun... and would probably have to work really hard to get anywhere.
This could eat into your overall reliability as you might be flogging on the 3.7L whereas you could simply cruise around effortlessly in the 4.7-5.7L with relative ease and comfort? I hear the 3.7L are reasonably reliable if not pretty thirsty motors. I had the older 3.9L in my Dakota 2WD which was a 5 speed. It was OK, the 5 speed auto sucked however...gutless.