I just finished a similar problem on my 2000 XJ. I had lost one blower speed months ago, then another, so when it quit working even on high speed, and I read up on it, I looked first at the resistors. When they checked OK, and so did the relay, and the motor ran fine when I touched a hot lead to the motor lead from the relay with the black and tan wire to the resistors grounded, it had to be the switch.
Took me a while to finally find some advice on how to access the control panel. The old Jeep still looks pretty good so I didn't want just tear the dash panels trying to get to the access. Turns out the surround just snaps in and out, after I spent a couple hours searching the net for the magic method.
Once that was out of the way, four screws and unplugging the connections to the backside put the control in my hands. The copper strips on the backside of the control panel that are supposed to remain electrically separated had accumulated enough of the factory dielectric grease that it had eventually filled the gaps between some of the strips.
I cleaned the whole area of grease, then re-applied a little of it very sparingly. Reassembled everything and it works just like brand new!
Two weeks of stumbling around learning about the whole system and messing with it in my spare time, and it would have been a half-hour job had I started on the right end and known how to access the panel.
Oh well, hope this helps someone else.