I have to answer your question with some other thoughts (that you probably won't like
1. Why do you think a 4-banger will get substantially better mileage? You're still pushing a brick through the air. I doubt the gasoline savings would ever pay for the engine swap.
FWIW, I have a 4-banger TJ (3 of 'em actually) and get the same mileage as my friends with 6 cylinder Jeeps. It's an aerodynamic thing, not a fuel efficiency thing.
2. You'll need to regear your axles for the 4-banger. It likely won't push the gears you currently have, especially for the amount of on-road driving you say you do.
My suggestion would be to buy a cheap beater daily driver and use the Jeep for playing. It probably won't cost any more than what you'd spend doing the engine swap and axle regear.
Either that or put a fuel injection setup on the 304. That would help your mileage too.