I think everyone's does this. The voltage reading is showing voltage after draw. So your battery should still be seeing 13.5 volts, even though your cigarrette lighter only shows 12.4.
The blower has a lot of draw as does the e-fan. If you switch it from a/c to vent, my voltage drop is usually just a little better, but that's because the a/c clutch doesn't kick in, and the e-fan doesn't turn on. But that blower is turning away, so I've always just blamed the blower motor. It's not bad, just power hungry.
1996 XJ Classic
My sig got too long. So we'll just say it's Friggin' Sick.