MyJeepRocks.com forum has members from all over Southern California that go from mild to wild, if there is a place to find some fellow wheelers to go out with for desert exploring, some fun in Big Bear, some wheelin at Cougar buttes, the Hammers, or in the Red Rock area, then this place is pretty much the ticket. For something closer to San Diego try www.justruns.com
is more strictly organized, but they do quite a few runs east of San Diego and in the Truckhaven area which can be thoroughly enjoyed even in a stock Jeep.
Another possibility is
More Toyota oriented, and they are more into soft core trails and trecking
they have several pretty cool get together's each year.
Still plenty of fun 4 wheelin in So Cali