You can buy replacement bolts and nutserts from the dealer (or one of the online resources for mopar parts), the part numbers are listed in this thread:
If the part numbers for the bolts are not there, they should know what to give you after giving them the nutsert part number. You could also just get the bolts from a local hardware store.
If you are able to get the bolt out of the loose nutsert, and the old loose nutsert is still in good condition you can try to use the tool to "tighten" up the nutsert, and then tack weld it. If the nutsert looks to be in pretty bad condition, I'd probably just replace it to save yourself a headache down the road. It's too easy of a job to put off.
You can get nutserts that will work from a hardware store, but the ones from the dealer are usually better quality than what you get elsewhere. All of those details are covered in the thread linked above.
A little tip if you can't get the bolt out of the old loose nutsert. If you loosen the others on that side of the skid plate and use a crow bar (or similar tool) to pry down on the plate it can sometimes create enough friction to allow the bolt to turn out of the nutsert.