Couple of suggestions but first, Bruce's standard disclaimer: "if you can use it, (the advice) use it - if not, throw it out."
Based on what you're telling us, THEY are the ones who fornicated with the family puppy, not you.
First, I'd call the bank and find out if there is any protection on your debit card. If it's a Visa, for example, there probably is.
Then, I would call that vendor and ask to speak with someone in management. (Did you get the name of the person with whom you spoke originally?) When you get a supervisor on the phone, KEEP YOUR COOL.
In the most cordial tone of voice you can muster, and without being accusing, simply state that Larry, Moe, or Curly (the sales person) agreed that you would not be purchasing from his company and that no sale had been completed.
Very politely tell the member of management what happened and that you will return the merchandise freight collect. Without hesitating, ask him if he has a preferred shipper. If they truly are a stand-up operator, he'll agree to that arrangement.
(Make it clear that there are to be NO further charges to your card.)
If the outfit is not that "stand-up" and expresses reluctance to do the right thing, tell him that you really hate to see this thing end badly but he may leave you no choice but to contact his local Better Business Bureau. Most companies actually care what gets reported to their local BBB. Matter of fact, I would place a call to that particular BBB up front and enquire whether or not they know of any complaints.
If the BBB idea doesn't take hold, then simply inform him that you've no other choice but to file a small claims complaint in court. Contact a local attorney and enquire about that route. Usually the local DA's office can advise on a "small" matter such as yours without charge.
I apologize for the lengthy response but I hate to see anyone get shafted due to incompetence or misunderstanding.
I'm sure there are others on the Forum who can offer better specific advice if the thing deteriorates into a legal matter. Hopefully it won't, though.
Let us know how it turns out.