Holiday booking nightmare - can I get a refund?

Sorry this is lengthy but I’ll try my best to get the key points out as brief as I can.

I booked a holiday via telephone through Icelolly, which took me through to global hoppers.
They told me what I wanted wasn’t possible so tried bombarding me with numerous other resorts.
I mentioned what I was after (temperature, baggage, flights etc) and he intrigued me with one hotel.
He told me what seemed to be all I needed to know, and he negotiated an only just acceptable price with me, I clarified that this was the total cost and he said yes. Then asked me to pay. I reeled off my debit card numbers.
After he’d processed and taken a £200 deposit he then proceeded to tell me about some further admin fees which totaled up to £120. I wasn’t happy from this moment as it made the total cost of the holiday more than I could afford. He made it clear there was no going back as the deposit was processed. I swallowed my own stupidity and carried on with the call. He went through all the recapping of the booking and then told me I’d need to pay extra £60 for some baggage as it didn’t include any baggage even though I’d already mentioned I’d need baggage. He also told me that I don’t get a 14 day cooling off period because I’ve booked someone’s “late cancellation”.
He never read any proper terms and conditions before asking me to pay. He also lied to me about the expected temperature for the resort at the time of holiday.

I’ve contacted the company by email asking for a refund and free cancellation because I’ve been mis-sold this holiday. I’ve also sent the email to ABTA but not sure how successful this will be.
Does anyone know if I’ve got a leg to stand on regarding getting any refund?

I’m really not able to spend an extra £180 than I was led to believe on a holiday, when he wasn’t even honest about it with me. Also he says if I cancel I lose me £299 and have to pay £79 for each of the 4 travelers cancellation fee (meaning I’d lose a total of £516 for being mid-sold a holiday)