Damage by delivery vehicle

A delivery driver knocked on my door and asked me how he could get the delivery to our door as we have a gravelled drive and he said that pallet won't go over the gravel at all. So I suggested backing onto the drive and unloading the pallet onto the grass area near the house. This involves driving through our gate, onto the gravel drive. He asked me if we had any pipes that could be damaged and I said no.

This gate has had hundreds of very large vehicles over the years drive through it, including removals vans. I heard a crack and looked out the front door to see him reversing into, and knocking, the fence post. I ran out, shouting to him to stop, but he initially carried on going, causing further damage. He eventually saw me and got out.

I said, "I assume you have insurance" and he said, "well I did ask you about going on the drive"
I said, "you asked me if we had pipes that could be damaged! You didn't ask me if it was okay to reverse into our gate and break it!" He also told me I was lucky because he was actually one of their more helpful delivery drivers. :rotfl:

I took a picture and he told me that the vehicle has a camera too. Then I went inside to get my Husband. When he came back the guy had tried to move the fence post back into position, so it wouldn't look so bad. It's a wooden gate that is now chipped, bent and doesn't line up properly when closed. The fence post into which the gate goes, was set in concrete and is now completely wobbly.

My Husband called the company to get their insurance details, but they said because we gave him permission to go onto the drive it's basically our problem. They said they'll get back to us, but this company is known in the local area for being an unethical company (don't pay their staff, that kind of thing) and I'm not convinced they will.

Wondering if anyone knows what rights we have and what's the best thing to do?