Transfered ISA without using the bank form!!!!?

lisyloo

without using a form suggested by the bank

The transfer must be done by the provider so that the accompanying paperwork goes with it.

I would start with a formal complaint to the bank (go through the official procedure on their website or in their paperwork). This might seem OTT to complain but this goes to a specific "customer services" department you tend to have more leeway. It also gives you rights to go to the ombudsman. Tell them what you want them to do.
Personally I think you have to accept responsibility for what you did, but I don't think they should have returned funds by cheque, it should have gone bakc to the original ISA. You might have more luck arguing the second point.
You can ultimately complain to the free and independent FOS but you need to start with the bank.

I know it seems better to ask someone nicely, but putting it on a formal footing means it gets through to people that might be able to help rather than front line operators/managers who are not in a position to do anything,