Techies, what's with the RBS computer?

LincolnshireYokel

bazster wrote: »

This is all absolutely spot-on. These systems are not only immensely complex, but their core components are creaking antiques running on IBM or ICL mainframes, and 100%-proprietary. The only people who can spot potential problems before they arise, or fix them quickly, are programmers and operators who have been working there for decades, and for these people the job is not so much science as it is intuition: these systems have seeped into their DNA, and they don't so much diagnose problems as feel them with the seat of their pants.

So where are these people now? Retiring or dying for years, and not being replaced because no bright young IT graduate wants to work with this carppy old technology, let alone spend the years it takes to approach the kind of sympathy with the systems that the oldies had. And as if that weren't bad enough, what do you think happens to the remaining old stalwarts when the entire operation gets shifted to Bangalore?

The last company I worked for was a classic example of this.

A medium company with a factory, 200 employees, 60 of them office staff - in fact they are a well known compost manufacturers, and an IT department consisting of 4 guys, the head of IT is the old company accountant, as IT literate as the average Cocker Spaniel.

The computer system is unbelievable. When I wen to work for them in the early 2000's , it was like walking back into 1970. They had 132 col line printers, and Laserjet 4's. The main computer system was a 1980's minicomputer the size of a Ford Focus. They had had to custom write interfaces to get stuff like Office to work, some accountancy tasks still had to be done on 11 inch green screen monitors. A lot of the main stuff was written in ICL Job Control Language, using Minimac o/.s and MUMPS.

Whenever anything went wrong, a guy about 60 years old up in Scotland had to be summoned at the cost of £800 a day to fix it. I got talking to him one day, he said as far as he knew, he was the only person in the UK who knew the system and could fix it, and he was about to retire.

They had about 80 PC's, every single was was different, there was no standard images. They had PC's ranging from Win 98 to Vista, using Ethernet and Token Ring , on CAT 5 and Coaxial wiring They didnt even have roaming profile implemented, so you couldn't even move desk and log on somewhere else if your PC failed. Every single software system - accounts, budgeting, purchasing, production, transport, was separately written, at different stages, in different languages, and there was no automatic integration or cross module data transport. It all had to be done manually by exporting to some sort of intermediate file (such as CSV) and then sucked in again. This took huge effort to avoid data duplication and conflicts.

I offered my professional opinion that what they need was to grit there teeth and shell out £200k for SAP, but it fell on deaf ears. Instead they stumble along patching the holes and paying vast amounts of overtime and Agency Fees keeping this inefficient, unreliable, ancient, system limping along, which waste vast amounts of money and staff time correcting errors, mistakes and manually doing things that should happen automatically.

Even the dept i worked in was crippled by it - if a customer rang you, you could only locate them on the system with there account number, which sometimes took several minutes because there was no type of fuzzy search or wildcard search. Eventually I got the IT guy to write a postcode search, which made it quicker, but I got bollocked for it cos they charged us £100 to write it. Even if you got the account number quickly, you couldn't see onscreen a list of previous orders, you could only do an SQL query and direct the output to the Line printer in the far end of the building. Vast amounts of staff time were wasted trying to do stuff which on systems like SAP you would do automatically in seconds.

Non technical bosses do not understand the implications of trying to rely on rubbish IT systems, and you cant make them see because spending the money doesnt produce instant visible results in savings.

agrinnall

bazster wrote: »

So where are these people now? Retiring or dying for years, and not being replaced because no bright young IT graduate wants to work with this carppy old technology, let alone spend the years it takes to approach the kind of sympathy with the systems that the oldies had. And as if that weren't bad enough, what do you think happens to the remaining old stalwarts when the entire operation gets shifted to Bangalore?

As you suggest, many of them have been made redundant in the last 3 years and their jobs (in theory) exported to India. Some of those people have moved to other jobs (for some reason quite a few have ended up in retail, probably because the stress levels are a lot less), some are back doing the job they were before at contract rates, and some have taken the money and run.

[Deleted User]

agrinnall wrote: »

As you suggest, many of them have been made redundant in the last 3 years and their jobs (in theory) exported to India. Some of those people have moved to other jobs (for some reason quite a few have ended up in retail, probably because the stress levels are a lot less), some are back doing the job they were before at contract rates, and some have taken the money and run.

The others are milking the system as private consultants since they're the only ones who know how to stop the deck of cards from collapsing

chunter

Oddly enough RTE radio are just talking about this as i'm reading this.
Exactly what's been said here. Software update caused a cascade failure.
Held together by antique software sticky tape.
Outsourced software company possibly doesn't show same commitment.
Poor resilience systems with RBS.

alanwsg

Someone's been talking to The Register ....

http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_went_wrong/

Owain_Moneysaver

paddyrg wrote: »

I agree that we're not just talking banks here, but all large systems. NHS, HMRC, DWP etc (and pretty much by definition any government IT project) will be huge and complex and evolved over time. And it will be a 'when, not if' before the next problem.

Banking is slightly different as it has huge transaction volumes and there isn't any scheduled down-time.

NHS, HMRC, DWP close at weekends and for christmas holidays, and if they lose a day's processing they can catch up over a few days or even weeks if need be.

Banking systems are just as busy at weekends (in terms of retain EFTPOS and ATMs, possibly even busier) and there isn't any quiet period where they can recover. If RBS had said "all accounts will be frozen for 3 days while we recompile and reindex everything" then they'd probably have recovered quicker. But they can't do that, so they haven't.

[Deleted User]

Owain_Moneysaver wrote: »

NHS, HMRC, DWP close at weekends and for christmas holidays, and if they lose a day's processing they can catch up over a few days or even weeks if need be.

Want to tell that to the A&E department of your local hospital?

I'd love to get an idea of the throughput of a banking system. I've been working on a soon-to-go-live hospital lab system, and we've estimated a throughput of 50,000 messages a day (24hr service) which is quite high in clinical terms. Banking must be so much higher.

Ah, integration. I wonder if anyone in finance uses InterSystems Cache or Ensemble?

Billy-no-Money

Owain_Moneysaver wrote: »

NHS, HMRC, DWP close at weekends and for christmas holidays, and if they lose a day's processing they can catch up over a few days or even weeks if need be.

And here was me thinking banks close overnight and most of the weekend!

Government departments aren't that different and while most are behind the banks in terms of providing services online they're rapidly heading that way. Call centres need IT support so the services have to be available way beyond office hours; and believe me if a big department like DWP were without systems for a few days heads would roll - whatever you might think of the welfare system, that kind of outage would leave thousands without food with a real risk of disorder (pasty riots anyone?).

LincolnshireYokel

Billy-no-Money wrote: »

And here was me thinking banks close overnight and most of the weekend!.

The way it used to work (dont know if its still the case) was when banks close at 5 oclock or whatever, they then have about 4 hours hours to prepare and send there transactions files to the clearing banks, who then process all the transaction from the day before, in the small hours. Then they send back new transaction file. In the meantime, as the clearing banks are running, the end banks process the received transactions from yesterday. The point is a most of the worlds actual banking is done in the small hours.

prowla

Well, looks like it's an offshored service then - server the sods right. Something costing less doesn't necessarily make it cheaper.

The problem with a batch scheduling app is that it tells lots of systems what to do and when, so if it told the wrong ones at the wrong time then it could wreak havoc. Data could have been purged before it was used, transactions could have been performed in the wrong order, backups could have been done with the applications in the wrong state, data feeds could have been broken or duplicated, and so-on.

It is possible that it ran amok and they offlined it to stop whatever it was doing, and it's taken them the time to validate things before opening it up again.

(I'll have to check our One account and see if there is anything suspicious there, and also that any scheduled payments weren't missed.)