Hahahaha at natwest trying to blame a indian worker!!

sirmarcus

zxspeccy wrote: »

Pure media speculation which has been denied by RBS.

Er no..they have not denied it. They are just refusing to comment and/or confirm.

zxspeccy wrote: »

The main RBS IT Centre is based out of Edinburgh and not India. It is also highly unlikely come lowely IT Tech Support guy could mistakenly delete a whole days data file, and then delete the back up file which is what is being claimed.

Not correct. The Indian person is alleged to have backed out the failed CA-7 (which is used by RBS to schedule the hundreds of batch jobs that run nightly in order to update customer accounts) upgrade but made a huge error by deleting all of the scheduled batch jobs as part of the backout. He/she did this on both the live and backup systems.

zxspeccy wrote: »

What this story is trying to raise is the fact that RBS has outsourced a lot of it's work to India. Staff in th UK are being made redundant whilst staff in India are being recruited on less that half the salaries that the Bank would have to pay in the UK.

No is is not a story. If you look at the websites that have reported this, they have sources with inside knowledge of IT within RBS. The papers are just highlighting that RBS Senior Exec made a strategic decision to outsource key IT skills and services to offshore, which they will probably be regretting now.

It is also worth noting that they use complex legacy mainframes to run the batch jobs and do not have sufficient IT skills within RBS to fully understand the nuances of the mainframe applications, which has only added in the delay for fixing the problems.

zxspeccy wrote: »

No doubt RBS will be forced to disclose what has happened to the FSA, but not until the whole issue is resolved first..

Totally agree and be interesting to see what spin, if any, they attempt to put on it. If they were to be honest, they would confirm the press reports as being factually accurate.

I also see that there is a report in the FT saying that RBS are thinking about suing the 3rd-party supplier of CA7. Both RBS and the supplier are currently refusing to comment or confirm to the press. I think this would be a total disaster for RBS cos they would have to show what design, testing, coding, infrastrucure builds, backout plans, implementation plans, etc. they had for the upgrade. From wot has happened here, their IT processes and methodologies were sadly lacking and IMO they will be laughed out of court !

zxspeccy wrote: »

I work in the Banking industry and although mistakes like this can happen, the time taken to resolve the matter would seem completely unacceptable. That said I do think that some praise should go to the bank staff that, through no fault of their own, are at the sharp end of this problem. They have a very tough job and one I would certainly not envy, and from what I have read they have been doing their best for customers in difficult circumstances.

Agreed. I have worked within the Banking IT industry for many years and have first hand experience of working with off-shore staff; including India. You just make sure that you focus them on simple and non-complex tasks; for example 1st-level help desk IT support. You shouldn't give them access to complex and/or business-critical IT systems without having skilled and experienced onshore staff to monitor them.

This will be a lesson for RBS and I hope other banks as well that have attempted a similar outsoucing model as RBS.

holyroller

zxspeccy wrote: »

Well the money is all back now isn't it, at least for NatWest and RBS customers at least. In any case customers balances are just numbers on a computer screen, so if a Bank was in trouble it wouldn't need to come up with an elaborate plot like this to hide any losses.

However I would agree that there is more to this than meets the eye.

I have a natwest bank account and I only started having problems yesterday. The money isn't all back and things are still a mess regardless of what they say.

On another note, I use to work in the IT department for a big bank during a big migration of data and even when things went trouble free it took days to complete due to the sheer volume of data (and this was for one department), not to mention the issues caused by people still using the system and then the backups of the system slowing everything down. We worked pretty much every weekend and usually didn't leave till 1am.

For a major issue like this you will have loads of IT staff literally working through the night to get it sorted. If they get it all fixed by beginning of next week then they are doing well, but you can bet there has been a lot of all-nighters for the staff in the I.T. department.

holyroller

sirmarcus wrote: »

Not correct. The Indian person is alleged to have backed out the failed CA-7 (which is used by RBS to schedule the hundreds of batch jobs that run nightly in order to update customer accounts) upgrade but made a huge error by deleting all of the batch jobs as part of the backout. He/she did this on both the live and backup systems.

When I worked in the IT department of a bank we didn't have access to both the Live and Backup system, so it would seem odd that someone running a batch job would actually have enough permissions to access the backup system to delete it. Large corporations like Scape Goats for situations like these.

sirmarcus

holyroller wrote: »

When I worked in the IT department of a bank we didn't have access to both the Live and Backup system, so it would seem odd that someone running a batch job would actually have enough permissions to access the backup system to delete it. Large corporations like Scape Goats for situations like these.

I am assuming the backup system, which the websites are referring to, is a hot-standby live system that they can cutover to immediately if they experience problems on the primary live system and, for info, we had access to both within the banks that I have worked for.

fallen121

There might be a nugget of truth in this story.

Fact: RBS have a large IT Centre in Edinburgh

Fact: Not all the Indian outsourced staff operate out of India. Many of them have relocated to Edinburgh and work out of the large RBS IT Centre in Edinburgh.

Fact: A lot of misunderstandings arise between some of the Indian Contractors and some of the local native Scots because despite a common language, the accents on both sides make for a lot of confusion. Therefore the two cultures tend to keep themselves to themselves.

Fact: Mistakes sometimes happen because of slavish adherence to work instructions which are sometimes followed a bit too literally!

My conclusion therefore is that even if this is a "domestic mistake" made in Edinburgh, outsourced staff from India could still have been actively involved.

I share the confusion of others that one individual should hardy have been invested with quite so much responsibility at every stage of the process, that processes should have been checked, and that backups ought to have existed.

Mistakes of this kind sometimes happen because backout procedures differ according to what day of the week it is, and I have seen instances where a weekday backout procedure was followed at a weekend or on a Bank Holiday. But this hardly goes any way towards explaining the magnitude of what occurred.

Debt_Free_Chick

I've posted on another related thread and have unashamedly simply copied that post here.

Source at the end ....

"CA7 was updated to a new level last weekend. It was causing problems with the schedules on Monday so was backed out on Tuesday. The backout caused the "queue" of jobs for that batch to be unexpectedly deleted. "

This is consistent with comments of RBS spokesman on Channel 4 news last night.

"RBS wasn't actually affected directly as the schedule is on a different LPAR - however focus was on NWB & UB and normal everyday errors on RBS side were not getting dealt with."

This is consistent with my own experience, as an RBS customer i.e. not impacted at all. LPAR is, AIUI, a separate partition on the Bank's mainframe computer i.e. it was isolated from the partitions affected, which were for NatWest & Ulster Bank.

"Not sure why DR wasn't invoked to be honest, but it wouldn't have made much difference as we'd still have been hours behind on our batches."

" I'm sure that the discussions were had about DR and there must have been a good reason why it wasn't invoked. In truth DR is great in certain circumstances if it works, but in this one it wouldn't have. Obviously I can't go into too much detail here, but although the outsourcing to India did not cause the initial problem, the recovery has prompted questions to be asked as to who thought dumping over 50% of the UK support staff was a good idea - indeed people made redundant less than a month ago have now been offered short term contracts to come back and help. Unfortunately, the balance sheet and pressure from the government to become independent again hasn't helped in the outsourcing/offshoring decision and I speak as a someone who was due to have been made redundant two weeks ago. "

DR = Disaster Recovery

Source: as is obvious from the above, someone actually involved at RBS, posting on another forum elsewhere.

Fascinating stuff - well, to a saddo like me who simply has to "understand and rationalise" everything in the World :rotfl:

At a guess, backing out the CA7 upgrade seems to have resulted in some unexpected issues - like deletion of the jobs in the queue. Normally, a "restore" is the first option anyone would try, but it seems like the restore option didn't work. Perhaps this is why RBS is looking to see whether it has a case to sue CA Technologies? Perhaps CA's code caused the unforeseen deletions resulting from the rollback?

BTW this guy is on his 5th 12-hour shift in four days :eek:

innovate

fallen121 wrote: »

I share the confusion of others that one individual should hardy have been invested with quite so much responsibility at every stage of the process

That confusion mainly arose from pure speculation on the forum of 'The Register', from whence it morphed into a "true" story in the Sun and the Daily Mail.

It will be some time, if ever, before the public at large will get an RBS-confirmed explanation - and then it's likely that not everybody will understand it because it will inevitably involve technology and processes the vast majority of people will be unfamiliar with.

I am pretty certain that no Sun or Daily Mail journalist will have the skills required to really understand it. Same goes for most of the Press, with the exception of one or two of the more reputable IT mags. Of course that won't stop any of them from spinning their theories, and it won't stop their readers from making whatever they want to make from it.

Mandelbrot

There is enough on the Register/Computing/etc forums to be fairly sure now of what happened. (And it can be deemed consistent with the latest RBS PR spin, up to a point; though they clearly have not fixed it all yet.)

The information from certain posters (admittedly anonymous - for obvious reasons) sounds fully plausible and explains the type of problems faced by some customers but not others, and how the system is being slowly pieced back together.

But quite frankly, the more you read about it, the more worrying it is.

andrinaperry

my account still has not been updated since 19th june bit worried my salary goes in tommorrow

sirmarcus

Debt_Free_Chick wrote: »

At a guess, backing out the CA7 upgrade seems to have resulted in some unexpected issues - like deletion of the jobs in the queue. Normally, a "restore" is the first option anyone would try, but it seems like the restore option didn't work. Perhaps this is why RBS is looking to see whether it has a case to sue CA Technologies? Perhaps CA's code caused the unforeseen deletions resulting from the rollback?

Even if this is true, RBS should have fully tested the restore on a live-alike test environment so I don't see how they will get this to court when their processes appear to be totally inadequate.