Natwest Technical Issues

[cottager]

Letter from Stephen Hester to the chairman of the Treasury Select Committee
29 June
(extracts only)

On the original incident:
'The initial reviews we have carried out indicate that the problem was created when maintenance on systems, which are managed and operated by our team in Edinburgh, caused an error in our batch scheduler. This error caused the automated batch processing to fail on the night of Tuesday 19 June. The knock-on effects were substantial and required significant manual interventions from our team, compounded because the team could not access the record of transactions that had been processed up to the point of failure. The need to first establish at what point processing had stopped delayed subsequent batches and created a substantial backlog. It is not clear at this stage why that record was not available. Consequently, a significant number of customer account balances did not update as they should have from Thursday 21 June.'

On Ulster Bank situation:
'Ulster Bank has been more heavily affected, which we particularly regret, as it is, in part, dependent on NatWest systems. While progress is being made we are being purposefully cautious in our approach to ensure our solutions are sustainable. We expect the Ulster Bank situation to have improved substantially by early next week. We continue to work closely with the Central Bank of Ireland and regularly update them on progress. We are working day and night to process the backlog of transactions for Ulster Bank as quickly as possible though some level of inconvenience is likely to continue into next week.'

Read full text of the letter here:
http://www.rbs.com/news/2012/06/step...sponse-AT.html

The text of the Treasury Select Committee's original letter (to which Hester was responding) should be available from a link on this page:
http://www.parliament.uk/business/co...uter-failures/
but instead of a PDF opening I only see gobbledygook -- don't know if it's just me (a link to another letter, written to the FSA, opens fine).

[stormymonday]

Originally Posted by cottager

Letter from Stephen Hester to the chairman of the Treasury Select Committee
29 June
(extracts only)

On the original incident:
'The initial reviews we have carried out indicate that the problem was created when maintenance on systems, which are managed and operated by our team in Edinburgh, caused an error in our batch scheduler. This error caused the automated batch processing to fail on the night of Tuesday 19 June. The knock-on effects were substantial and required significant manual interventions from our team, compounded because the team could not access the record of transactions that had been processed up to the point of failure. The need to first establish at what point processing had stopped delayed subsequent batches and created a substantial backlog. It is not clear at this stage why that record was not available. Consequently, a significant number of customer account balances did not update as they should have from Thursday 21 June.'

I am afraid that Hester still appears to be speaking with a forked tongue. I have done a few mainframe software upgrades in my time and I can not see anyway that a third party batch scheduler upgrade alone no matter how bug ridden or ineptly implemented would deny an IT department access to the 'record of transactions' since that processing should be maintained by the company's own software, reading its own data files and written to its own output reports and logs. In order to lose that information someone in house or some program written in house must have deleted it. We are still not being told the full facts here. This is a complete IT Governance failure not just a software 'glitch' caused by an aborted upgrade.

[harveysnana]

Well just checked and the missing cash paid in last monday has finally appeared in my account.
Good job as all the direct debits due to go out monday havr already been taken...which is usual procedure.
It looks like my account is back to normal.

Fingers crossed for anyone who still isn't back to normal.

Xxx

[rb10]

Originally Posted by cottager

The text of the Treasury Select Committee's original letter (to which Hester was responding) should be available from a link on this page:
http://www.parliament.uk/business/co...uter-failures/
but instead of a PDF opening I only see gobbledygook -- don't know if it's just me (a link to another letter, written to the FSA, opens fine).

The letter states:

27 June 2012

Dear Stephen

The Treasury Committee is extremely concerned about the current crisis at RBS.

Your first priority must be to resolve this crisis, caused by the failure of your computer systems. We would appreciate a report on how close you are to achieving this, both in the UK and for your operations in Ireland. We would like this by the end of the week.

We would be grateful for a more detailed response, preferably by the end of next week (unless you are still fire fighting), in answer to the following questions:-

• What caused this problem?
• What implications has it had for your operations?
• How many, and what proportion, of your customers have been affected (by brand and type of customer)?
• What weaknesses does this reveal about your capacity to identify risks and construct contingency plans?
• What initial assessment have you made about the quality and the apparent shortcomings of the bank's response?
• What assessment have you made of the enduring risk to your computer systems?
• Have staff savings or outsourcing, of your IT or other functions, contributed to RBS's difficulties?
• What effect has this had on the Government Banking Service system for which you and Citigroup are responsible?
• What plans have you put in place for compensation a) to your customers, b) to others affected?
• What criteria will be used to access compensation?
• What have you done to ensure that credit scores of your customers are not adversely affected as a result of your problems?
• What affect has this crisis had on the level of customer switching?

I will also be writing to the FSA and will send you a copy.

I will be placing this letter, and your response, in the public domain.

ANDREW TYRIE MP
CHAIRMAN OF THE TREASURY COMMITTEE

[stormymonday]

I am still struggling to understand how RBS\Natwest\Ulster Bank Batch processing staff could have lost access to its 'record of transactions'

The banks database itself should keep a history of all accounts transactions.

The databases internal logs would have before and after images of all inserted, updated or deleted records.

Properly written batch programs should also have all have inbuilt processing history reports, log displays of counts of records read, updated inserted and deleted, discard files to pick up rejected inputs, commit points details written to output , restart records for use in case of processing failure, formatted dumps from program ABENDS etc

Is Stephen Hester seriously suggesting that his IT department has lost access to all of this information ?

[scattykat]

Originally Posted by 1111

Hi, anyone know if payments are still being cleared on a saturday?

I've had a dd go out today and money go in today

[runninglea]

my cheques have gone in from last monday as have the business cheques.

That is a relief

All dd have been took so needed the cheques to clear to pay the dd

Hope other people get some luck today and actually receive their monies before all the monthly dd get taken.

This is the weekend when most dd will go out and when we need are cleared funds actually in our accounts

[paul_h]

After a few days of things looking reasonable, I have a new problem... and I'm starting to get a bit twitchy again.

I'm now waiting for a rather large business cheque which is overdue. However, the company concerned also banks with NatWest and is unable to issue the cheque due to the failure of internal money transfers.

They can't even give an estimate when the cheque may be issued, as NatWest is giving them no indication of when the missing money may appear.

[huckster]

Well I suppose if they don't know their true financial position, then they can't issue large cheques.

May be worth getting something in writing from this company, as to how they have been affected, why they can't issue the cheque and when they expect to be able to do so.

If you want NatWest to be in a position to look any financial loss you have suffered as a result of their failures, then you will need to get some evidence together.

[Deepfatfriar]

Originally Posted by stormymonday

I am afraid that Hester still appears to be speaking with a forked tongue. I have done a few mainframe software upgrades in my time and I can not see anyway that a third party batch scheduler upgrade alone no matter how bug ridden or ineptly implemented would deny an IT department access to the 'record of transactions' since that processing should be maintained by the company's own software, reading its own data files and written to its own output reports and logs. In order to lose that information someone in house or some program written in house must have deleted it. We are still not being told the full facts here. This is a complete IT Governance failure not just a software 'glitch' caused by an aborted upgrade.

I heartily agree with all this. Having been in a similar environment it appears that the basic rule of ensuring you can get back to square one before you undertake any upgrades has been ignored.

The notion that a piece of data(the schedule) a fundamental part of the banks daily business can be "just deleted" shows a total dereliction of duty and standards.

Some very severe questions need to be asked about the bank not only in it's technical standards but also those allegely associated with risk management and business continuity.

[paul_h]

Originally Posted by huckster

Well I suppose if they don't know their true financial position, then they can't issue large cheques.

To be precise, they can't issue the cheque because they are having problems transferring funds into their settlement account - the money has been transferred, but has not reached the destination account. I am sure I'm not the only one in this position, the company concerned handles large numbers of similar transactions.

[metalstardust]

RBS One account

Transaction history is screwed.
Transactions with no descriptions.
Salary paid in a day to early but a second uncleared salary showing for the correct day.

I work in IT and I know what a bunch of muppets our off shore resources are.

Off shoring does not save you money. RBS deserve every thing they get for this mess. Bring your IT back to the experts in the UK.

[Browntoa]

Originally Posted by Browntoa

no wages for me ,a large direct debit taken twice , once yesterday and again today

just about to ring up and question the direct debit , sadly sort of expected the wages not going in .....sigh

UPDATE:- wages appeared just now ???

direct debit probelm logged by them and to be checked

one account home page now says they are "aware of double entries but please be assured that the money has only been taken once"

pinch of salt comes to mind , not convinced

also getting no descriptions on some transactions as above

[cottager]

Originally Posted by Agricolae

Just a heads up - not all cheques will have shown up by tomorrow. This was the plan but as with so many other disappointments the time for all cheques to show in accounts has been pushed back.

Thanks for explaining.

Do you know what the situation is in reverse? -- cheques we have written to other businesses where we don't have monthly accounts. We just seem to have one or two more than usual which haven't been presented yet.

I suppose in one way it's quite nice these amounts are still in our account, but on the other hand I prefer it when they're presented/deducted quickly so I don't have to keep making allowances for them still to go out.

As it is, without contacting the payees I don't know if they're stuck in the system as part of the NatWest problem, or simply haven't been paid in yet.

If they have been presented, but NatWest haven't been able to process them, are they queued up to be dealt with, or will they have been returned to payees' banks for any reason?

[jimmyay]

My account is NOT up to date despite being told on Thursday it would be on Friday, then Friday it would be this morning.

The very nice but totally -in- the- dark employee i spoke to this morning could not confirm when the cheques for several thousand pounds i paid in in person at the bank last week would be paid , saying she had no record of them on their system but that i could get a "cash advance" against them from the branch. Not a great way to manage money is it . All my direct debits came out today and have bounced including my mortgage.

I am absolutely disgusted. I paid in money on the basis that the problem they had affected people who had paid in money between 18-20 June. This is incorrect. The problems are still outsanding and personally with my account, getting worse.

This is not a glitch, it is a huge corruption of and failure of their data probably from being hacked or a bug i would imagine, deep inside their systems. I am desperately worried about the integrity of information.

Just goes to show that credit in the bank is not the same as cash in your hand. It can be wiped and then its up to you powerless to plead your case. Abominable. Where is MY money Natwest?

[cottager]

Originally Posted by rb10

The letter states:
...

Thanks rb10. Obviously just me who can't see it for some reason. Tyrie's clearly expecting some more detailed answers: a pity it's not going to be before the end of next week, and then only 'preferably' (and then only if they're not still 'fire-fighting'). I hope there'll be a TSC hearing in due course.

[Debt_Free_Chick]

Originally Posted by paul_h

To be precise, they can't issue the cheque because they are having problems transferring funds into their settlement account - the money has been transferred, but has not reached the destination account. I am sure I'm not the only one in this position, the company concerned handles large numbers of similar transactions.

Hoping they're not using this as an excuse, though. If they go t!ts up, you'll not get paid at all!

[lexilondon21]

Just an update was told that my cheque (which shows a debit on my account rather than a credit and has wiped all available funds) would be sorted by today.

Have been told they are working on clearing the backlog and it should be resolved by Tuesday- am doubtful.

Have cancelled all my direct debits for this month and told them I would pay them directly as soon as funds clear. All the companies I have spoken to very, my gym, credit card, pet insurance etc have been extremely understanding.
x

[paul_h]

Originally Posted by Debt_Free_Chick

Hoping they're not using this as an excuse, though. If they go t!ts up, you'll not get paid at all!

I'm certain that's not the case, as a company, they are about as bona fide as it gets. I also know that they bank with NatWest, from previous cheques which ironically always clear within 24hr being same bank.

[runninglea]

Originally Posted by Debt_Free_Chick

Hoping they're not using this as an excuse, though. If they go t!ts up, you'll not get paid at all!

Its probably true as the company where I work paid a bacs run for 56k on 21st june and then same amount went out on 22nd.

Still waiting for the reversal to happen.

This has resulted in company being 30k overdrawn and thus not being able/willing to do any more payments.

Even if we pay by cheque then it will just make us more and more overdrawn.

Suppliers have so far been willing to beleive us but have told them we will hopefully do a bacs run on tuesday.

This is not looking likely