We'd like to remind Forumites to please avoid political debate on the Forum... Read More »
We're aware that some users are experiencing technical issues which the team are working to resolve. See the Community Noticeboard for more info. Thank you for your patience.
📨 Have you signed up to the Forum's new Email Digest yet? Get a selection of trending threads sent straight to your inbox daily, weekly or monthly!
Techies, what's with the RBS computer?
Options
Comments
-
You would hope so, but this is more like openheart surgery - they have done several days business since this started but the problems continue. So it is not a case of simply switching off and starting with known good data, becuase that data, although once good, is now stale. The system has to be put right while still running.cookie_monster wrote: »Can they not do a system restore? :rotfl: or maybe ctrl alt delate :beer:
I begin to suspect that someone questioned why certain data was being collected and kept in a certain way without ever being used. Stopping collecting that data would ultimately reveal why it was being kept ....Hi, we’ve had to remove your signature. If you’re not sure why please read the forum rules or email the forum team if you’re still unsure - MSE ForumTeam0 -
I am with One Account - owned by RBS. My online bank account is frozen up to Thurday, I need to pay a holiday balance but it won't process my payment. Fix it RBS!
I blame Apple, apparently RBS didn't upgrade to the latest version of iBank and then tried to run a web page with Flash on!0 -
Little story...
Once upon a time xyz bank used paper-based systems, which worked fairly well but were extremely manually intensive and painfully slow. Skipping through the mechanical adding machines which helped speed up the adding up process, early computers could be used to store account information and speed up the processing.
These computers were so successful, as they increased in power, new functionality was added to the systems. As the core of the system was already up and working reliably, adding new functionality meant interfacing with a system that was already defined, so was a retrofit. Technology moved on, so the new system was written and ran on new hardware, with a different OS, in a different language. Layer upon layer of retrofits to accommodate ATM's, internet banking, ISAs, pensions, debit cards, etc etc mean the systems are a hodge-podge of technologies spanning decades. Success means buying up other banks, then integrating their completely different systems together. Complexity doesn't just double, it is squared!
Some ancient servers get virtualised onto newer ones, but all the time some parts of the kit is going out of warranty/support so the systems get patched together, layer upon layer. Jobs run on schedules hoping previous jobs have completed as they depend on that day's data, yet something else further downstream relies on the data of the job you're just running. In the end it becomes so complicated that no single person, no single department even can completely map out how it all works. And as long as it does all work, it is business as usual.
And because it does all work ok most of the time, the business starts to assume that the people who look after it must be surplus to requirements - get rid of them and profits go up! So trim them right back, or offshore them, and reap the rewards.
The board are not technical people, and have no way to understand how complicated their IT systems are. As a comparison, if you show a duck a page of algebra, it has no way to even comprehend how complicated it is - this is the position the board and shareholders are in. They literally cannot comprehend how complex their systems are. And if you cannot comprehend something, it can be easier to discount it.
So you end up with an insanely complicated system which is so complicated with dependencies on hardware, software, timed events, transactionality (ie if one account debits £50 another credits £50 - the £50 never gets lost or duplicated), data mappings, datasets being imported and exported in hundreds of formats, each format with different codepages and mappings... as said above many IT people believe systems are just too complicated to ever work!!
Eventually *something* goes wrong. That *something* propagates harmlessly through the system until one corner breaks undetected, which causes other problems in the data integrity, which get backed up as a part of everything whilst the rest of the world carries on. Left unchecked for a few days, weeks or months the system is critically ill, but the good data is infected by the bad data, chokes and falls over in a slow but critical way.
Somebody notices, a fix is planned. Unfortunately the good data is so choked up with the bad, the only way to make sure every one of the BILLIONS of records and data sets are correct is to go back to before the *something* and to run every single job in order with the revised code, through all the complex systems. And that is a HUGE job, taking days.
That's my hunch, anyway. Doesn't matter what the *something* was, it could be a magic date, it could be a stray neutrino kicking a 1 to a 0, it could be a small, simple dataset sent from an overseas subsidiary that was entered on Excel of a Mac instead of a PC, and someone copy/pasted without realising the date problem, it could have been one of a zillion things.
This is the problem with such complex systems, the opportunity of a small error to cause massive damage is huge. Frankly it is inevitable. It is when, not if.0 -
-
They should try switching it off and then on...0
-
Little story...
Once upon a time xyz bank used paper-based systems, which worked fairly well but were extremely manually intensive and painfully slow.
.....
This is the problem with such complex systems, the opportunity of a small error to cause massive damage is huge. Frankly it is inevitable. It is when, not if.
An extremely well-articulated description of what's going on not just in our banks, but lots of other major enterprises and government departments. It's about data, and that data is spread all over the systems and indeed those of partner organisations. Once a fault has rippled across multiple systems it's not simply a case of restoring from a single backup and besides, the backup itself may well be corrupt.
This is one of the big threats facing our society now, and it's nothing to do with hacking, viruses etc. The generation that understood these systems has already started to retire (many taking early retirement); add to that offshoring and the fact that complexity keeps increasing and you've got a toxic mix.Long-haul Supporters DFW 120
Debt @ LBM (October 2007): £55187
Debt Now (April 2014): £0
Debt-free-date: [STRIKE]July[/STRIKE] April 2014 :j:j:j0 -
Does anyone think that perhaps someone at RBS believed the telephone caller from the Indian sub-continent, who told them they were infected, and then went to the suggested website and installed Logmein?:rotfl:
Sorry for the looooong sentence!Move along, nothing to see.0 -
Billy-no-Money wrote: »An extremely well-articulated description ...
... The generation that understood these systems has already started to retire (many taking early retirement); add to that offshoring and the fact that complexity keeps increasing and you've got a toxic mix.
Thank you for your kind words!
I agree that we're not just talking banks here, but all large systems. NHS, HMRC, DWP etc (and pretty much by definition any government IT project) will be huge and complex and evolved over time. And it will be a 'when, not if' before the next problem.
As a profession, IT professionals have been too successful! We say 'yes, we can do that' and enjoy the challenge. The business then assumes it must be easy and dreams up new changes. And that is how it should be - those changes give a cometitive edge, which means more business, which means more money, so everyone gets paid. IT is the tail, not the dog. But every single change, including the ones that seem small sometimes, have a cumuative effect on complexity and stability/maintainability. It becomes one big game of buckaroo!
You're so right that a lot of the original architects are leaving the industry, and that fresh graduates seem like a good idea as they are cheap. But as with any discipline, it is the years of practice that give you good solid work. Uni cannot even start to give students a clue how real-world IT systems really work. Most will never step inside a datacentre, most will never even understand how to build ruggedness into their systems (eg messaging for asynchronous processing - and the benefits and dangers thereof - and marshalling limited system resources appropriately...in a massively parallel situation), but be passablly capable with whatever the language of the time is. Why would you ever need to rebuild the indexes on an Access database or a spreadsheet - yet large data systems require some deep specialities like this which fresh graduates could never be ready for fresh from Uni, whatever continent they are on.
Now this is where some companies have an edge over others. Companies full of techie types with an appropriate corporate culture are all very aware of how complicated systems get and tend to be quite defensive of core systems whist completely aware they may need a lot of attention. Non-techie businesses often see IT as an inconvenience and as something to spend as little on as possible if the systems are still working.
I know of one company who turn over millions with an IT dept of 2, 1 of which isn't very switched on and knows nothing of the systems beyond patching up email boxes. The business refuse to spend any money beyond firefighting on their IT systems, yet their whole businesss is built around one overcomplicated and terribly written system that is kept alive artificially. It is so delicate it is unmaintainable, so every piece of data coming in is scrubbed for purity before being allowed in (and it arrives in multiple spreadsheets from customers with 18000 elements in each one, so once in a while... ).
Their chief techie points out to the business he is leaving in 2 years and that this system needs sorting, he has even written an update already, it is waiting in the wings, but that company will not spend the time and effort into implementing and testing the replacement system (which is massively simpler for many reasons). The servers run in a cupboard with a large domestic air conditioner which drains its water into a garden rain butt, which is emptied every few day manually into plastic bottles which are carried to the loo. You can see the tidemarks on the carpet where people forgot for a long weekend). If I was a shareholder I would be terrified and livid, but the CEO is likely to split and run just before the techie, leaving the company in the doodoo.
This is a case of a relatively simple complex system not receiving the love it needs where it is when, not if it fails catastrophically. And no amount of shouting when there is a problem will compensate for the fact it is down, and if it even can be revived (it really is complex and uses extinct technologies waaay out of support) it will take days. They may as well empty their desks, in fact, as when it goes, they are effectively jobless overnight - but resolutely refuse to spend the time and money defensively. Crazy, but scarily common.
And this attitude is multiplied across business of all sizes the world over. Take the money today and hang tomorrow. Indeed it is the same greedy/short term attitude (but with money not data) that led us into the recession we are all currently enjoying. Thanks, short-termists!
What should we do differently? Listen to the techie guy. Learn his language if you have to. When he tells you your systems are too complex to be reliable, think what it'll cost you if things go wrong. You can have wonderful service contracts and insurance out your ears, but the reputation damage is extreme, if you caan recover at all.
Mature engineers like to be paid, but love their systems more. Their systems are their gardens carefully tended and weeded, they hate thinking they may be at risk. The young ones are busy showing off their toys whilst the solidity of the systems rests with the old dogs. Listen to them, and listen carefully. And do everything you can to keep the old boys around - you don't realise that they hold your company's beating heart in their hands!
Hmmm sorry, bit of a tirade, less eloquent than above, less focussed. Just talking aloud ;-)0 -
Billy-no-Money wrote: »An extremely well-articulated description of what's going on not just in our banks, but lots of other major enterprises and government departments. It's about data, and that data is spread all over the systems and indeed those of partner organisations. Once a fault has rippled across multiple systems it's not simply a case of restoring from a single backup and besides, the backup itself may well be corrupt.
This is one of the big threats facing our society now, and it's nothing to do with hacking, viruses etc. The generation that understood these systems has already started to retire (many taking early retirement); add to that offshoring and the fact that complexity keeps increasing and you've got a toxic mix.
This is all absolutely spot-on. These systems are not only immensely complex, but their core components are creaking antiques running on IBM or ICL mainframes, and 100%-proprietary. The only people who can spot potential problems before they arise, or fix them quickly, are programmers and operators who have been working there for decades, and for these people the job is not so much science as it is intuition: these systems have seeped into their DNA, and they don't so much diagnose problems as feel them with the seat of their pants.
So where are these people now? Retiring or dying for years, and not being replaced because no bright young IT graduate wants to work with this carppy old technology, let alone spend the years it takes to approach the kind of sympathy with the systems that the oldies had. And as if that weren't bad enough, what do you think happens to the remaining old stalwarts when the entire operation gets shifted to Bangalore?Je suis Charlie.0 -
Man, i love the smell of antique code. It's got that musty "GO TO 20" feel to it
I worked as a graduate for a firm that sold enterprise-level software... i think it took me 2 years to get a good idea of how it worked and after 4-5 years to really 'feel' how it worked
Stupid 4000 table system with layer upon layer of interacting modules and servers, all on a proprietary language0
This discussion has been closed.
Confirm your email address to Create Threads and Reply
Categories
- All Categories
- 350.9K Banking & Borrowing
- 253.1K Reduce Debt & Boost Income
- 453.6K Spending & Discounts
- 243.9K Work, Benefits & Business
- 598.8K Mortgages, Homes & Bills
- 176.9K Life & Family
- 257.2K Travel & Transport
- 1.5M Hobbies & Leisure
- 16.1K Discuss & Feedback
- 37.6K Read-Only Boards