using AI for pension projections

Nebulous2

https://forums.moneysavingexpert.com/discussion/comment/82131750#Comment_82131750

Thanks - I think. I'm not a professional AI user, its a toy for me - but I think you've given me enough pointers to see where I might have gone wrong.

I wanted to walk up a hill not cycle it.

Coincidentally I did a lot of cycling, live in an area with a myriad of small roads, and at one point used to plan a cycle ride to fill in gaps in my strava heatmap. If I go back to that I could try giving access to all my hundreds of garmin and strava rides to see how well it could plan rides.

I'm a layman, with limited access to AI tools. I pay for microsoft 365 so get copilot use as part of that. I have a recent model google phone, so get a better gemini than people who don't have a pixel phone.

I ran both scenarios through both of them. Both performed really well in France, and abysmally at home. There are a lot of hillwalking resources available, with detailed routes, times, difficulty levels etc. Most of them would assume initial access by car, there would be very limited detail about arriving by public transport. There would also be quite a lot of stuff on the internet about bus services, getting round Scotland by bus. What would be more difficult to find is an interaction between the bus route and the walking guide. I gave copilot access to the bus timetable, had it open and told it to refer to it, but don't know what else I could have done.

So to come back to the initial point of this thread - I've kind of gone down a cul-de-sac by describing my own frustrations. I've tried some simple pension / finance planning using AI. Giving details of my own situation and asking where I should put some available money. ISA v SIPP type of stuff. It came out very strongly in favour of an ISA. I then realised it hadn't considered the tax-relief available in the SIPP. There were other errors, but that is the only one I can remember. When I challenged this it altered the figures, but still doubled down on its preference.

My issue with that is that I knew it was wrong - I haven't been on MSE this length of time without picking up some knowledge. What about people who don't know that? We get a lot of people on here, often well-paid educated people with a very rudimentary knowledge of personal finance. ISA v SIPP is a fairly common question.

If I can only trust it when I already know substantial parts of the answer - and can correct it repeatedly to shape its response - what use is it to me?

butterflymum

If I can only trust it when I already know substantial parts of the answer - and can correct it repeatedly to shape its response - what use is it to me?

It keeps your thinking skills oiled and your grey matter active…I know you would rather it be useful for the task you set it, but these side uses, have worth in their own right. As with anything new, go in with eyes wide open, as you have done, and accept there will be shortcomings, as you have done, but at the same time consider points or aspects arrived at that you may not have thought of, but which just might be useful……then do your own further research on those 😁.

gm0

AI isn't solid on the tax code. And a consumer tool which is very accurate and highly optimising for minimising your own tax isn't on HMRC's list of desired xmas gifts.

It has long been the case that general investment and finance books exist. And in the larger US market there is a body of literature and web content which at a point in time at least covers us taxes 401k roth conversions, tips and many other things. And then goes off as the tax code gets added to year on year on year.

UK focused equivalents are (relatively) few and far between. Hat tip notwithstanding to monevator, pensioncraft, bogleheads of old etc.

It would seem obvious (to me)

That the rule here which applies is "garbage in garbage out". LLMs are not going to be good on the probability of symbols in sparsely populated areas of the training data used to bake the x bn parameter model. Letting them google and loop around a notepad document of unstructured text (as is the current pattern of agentic AI) - won't solve that. For the same reason

I find a chat with an LLM occasionally helpful around strategy and options and things to consider. Would I trust calculations or the gushing "you are so right already" endorsement of anything you mention initially. Of course not. Stevenson's rocket. Or the dog in a tutu. It's interesting it does it at all. One day it will be something. We may go via the APT tilting train (as parked mouldering at Crewe).

af1963

So you need to give your favourite LLM a data source and tell it how to interpret it. If I wanted to plan a cycling route around Scotland and a bus route, I'd give Claude Code access to my computer's Chrome browser and tell it to go ahead and use suitable tools to access, for example, your bus route, Strava, Veloviewer, Google Maps, cycling route reviews and so on. Then I'd get it to create me route guides that I can read on the screen to see if they make sense. Once I am happy with them, I'd get it to create the route or GPX files for my bike computer and transfer them to it.

Seems like more work than just reading up on it and planning it yourself. Especially as you need to "read them on screen to see if they make sense" after it's done.

OK: if there's a repetitive task that a tool can be trained once to subsequently perform on many occasions, it may justify the extra effort.

Where a fact is not known, an LLM may make something up that sounds plausible.

And if you're asking about something you couldn't already do for yourself, how would you ever know ?

ComicGeek

A client of mine changed something part way through building a new house. Didn't tell anyone, but the Building Control inspector noticed the problem at the end of the project.

Client decided to try chatgpt for a response - it produced a lovely little report confirming what was required to correct the non compliance, and Building Control accepted it. The problem was that the chatgpt response was wrong - but neither the client or inspector had sufficient expertise in that field to realise it was wrong, and weren't happy when I pointed out that it was wrong.

Client's conclusion is that they can save money by not appointing me in the future to advise them on things, as they will just rely on chatgpt. My conclusion is that I can't wait to retire and have contact with (fewer) idiots.

All I know after working for decades as a consultant is that I can ask a question and get 10 different answers from 10 highly qualified and experienced people - and it doesn't mean that 9 are wrong and only 1 is right, sometimes none are right and sometimes 10 are right.

Dead_keen

AI isn't solid on the tax code

Agreed. I use ChatGPT generally for tax stuff (typically 5.6 Sol Light to start with) and it is surprisingly good at UK tax. But equally it gets things wrong big time because it chooses the easy answer or misses nuances.

I also use other models with my own KG and RAG for relatively narrow areas of tax. When I use weaker models with this, I get very good answers. And I have the experience to judge those answers.

Especially as you need to "read them on screen to see if they make sense" after it's done.

If I do a cyclosportive or ultra or whatever, I am going to look at the map first to see where the cafe stops / aid stations are first rather than just plug them into my bike computer / gps and hope I don't run out of cake / water.

OK: if there's a repetitive task that a tool can be trained once to subsequently perform on many occasions, it may justify the extra effort

It doesn't take long to do it these days. With a bit of experience, a lot can be done in five to ten minutes for something straightforward, but if you want to take account of personal preferences, it takes longer.

For example, I have a personalised travel agent. On a dog walk, I'lI talk to my phone and say I want to go to Copenhagen for three days later this month, it asks a few questions and I talk back to it. A few minutes later it suggests a priced itinerary or two based on my preferences (e.g. flights are cheaper on Friday but the hotels are more expensive so do you want to go midweek, leaving Tuesday, especially as it looks less likely to rain). To do so it looks at my calendar for dates I can't do, knows the airports around me and how I can get to them, actual flight times and prices, actual hotel availability and prices, what's on in the city, the weather and so on. At the moment I book the flights and hotel myself but on other things it completes online forms for me (with me doing the captcha and clicking submit). It then produces a detailed itinerary, maps and suggestions for each page on a private website. I can then chat to it about seeing stuff (no to the little mermaid, yes to the Rosenborg if it is not raining).

Does it take the fun out of it? Not for me. Have you tried using skyscanner trying to go from London to Basel to Copenhagen to Bristol when the site says the price of the first flight is £28 but are actually £189 when you click through and there are no direct flights on a Tuesday but you could go via Amsterdam if you are happy to wait 4 hours in the airport or Frankfurt if you are happy to wait 3 hours. For me, the fun bit is deciding where to go and then going there. Not the admin of deciding what flight to get. I'd rather do a short voice note saying what I want and then, five minutes later, get some sensible suggestions and really start planning.

Going back to the AI for financial projections, it's similar. Spend the time getting it set up properly with Claude Code or Codex (not just ChatGPT or Claude's chat), get the data sources in automatically, test it and refine it. Do it right and there will be no maths errors or hallucinations. But that will take more time and inclination than most people want to spend.

phlebas192

One of the main problems with AI is that the consumer variants are programmed to ingratiate itself with you. So if you ask it to validate what you have come up with then it will likely come up with reasons to agree with you.

OTOH, if you presented your intention as that of someone you disagreed with and asked the AI to find reasons why the idea was wrong then it just might come up with good arguments against your reasoning.

Hint: anything related to the so-called 'SWR' is based on the very obvious survivorship bias of using US (or UK) data. Repeat the analysis using other early 20th century major economies such as Russia, Germany or Japan and see what the results would have been.

Mistermeaner

I tried using Gemini to do some pension planning and it was awful. It got basic things wrong like the age I could access my pension. When I corrected it I got the “oh yes you’re right …” response and it would re do but still had multiple errors , like basic errors eg missing off state pension in later years

essentially if it can’t get basic stuff right I have no faith in any of it so back to the manual spreadsheet for me

imo owning the spreadsheet and the formulas is key as that it what helps you really understand your numbers and also tweak and play meaningfully …. Build one have a play see what it says. A month later start from scratch build another - look at both for consistency etc

Ai can be useful for certain specific things to help you eg “provide a table for global stock market growth per annum for each year from 1950 onwards” etc

Cobbler_tone

As AI is a hot news topic, I revisited this and I don't think this thread will age too well. It is the equivalent of people currently playing with an abacus. Agentic AI won't be writing your project plan, it'll be launching it.

I think we have heard and seen enough that not all developments are likely to have positive outcomes, with some unintended consequences.

snowlaser

If I can only trust AI when I already know substantial parts of the answer - and can correct it repeatedly to shape its response - what use is it to me?

But that's exactly how you'd use a human advisor as well - you might say to a travel agent "I want to go on holiday to a nice hot beach in Europe" And they say "how about Spain?" and you go "oh no I've been to Spain a lot, what about something a bit different?" and they say "how about Greece then?" and you keep chatting and refining the answer until you get what you want.

If you simply went into the travel agent, said "I want to go on holiday to a nice hot beach in Europe" and then got cross when their first suggestions wasn't EXACTLY what you wanted and declared all travel agents do is give you errors and hallucinations that would be missing an opportunity.