‹ BackHN Continuity

Thread

AI chatbots give wrong answers to financial queries 'most of the time'

157 points · 89 comments · 1vuio0pswjnm7

  1. ozgung · · focus · HN ↗
    This will be the same story for every industry again and again. AI is not good in x=Finance because models were not RL trained heavily on x=Finance capabilities yet. This is only because Big Labs have finance benchmarks lower in their priority list. Their first priority was solving programming because that gives the best leverage at this stage. As a side-effect they were able to solve a Millennium Prize problem, since theorem proving was also code.

    So finance advisors in the comments section of FT are falling for the classical pitfall. They assume there is something fundamentally wrong with “AI chatbots” that they can’t do finance ever. They mistake the current products in the market for the technology itself. In near future someone will release “Claude x=Finance” and their world will shatter.

    1. otabdeveloper4 · · focus · HN ↗
      Programming hasn't been solved by LLMs. AI chatbots give wrong answers to programming queries most of the time too.
      1. high_na_euv · · focus · HN ↗
        Not really

        I do find them useful when querying like "how todo xyz in abc"

      2. qgin · · focus · HN ↗
        Not most of the time
        1. johnnienaked · · focus · HN ↗
          More often than it's worth most of the time
    2. anfogoat · · focus · HN ↗
      > AI is not good in x=Finance because models were not RL trained heavily on x=Finance capabilities yet.

      Not sure what the "financial queries" here amounted to but I find it hard to believe LLMs will ever be to finance what they are to programming. Past a point, information related to the former is gatekeeped behind private institutions with special government granted privileges, while information related to the latter is freely available and open to anyone.

      1. ImaCake · · focus · HN ↗
        Plenty of code is not publically available, and yet that has not stopped them from RLHF'ing good coding bots. You can surely see the parallels to any other field including finance.
        1. shim__ · · focus · HN ↗
          Lots of bad(internal) code isn't accessible, the good open source stuff is quite the contrary is true in finance, lots of amateurs are sharing their thoughts where as actual deep analysis is kept to internal presentations.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.