‹ BackHN Continuity

Thread

What happens when you analyze your favorite college football team like the CIA?

64 points · 35 comments · adam

  1. ungreased0675 · · focus · HN ↗
    Fascinating project. How are the probabilities being measured? I’m skeptical of AI analysis because my past experience is that it appears to be analyzing, but behind the curtain there’s not even a fraction of the same rigor a human analyst would have done. In other words It’s very shallow but looks deep.
    1. adam · · focus · HN ↗
      In this model the probabilities were measured by an ensemble of LLM's. We give them context and ask them to follow a specific methodology for making a forecast, and then aggregate their forecasts into a "crowd" forecast. We can get input from humans as well to compare/combine them further, but in this particular instance I just used our AI Forecaster.

      You're right that a human analyst - especially a very experienced one - is going to be able to still do a better job of analysis most of the time, but the AI is useful for having loops to re-examine the question, consume and curate new information, etc. We've built the system with that assumption: let humans do what they're best at, let AI do what it's best at...

      1. ungreased0675 · · focus · HN ↗
        It seems very likely then that the results are hallucinations. You can’t prompt an LLM to follow a certain analytic methodology, it can’t.

        But, it can simulate the output of a methodology, which is not the same thing.

        1. toyg · · focus · HN ↗
          I think this comment encapsulates the fundamental state of AI at the moment.

          The industry believes very strongly that this simulation of methodology is basically as good as the real thing (or at least "good enough" in most cases).

          This belief is the fault line between the bullish and the bearish; it requires a leap of faith that not everyone is willing or capable of taking.

          1. ricksunny · · focus · HN ↗
            fake it till you make it. (looks around at real life).
            1. lukan · · focus · HN ↗
              It seems like it. My agents do real work and the code works.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.