‹ BackHN Continuity

Thread

AI coding has made CI a bottleneck, so we reworked ours to keep up

317 points · 408 comments · julian_digital

  1. torben-friis · · focus · HN ↗
    Here's my constant question:

    Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.

    Why have we not seen an improvements in products?

    While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.

    Is everyone just running full speed in circles or something?

    1. TrueDuality · · focus · HN ↗
      A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.

      Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.

      We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.

      We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.

      These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.

      1. jurgenburgen · · focus · HN ↗
        Cost is stable (or rising?), user-facing delivery is sameish, (some) engineers seem happier because they are allowed to gold-plate and prepare for future “human development of the hard-things”.

        Is that a fair summary?

        1. baq · · focus · HN ↗
          > Is that a fair summary?

          Not op but no.

        2. theshrike79 · · focus · HN ↗
          Cost is stable, what users see is "the same".

          Engineers are happier because they can spend more time on that extra round of polish that was just filed into low-priority "fix when time" Linear tickets.

          QA is happier because the stupid bulk operations are handled with AI and they can focus on the hard to find stuff that tickles the QA mind in a special way =)

          Same stuff, but with better (in-house) tooling and (most) people are happier. The Code Artesans aren't, but it's mostly their problem and they'll find companies that only do fully manual programming for specific niches.

        3. JimDabell · · focus · HN ↗
          Them:

          > a direct and meaningful impact on the quality and availability of our services.

          You:

          > user-facing delivery is sameish

          I don’t see how you can think that is a fair summary or a good-faith argument.

          1. cc62cf4a4f20 · · focus · HN ↗
            If the user doesn’t perceive/value the measured improvements in quality or availability, then it is legitimately sameish.

            Just because it can be measured doesn’t mean it matters.

            1. someguynamedq · · focus · HN ↗
              Quality and availability are directly user-facing. You're reaching to rationalize your beliefs
            2. JimDabell · · focus · HN ↗
              > If the user doesn’t perceive/value the measured improvements in quality or availability, then it is legitimately sameish.

              Right, but that is not what was said, so “summarising” it in that manner is not a good faith representation of their argument.

        4. ndriscoll · · focus · HN ↗
          At least with my team costs are down thanks to performance improvements, but that will depend on being large enough for that to make a difference.
        5. TrueDuality · · focus · HN ↗
          Internal costs are rising directly due to AI spend, we're still figuring out cost control and budgeting around this but are expecting this to basically become a new per-employee cost to factor in.

          I would say internal morale and productivity has improved. We have explicit policy guidance and a general collective distaste of pasting AI responses which is what usually sours me on interactions with people that are fully gargling the kool-aid. My CI experience has gone down from ~1.5 hours to less than five minutes and I can actually run our entire platform locally on my laptop again. My personal experience has improved at least.

          Externally, you're probably right that we're not really capturing user sentiment well about how our company's changes are impacting them. I can concretely say that there were several customers that were disappointed that our public change log notes have been about the same length. They were expecting us to go through our feature backlog faster because we we're starting to use AI.

          On the other hand, we have had fewer outages, reached our fifth 9 consistently over the past four months which is a new company record in the 11 years its been tracked. We have had fewer support tickets, our customer retention rate is the highest it has ever been in the company's history, and we're growing our internal teams. Based on the evidence I have available to me, I would say that we're healthier than before both from a user sentiment perspective and an internal morale perspective.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.