Thread

AI coding has made CI a bottleneck, so we reworked ours to keep up

317 points · 408 comments · julian_digital

torben-friis · 1790026444 · focus · HN ↗
Here's my constant question:
Everyone's going so fast that they keep hitting walls. Review, CI, product asking for things, whatever.
Why have we not seen an improvements in products?
While every post and thread feels like a 90's wall street office, the new android and iphone ship with fewer features than usual. No indie guys come up with a linux-sized alternative OS. Switch 2 remains unhacked. Windows takes 3 seconds to show the right click menu.
Is everyone just running full speed in circles or something?
1. TrueDuality · 1790027434 · focus · HN ↗
  A good chunk of what my company has been doing with AI falls into either burning down our known tech-debt and "easy wins" that no one ever had the bandwidth to approach... And improving / automating our processes. The former is having a direct and meaningful impact on the quality and availability of our services.
  Our QA, formerly a fairly frequent blocker of all our releases, are doing more in-depth reviews and catching issues earlier in our release process. They have become unblocked to the point they are actively chasing down work that starts to slip.
  We have cleaned up and tuned both our security alerts and operations logs and improved our tenant isolation in our service in a way that makes customer and formal audits SIGNIFICANTLY easier.
  We're setting ourselves up for faster human development of the hard-things. Our development environment and infrastructure are faster, cleaner, more auditable processes, and cheaper overall to operate.
  These fixes mostly don't show up in our product change logs, and definitely don't fall into "new features". It would largely be invisible to the outside world, but our costs are going down (though to be fair, not offsetting the spend on AI to date), internal productivity has improved, operational incidents are down, and customer satisfaction is up.
  1. lazyasciiart · 1790038889 · focus · HN ↗
    How is dev satisfaction?
    skulk · 1790039259 · focus · HN ↗
    doesn't really matter now, does it? Someone's gonna do the job, whatever it looks like.
    consumer451 · 1790040098 · focus · HN ↗
    A luxury of the past, for >90% of devs?
    consumer451 · 1790060635 · focus · HN ↗
    I want to add to this, because it's certainly not all doom and gloom for me, yet.
    If you are product-driven, life is pretty good, and maybe has never been better? (For the moment)
    netdevphoenix · 1790066000 · focus · HN ↗
    >If you are product-driven, life is pretty good, and maybe has never been better
    It has certainly been better for junior devs, mid devs, anyone looking for zirp era salary growth and supply demand ratios. For a senior code-adverse coaster looking forward to see his static salary get eaten by inflation, life has indeed never been better.
    consumer451 · 1790067243 · focus · HN ↗
    Yeah, at Corp, that makes sense. I was thinking more along the lines of scrappy startup silliness. "I am one person and I deal with any and all code to get my product built."
    From that POV, has it ever been better, assuming you can rise to the top of an ever more crowded market?
    netdevphoenix · 1790067940 · focus · HN ↗
    > "I am one person and I deal with code to get my product my built." From that POV, has it ever been better, assuming you can rise to the top of an ever more crowded market?
    Building a product is easier now, same way making a photograph is easier now. Making a reliable and high income living out of it though, it looks increasingly uncertain.
    In a world with abundant music and photography, just saying that you made your music using Reason/ProTools or you made your photograph with a Leica isn't enough. I believe the same will happen with software. Sales, product design, customer service and operational aspects like uptime will matter more than the software itself. Ironically, this is the kind of stuff the HN type highly dislikes.
    consumer451 · 1790070025 · focus · HN ↗
    > I believe the same will happen with software. Sales, product design, customer service and operational aspects like uptime will matter more than the software itself. Ironically, this is the kind of stuff the HN type highly dislikes.
    Agreed. It's not even a new thing. They tried to teach me/us long ago, starting by putting it into words we might more easily understand, such as "Customer Development" lol.
    When I realized that it was the sales guys that will have the power in the mid-term new world, I got genuinely upset. However, maybe they always had the power and I just didn't want to accept reality.
    Now, I believe the most desired hire in the near future will be the PMM.
    netdevphoenix · 1790073687 · focus · HN ↗
    What is PMM?
    sgarland · 1790077688 · focus · HN ↗
    Percona Monitoring and Management [0].
    ...or not. Probably not.
    0: <a href="https://github.com/percona/pmm" rel="nofollow">https://github.com/percona/pmm
    disgruntledphd2 · 1790080304 · focus · HN ↗
    It's probably Product Marketing Manager. So essentially selling the new stuff to new and old clients at scale.
    r_lee · 1790078244 · focus · HN ↗
    I'm guessing PM mistyped
    daveidol · 1790092903 · focus · HN ↗
    No it’s Product Marketing Manager. A common title and one part of the holy trinity (PM, EM, PMM) at bigger companies.
    brookst · 1790080158 · focus · HN ↗
    Some orgs use PMM for product management and marketing.
    consumer451 · 1790099757 · focus · HN ↗
    > A Product Marketing Manager (PMM) in tech is a strategic bridge between the product, sales, and marketing teams who defines who a product is for and why it matters.
    A PMM with a tech background must be worth their weight in unobtainium at this point.
    someguynamedq · 1790083451 · focus · HN ↗
    They always had the power.
    zerr · 1790067697 · focus · HN ↗
    If you are product-driven why not go into Product Ownership roles? It certainly will be more satisfying compared to prompt "engineering" clerk "career".
    consumer451 · 1790068008 · focus · HN ↗
    Well, my life is currently both, as I am on the smallest possible tech and product team. To me, extrapolating on having done this since Sonnet 3.5 to now, there will be no tech aspect to the job soon enough. There are many hundreds of billions invested in making that a reality, and they are succeeding.
    Until ~9 months ago, I spent most of my time being a "prompt engineer." Today I join a meeting, ten minutes later I receive the transcript. Then, I run a custom skill in my project, and I get Jira epics and stories to triage that are nearly perfect. Then, I run the second skill orchestration skill... some babysitting... and ~85% of the time that is all I need to do. Docs, code, unit and e2e, great UX... all there after every meeting, and basically two commands on my part.
    I see two to three to maybe five years before anything I have to offer, in any capacity, is completely cut out of the picture.
    I honestly don't understand how everyone is not on this same page. The labs are going to eat it all. The only reason I see for them to talk about "pausing," is because they finally realized that they are going to collapse the entire service economy around themselves at this rate: aka, the USA.
    zerr · 1790070122 · focus · HN ↗
    Not sure how long the current subsidized pricing would last. Also, the vast majority of software shops can't afford those subsidized prices anyway.
    baq · 1790076676 · focus · HN ↗
    It’s 10 devs with a small ai budget each vs 1 ex-dev now PM with a large ai budget - the choice is quite obvious for any decision maker who counts time and money
    brookst · 1790080348 · focus · HN ↗
    But assuming that one ex-dev with a large AI budget is highly profitable, why wouldn’t you convert the other 9 and give large ai budgets as well?
    I guess if the company has no opportunities for growth so they only need exactly as much output as one team? But that sounds like a company that’s doomed anyway, regardless of ai.
    baq · 1790083987 · focus · HN ↗
    You need to grow customers or contracts by 10x to fill the new pipeline and that's an impedance mismatch. Easier to let 9 go and hire later, especially since everyone else will be doing the same thing and there'll be a rather large pool of talent.
    How you identify talent in this new world is a different kind of a problem which I don't think people figured out still and won't for quite a while.
    brookst · 1790168945 · focus · HN ↗
    If an org suddenly has 10x the production capacity for the same price and can’t figure out how to sell it profitably, it is mismanaged and will die. That’s very much the case for a lot of businesses for sure, but there have been filters before (the internet, for instance) and we survived.
    baq · 1790076595 · focus · HN ↗
    Yeah it’s going to be very rough for any knowledge worker unless the governments decide it isn’t ok for the general public to be allowed to use the tech.
    It’s actually rough today with astra and fable, it’s just not been diffused enough. Tech workers like us see the writing on the wall, but a lot of others are blissfully ignorant.
    weakfish · 1790093363 · focus · HN ↗
    I honestly don’t understand comments like this because in my work, this would be a disaster. And before I get the comments about my harness/skills etc, I’ve tried many tools and harnesses and skills and all that earnestly and in good faith. I find use in it for doing the grunt typing labor, but letting it loose in ways described above have only ended in spending much more time cleaning it up than if I just did the work myself.
    I hate to sound pretentious, but I wonder if it’s a difference in complexity of work and problems being solved.
    consumer451 · 1790193673 · focus · HN ↗
    I just reviewed this thread, and thanks for an opening to say something I had realized I missed.
    > I hate to sound pretentious, but I wonder if it’s a difference in complexity of work and problems being solved.
    It is about complexity, at least for me. I am working on b2b SaaS.
    There are times where even using LLM assistance, I spend weeks or months working on a tough problem.
    However, the <show product get feedback> loop is now nearly entire automated, when it does not involve some actually complex problem, which are most of the meetings.
    nradov · 1790044572 · focus · HN ↗
    Which devs are you asking about, the humans or the AI agents?
    Tempest1981 · 1790049973 · focus · HN ↗
    Start a few dozen agents, and take a long lunch, right? No?
    UltraSane · 1790050772 · focus · HN ↗
    I assume they are very satisfied with their paycheck.
    notyourwork · 1790053392 · focus · HN ↗
    No better:
    arathis · 1790056448 · focus · HN ↗
    If we cared about the satisfaction of the worker, we’d never have fast food.
    brookst · 1790080511 · focus · HN ↗
    If we cared exclusively, you mean.
    As soon as you care about multiple things that are sometimes in tension, this kind of syllogism collapses.
  2. verpeteren · 1790057923 · focus · HN ↗
    This is the way!
  3. jurgenburgen · 1790066157 · focus · HN ↗
    Cost is stable (or rising?), user-facing delivery is sameish, (some) engineers seem happier because they are allowed to gold-plate and prepare for future “human development of the hard-things”.
    Is that a fair summary?
    baq · 1790068418 · focus · HN ↗
    > Is that a fair summary?
    Not op but no.
    theshrike79 · 1790070651 · focus · HN ↗
    Cost is stable, what users see is "the same".
    Engineers are happier because they can spend more time on that extra round of polish that was just filed into low-priority "fix when time" Linear tickets.
    QA is happier because the stupid bulk operations are handled with AI and they can focus on the hard to find stuff that tickles the QA mind in a special way =)
    Same stuff, but with better (in-house) tooling and (most) people are happier. The Code Artesans aren't, but it's mostly their problem and they'll find companies that only do fully manual programming for specific niches.
    JimDabell · 1790073572 · focus · HN ↗
    Them:
    > a direct and meaningful impact on the quality and availability of our services.
    You:
    > user-facing delivery is sameish
    I don’t see how you can think that is a fair summary or a good-faith argument.
    cc62cf4a4f20 · 1790083043 · focus · HN ↗
    If the user doesn’t perceive/value the measured improvements in quality or availability, then it is legitimately sameish.
    Just because it can be measured doesn’t mean it matters.
    someguynamedq · 1790083333 · focus · HN ↗
    Quality and availability are directly user-facing. You're reaching to rationalize your beliefs
    JimDabell · 1790085116 · focus · HN ↗
    > If the user doesn’t perceive/value the measured improvements in quality or availability, then it is legitimately sameish.
    Right, but that is not what was said, so “summarising” it in that manner is not a good faith representation of their argument.
    ndriscoll · 1790081628 · focus · HN ↗
    At least with my team costs are down thanks to performance improvements, but that will depend on being large enough for that to make a difference.
    TrueDuality · 1790106974 · focus · HN ↗
    Internal costs are rising directly due to AI spend, we're still figuring out cost control and budgeting around this but are expecting this to basically become a new per-employee cost to factor in.
    I would say internal morale and productivity has improved. We have explicit policy guidance and a general collective distaste of pasting AI responses which is what usually sours me on interactions with people that are fully gargling the kool-aid. My CI experience has gone down from ~1.5 hours to less than five minutes and I can actually run our entire platform locally on my laptop again. My personal experience has improved at least.
    Externally, you're probably right that we're not really capturing user sentiment well about how our company's changes are impacting them. I can concretely say that there were several customers that were disappointed that our public change log notes have been about the same length. They were expecting us to go through our feature backlog faster because we we're starting to use AI.
    On the other hand, we have had fewer outages, reached our fifth 9 consistently over the past four months which is a new company record in the 11 years its been tracked. We have had fewer support tickets, our customer retention rate is the highest it has ever been in the company's history, and we're growing our internal teams. Based on the evidence I have available to me, I would say that we're healthier than before both from a user sentiment perspective and an internal morale perspective.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.