‹ BackHN Continuity

Thread

The problem is not AI code, but not knowing about system architecture or intent

388 points · 240 comments · zazuke

  1. zero_shift · · focus · HN ↗
    I was reflecting on this on Saturday in an unformed way, trying to trace the lineage of a decision made at work.

    The code change itself doesn't specifically matter. But suffice to say, it was about an AI feature in one of our products.

    The code was stamped by Claude driven by a prompt. The prompt was for a ticket generated with the Atlassian AI integration. Atlassian had digested docs made with AI. The docs came from strategy memos I'm 90% sure were written entirely by Claude.

    The strategy was chosen by management at the urging of exec leadership. The execs now communicate mostly via AI written memos. I do not know how they make decisions, but they reference tech influencers, market conditions, customer expectations.

    This gave me pause. Who had actually made the decision then? Arguably there has been several layers of human review, but the actual source of the decision was hard to pin down.

    We were not building the feature because we wanted it. We were building it because we thought other people expected it.

    Perhaps reflecting on the state of the market, I thought, could indicate who was actually in control.

    Where do investor and customer expectations come from in 2026? It is very murky, at least in tech. There appears to be hype. Some hype comes from true believers, some comes from cynics. But both respond to market incentives that reward bigger and bigger claims.

    Where does the market's "action" come from? What is the driver?

    Investors do not really seem to understand what the tech is or its limitations. Some are passive operators. Others are just responding to the overall froth and speculation in the market - which becomes a runaway feedback cycle.

    This left me lost.

    Nobody in this ecosystem, I thought, is actually in control here.

    Nobody is actually orienting work and action to real, concrete goals. It's all based on speculation and anxiety about the future.

    So it is not only that nobody understands what the code does. It is that we cannot, or at least I cannot, explain the motivation. There doesn't seem to "be" any form of "intention" in this environment.

    It has all been hollowed out, replaced either be inscrutable machines, or inscrutable incentives.

    Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.

    1. BOOSTERHIDROGEN · · focus · HN ↗
      For me, at this company, which is struggling to develop a new value proposition, everyone is unfortunately using AI in a blatant way. It’s even worse when executives and management pressure us to move fast. Presentations, documents, and mockups all use AI, rinse and repeat, feeding it context, but somehow, nothing is moving. It’s just staying static.
    2. visarga · · focus · HN ↗
      The invisible hand of the market is in control, and we don't understand it either.
      1. Isamu · · focus · HN ↗
        You could look at it as a kind of evolutionary pressure, where the AI driven companies that have no tether to reality are eventually extinct. Companies that are able to get a handle on the direction they are being dragged may have a fighting chance of survival.
      2. [deleted] · · focus · HN ↗

        [deleted]

    3. ashleyn · · focus · HN ↗
      Anecdotally, I know people in the industry who tell me their job is "so easy" now because all they need to do is be a meat proxy for claude and take home the paycheck.

      Question from someone written in AI? Just answer it in AI and send it back. What was it actually about - who cares? Bug comes in? Post the jira link in claude and don't even bother prompting anything else. If something critically breaks - well, eh, we'll deal with it then. FIRE (early retirement), a prediction their layoff is inevitable, and investing aggressively so you can finally escape actually working are often invoked in the same breath. Everyone feels like they're just trying to punch the drywall and grab as much copper wire out of the walls as they can until they're finally let go and/or the whole company fails.

      There's a great deal of nihilism and cynicism in the industry currently, and it feels like LLMs are just greatly enabling it. Where you would've done a halfassed job previously, you'd now do an unchecked AI job.

      1. sarchertech · · focus · HN ↗
        > prediction their layoff is inevitable, and investing aggressively

        If that happens on a large enough scale those investments won’t be worth much.

        1. praptak · · focus · HN ↗
          The problem with the tragedy of the commons is that everyone knowing it will happen is not enough to prevent it from happening.
      2. augment_me · · focus · HN ↗
        Well it's the incentive given. A bit like the original comment writes above. If your performance and impact are measured by charts, meeting outputs, some kind of abstract KPIs and other means that traditionally kind of worked because people had agency, you are incentivized to just slopmaxx it. Otherwise you will fall behind your colleagues who slopmaxx and have a better number on their abstract KPI.

        If you try to do good work, you won't be able to keep the pace with the slopmaxxers, which will mean you get laid off earlier. You will also be swamped in slop review.

        If you get called out on some issue or shitty implementation, you can just make Claude abstract it away behind more complexity to the point where people have a hard time doubting you because they don't have time to get into the details and verify things.

        Grab as much as you can, invest in immovable property and other shit that has value after a stock crash and enjoy the ride

      3. otikik · · focus · HN ↗
        > Everyone feels like they're just trying to punch the drywall and grab as much copper wire out of the walls as they can until they're finally let go and/or the whole company fails

        That’s very graphic.

        The thing is, I already felt a bit like that before AI. It’s just money. This quarter’s. The rest doesn’t exist. Make flashy features faster and you’ll be promoted. Make things carefully so that they last and can be maintained easily, and you are be ignored.

        I say this not as an engineer that tried to write maintainable software and now is butthurt, but as an (ex)manager who tried to promote such people. And now is butthurt. I found myself telling my engineers that if they wanted to get promotions they had to prioritize the shiny and skip the rest.

        I hated it.

        Now I am back to Engineering. I do use AI heavily at work because no one cares but if my output is “too slow” I will look bad. I at least try to raise concerns when I see them. The answer tends to be “yeah, we can’t afford to do that properly now”.

        I use AI way more sparingly for my personal open source stuff. Because I care.

      4. mhurron · · focus · HN ↗
        > a prediction their layoff is inevitable

        It probably is. Where I work it has been made clear, as in actually stated by leadership, that any process that does not include AI input is to be considered broken and needing to be fixed. It doesn't matter if it works, if it is 100% human made and maintained, it is broken and needs to be fixed by injecting AI in a critical place. You should be in a position that if you lost access to the AI tools, you are unable to proceed.

        This is the step towards replacing people. You don't need highly paid people in that process, you just need someone who can speak a language the AI tools can transcribe. This is no different than moving from a codebase or process that is tightly coupled to a specific technology to a more generic process so that you can easily and quickly change the backend tech on a whim. The technology being removed is the people. The AI is important, you are not.

    4. yearesadpeople · · focus · HN ↗
      Exactly the same issues we are facing right now. And, well why wouldn't it end up like this? The promises of going 'faster' while ignoring the organisational processes capability to handle it is - and continues to be - the greatest delusion of this new age. But, even in the face of all of this, we are seeing leaders asking for more. At some point the culture will collapse in on itself, and no one will understand anything anymore.
    5. techpression · · focus · HN ↗
      Very good post, thanks! You remind me of the countless times I've heard "we need to be AI native", "use AI", "pass it through AI for review".

      AI has become the thing you do, and what you do it with, to achieve it. It's a self-fulfilling chicken that is an egg that is a chicken.

    6. saadn92 · · focus · HN ↗
      > The execs now communicate mostly via AI written memos

      It bothers me to no end when I get an AI written response, especially from the executive team or any one of my co-workers

      1. wholinator2 · · focus · HN ↗
        Professors in grad school are also simply replying with chatgpt answers to any and all questions, including reviewing papers before they're submitted. I've learned to simply cut the middle man and ask the AI myself, though i cannot afford the pro subscription.
        1. otikik · · focus · HN ↗
          You could tell them: “ChatGPT found 3 minor issues with this paper, that I have since addressed: issue a, b and c.
    7. cess11 · · focus · HN ↗
      It's like old fashioned Wordpress, edited by someone directly in production, on a shared host that probably is infested with who knows what, with little to no oversight except inscrutable emails, which in the old days were like a CEO of a small business emailing 'hey, the thing we spoke about on the phone, is it coming?' and no actual document trail.

      Except for the enterprise customer and at enterprise prices.

    8. epgui · · focus · HN ↗
      To put a finger on a word: David Hume’s “is vs ought” problem.

      Data can describe to you what exists. But it can’t tell you what you value.

      What you describe is people who can’t tell the difference, and who let the machine (data) make the value judgments.

      1. xhevahir · · focus · HN ↗
        I don't think the problem OP has is anything so metaphysical as the fact/value distinction. This is a business, after all. It has goals (e.g., making money) that the model surely can grasp. It sounds to me more like a breakdown of organizational control owing to a lack of transparency in the tools and a general ignorance among the management.
      2. godwinson__4-8 · · focus · HN ↗
        Imo, most companies ought not to exist.

        The parent comment is interesting, but ultimately I think in this case, the AI is actually revealing something about the true nature about their place of employment, a nature that has always been there versus some mutation caused by the prevalence of the AI itself.

        A lot of money can be made purely algorithmically. Think market markers or other algorithmic trading. The ought vs is divide is quite narrow here. It's not a moral question, the "value" is in the money to be made. It's actually not a great example to invoke Hume's problem. Many companies essentially are chasing a similar spread, its just less obvious. Few people ever ask what "ought" to exist. If the power of AI makes more businesses operate more reactively and algorithmically, because of more data or processing power or w/e that really is probably in keeping with their alignment and goals. Because the ultimate ought for a company is we ought to be making more money. So, in many ways the ought is really not that interesting, the is is satisfactory provided the return on whatever their version of a spread is keeps improving.

        The number of companies that actually "invent" useful things and thus ask even vaguely meaningful "ought" questions are extremely slim. The vast majority of employees are, at best, accessories to these questions, even in software where even before AI many of us were not doing very interesting work. There is a lot of essentially rebuilding your competitors same layers on top of common libraries and standards where the actual interesting work is done. Really not unlike asking AI to cook you up a boilerplate by leveraging the vast work of a fraction of SWEs who maintain OSS tools. It's the same pattern and the same sort of behavior, just now your "layering" is becoming automated to the point of irrelevance.

        What the parent misses - the real promise of AI is paradoxically, that it will allow more people to ask actually interesting ought questions as AI owns more of the spreads. In the same way a human does not compete with an algorithmic trader, and at some level, really doesn't care. The more algorithmic your business becomes, the less any individual human "value judgement" matters. And really this is desirable, because again, most companies are not asking interesting value questions anyway. The end state of this you are missing is these companies are going to cease to exist. In the optimistic case this will free you up to ask more interesting value questions - like how do I value all my UBI enabled free time. In the less optimistic case your value judgments will be more dire - like who do I sacrifice given the Terminators are at the door and we only have x quantity of supplies left.

        This is the other paradox. When questions of what ought to happen are of paramount importance, you are probably finding yourself in a very undesirable situation. It's easy to valorize the ought problem from a distance, it is much much harder to actually engage with it when it actually matters. In many ways, the relative luxuries of society and civilization are derived from taking such questions out of most of our hands. This is (perhaps surprisingly) true even as you climb the ladder of power:

        I used to think that if there was reincarnation, I wanted to come back as the President or the Pope or as a .400 baseball hitter. But now I would like to come back as the bond market.

    9. danielmarkbruce · · focus · HN ↗
      In many (most perhaps, but hard to know personally) companies, most people don't really understand the market they are in, the competition, their own products, the customers etc well enough to actually make good decisions. They also aren't likely to be around (or held responsible) for decisions as they play out over years. So what has historically happened is that people use proxies for good decisions and understanding - which are clever sounding documents and presentations.

      This is a long winded way of saying "people made up clever/sensible sounding stuff". Now it's easier to do it with AI so the problem is worse. However, I'm not sure what you were looking for was ever really there - the "inscrutable machines" and "inscrutable incentives" were always quite inscrutable.

      1. ryaniscool · · focus · HN ↗
        Yes, people can be bad at their jobs. Yes, they can work in an industry and not know anything about the market but I refuse to believe that most people in most companies are just stumbling around faking it. I've never worked in a place like that. People like this exist but there are only a handful of people like this at every job.

        The hyperbole feels like it's been contrived to fit into the classic AI counterpoint: But humans do this too!

        1. danielmarkbruce · · focus · HN ↗
          It's more nuanced than that - I never said faking it, people usually believe what they write in those documents/ppts. And they do actually sound clever. Next time you see a decision of any magnitude, especially something which sounds "strategic" - trace the decision back as far as you can, consider the question from several angles, and really question it. You will see gaping holes, decisions made using "frameworks", anecdotes, hopes and dreams.
          1. ryaniscool · · focus · HN ↗
            I can't argue that people don't always consider every possible angle and nth-order consequence to a decision before making it. AI doesn't do this either. Also, this seems different than your original position.

            It still sounds like what you're saying is people are bad at decision making so AI isn't doing that much worse than people. That's very bizarre thinking because it opens the door to AI making decisions for us.

            1. danielmarkbruce · · focus · HN ↗
              I'm saying something even worse - humans are bad at decision making, we usually don't even most of the context required to make good decisions so we use heuristics to make decisions, and proxies for "good decision making".

              If the above appears to be bizarre thinking, take a couple extra laps around the block.

              Now I'm not saying AI is better, in fact above I mention that it's worse - but don't hold onto some nostalgic idea that decisions were made by people with lots of context thinking rationally and incentivized properly.

        2. TheOtherHobbes · · focus · HN ↗
          It's not counterpoint, it's a fact. Can you say with your hand on your heart that the typical user experience was acceptable before AI arrived?

          The reply was always "It's hard, and there are always going to be bugs."

          But it's not just about bugs. It's about systems that are hard to use, poorly designed, and opaque. And sometimes the opacity is deliberate. There are dark patterns, outright lies about what happens to data, and more or less obvious grifts.

          Was any of this truly great before AI arrived?

          Is it an accident that you can't kill an MS365 subscription if you have more than X amount of GB on OneDrive, but the actual usage includes all your spam and email attachments, and it doesn't show up unless you know where to find it, and the location is very non-obvious, and so is the cleanup and deletion process?

          Or that the eBay fee structure for international sales is utterly incomprehensible without automated help?

          Or that if you select Subscribe and Save on Amazon your cancellation date is something like two weeks before the next delivery?

          Or that if you sign up to Discord you need a mobile and an email for activation, but the site doesn't tell you this, and nor do tech support, who insist that a mobile is optional?

          And software quality - services down, records lost, records stolen, photos and documents deleted - is a whole other layer on top of that.

          AI is a moderately good solution to the first set of problems, because it can search and assemble information far more quickly than you can. So IME it's pretty damn good at tech support - not perfect, but better than DIY in many cases.

          Software quality? We'll see what happens. If tech stacks collapse over the next couple of years we'll know AI was terrible thing.

          I'm in the 'Too early to tell' camp. There's a fair chance they might. But if so it will be because of poor testing and design, not because of code review or lack of hand-rolled code. And it's not completely clear that quality levels wouldn't have dropped anyway.

          1. skydhash · · focus · HN ↗
            > Can you say with your hand on your heart that the typical user experience was acceptable before AI arrived?

            I remember using Windows XP (SP3) and Windows 7, Photoshop 7 and CS3, Office 2007, Blender, Winamp, Linuxmint and my computer was a joy to use (with HDD and slow ram)

            The internet was mostly a repository of knowledge and communication. I was online maybe 30 minutes every few days. This was around 2010.

          2. ryaniscool · · focus · HN ↗
            AI is here and things don't seem like they are getting any better any faster than they were before. Some people get it wrong and some people get it right...just like before.

            Also, you're completely missing my point. The author is talking about his company inadvertantly offloading decision-making to AI. The comment I was responding to was like "Well, it won't do that much worse than people". That type of thinking is a little dangerous to the future of our species.

      2. Supermancho · · focus · HN ↗
        > most people don't really understand the market they are in, the competition, their own products, the customers etc well enough to actually make good decisions

        "Product" studies this data and tells engineers what features they want. I am rarely instructed to A/B anything...excepting when it's defensive, to ensure the first rule of "don't interrupt the flow of money" is upheld, if at all. Most of the features are obvious improvements anyway (determined from plain conceptual planning, manual testing, and personal usage).

        ofc I don't understand the customer or market or anything else. I'm paid with the expectation that I'm not going to share an opinion about it, especially since am never exposed to the raw data and inner circle decisioning except during a quarterly meeting...maybe. This is part of why AI is so successful. Humans in large organizations are specialized with little creative input and lots of mechanical process. Coding is largely a mechanical black box (turing machine) from the outside looking in. It works seamlessly because I don't need to know about the things product wants, so the AI doesn't know and everyone carries on faster than we were before.

    10. T-RN-R · · focus · HN ↗
      This is why we are building Origin (originhq.com). Having a record of every prompt, tool call and model response enterprise wide allows you to answer these kinds of questions.
      1. [deleted] · · focus · HN ↗

        [deleted]

      2. moogly · · focus · HN ↗
        No human will read that though.
        1. genxy · · focus · HN ↗
          It is for execs
    11. igregoryca · · focus · HN ↗
      This resonates, but also, I'm not convinced anyone can truly understand the workings of any society, or market economy, or what have you.

      Who's in control? Everyone is, to some extent. And no one is: when you're hungry for food, are "you" in control of that? You can consciously repress your impulses to go eat something, but your mind didn't create those impulses.

      Human societies develop impulses and minds of their own, emerging (weakly) from the impulses and minds that comprise them, and they make decisions in mysterious ways.

      Of course, it sure is nice when we can come up with a compelling story for the motivations behind something. Easier said than done…

      1. [deleted] · · focus · HN ↗

        [deleted]

    12. aaroninsf · · focus · HN ↗
      One of the more thoughtful observation's I've seen in this "space," one which offers an unsettling counterpoint to the claim in the original post that "AI Can't Drive Itself."

      To your point, AI can drive itself; it's just that the fashion in which control occurs is distributed. Which is to say, the process (e.g. a feature being implemented, in some way, or at all) is not spontaneously occurring: it's emergent from the mesh of AI automation.

      Niceties aside, it's pretty clear that humans are not in the loop in a meaningful way, much of the time. What emerges may hence not well be not aligned with business or technical needs. What it is aligned to may be impossible for we humans to discern.

      Who controls the way a forest grows?

      Ask a tree, get an answer, but don't forget, that's not the answer.

    13. jmoggr · · focus · HN ↗
      Ultimately competitive pressure is what drive decisions.

      > Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.

      yes, and it is the exact same system that is producing the "misaligned" superintelligence. Funny how that works, and begs the question: exactly how are you supposed to avoid building "misaligned" superintelligence?

      At some point deferring all decisions to AI will be the competitive thing to do, regardless if its aligned or not.

    14. RGS1811 · · focus · HN ↗
      The word I have for this is “commitment laundering”. People pass around AI artifacts that nobody has necessarily read or considered, and the invented assumptions and tagalong commitments just keep piling up. They look like they’ve got real provenance but nobody can say what is being done or why.
      1. saltcured · · focus · HN ↗
        Right up there with responsibility/culpability/liability laundering. These are things being shrugged off and externalized.

        And the complement is credit/provenance laundering. These are things being misappropriated.

        The grift often does both of these with the same sleight of hand, and this is what gets accelerated with the new tools and cavalier culture around everything.

    15. cjbgkagh · · focus · HN ↗
      Most people are trend followers and what trends they see decides what trends they follow. It sounds like the algorithms that chose which influencers to promote made the decision, which was already somewhat true before the current AI trend. This happened when there was a big push from deterministic feeds to algorithmically ranked feeds by facebook and twitter. I guess an argument could be made that the people who chose what content to consume feed the algorithm but really there is a heavy hand on the scales that tilts the algorithm in favor of various things.
    16. barbazoo · · focus · HN ↗
      > The code was stamped by Claude driven by a prompt. The prompt was for a ticket generated with the Atlassian AI integration. Atlassian had digested docs made with AI. The docs came from strategy memos I'm 90% sure were written entirely by Claude

      Was there never a developer in the loop? I hope my org won't give up control of their business to an AI like this soon, sounds like a nightmare to figure out what's going on.

    17. GuinansEyebrows · · focus · HN ↗
      > Nobody is actually orienting work and action to real, concrete goals. It's all based on speculation and anxiety about the future.

      the simulation has become simulacra! cosmic horrors abound.

    18. Terr_ · · focus · HN ↗
      I see this is code in a smaller level. It lands on an alternative but in the end nobody actually chose it, not even by personal taste.

      Yet everyone assumes that, like before, there must be a reason.

      Worse, we can't tell apart real decisions choices from the dream-machine side effects.

    19. sillyfluke · · focus · HN ↗
      Thank you for taking the time to write this. Unfortunately, there are still people on this forum who argue that a turtles-all-the-way-down approach will somehow work out in the end and that the best way to verify LLM work is to add another layer of AI agents, whereas the only way to stop it is to ensure that the entire thing is built on some human verified last line of defense. Unfortunately, the higher up you go the more insecure people get when you ask for their supporting sources. They'll try to hand wave it saying they researched it or got others to research it for them instead of owning up to deferring the decision to AI. You know whats even scarier? When a C-suite doesn't try to hide the fact that they had the AI make the decision. Then it's just time to leave, or alternatively time to turn your brain off and collect your pay check until the music stops.

      >Perhaps reflecting on the state of the market, I thought, could indicate who was actually in control. Where do investor and customer expectations come from in 2026?

      This looks like ill-fated Gartner driven development on crack. Where does customer expectations come from in 2026? How about from your customers?

      If you're doing enterprise software and talk to your users, you'll end up learning that the actual users of your software don't use 80% of your features. You might even learn they don't use your software at all, and that the software was mandated top-down from the C-suites or that the execs in charge forgot what purpose your software served but are too insecure to question it, unless and until there is a sudden pressure to cut costs.

    20. tarsinge · · focus · HN ↗
      From my experience in big and medium sized companies or even startups I would say that this was already the state of things before AI. Most leaders and top management were following internal committees, that were following consultants recommendations, that were themselves recommending what others were doing and was hype and/or following Gartner like "studies". The motivation was equally dubious, and many developers already had problem with management and product direction, but many didn’t bother to question it. AI just makes it obvious.
    21. AIorNot · · focus · HN ↗
      this is a good summary of the problems of AI decision making, but its assuming corporate decision making before was any better.. its probably about the same

      ie we have companies making decisions just as badly before AI as with AI..

      You need human vision and leadership to move an org.. AI is just a tool at this point

    22. fallinditch · · focus · HN ↗
      > Nobody in this ecosystem, I thought, is actually in control here.

      Profound, your comment provoked this thought: we know that AI in organizations is disruptive, but we still don’t understand some of the shifting patterns of power, communications, agency, etc.

      If AI is not well-suited for traditional models of stratified management then maybe we need different models of bureaucracy?

    23. ericmcer · · focus · HN ↗
      That is a great example, and kind of frightening how once the market decides to start pumping money into something our world morphs to meet the flow of cash.

      AI is definitely impressive but are LLMs the true path to AGI? Is AGI the path to a better life/world? These tools are really impressive but it would be nice if they did something actually useful. Can a rogue OpenAI agent draw investor attention by curing a disease instead of hacking into the CIA?

    24. uxhacker · · focus · HN ↗
      So AI is already in control with humans just rubber stamping the output?
    25. pfannkuchen · · focus · HN ↗
      So it’s tulips all the way down?
    26. Gravityloss · · focus · HN ↗
      I just today found a term to describe what is happening: gradual disempowerment. I think it's a likely path. Everybody, because of competition, needs to end up giving control to AI.
    27. perrygeo · · focus · HN ↗
      > This gave me pause. Who had actually made the decision then? Arguably there has been several layers of human review, but the actual source of the decision was hard to pin down.

      Cynically, this might be the real reason managers and investors love AI. It diffuses responsibility. No one is accountable. "Oops the AI messed it up" is a convenient excuse for bad management.

    28. kybernetikos · · focus · HN ↗
      Actually reminds me a lot of 'Cube'

      > there is no conspiracy. Nobody is in charge. It's a headless blunder operating under the illusion of a master plan.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.