The problem is not AI code, but not knowing about system architecture or intent
Thread
Unofficial Hacker News client; not affiliated with Y Combinator.
The problem is not AI code, but not knowing about system architecture or intent
Unofficial Hacker News client; not affiliated with Y Combinator.
zero_shift · · focus · HN ↗
The code change itself doesn't specifically matter. But suffice to say, it was about an AI feature in one of our products.
The code was stamped by Claude driven by a prompt. The prompt was for a ticket generated with the Atlassian AI integration. Atlassian had digested docs made with AI. The docs came from strategy memos I'm 90% sure were written entirely by Claude.
The strategy was chosen by management at the urging of exec leadership. The execs now communicate mostly via AI written memos. I do not know how they make decisions, but they reference tech influencers, market conditions, customer expectations.
This gave me pause. Who had actually made the decision then? Arguably there has been several layers of human review, but the actual source of the decision was hard to pin down.
We were not building the feature because we wanted it. We were building it because we thought other people expected it.
Perhaps reflecting on the state of the market, I thought, could indicate who was actually in control.
Where do investor and customer expectations come from in 2026? It is very murky, at least in tech. There appears to be hype. Some hype comes from true believers, some comes from cynics. But both respond to market incentives that reward bigger and bigger claims.
Where does the market's "action" come from? What is the driver?
Investors do not really seem to understand what the tech is or its limitations. Some are passive operators. Others are just responding to the overall froth and speculation in the market - which becomes a runaway feedback cycle.
This left me lost.
Nobody in this ecosystem, I thought, is actually in control here.
Nobody is actually orienting work and action to real, concrete goals. It's all based on speculation and anxiety about the future.
So it is not only that nobody understands what the code does. It is that we cannot, or at least I cannot, explain the motivation. There doesn't seem to "be" any form of "intention" in this environment.
It has all been hollowed out, replaced either be inscrutable machines, or inscrutable incentives.
Ironically it rather resembles the kind of "misaligned" superintelligence we are supposed to be avoiding.
epgui · · focus · HN ↗
Data can describe to you what exists. But it can’t tell you what you value.
What you describe is people who can’t tell the difference, and who let the machine (data) make the value judgments.
godwinson__4-8 · · focus · HN ↗
The parent comment is interesting, but ultimately I think in this case, the AI is actually revealing something about the true nature about their place of employment, a nature that has always been there versus some mutation caused by the prevalence of the AI itself.
A lot of money can be made purely algorithmically. Think market markers or other algorithmic trading. The ought vs is divide is quite narrow here. It's not a moral question, the "value" is in the money to be made. It's actually not a great example to invoke Hume's problem. Many companies essentially are chasing a similar spread, its just less obvious. Few people ever ask what "ought" to exist. If the power of AI makes more businesses operate more reactively and algorithmically, because of more data or processing power or w/e that really is probably in keeping with their alignment and goals. Because the ultimate ought for a company is we ought to be making more money. So, in many ways the ought is really not that interesting, the is is satisfactory provided the return on whatever their version of a spread is keeps improving.
The number of companies that actually "invent" useful things and thus ask even vaguely meaningful "ought" questions are extremely slim. The vast majority of employees are, at best, accessories to these questions, even in software where even before AI many of us were not doing very interesting work. There is a lot of essentially rebuilding your competitors same layers on top of common libraries and standards where the actual interesting work is done. Really not unlike asking AI to cook you up a boilerplate by leveraging the vast work of a fraction of SWEs who maintain OSS tools. It's the same pattern and the same sort of behavior, just now your "layering" is becoming automated to the point of irrelevance.
What the parent misses - the real promise of AI is paradoxically, that it will allow more people to ask actually interesting ought questions as AI owns more of the spreads. In the same way a human does not compete with an algorithmic trader, and at some level, really doesn't care. The more algorithmic your business becomes, the less any individual human "value judgement" matters. And really this is desirable, because again, most companies are not asking interesting value questions anyway. The end state of this you are missing is these companies are going to cease to exist. In the optimistic case this will free you up to ask more interesting value questions - like how do I value all my UBI enabled free time. In the less optimistic case your value judgments will be more dire - like who do I sacrifice given the Terminators are at the door and we only have x quantity of supplies left.
This is the other paradox. When questions of what ought to happen are of paramount importance, you are probably finding yourself in a very undesirable situation. It's easy to valorize the ought problem from a distance, it is much much harder to actually engage with it when it actually matters. In many ways, the relative luxuries of society and civilization are derived from taking such questions out of most of our hands. This is (perhaps surprisingly) true even as you climb the ladder of power:
I used to think that if there was reincarnation, I wanted to come back as the President or the Pope or as a .400 baseball hitter. But now I would like to come back as the bond market.