Thread

1. continuational · 1786022679 · focus · HN ↗

   It's kinda funny there is still software coming out whose security model is "constantly ask the user for permission, and hope they never make a mistake".

   It's been tried so many times before, and it never worked.

   1. Aurornis · 1786026300 · focus · HN ↗

      Harnesses like Codex support having a separate agent perform reviews on commands to try to identify malicious or broken commands. Some people turn it off because they either don’t understand or don’t want to spend the tokens on it.

      The common harnesses also have some sandbox functionality, which although imperfect actually does help contain the blast radius for a lot of things.

      The common harnesses also support remote development over SSH, which I and many others use to contain development to a virtual machine.

      If your complaint is that LLMs can execute tool calls then you’re never going to be happy with any of these solutions and this turns into another generic anti-LLM complaint.

      1. acedTrex · 1786026463 · focus · HN ↗

         "Lets have the system that fails sometimes that we are trying to ensure does not fail check it self"

         This is such an unserious approach.

         1. Aurornis · 1786026564 · focus · HN ↗

            A separate model with separate context is used for review.

            Like I said above, some people will never be happy with LLMs being allowed to do anything and nothing is going to make them happy about it.

            It’s only fair to discuss what the real current status of these systems is. Every time I highlight that things are actually being done, the goalposts move again. There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.

            1. acedTrex · 1786029063 · focus · HN ↗

               > A separate model with separate context is used for review.

               Thats fine, theres still a chance it fails.

               > There is no possible solution which will satisfy someone who has zero tolerance for letting an LLM execute tool calls because they will always find something.

               This is generally correct, security goes completely out of the window with this stuff. It will/currently is a security disaster and theres no actual solution to it.

               1. [deleted] · 1786032078 · focus · HN ↗

                  [deleted]

         2. wang_li · 1786027306 · focus · HN ↗

            "Let's make sure our model fails sometimes so that we can bill more for a second agent to validate, sometimes correctly, the work of the first model."

            1. Aurornis · 1786027640 · focus · HN ↗

               If you’re implying that the LLM companies are trying to train their models to make malicious tool calls so they can collect a few more tokens on the review, then I don’t know what to say. I guess threads like this are just a breeding ground for conspiracies now?

               1. wang_li · 1786028297 · focus · HN ↗

                  I think that solving a problem they have that has a side effect of generating more revenue is not lost on them. Particularly in the situation where the cost to improve the models goes vertical for minimal improvement in an asymptotic fashion. The decision to spend 6-7 figures regularly on training or bill every customer 3-4 figures per month isn't hard. While I don't think they will explicitly impair the models, they will certainly make choices that externalize the shortcomings of the model to their customers.

                  1. Aurornis · 1786029179 · focus · HN ↗

                     > I think that solving a problem they have that has a side effect of generating more revenue is not lost on them.

                     I think you’re overestimating the revenue generated by this. Having a separate LLM with a cached input prompt check commands is a trivial adder. The only reason it comes up is because they explain to users that it comes out of their plan. So someone on a $20/month plan is going to hit their limits marginally, though mostly negligibly, faster.

                     If you think they’re sitting in a conference room scheming about making their main models worse on purpose to collect a few extra cents, that’s just baseless conspiracy. They have more to gain or lose based on main model performance.

Unofficial Hacker News client; not affiliated with Y Combinator.