Love pi. I tried to run some local models and pi was the only one that actually worked decently because it didn’t have a gargantuan system prompt that would take minutes to prefill on my scrawny ass laptop.
Been running it almost barebones vanilla for a couple of months. Just a bunch of basic extensions and some skills.
Now, if only they could fix the very annoying bug of the history jumping back at the beginning if I am not a the end while the model is reasoning that would great.
> Now, if only they could fix the very annoying bug of the history jumping back at the beginning if I am not a the end while the model is reasoning that would great.
It's so interesting because Claude Code used to have this bug a long time ago, but it was eventually fixed. Strange that they both had/have the same issue.
I don't think it's necessarily a bug per se, but a central tradeoff in system prompt length between well-documenting the environment (harness specifics, exposed tools, tool use instructions etc) to the llm, vs the initial prompt stage ("prefill") growing so large that it results in an unpleasant lag to first response, and reduced available context, which is most noticeable with open models on resource-constrained consumer hardware.
You can use llm to optimize some of this, I condensed the tool descriptions of some larger LM Studio plugins to shrink prefill by almost 10k tokens. But there's a soft limit to this, if you don't want to under-document available tools and let the model guess (/behave unsafely).
One optimization around this is called "smart tool selection", which only sends tool descriptions when the model indicates need for a certain tool (suite), not all of them upfront.
FacelessJim · · focus · HN ↗
Been running it almost barebones vanilla for a couple of months. Just a bunch of basic extensions and some skills.
Now, if only they could fix the very annoying bug of the history jumping back at the beginning if I am not a the end while the model is reasoning that would great.
kelnos · · focus · HN ↗
It's so interesting because Claude Code used to have this bug a long time ago, but it was eventually fixed. Strange that they both had/have the same issue.
smokel · · focus · HN ↗
Or is it a non-trivial bug that requires a lot of refactoring, and could introduce a lot of new bugs? That would require careful review from a human.
The latter may well be a reason why I don't see extreme productivity gains in larger brown-field projects.
(Disclaimer: I see enormous benefits in one-off greenfield projects.)
kekebo · · focus · HN ↗
You can use llm to optimize some of this, I condensed the tool descriptions of some larger LM Studio plugins to shrink prefill by almost 10k tokens. But there's a soft limit to this, if you don't want to under-document available tools and let the model guess (/behave unsafely).
One optimization around this is called "smart tool selection", which only sends tool descriptions when the model indicates need for a certain tool (suite), not all of them upfront.