It could be used for coding if you gave it an AST.
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
I've implemented tree-sitter in pi before, and while it works, I have no real proof it saves me tokens, or is more accurate. I think a better implementation is a model that's trained for AST's, not just "use tool, see what happens".
I'd love to do research on this when I have the time.
That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA
vatsachak · · focus · HN ↗
If you work at TypeSafe please try this.
Side note: This is probably how LLMs would perform with better encoders and next-latent prediction, so eventually those will beat this architecture out. Still amazing though.
ramon156 · · focus · HN ↗
I'd love to do research on this when I have the time.
vatsachak · · focus · HN ↗
That's what I was insinuating through "better encoder"; the model creating more efficient representations of ASTs using something like JEPA