Thread

Laya on Mac M4 CoreML Offline

174 points · 34 comments · putna

Loading the complete thread in the background. This saved snapshot is available now. Refresh

PaulRobinson · 1789924370 · focus · HN ↗
Local LLMs are the future, and one of the reasons I think the data centre furore is just going to end in a market crash.
LLMs that can reliably be used for control problems are the future, and I think classic/deep RL has generally been overlooked for years for a whole host of problems by wider industry because it felt inaccessible. The first thing I thought of when I saw Jev (and then Laya), was "this might move the needle in a really, really interesting way".
Local LLMs that can reliably be used for control problems smash through a lot of barriers I'm interested in, and this intrigues me a lot. Guess I'm about to become a big Laya fan if it can run on this kind of hardware to this performance.
1. frag · 1789924838 · focus · HN ↗
  that's not a local LLM. If it's local, it doesn't matter in this case. Laya is a System 1 "AI", namely works like a classifier, given a state and questions, it shoots probabilities for each. I publish an episode tomorrow about Laya and Typesafe AI on <a href="https://www.youtube.com/@DataScienceatHome" rel="nofollow">https://www.youtube.com/@DataScienceatHome
  Stay tuned ;)
  1. putna · 1789933271 · focus · HN ↗
    cool, will check it
  2. EagnaIonat · 1789962049 · focus · HN ↗
    I’ve found they start to fail the more classifications you have, long before your typical ML classifier.
    To me it’s like a solution looking for a problem that is already solved.
2. bigyabai · 1789926247 · focus · HN ↗
  It won't. Laya is a finetuned version of Google's BeRT model, which is almost 10 years old right now.
  If BeRT had any potential to disrupt the datacenter buildout, it already would have.
  1. viraptor · 1789938533 · focus · HN ↗
    Modernbert is from 2024. It's also trained from scratch, not a fine tune.
3. ipsi · 1789932028 · focus · HN ↗
  The future for whom? The general public? Not a chance, no way, not unless it's able to run on a phone (anywhere from 20-40% of internet users, world-wide, are phone-only).
  For companies? I think that's a lot more plausible, as that's mostly just a question of money - is it cheaper to run and administrate our own models, or outsource that?
  For technically inclined users? I think that's unlikely unless they're able to operate on relatively cheap hardware while still being just as good as the hosted models. And by that I don't mean "a mac studio," that's far more money than I think is reasonable. A single RTX 5080, maybe, once memory prices start to drop.
  1. Izmaki · 1789934247 · focus · HN ↗
    Compare the games your average high-end smartphone can run to the AAA titles of the 2010's. It's not a matter of "unless it is able to" but "when it is able to".
    bigyabai · 1789950633 · focus · HN ↗
    That's going from 150w 720p gaming to ~15w 720p gaming in about 10 years. A typical inference cluster will draw well over 1500w to deliver even a small-ish 500b model at reasonable speeds/quantization.
    So, extrapolating from your gaming example, it will take smartphones only... *checks clipboard* ...100 years to achieve datacenter-level performance at the pace of 2010's improvements.
    mynegation · 1789951087 · focus · HN ↗
    Uh, my clipboard says 20 years
    bigyabai · 1789962130 · focus · HN ↗
    Mea culpa, but it's still a decent while.
    Izmaki · 1790178913 · focus · HN ↗
    We don't need datacenter-level performance in a handheld device just like we didn't need a storage room in our backyard for a mid-sized, tape-based NAS because technology did its thing and gave us something smarter, smaller and faster.
    Your argument is true if the number of parameters in an LLM is the only measurement for quality - a bit like number of bolts per aircraft or lines of code in software. I'd bet you that an 8b parameter LLM will in 5 years outperform the datacenter-level LLMs of today.
4. itemize123 · 1789967568 · focus · HN ↗
  no way. one is functionally (slightly overhyped) magic. another is better tool.
oezi · 1789924527 · focus · HN ↗
[dead]
tentacleuno · 1789924645 · focus · HN ↗
This looks like a local AI model playing Snake -- is that correct? The article offers no explanation.
1. ryuuseijin · 1789924789 · focus · HN ↗
  The linked github project [1] contains more information.
  [1]: <a href="https://github.com/mizorewww/laya-coreml" rel="nofollow">https://github.com/mizorewww/laya-coreml
2. putna · 1789933038 · focus · HN ↗
  Correct, the cli commands are just copy paste to run on your machine.
frag · 1789924870 · focus · HN ↗
Great job! Did you finetune your own Laya for the snake game or what?
1. putna · 1789936079 · focus · HN ↗
  its just an example how to run it locally. it takes the laya from official repo. you can try running it in 2 minutes
aidiveyt · 1789928338 · focus · HN ↗
[dead]
altano · 1789931545 · focus · HN ↗
[delayed]
1. WASDx · 1789932887 · focus · HN ↗
  The model is only 0.3B params so probably not much.
2. putna · 1789934829 · focus · HN ↗
  Physical footprint: 560.4M Physical footprint (peak): 778.0M
3. ImJasonH · 1789951842 · focus · HN ↗
  I had Fable one-shot the same demo from the same weights on an iPhone 15 Pro and it decides in ~40ms.
imranq · 1789934352 · focus · HN ↗
Based on my admittedly limited research, it seems like you should use Laya for much more deterministic tasks where you have some training data. It won't be as good as Jev for zero shot cases.
1. putna · 1789934891 · focus · HN ↗
  agree, too good to be true for one shot cases
2. jwpapi · 1789935095 · focus · HN ↗
  I think it can make a lot of sense to make a set with jev and then train laya on that set and do the rest with laya for saving $
  1. adinb · 1789964414 · focus · HN ↗
    I did that on a little macbook m4 last night on my model of the innate immune system—fine tuning took 15m or so. Just wish it had a larger context window
    jwpapi · 1789993723 · focus · HN ↗
    How about runpod
    adinb · 1790025966 · focus · HN ↗
    No need—small and light enough was able to do everything offline, though runpod would work fine, though it‘d be be a quick job
EgregiousCube · 1789939957 · focus · HN ↗
Isn't part of the Jev marketing that it has "terra-class intelligence"? I don't know how much it actually achieves that, but unless that's EXTREMELY wrong, it's hard to see how a 0.3B model could claim to be an OS Jev.
1. putna · 1789940670 · focus · HN ↗
  agree, maybe os jev is too strong of a wording
2. physicallyIllfr · 1789955307 · focus · HN ↗
  Jevs marketing has a lot to do with claiming credit for this pardigm, when the devloper of Layla was actually the first to do it. Its quite annoying to see another member of the OpenAI mafia so clearly rip off someone elses IP.
speedping · 1789940376 · focus · HN ↗
So cool. I've fired up pumas (energy monitor) and it seems to run almost fully on the neural engine and not the GPU so it plays really nicely with CoreML
brcmthrowaway · 1789949495 · focus · HN ↗
Why is Laya being shilled here? It doesn't have real intelligence backing it.
1. putna · 1790079990 · focus · HN ↗
  all examples is to inspire. For example i am inspired and know that i can run simple tasks on local machine offline. its just cool.
grejioh · 1789962121 · focus · HN ↗
[dead]
tc3oliver · 1790352784 · focus · HN ↗
[dead]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.