I just did a run with a benchmark I just used to test other models against. (It's about detecting irony in german soccer tweets). On my M5 Pro with 48GB it took over 30min to decide on just 100 tweets, the thinking definitely takes long.
It performed quite below Jev, but above other open decision models I tested (68 correct vs 79 correct for Jev - see [1]). I'm running it for the moderation benchmark as well, but that will probably take a few hours on my machine.
One of Nico's colleagues here, I'm working on an MPS port <a href="https://github.com/PostHog/jeeves/pull/1" rel="nofollow">https://github.com/PostHog/jeeves/pull/1 which should speed up M5 Pro performance quite a bit
TN1ck · · focus · HN ↗
It performed quite below Jev, but above other open decision models I tested (68 correct vs 79 correct for Jev - see [1]). I'm running it for the moderation benchmark as well, but that will probably take a few hours on my machine.
[1] <a href="https://tn1ck.com/blog/jevdit" rel="nofollow">https://tn1ck.com/blog/jevdit
robbie-c · · focus · HN ↗