Thread

Show HN: JevBench, a reproducible benchmark for typed decision models

154 points · 39 comments · florianstandhar

nzoschke · 1790113497 · focus · HN ↗
<a href="https://is-it-ai-slop.app.mintapis.com/" rel="nofollow">https://is-it-ai-slop.app.mintapis.com/ is a fun tool. Is the source or methodology for that in the github repo? I couldn't find it immediately.
We've been experimenting with Jev for classifying email, some thoughts here: <a href="https://housecat.com/blog/classifying-email" rel="nofollow">https://housecat.com/blog/classifying-email
Flagging AI written email is a much requested feature too.
1. 542458 · 1790123160 · focus · HN ↗
  Keysmashing my keyboard resulted in 86% confidence that the text was AI written. I don't think this is a particularly good classifier, I've never seen an LLM output "kad jfkhasljkdhf laksjhdf".
  Edit: If that's not realistic enough for you, the text "Hello world! My name is GravitasIsOverrated and I like coding and cooking. This text is 100% genuine, and not AI generated at all." results in 85% confidence that it's AI generated.
  More broadly, I don't know why this would work. Qwen/Jev/whatever doesn't magically have the ability to discern AI-authored text from non-AI-authored text, and will increasingly get worse at it as the hallmarks of AI-written text change.
2. ajs1998 · 1790123287 · focus · HN ↗
  I asked free chatgpt to give me some essays that will fool a slop detector and they all fooled this slop detector. Its best guess was "6% slop probability 87% confidence answered in 0.5 s for $0.000027" and yet it was 100% slop.
  I am very skeptical slop detectors will ever work.
3. florianstandhar · 1790162997 · focus · HN ↗
  Thanks! Yeah, the code is open: <a href="https://github.com/fstandhartinger/who-is-right" rel="nofollow">https://github.com/fstandhartinger/who-is-right. Email classifying is definitely a good Jev usecase, I agree

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.