I find this - or perhaps the title - a bit surprising.
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
Worth saying that GLM-5.3 isn't GLM-5.3-Flash's "big brother" the way one might think. GLM-5.3-Flash is not GLM-5.3 scaled down. While GLM-5.3 is based on GLM-5.2, and "every gain comes from post-training" (<a href="https://z.ai/blog/glm-5.3" rel="nofollow">https://z.ai/blog/glm-5.3), GLM-5.3-Flash uses a newly trained multimodal base model (<a href="https://z.ai/blog/glm-5.3-flash" rel="nofollow">https://z.ai/blog/glm-5.3-flash).
TuxSH · · focus · HN ↗
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
mariopt · · focus · HN ↗
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
seemaze · · focus · HN ↗
networked · · focus · HN ↗