Thread

Sonnet 5.5

884 points · 613 comments · D2OQZG8l5BI1S06

simonw · 1790621080 · focus · HN ↗
Pelicans. Sonnet 5.5 has the same problem as Opus 5.5: on "max" thinking effort it burned through 128,000 thinking tokens (taking 15 minutes to do that) and ran out before it had produced the final SVG.
<a href="https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F1d85a9be7f3ecce26e7f1569161a0d01" rel="nofollow">https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's how the thinking effort levels compare:
```
  low
  27 input, 1,623 output, thinking_tokens: 0
  1.6284
  Duration: 10138ms (10s)
  
  medium
  27 input, 1,796 output, thinking_tokens: 0
  1.7914 cents
  Duration: 11266ms (11s)

  high
  27 input, 2,334 output, thinking_tokens: 745
  2.3394 cents
  Duration: 17376ms (17s)

  xhigh
  27 input, 5,730 output, thinking_tokens: 2535
  5.7354 cents
  Duration: 41882ms (41s)

  max (failed to return response)
  27 input, 128,000 output, thinking_tokens: 128000
  $1.28
  Duration: 940617ms (15m 40s)
```
Low and medium both used 0 thinking tokens.
1. dennisy · 1790630346 · focus · HN ↗
  Does anyone really still care about these pelicans?
  Any model release it’s the top comment, I do not understand why.
  1. uncivilized · 1790631040 · focus · HN ↗
    Karma farming by parent commenter and HNers’ tendency to upvote low quality content (not dissimilar to other social media networks)
    conception · 1790634509 · focus · HN ↗
    Is this low quality content relative to most HN comments?
    uncivilized · 1790640612 · focus · HN ↗
    Very few HN comments are high quality
  2. simonw · 1790631645 · focus · HN ↗
    Mainly because they're funny, but it's also because I try pretty hard to make the comment more interesting than just "here's a pelican". In this case I used the pelicans to talk about the 128,000 token limit bug at "max" and share comparative pricing.
    In the GPT-6 comment I included full visual comparison grids: <a href="https://news.ycombinator.com/item?id=49805509#49806126">https://news.ycombinator.com/item?id=49805509#49806126
    For DeepSeek v4.1 Flash I identified that the OpenRouter reasoning levels are mapped to a smaller set of levels for that model: <a href="https://news.ycombinator.com/item?id=49639090#49645591">https://news.ycombinator.com/item?id=49639090#49645591
  3. kennyadam · 1790632083 · focus · HN ↗
    Agreed. It was a creative and unique test for a while. Now, no offense to the author, it feels like every conversation about a new model is dominated by the pelican on a bike posts as they always become the top comment.
    simonw · 1790632436 · focus · HN ↗
    You can click the little [-] icon next to the post to collapse the entire sub-thread. I do that all the time.
  4. marktolson · 1790632941 · focus · HN ↗
    It's an easy way to compare the coding and creative strengths of models. I prefer them over reading a tabular comparison of benchmarks which you have no real insights into.
  5. Glemmlko · 1790634262 · focus · HN ↗
    Because hn has some kind of a community and not every comment is gold (see yours for example) and people are able to skip comments if they don't enjoy them?
  6. mvdtnz · 1790639933 · focus · HN ↗
    I can't understand it. Clearly someone cares because like you say the comments are always upvoted. But why anyone cares I simply don't know. It just feels like attention seeking behaviour to continue posting it.
    simonw · 1790640746 · focus · HN ↗
    Isn't posting any comment on a forum like Hacker News "attention seeking behavior"?
    mi_lk · 1790652808 · focus · HN ↗
    Did you ignore the continue part that compounds the attention seeking?
    simonw · 1790655382 · focus · HN ↗
    Linking to <a href="https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F1d85a9be7f3ecce26e7f1569161a0d01" rel="nofollow">https://tools.simonwillison.net/markdown-svg-renderer?url=ht... should be pretty inoffensive (I started habitually linking to that after people kept complaining about linking to my blog) - that page renders Markdown with SVG embedded in it, but doesn't link to the rest of my site at all.
  7. ceroxylon · 1790644152 · focus · HN ↗
    This is a community and it is an inside joke at this point, it wouldn't be a proper model release without Simon's pelicans.
    I find it useful (as well as a fun art project).
  8. mi_lk · 1790644981 · focus · HN ↗
    Sick of the cheap shot
  9. permalac · 1790664243 · focus · HN ↗
    I would not say I care, but I do have curiosity. I expect one day they will start adding some textures or something like that.
  10. dramebaaz · 1790669147 · focus · HN ↗
    It would have taken me a while to stumble upon this "running out of tokens" on MAX thinking issue without his trials and post. I've seen the pelicans for years now, and if they stopped coming for some reason, I would probably go to his site to catch up on recent models and findings. So I don't mind them
  11. krzyk · 1790673220 · focus · HN ↗
    I do, it gives some fun comparison between models.
    You can also check for any kind of degradation of them - you have the prompt, it doesn't use much $.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.