Thread

Sonnet 5.5

884 points · 613 comments · D2OQZG8l5BI1S06

simonw · 1790621080 · focus · HN ↗
Pelicans. Sonnet 5.5 has the same problem as Opus 5.5: on "max" thinking effort it burned through 128,000 thinking tokens (taking 15 minutes to do that) and ran out before it had produced the final SVG.
<a href="https://tools.simonwillison.net/markdown-svg-renderer?url=https%3A%2F%2Fgist.github.com%2Fsimonw%2F1d85a9be7f3ecce26e7f1569161a0d01" rel="nofollow">https://tools.simonwillison.net/markdown-svg-renderer?url=ht...
Here's how the thinking effort levels compare:
```
  low
  27 input, 1,623 output, thinking_tokens: 0
  1.6284
  Duration: 10138ms (10s)
  
  medium
  27 input, 1,796 output, thinking_tokens: 0
  1.7914 cents
  Duration: 11266ms (11s)

  high
  27 input, 2,334 output, thinking_tokens: 745
  2.3394 cents
  Duration: 17376ms (17s)

  xhigh
  27 input, 5,730 output, thinking_tokens: 2535
  5.7354 cents
  Duration: 41882ms (41s)

  max (failed to return response)
  27 input, 128,000 output, thinking_tokens: 128000
  $1.28
  Duration: 940617ms (15m 40s)
```
Low and medium both used 0 thinking tokens.
1. TomGarden · 1790621785 · focus · HN ↗
  Where do you run sonnet/opus where you are limited to 128k, given they are both 1M context window models?
  1. simonw · 1790622677 · focus · HN ↗
    It's the output token limit, which has been 128,000 for Claude models for quite a while note
    croemer · 1790623863 · focus · HN ↗
    Pretty crazy that the model doesn't know that it needs to stop before it hits 128k output tokens. I guess it has no sense of how many tokens in it is? Wouldn't this be possible to work into the architecture?
    simonw · 1790624421 · focus · HN ↗
    I think this is a bug. I've not seen this problem from any of the other frontier models.
    NewJazz · 1790629999 · focus · HN ↗
    [delayed]

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.