Thread

Loading the complete thread in the background. This saved snapshot is available now. Refresh

1. nine_ch · 1790586816 · focus · HN ↗

   [dead]

2. HannaTao · 1790587150 · focus · HN ↗

   [dead]

3. majke · 1790594319 · focus · HN ↗

     8. 80.239.216.210.  0.0%    78  123.2  64.6  42.9 141.8  30.6
     9. vl202.zur-itx1-dist-1.cdn77.com. 0.0%    78   60.1  58.8  43.7 136.5  24.2
     10. 89-187-165-194.bunnyinfra.net. 0.0%    78   45.9  63.5  41.8 131.9  31.0
   

   so cdn77.com and bunny.net

   1. nine_ch · 1790594926 · focus · HN ↗

      Yes, that's bunny.net, it's named in the post. Their Zurich PoP appears to sit on CDN77's network, which is on their side of the setup, not something we configured.
4. jareklupinski · 1790594769 · focus · HN ↗

   > There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.

   "AI said it's all good. There are no attackers within our walls."

   1. nine_ch · 1790594906 · focus · HN ↗

      [dead]

   2. ErroneousBosh · 1790600435 · focus · HN ↗

      I sometimes work on a system so byzantine and poorly-documented with so many horribly obsolete systems - they have Windows XP machines all over the place - that there's no way it hasn't been popped.

      I think it's safe though.

      I think various groups have popped it, taken a look around with growing expressions of horror on their faces, and gotten back out as quickly as they could. Nah man, no way, not touching that, don't want my prints on this particular gun, there's no way this is legitimately how they've left it, this has to be some kind of a setup.

      1. BLKNSLVR · 1790603816 · focus · HN ↗

         Security by obsolescence layered with security by complexity.

         1. jareklupinski · 1790603949 · focus · HN ↗

            it's not enough to salt your hash; you must mold your Swiss Cheese

            1. ErroneousBosh · 1790621007 · focus · HN ↗

               It's encrypted using SHA-255; in theory half as secure as SHA-256 but in practice they spend ten times long trying to work out why nothing is working properly.

               1. jareklupinski · 1790627048 · focus · HN ↗

                  you can fit 255 into a byte tho, giving you room to do more encryption

                  1. ErroneousBosh · 1790680011 · focus · HN ↗

                     Being semiserious I do wonder how hardcore you could do encryption in 8-bit processors. Like, how hard would it be to keep "nation state" level cracking out of your private messages for a week or so?

                     You could probably generate elliptic curve keys pretty effectively even with an Apple II, at least enough that you'd have to throw a couple of dozen Israels worth of intelligence agency at it to pop it.

         2. cube00 · 1790607435 · focus · HN ↗

            Enterprise security in depth.
   3. cube00 · 1790605351 · focus · HN ↗

      Hope their logging kept up or else they'd never know if there was an intrusion. It wouldn't be the first time a DOS has been used to flood monitoring infra while the real attack takes place.
5. tshanmu · 1790594990 · focus · HN ↗

   "We found three concrete gaps during this incident, and we would rather be upfront about them than gloss over them." claudism?

   1. nine_ch · 1790595347 · focus · HN ↗

      [dead]

   2. trentnix · 1790601183 · focus · HN ↗

      Sticks out like a sore thumb, doesn't it?

      1. esseph · 1790615635 · focus · HN ↗

         [delayed]

         1. trentnix · 1790653374 · focus · HN ↗

            I read so many Claude walls-o-text I find my own writing ends up using the same tropes. And so, I end up "using AI" even when I don't.
   3. jiveturkey · 1790619133 · focus · HN ↗

      is it? i would write a sentence like this on my own. i guess informed by the thousands (or hundreds, at least) of PMs I have read or contributed to. the same material LLMs would have trained on.
6. someonebaggy · 1790595038 · focus · HN ↗

   Did AI write this?

   1. nine_ch · 1790595263 · focus · HN ↗

      Yes. The blog version was drafted with AI assistance and then edited and reviewed by us. The facts, numbers and timeline are ours and match the technical postmortem PDF linked in the post, which is the drier, more complete version. Happy to go into any of the technical details here.
7. BLKNSLVR · 1790605399 · focus · HN ↗

   Naive question: services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?

   In other words, if there are a bunch of services prone to amplification attacks, can traffic from these services be upstream-blackholed for the duration of the attack?

   If it's not traffic coming directly from an IoT botnet, which is probably where the source of the spoofed traffic that initiates the amplification, then isn't there likely a smaller, more manageable number of services responsible for the attack traffic?

   Or are we talking services that form the substrate of the internet that have inherently exploitable protocols that it would take a large herd of organized cats in order to update in a way that doesn't break the internet, and will still take ~10 years?

   I still think in IPv4, so this may be a stupid question, but it's it known how many unique IP addresses were attempting to connect in the space of that time, and then it's there logging to identify those with unusually large amounts of individual traffic?

   1. nine_ch · 1790607519 · focus · HN ↗

      [dead]

   2. toast0 · 1790607662 · focus · HN ↗

      Ten years ago, when I worked on stuff that attracted DDoS, the memorable vectors were UDP chargen reflection and wordpress pingback reflection.

      Wordpress does get lots of patches, but I don't know how you really fix pingback, but it was easy enough to look for user-agent WordPress and drop requests before serving large files (or really anything... what do I have that WordPress should request). For a smaller site, the volume might have been high enough to overwhelm TLS handshaking, which is harder to solve.

      Chargen, wow. There's pretty much zero need for it to be on the internet. There's no need for anyone to run it. But evidence showed many instances running and it seemed to be the implementation Microsoft shipped in the Services for Unix package. Someone was trying to blocking the reflected traffic, but the servers were sending 64k responses (!) and that was being fragmented, and they only dropped the first fragment... fun times.

      I didn't spend time trying to get the hosts involved to stop sending this garbage... Writing abuse reports is herding cats, and networks that would be responsive probably already have taken these senders offline. I was also seeing short duration attacks consistent with people trying the free tier of DDoS as a service... so dealing with 90 seconds of garbage every once in a while was no big deal (as long as fragment reassembly didn't knock the machine over)

   3. jiveturkey · 1790619251 · focus · HN ↗

      > services that can be used for amplification attacks, are they constantly getting patched to prevent the latest iteration of attack type?

      unfortunately, no.

8. cube00 · 1790605928 · focus · HN ↗

   > There was no unauthorised access and no compromised systems. This was an overload attack, not an intrusion.

   Hopefully your logging infra is rock solid and nothing has been dropped in the flood. It wouldn't be the first time a DOS was used to mask the actual attack by overwhelming the monitoring infra.

   > Use a CNAME or ALIAS record instead of an A record. An A record ties your domain to one specific IP address on our platform. That fixed binding was exactly the problem during the attack: wherever we could change the address on short notice, availability could be restored, wherever we could not, only the blunt measure remained.

   I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?

   1. beecasthurlbow · 1790606634 · focus · HN ↗

      > I don't understand how this helps. CNAMES have TTLs like A records and they eventually have to terminate at an A record somewhere so why pay for an extra hop?

      I assume the customer controls the domain DNS records here rather than the hosting provider.

      CNAME record: hosting provider can change underlying IP freely.

      A record: hosting provider must get in touch with customer to change DNS.

      1. cube00 · 1790607271 · focus · HN ↗

         Thank you. I probably incorrectly read

         > we provide that customer’s connection to the internet

         As we provide the DNS infra for our customers.

   2. nine_ch · 1790607452 · focus · HN ↗

      [dead]

   3. jiveturkey · 1790619027 · focus · HN ↗

      indeed, this (A vs CNAME) is nonsense. I applaud this provider's transparency but they missed a 4th gap: inadequate network security expertise. The 3 gaps identified are perhaps not baseline but they are well understood hygiene. They shouldn't have had to learn these things as a result of an incident.

      and then up the stack a bit, they also are confused about DNS' role in attack mitigation. they should just strike that part of the PM entirely.

      that said, their reaction time is amazing. this would have included live troubleshooting during an ongoing incident! criticism aside, i wouldn't hesitate to use them if I wanted EU service.

      1. nine_ch · 1790679347 · focus · HN ↗

         [dead]

9. stavdavid · 1791110621 · focus · HN ↗

   [dead]

Unofficial Hacker News client; not affiliated with Y Combinator.