‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. arjunchint · · focus · HN ↗
    their static ip's were initially good and didn't get flagged, but now most sites are recognizing their ip ranges and blocking.

    Muse's utility has significantly dropped with the blockages.

    To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage.

    1. Gareth321 · · focus · HN ↗
      Most residential proxies are already far more blocked and rate limited than any Meta IP. The internet is becoming a very weird place, where individual and "trusted" personal IPs are becoming a kind of commodity. Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised. AI analysis is turning this up to 11.
      1. simoncion · · focus · HN ↗
        > Some sites are already scoring IPs based on usage activity - like a credit score. It's only a matter of time until this data is collated and commoditised.

        Spamhaus is nearly thirty years old and the notion of electronic distribution of IP and domain "reputation" lists is at least that old.

        I'll bet my hat that the Internet "advertising" [0] industry has been calculating and determining the reputation of individual households (if not individual users) for at least a decade.

        [0] The scare quotes are because its primary purpose these days is for dragnet private-sector surveillance.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.