‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. arjunchint · · focus · HN ↗
    their static ip's were initially good and didn't get flagged, but now most sites are recognizing their ip ranges and blocking.

    Muse's utility has significantly dropped with the blockages.

    To become truly useful again they will need to use residential proxies, but I can't see them use those due to the risks and reputational damage.

    1. wraptile · · focus · HN ↗
      Every time a new tool launches there's a good window where it can act as a scraping proxy. Back in the 2010s I used Google Translate for years to scrape hard targets like LinkedIn but these windows are much shorter these days as scraping is so much bigger.

      One thing with Muse though is that you can scrape Meta's own sites which are currently all going under login walls and restricting discovery/search entirely.

      1. STRiDEX · · focus · HN ↗
        similarly, gemini via google ai studio will happily run workloads over youtube that would be very annoying to run at scale. Especially if you needed to download the video.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.