‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. kees99 · · focus · HN ↗
      Some webshops (e.g. Aliexpress) already soft-block Chromium (at least Chromium-on-Linux). I.e. such users see larger-than-usual share of captchas.

      Assuming "more people catching onto" this, expect Cloudflare and most everyone else to follow the suite.

      1. kotaKat · · focus · HN ↗
        And the worst part is humans just getting more cognitively demanding captchas.

        Some days I'm literally fatigued enough I can't clear the overly complex hCaptchas these days I keep getting forced upon me and bots sail right through this crap.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.