‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. kees99 · · focus · HN ↗
      Some webshops (e.g. Aliexpress) already soft-block Chromium (at least Chromium-on-Linux). I.e. such users see larger-than-usual share of captchas.

      Assuming "more people catching onto" this, expect Cloudflare and most everyone else to follow the suite.

      1. dvt · · focus · HN ↗
        Are you saying they're blocking the `User-Agent`? Because that's trivial to bypass. I'm not exactly sure what you mean by "Chromium" because that's a whole class of browsers (Edge, Brave, Chrome, etc. are all forked "Chromium" browsers).
        1. kees99 · · focus · HN ↗
          Aliexpress specifically uses a javascript blob to do detection, and quite a sophisticated one. There was a recent story about that blob messing with bluetooth headphones.

          > not sure what you mean by "Chromium"

          I meant that literally. This thing: <a href="https:&#x2F;&#x2F;www.chromium.org&#x2F;getting-involved&#x2F;download-chromium&#x2F;#chromium" rel="nofollow">https:&#x2F;&#x2F;www.chromium.org&#x2F;getting-involved&#x2F;download-chromium&#x2F;...

          1. dvt · · focus · HN ↗
            Here&#x27;s my agentic Chromium browser browsing Aliexpress[1]. Usually I&#x27;d run it in headless mode, but wanted to test and make sure I&#x27;m getting the website proper.

            It searched for &quot;Aliexpress&quot; on Google and clicked the link, hence the Google UTM params.

            [1] <a href="https:&#x2F;&#x2F;imgur.com&#x2F;a&#x2F;eNulg11" rel="nofollow">https:&#x2F;&#x2F;imgur.com&#x2F;a&#x2F;eNulg11

      2. kotaKat · · focus · HN ↗
        And the worst part is humans just getting more cognitively demanding captchas.

        Some days I&#x27;m literally fatigued enough I can&#x27;t clear the overly complex hCaptchas these days I keep getting forced upon me and bots sail right through this crap.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.