‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. onion2k · · focus · HN ↗
      (using a custom side-loaded plugin that talks over websockets to a "driver")

      Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

      1. dvt · · focus · HN ↗
        It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
        1. raffraffraff · · focus · HN ↗
          Does remote debugging port itself get detected, or does the presence of a webdriver connection to the port get detected? I believe it's the latter.

          I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.

          Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)

          1. dvt · · focus · HN ↗
            I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.