‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. onion2k · · focus · HN ↗
      (using a custom side-loaded plugin that talks over websockets to a "driver")

      Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

      1. dvt · · focus · HN ↗
        It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
        1. LunaSea · · focus · HN ↗
          Interesting, do you know what exactly changes on said root DOM node in case the remote debugging feature is enabled?
          1. dvt · · focus · HN ↗
            `navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
            1. Cakez0r · · focus · HN ↗
              What model did you use to build this? I recently looked in to building something similar and got refusals.
              1. dvt · · focus · HN ↗
                I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.
            2. LunaSea · · focus · HN ↗
              Ah right, there are stealth plugins to remediate these types of hints I believe.
        2. raffraffraff · · focus · HN ↗
          Does remote debugging port itself get detected, or does the presence of a webdriver connection to the port get detected? I believe it's the latter.

          I've had success launching the browser and using dumb dumb methods to get around the captcha before attaching playwright.

          Dumb dumb methods = wmctrl, xdotool, bash (work fine for Cloudflare's "are you human" check)

          1. dvt · · focus · HN ↗
            I think it's the latter, and that's why I think using a "normal" non-debug browser is the best idea because it's essentially undetectable. And for the record, you can technically control Chromium using IPC if you feel like adding that feature & fully rebuilding it from scratch.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.