‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. onion2k · · focus · HN ↗
      (using a custom side-loaded plugin that talks over websockets to a "driver")

      Is that necessary? You could start Chromium with an open debug port and use Chrome Devtools Protocol to send commands.

      1. dvt · · focus · HN ↗
        It is, because `--remote-debugging-port` is detected via the root DOM object (and that can't be changed unless you want to recompile Chromium), so for example, if you try doing a Google search with debugging enabled, you'll get blocked (usually just by being served a blank page).
        1. LunaSea · · focus · HN ↗
          Interesting, do you know what exactly changes on said root DOM node in case the remote debugging feature is enabled?
          1. dvt · · focus · HN ↗
            `navigator.webdriver` afaik, but I think there's a couple more, I did some research on it a few months ago. If using Playwright/Puppeteer, they also have a few special things injected that you have to strip.
            1. Cakez0r · · focus · HN ↗
              What model did you use to build this? I recently looked in to building something similar and got refusals.
              1. dvt · · focus · HN ↗
                I built the POC by hand (just had like 3 things it could do: scroll, click, type), and then Codex used my established patterns to make it slightly more general and cleaned it up.
Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.