‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. bayindirh · · focus · HN ↗
      Thanks for letting us know that we need a new layer of detection systems.

      Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.

      Humans are interesting creatures.

      Edit: Please before assuming that I'm assuming things, this is an observation I'm making over time. It's possible that I'm in a bubble, but it's not a sample size of 1 (i.e. The comment I replied only).

      1. Cakez0r · · focus · HN ↗
        You're framing this as if people are deliberately making decisions that they believe are unethical. The reality is that people have different ethical frameworks. For example, I believe that there is no ethical distinction between whether a web request originates from a browser or from an LLM on my behalf.
        1. cyanydeez · · focus · HN ↗
          There's two types of cares:

          1. People like other people.

          2. Businesses need to sell product

          The internet mixes those people, and an Agent basically pushes the signal to noise ratio that businesses have relied on the intenet to basically zero.

          If the internet just allowed indescriminate traffic, neither #1 nor #2 survives, and while it's interesting to think the value is, it certainly isn't anything we understand.

          Maybe you think the value will still exist, but some how agents will replace all value with equivelents. That's an argument, but I don't think it will.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.