‹ BackHN Continuity

Thread

Meta's Muse is fantastic for web scraping

60 points · 74 comments · STRiDEX

  1. dvt · · focus · HN ↗
    I built an AI "web harness" running on a sandboxed Chromium (using a custom side-loaded plugin that talks over websockets to a "driver") to basically do anything a normal user could do in a browser. It totally bypasses any and all bot measures and only gets the ones you yourself would get as well (and passes those successfully, e.g. Cloudflare checkbox or those annoying OCR puzzles).

    Not sure if I should release it, but I'm sure more people are catching onto the power of agentic browsing.

    1. bayindirh · · focus · HN ↗
      Thanks for letting us know that we need a new layer of detection systems.

      Also it’s great(!) to see that we’re going from “but ethics” to “I got mine, who cares”.

      Humans are interesting creatures.

      Edit: Please before assuming that I'm assuming things, this is an observation I'm making over time. It's possible that I'm in a bubble, but it's not a sample size of 1 (i.e. The comment I replied only).

      1. Cakez0r · · focus · HN ↗
        You're framing this as if people are deliberately making decisions that they believe are unethical. The reality is that people have different ethical frameworks. For example, I believe that there is no ethical distinction between whether a web request originates from a browser or from an LLM on my behalf.
        1. Kepten-Hook · · focus · HN ↗
          Another person replied already and I agree with them, but also I think you need to see this from the perspective of the people operating the service you are accessing.

          As an example, imagine a really small community maintaining a small site/wiki/cms/forum that has the ultimate goal of promoting human relationships around a common interest (let's say retro computing as an example, but it could be anything). There are many many many such communities on the internet.

          It's very improbable you will specifically instruct your LLM to access their site, it's way more probable it will happen without you even knowing, as a result of you doing some /deep-research or something. And not only that, but your LLM will probably spawn a ton of agents to gather as much information as possible in as little time as possible. A torrent of requests will go at this community's site, effectively killing it. They are a small community, they use their spare time and money to maintain something to serve them, they don't have the resources to serve your LLM and until you showed up they probably never even had to think about Cloudflare. They are certainly not against you getting the content, but they don't want you causing them issues either and you just did.

          End result? Your LLM (effectively you) DoSed a small community's site. You caused harm. Could you have caused similar harm if you were doing it on your own? Sure. But you would have done it on purpose, not accidentally while instructing your LLM to do something else.

          This isn't a made up story, it has happened already more than 1 times.

          So the question is, now that you know your LLM can cause harm without you even knowing it, how does this change your stance?

          1. mvt67 · · focus · HN ↗
            Already have whatsapp and signal for this. Hardly need a website for these thing anymore.
            1. bayindirh · · focus · HN ↗
              So shall we build more walls around our content and make it anti-FAIR?

              We can stop putting information in the open in any form, as well.

              FAIR: Findable, Accessible, Interoperable, Reusable.

              1. mvt67 · · focus · HN ↗
                Why do you build a wall around your house genius?
                1. bayindirh · · focus · HN ↗
                  A public website is not a house, it's something between a big board in the city square or a community center where people can meet an interact.

                  Considering that, shall we paint the boards black or lock the doors to community spaces?

                  Also, while I assume this is not your first account here, these are worthy of reminding:

                  > Be kind. Don't be snarky. Converse curiously; don't cross-examine. Edit out swipes.

                  > Throwaway accounts are ok for sensitive information, but please don't create accounts routinely. HN is a community—users should have an identity that others can relate to.

                  For more, please refer to <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;newsguidelines.html

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.