Thread

Hister: A private search engine for the pages you visit and the files you keep

741 points · 202 comments · bookofjoe

taude · 1789669128 · focus · HN ↗
Kind of related to this in that I built it to hoard knowledge from web pages I've visited along with implementing a Karpathy-style LLM Wiki, but the knowledge is collected automatically from sources I browse.
I have it up on GitHub, but I don't think anyone should use my implementation.
Loosely, what I built:
* On each of my machines I have a cron job running that looks at all my web browser history (usualy it's inspecting the brower's SQLlite across firefox and chrome). If it matches my rule list: hacker news stories, certain reddits, etc. it'll grab the page, convert to markdown and drop in my Obsidian Vault incoming.
* It has a whole de-duping architecture since I might open the same page on multiple machines. Uses the CloudFlare SQLITE D1 storage for tracking the processed links.
* it'll then trigger the LLM to do some Karpathy wiki style taxonomy assignment to the articles, organize them, create an index etc.
It's then available for my "bot" stuff to do writings for me.... I will probably write more about it at some point. I'm not certain it's totally useful and not just a yak-shave on hoarding knowledge.
Ai-drafted article on this [1]
Example AI-Drafted article based on some discussions the other day on Ollma vs LLama.cpp [2]
[1] <a href="https://taude.xyz/posts/how-archivore-turns-browsing-into-a-wiki/" rel="nofollow">https://taude.xyz/posts/how-archivore-turns-browsing-into-a-...
[2] <a href="https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-on-a-mac/" rel="nofollow">https://taude.xyz/posts/skip-ollama-run-llama-cpp-directly-o...
1. devsda · 1789671280 · focus · HN ↗
  Webpages used to have meta keyword tags until spammers hijacked them for irrelevant keywords.
  I think browsers can play a part in building a local search index for URLs based on those keywords the page declares and cross verify/accept only those that are in prominently visible content, or may be delegate to an external engine(like LLMs) via an extension etc. This is particularly useful for cases where full text indexing is not feasible or desirable.
  I doubt Google will ever add such feature in chrome though.
  1. marginalia_nu · 1789672384 · focus · HN ↗
    Meta keyword tags have always sucked even without malice. Because they're invisible, authors frequently forget to update them, copying <head> boilerplate between documents.
    Exoristos · 1789677795 · focus · HN ↗
    "I can't practice minimum quality control as a web dev" doesn't sound like a problem with the standard.
    marginalia_nu · 1789718689 · focus · HN ↗
    When designing a standard like this you need to apply systems thinking.
    The poor data quality is a problem for anyone who wants to use the tags, which has seen everyone almost universally reaching for other solutions. Search engines have preferred anchor tags, bookmarking solutions have applied user tagging.
    With keyword tags it's been a vicious circle of poor data quality and neglect since day one. Even in documents from the early 1990s when people were really trying, the data quality is inconsistent at best.
  2. z3t4 · 1789724458 · focus · HN ↗
    Did irrelevant keywords ever work for the spammer? You could add "naked" as a keyword because it had a lot of searches compared to "compiler". Now what happened to those looking for naked cats when they got to your website about code compilation. It's not like they would stop and think "oohh I'll look at this instead"

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.