‹ BackHN Continuity

Thread

WalShadow: Sub-second Postgres replication to ClickHouse from physical WAL

63 points · 10 comments · spathak

  1. rgbrgb · · focus · HN ↗
    would this work with a postgres hosted on supabase? for our company brain mcp thing we mirror postgres data (along with server logs, banking data, slack, etc) in clickhouse but took a very inefficient polling approach since it works with any read-only db connection string. always looking to improve that and get rid of latency though.
    1. ZiiS · · focus · HN ↗
      Not this quick but better then polling: <a href="https:&#x2F;&#x2F;supabase.com&#x2F;features&#x2F;supabase-pipelines" rel="nofollow">https:&#x2F;&#x2F;supabase.com&#x2F;features&#x2F;supabase-pipelines
    2. saisrirampur · · focus · HN ↗
      WalShadow requires direct access to the physical WAL, and most managed Postgres providers don&#x27;t allow that. It works with ClickHouse Managed Postgres or self-hosted Postgres. I talk about this in the blog:

      Physical WAL is key to WalShadow’s architecture, but most managed Postgres services don’t expose it to customers, making it impossible to use WalShadow. ClickHouse Managed Postgres manages both sides of the stack, allowing us to integrate WalShadow directly into the Postgres replication layer and provide a native path from Postgres WAL to ClickHouse.

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.