Thread

Drawgent: Coding agent on a live Excalidraw canvas

176 points · 46 comments · parasitid

Loading the complete thread in the background. This saved snapshot is available now. Refresh

parasitid · 1790438195 · focus · HN ↗
video <a href="https://lnkd.in/p/ez4EZede">https://lnkd.in/p/ez4EZede
armanj · 1790443043 · focus · HN ↗
I've had a fairly thorough exploration of how I can give my agent a whiteboard so we can work on architectures together. To my surprise, current solutions (including Excalidraw) were not good enough and didn't deliver what I wanted. I ended up finding Mermaid to be the most agent-friendly medium and coded an Obsidian plugin for it. It works okay; we can work on the same doc while I bring my own AI agent, and we can brainstorm together. <a href="https://community.obsidian.md/plugins/mermaid-relay" rel="nofollow">https://community.obsidian.md/plugins/mermaid-relay
1. jmtulloss · 1790449470 · focus · HN ↗
  Did you play with TLDraw at all? I’ve had some success with it but haven’t been using it for that particular use case.
  1. armanj · 1790458405 · focus · HN ↗
    i found it a bit later than others and didn't really explored it deep; but yeah still couldn't find what i was looking for out of the box
2. mncharity · 1790458467 · focus · HN ↗
  Fwiw, also on the front page is Show HN: Reladraw – A diagram language where you decide where to place things[1], which adds relative placement constraints, and provides agent skills.
  [1] <a href="https://news.ycombinator.com/item?id=49858513">https://news.ycombinator.com/item?id=49858513
3. andrewkhorzx · 1791180880 · focus · HN ↗
  Same problem. Disclosure: that's why I made Pastefield (pastefield.com). What finally helped was a board the agent can actually read eg: what’s on it, what’s nearby, what I have selected so I don't need to babysit it too much and can convey my thoughts in text without worrying.
4ndrewl · 1790444652 · focus · HN ↗
YMMV, but the value I get from producing a diagram is derived from the thinking. Thinking about what I'm trying to draw leads to understanding about where my/my team's knowledge is poorer, what assumptions we're making, etc.
1. johongo · 1790445478 · focus · HN ↗
  I don't remember who said it, but someone mentioned that mathematics papers don't read well because the models don't know which parts to emphasize and which parts to gloss over. I would imagine similar tendencies here.
2. sbarre · 1790445626 · focus · HN ↗
  Yeah I tend to make my diagrams either as I go, as exploration, or after I'm done figuring things out as a summarizing artifact (and agents are great at using Mermaid to do that second one).
3. qudat · 1790447783 · focus · HN ↗
  Yes I agree! Drawing helps me think.
4. augment_me · 1790453591 · focus · HN ↗
  I found a big part of the appeal with tools such as Excalidraw is that people are still more willing to engage with your content because they think that its made by hand.
  Compare a piss-tinted LLM-generated diagram with 80 pipes, flows and generic names everywhere to a seemingly hand-drawn sketch.
  I have had my Exaclidraw connected to an agent through their MCP and I have gotten way more attention/time/reads on both blogs and papers because of the illusion of being human-made/hand-drawne making it more palatable.
  1. msy · 1790456871 · focus · HN ↗
    ‘I found a new way of wasting people’s time with my slop’ god this is bleak.
    augment_me · 1790457791 · focus · HN ↗
    Attention is the only currency when you have free information generation. Take it with whatever means, or someone else will.
    dustbunny · 1790459848 · focus · HN ↗
    The mindset that "everyone else is doing it" or "if I don't do it someone else will" is common amongst psychopaths and it reveals a deep cynacism and lack of consciousness. Often, the person is wrong as well.
    augment_me · 1790460636 · focus · HN ↗
    Calling the observation psychopathic doesn’t address the incentive it describes.
    Readers have limited time at the same time that there exists unlimited free information generation, and use presentation as a shortcut for deciding what deserves attention. A hand-drawn aesthetic can benefit from that shortcut whether the underlying work is thoughtful or empty. Producing the diagram manually doesn’t settle that question either.
    LeBit · 1790462034 · focus · HN ↗
    This is close to saying LLM users are psychopaths.
    4ndrewl · 1790506204 · focus · HN ↗
    More like that content creators are psychopaths?
5. parasitid · 1790489559 · focus · HN ↗
  the whole idea is that, as i'm very bad a making good drawings by hand, here i can sketch them in a quick and ugly manner and get them instanlty properly arranged and iterate: <a href="https://tangled.org/yanndegat.tngl.sh/drawgent/raw/main/docs/network-overview.png" rel="nofollow">https://tangled.org/yanndegat.tngl.sh/drawgent/raw/main/docs...
bad_haircut72 · 1790444683 · focus · HN ↗
Posted this somewhere else aswell but I've had good success with <a href="https://whiteboard-mcp.com" rel="nofollow">https://whiteboard-mcp.com - it seems to work better than other free alternatives for things like architecture diagram creation
jimmySixDOF · 1790444912 · focus · HN ↗
tldraw has a lot of programmatic access I have not compared it to excalidraw but there is still no equivalence between the model capabilities to build amazingly detailed infographics and diagraming tool use which is a gap I hope will narrow
tetrisgm · 1790445533 · focus · HN ↗
I’ve been trying to find the best practice to connect a person’s existing ChatGPT/Codex / Claude accounts so they can be used in an app for the agent in the app.
Any advice? Everything seems clunky, even the best MCP efforts
1. brazukadev · 1790448036 · focus · HN ↗
  best approach is to create an webapp that uses WebMCP and then use ChatGPT Desktop. you open the app using it and it and it can use all the tools. Then you implement navigation in the tools and you will see ChatGPT using your app pairing with you.
  1. tetrisgm · 1790454004 · focus · HN ↗
    Right I've already got MCP integration but it seems quite clunky. Do you have an example of an MCP implementation that you like? and maybe an example of code you used that did the integration nicely? Would love some pointers
2. jascha_eng · 1790448289 · focus · HN ↗
  MCP is the answer but it's only really worth it if you product is reasonably complex e.g. n8n has a community MCP fully for this reason.
  1. tetrisgm · 1790453997 · focus · HN ↗
    Right I've already got MCP integration but it seems quite clunky. Do you have an example of an MCP implementation that you like? and maybe an example of code you used that did the integration nicely? Would love some pointers
    jascha_eng · 1790455720 · focus · HN ↗
    I played around a bit with pencil.devs mcp integration before: <a href="https://www.pen.dev/" rel="nofollow">https://www.pen.dev/
    Which was at least fun for a bit although I failed to really see the use after models became better at designing themselves.
seemaze · 1790445845 · focus · HN ↗
Excalidraw offers their own open source first party MCP endpoint[0] and server[1]:
[0]<a href="https://mcp.excalidraw.com" rel="nofollow">https://mcp.excalidraw.com
[1]<a href="https://github.com/excalidraw/excalidraw-mcp" rel="nofollow">https://github.com/excalidraw/excalidraw-mcp
1. eshack94 · 1790465833 · focus · HN ↗
  Oddly, it seems like it hasn't been touched in 6 months (and last release 7 months ago). I wonder if it's been abandoned, or simply deprioritized in favor of other work.
  1. kamranjon · 1790491928 · focus · HN ↗
    Is the excalidraw api constantly changing?
    QuantumNomad_ · 1790496537 · focus · HN ↗
    If the software has not been updated in the last 5 minutes, is it even maintained anymore? /s
    carlmr · 1790520330 · focus · HN ↗
    If we're talking about something old, sure, but MCP and AI is new enough that any software interfacing with it that hasn't changed in 6 months it's a fair suspicion that it's abandoned.
AM1010101 · 1790446686 · focus · HN ↗
What frustrates me about many projects is it's never been easier to just make a quick video or image or gif and put it in the README to show what the value is. This was at least somewhat complex before AI but now you can just paste an image and have the AI put it in your README.
I guess the frustrating bit is that the readme is 100s of lines long. AI is already saving you so much time on a project like this but you still make me clone and spin it up if i want to see it for real.
I get that open source is a gift and you’re not obliged to do anything I say but please consider taking the extra few minutes on your next project.
1. taylor-tg · 1790448073 · focus · HN ↗
  To that end, I've found it super easy to tell $LLM to make me a video showcasing the main features of whatever I'm building, have it write subtitles and give it a voice over. Obviously for a README a simple gif with burned in subtitles/annotations would be more useful.
  Especially if the project is web-based it can just spin up playwright and record its actions.
2. bityard · 1790448129 · focus · HN ↗
  Wow, I just can't agree with this take at all. The README is only 161 lines (very short for a git repo README!) and took me about 15 seconds to skim and decide whether or not it was interesting enough to keep the tab open.
  Most of the project pages I have seen with videos either animate things at hyper-speed, meaning you can't actually follow is what happening, or drag on for minutes just to communicate something that I could have read in seconds. My time is more valuable than that.
  Images are cool, however. We should do more of those.
brumar · 1790449098 · focus · HN ↗
This is crazy. My claude chess stuff is currently near yours on the front page and noticed your post. I have a project very close than yours that I hesitated to share. From a quick glance, we went for a similar approach. I just open sourced it so that you can compare implementation notes. <a href="https://github.com/brumar/whiteboard-agents" rel="nofollow">https://github.com/brumar/whiteboard-agents . It's not thoroughly tested but can be interesting to check.
dtkav · 1790454089 · focus · HN ↗
I am working on a similar thing for jsoncanvas (.canvas files in Obsidian). My takeaway is that agents are heavily trained on specific formats (e.g. svg), and are conversely pretty bad at niche formats. My strategy was to create a native renderer [1] so that at least they can rapidly iterate "visually" by reading svg and png renders instead of trying to one-shot.
I have yet to see an AI generated visual diagram that doesn't feel like slop though... I'd be very interested in examples of actually-good diagrams workflows if anyone has seen them.
[0] <a href="https://jsoncanvas.org/" rel="nofollow">https://jsoncanvas.org/
[1] <a href="https://github.com/dtkav/canvas-native" rel="nofollow">https://github.com/dtkav/canvas-native
augment_me · 1790454102 · focus · HN ↗
I have a personal experience that there is a lot of attention be gained/exploited with tools such as Excalidraw because people still don't understand that this is possible to automate, and people with an LLM-negative bias that are actively against generated content will still give attention to such tools because it looks hand-drawn.
For example /r/art banned AI-generated content without a process due to this anti-automation bias, however you can now generate the process by using MCP-server tool calls into the software.
So there is a big attention market as long as you can keep up with things that seemingly looks like they had human effort.
0xdeafbeef · 1790454419 · focus · HN ↗
<a href="https://likec4.dev/" rel="nofollow">https://likec4.dev/ It's a bit different usecase, but llms are proficient with it
ramoz · 1790460787 · focus · HN ↗
I think many folks are underutilizing or underestimate the power of HTML.
There's been a few of these things on the front page lately. Including false "whiteboards" that are DSL abstractions over things like React Flow...
I've done quite a bit with Excalidraw MCP and the models had to deal with a lot of JSON data and estimating/computing with bounding box and pixel point numbers.
But with HTML, the agents get the natural semantics/styling capabilities and basically zero abstraction. The recent models can draw exceptionally well any form of logical/conceptual/flow diagram.
1. Marlinski · 1790586939 · focus · HN ↗
  I did exactly that <a href="https://lookwhatidid.xyz/" rel="nofollow">https://lookwhatidid.xyz/
hostdefi_dev · 1790475237 · focus · HN ↗
[dead]
ld_path · 1790480220 · focus · HN ↗
Might actually make my rough Excalidraw layouts useful for scaffolding. The live canvas interaction is a neat touch.
Tbarlow · 1790481844 · focus · HN ↗
Having an agent sketch out component ideas visually on Excalidraw seems like a genuinely useful way to kickstart UI projects.
Driftbench · 1790483992 · focus · HN ↗
Curious how deep the code generation goes from visual cues. Could be amazing for quick prototypes if it's smart enough.
thimble_io · 1790485923 · focus · HN ↗
Interesting demo, but does it really speed up coding more than just typing? Seems like a novel UI for an LLM.
raesene9 · 1790492423 · focus · HN ↗
One other option, if you're an obsidian user is, just add the excalidraw plugin there, and then you can ask your agent to just create excalidraw diagrams as needed.
Definitely Opus 5.5 via claude code has no problems generating Excalidraw images with no additional software, just based on conversation.
jamespaynter01 · 1790495989 · focus · HN ↗
Am I the only psycho getting Claude to draw tikz images?
alescalaios · 1790503151 · focus · HN ↗
[dead]
yuvraj_ceisium · 1790514156 · focus · HN ↗
Interesting, i tried doing something similar with webmcp so you can edit the canvas directly from inside the codex.
<a href="https://www.youtube.com/watch?v=VaTTgnA9a8A&t=6s" rel="nofollow">https://www.youtube.com/watch?v=VaTTgnA9a8A&t=6s
shtse8 · 1790517152 · focus · HN ↗
[dead]
dominotw · 1790524728 · focus · HN ↗
these work like peter_adjusting_blinds.gif

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.