Thread

Qwen Image 2.1

740 points · 199 comments · jmillikin

mdp2021 · 1789912749 · focus · HN ↗
How do you use this model locally, similarly to using `llama-server -m <model>`?
(I mean: outside direct or substantial use of Python, and running the Neural Network in the most efficient way.)
1. embedding-shape · 1789912894 · focus · HN ↗
  Probably ComfyUI is one of the easiest way to get started with local image/video models. Or perhaps vLLM, if they have support for it already, would be something like `vllm serve <model> --omni --port 9080`
2. utopiah · 1789913176 · focus · HN ↗
  why not just as you suggested i.e. <a href="https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.html#llama-server" rel="nofollow">https://qwen.readthedocs.io/en/latest/run_locally/llama.cpp.... then get the result either via a UI or wget/curl it back?
  1. mdp2021 · 1789913434 · focus · HN ↗
    I am not sure that llama.cpp also supports image generation models.
    utopiah · 1789913646 · focus · HN ↗
    it's multimodal, see <a href="https://github.com/ggml-org/llama.cpp/blob/master/docs/multimodal.md" rel="nofollow">https://github.com/ggml-org/llama.cpp/blob/master/docs/multi...
    exe34 · 1789913964 · focus · HN ↗
    Multimodal doesn't guarantee input and output.
    > Currently, we support image, audio and video input.
    utopiah · 1789915865 · focus · HN ↗
    Seems I'm missing something. Does this model support other inputs?
    Image outputs are supported, videos I'm not sure but I don't think that's an output, just a preview of the equirectangular example, so, same question here, what does this model outputs that isn't supported?
    [deleted] · 1789916261 · focus · HN ↗
    [deleted]
    exe34 · 1789917509 · focus · HN ↗
    It's a diffusion model, completely different from autoregressive attention models.
    utopiah · 1789922089 · focus · HN ↗
    like <a href="https://github.com/ggml-org/llama.cpp/tree/master/examples/diffusion" rel="nofollow">https://github.com/ggml-org/llama.cpp/tree/master/examples/d... ?
    At the risk of stating the obvious llama.cpp isn't just about LLaMa as <a href="https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h#L13" rel="nofollow">https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... someone else pointed out.
    exe34 · 1789926056 · focus · HN ↗
    Aha I was wrong. Thanks for sharing that!
    utopiah · 1789926295 · focus · HN ↗
    no worries, I was wrong too, it is multimodal but only for inputs apparently, so for now there seem to only be text as output but no image as output
    mdp2021 · 1789917711 · focus · HN ↗
    Not all architectures are supported by llama.cpp . The GGUF format encodes the NN in a standardized way, but then you need code that can use that NN structure.
    I understand that llama.cpp could only output text, last time I checked (I do not know how to find a good source for that though).
    See <a href="https://github.com/ggml-org/llama.cpp/blob/master/src/llama-arch.h" rel="nofollow">https://github.com/ggml-org/llama.cpp/blob/master/src/llama-... , the
    enum llm_arch {
    ...
    utopiah · 1789922613 · focus · HN ↗
    I haven't used llama.cpp for image generation either but I recall an issue about it. Unfortunately I can't pinpoint it now and there is the older closed issue <a href="https://github.com/ggml-org/llama.cpp/issues/4408" rel="nofollow">https://github.com/ggml-org/llama.cpp/issues/4408 so unless mtmd supports also multimodal outputs out of the box safe to assume output is still limited to text.
3. fp64 · 1789913205 · focus · HN ↗
  on the linked GitHub page they list support Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V with links to each
  1. mdp2021 · 1789916744 · focus · HN ↗
    > Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V
    I think that's all Python (not a direct executable).
    You could just do (see the "Quick Start") four `pip install` and have a dozen lines script to generate the image. But `llama.cpp` and similar do not require e.g. installing Torch (or PyTorch) - you can use `llama.cpp` on a non-specialized machine.
    wgd · 1789920581 · focus · HN ↗
    "just"
    I don't think I have ever once run "pip install transformers" and had it work without three rounds of fiddling
    mdp2021 · 1789935270 · focus · HN ↗
    Or trying to install the whole of CUDA on machines that do not even have a GPU (not Nvidia, not anything past the embedded)...
    Yep, that's (also) what I meant ;)
    Lean, efficient... Also sensible and trouble-less.
    iugtmkbdfil834 · 1789938994 · focus · HN ↗
    The 'not nvidia' is no longer a deal breaker by itself.
4. Iolaum · 1789913813 · focus · HN ↗
  There is difussion.cpp which is intended for those types of models. I set up krea-2-turbo with the help of ChatGPT 2 months ago, if you have a capable computer that's what I would suggest once it becomes supported.
5. rwmj · 1789918110 · focus · HN ↗
  Additional question is what kind of local hardware would be required for this? 7B parameters sounds very light weight, but I'm not sure. (Edit: The download is 33 GB).
  Edit x2: As usual I'm in a twisty maze of pip packages that don't work together, with obscure errors about missing modules, even though I followed the instructions on the page to the letter. I really wish people didn't use Python for this stuff. A simple C/C++ program would be so much better.
  1. peri-cl · 1789922686 · focus · HN ↗
    It's about 16 GiB at Q8 quants (combining both the image and language parts). (Meaning, community quantized models from HuggingFace).
    I think it will technically run on anything that has enough memory. I just tried it on a standard laptop (dual-channel DDR5), and it took about 3 minutes for a 512x512. If you'd want to run it at interactive speeds, you would want a GPU (one which fits this in VRAM).
    > "I really wish people didn't use Python for this stuff. A simple C/C++ program would be so much better."
    You mean besides stable-diffusion.cpp ?
    rwmj · 1789926658 · focus · HN ↗
    > You mean besides stable-diffusion.cpp ?
    Yes, thanks, I didn't know about that. Will try it.
6. [deleted] · 1789918558 · focus · HN ↗
  [deleted]
7. nkhgfugjk · 1789918645 · focus · HN ↗
  I am using sd.cpp, which is the cousin of llama.cpp: <a href="https://github.com/leejet/stable-diffusion.cpp" rel="nofollow">https://github.com/leejet/stable-diffusion.cpp
  it already has day-0 qwen image 2.1 support!
8. peri-cl · 1789922347 · focus · HN ↗
  I've set it up on my local machine just now, as my first local image diffuser. I can confirm it's very easy.
  I tried stable-diffusion.cpp, following its compile guide here[0], and its Qwen Image-2.1 specific instructions here[1]. It works out of the box. I made a test pelican[2]. It took 3 minutes on a CPU.
  [0] <a href="https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/build.md" rel="nofollow">https://github.com/leejet/stable-diffusion.cpp/blob/master/d...
  [1] <a href="https://github.com/leejet/stable-diffusion.cpp/blob/master/docs/qwen_image_2.1.md" rel="nofollow">https://github.com/leejet/stable-diffusion.cpp/blob/master/d...
  [2] <a href="https://i.ibb.co/yMknC2K/output.png" rel="nofollow">https://i.ibb.co/yMknC2K/output.png
  1. mdp2021 · 1789935115 · focus · HN ↗
    Thank you! Can you please check how much RAM does it consume (and require)?
    peri-cl · 1789936851 · focus · HN ↗
    This is what the runtime reports, at Q8:
    total params memory size = 15645.19MB (VRAM 15645.19MB, RAM 0.00MB): text_encoders 7669.77MB(VRAM), diffusion_model 7331.05MB(VRAM), vae 644.38MB(VRAM), controlnet 0.00MB(N/A), extensions 0.00MB(N/A)
    mdp2021 · 1789938015 · focus · HN ↗
    That suggests that 16GB RAM will not be enough.
    In fact, like it appears in the reports above, it is "7b" as in
    > 7B parameters in its visual generation component
    It seems they calibrated the size to fill a 16GB VRAM near the limit. RAM requirements will vary.
9. leumon · 1789925683 · focus · HN ↗
  Unsloth Desktop is the easiest way imo. There are already gguf quants of this model, or simply wait until the official one comes out.
10. finnjohnsen2 · 1789929299 · focus · HN ↗
  I use opencode + <a decent saas llm> to set up all this new ai generation stuff. GLM-5.3 is my current gun. Safely inside podman containers too because I dont trust this fast moving python eco system at all. Never do I want this running on my main OS.
  I have FLUX.2 klein and dev, Ideogram, LaDA-Image and SenseNova locally. Works great. Ive never touched a file.
  The days of making container yamls myself is over. I read them but I dont edit anymore.
11. iugtmkbdfil834 · 1789938732 · focus · HN ↗
  I am on AI max 395, comfyUI+qwen models is all you technically need. With today's release, I just built a quick and dirty html that allow simpler prompt use and edits ( via headless comfyui ).. its not bad for a day's work, but a little too unpolished to publish. I would say, try comfyUI first ( complex, but it worked OOTB ).

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.