Thread

1. mdp2021 · 1789912749 · focus · HN ↗

   How do you use this model locally, similarly to using `llama-server -m <model>`?

   (I mean: outside direct or substantial use of Python, and running the Neural Network in the most efficient way.)

   1. fp64 · 1789913205 · focus · HN ↗

      on the linked GitHub page they list support Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V with links to each

      1. mdp2021 · 1789916744 · focus · HN ↗

         > Diffusers, ComfyUI, vLLM-Omni, SGLang, and LightX2V

         I think that's all Python (not a direct executable).

         You could just do (see the "Quick Start") four `pip install` and have a dozen lines script to generate the image. But `llama.cpp` and similar do not require e.g. installing Torch (or PyTorch) - you can use `llama.cpp` on a non-specialized machine.

         1. wgd · 1789920581 · focus · HN ↗

            "just"

            I don't think I have ever once run "pip install transformers" and had it work without three rounds of fiddling

            1. mdp2021 · 1789935270 · focus · HN ↗

               Or trying to install the whole of CUDA on machines that do not even have a GPU (not Nvidia, not anything past the embedded)...

               Yep, that's (also) what I meant ;)

               Lean, efficient... Also sensible and trouble-less.

               1. iugtmkbdfil834 · 1789938994 · focus · HN ↗

                  The 'not nvidia' is no longer a deal breaker by itself.

Unofficial Hacker News client; not affiliated with Y Combinator.