Thread

1. Lerc · 1790845274 · focus · HN ↗

   I'm rather surprised that it doesn't take the z-buffer as an input. I would have thought that would have provided useful information, it's one of the more useful forms of contolnet.

   1. rcarmo · 1790846155 · focus · HN ↗

      The official one seems to do, as well as other info from the engine (I think remember their mentioning LOD/UV map hints in one of the public demos, or articles, a few months back--or it might have been an Unreal Engine podcast)

      1. strangecasts · 1790849930 · focus · HN ↗

         The technical report suggests it does not use the depth buffer: <a href="https:&#x2F;&#x2F;research.nvidia.com&#x2F;labs&#x2F;adlr&#x2F;files&#x2F;DLSS5_Report.pdf" rel="nofollow">https:&#x2F;&#x2F;research.nvidia.com&#x2F;labs&#x2F;adlr&#x2F;files&#x2F;DLSS5_Report.pdf

         &gt; The inference interface uses the engine-rendered RGB image as a dense, registered observation of visible scene appearance. It provides dense, pixel-aligned evidence for object support, occlusion boundaries, composition, and local material properties; engine motion vectors separately provide temporal correspondence.

         &gt; Existing image generative models commonly rely on text embeddings, exemplar images, or spatial control fields such as depth, edges, segmentation, and pose [...] These conditions are effective for general-purpose generation and editing, but they do not uniquely determine the object identities, materials, visibility relationships, lighting decisions, and pixel-aligned detail contained in an engine-rendered frame. DLSS 5 is therefore conditioned on the rendered frame itself.

         1. mikepurvis · 1790866813 · focus · HN ↗

            Obviously they tried it multiple ways have brought receipts, but nonetheless it seems surprising that it wouldn&#x27;t be of benefit to bring as much of that kind of metadata to the model as possible. You&#x27;d think depth and segmentation in particular would basically just be a straight shortcut without which the model spends its own time and effort re-deriving that stuff.

            I&#x27;d also be interested in how post-processing fits in with this. Like if you&#x27;ve got weather effects, film grain, tone mapping, etc, I would have thought the model would do better working on the image before those processes.

            1. strangecasts · 1790868484 · focus · HN ↗

               I think it has more to do with what kind of data they have access to at runtime - IIRC DLSS upscaling has only required the previous frames and motion vectors, so requiring depth buffers would mean it was no longer a &quot;drop-in&quot; replacement

               &gt; I&#x27;d also be interested in how post-processing fits in with this.

               I think screenspace effects like film grain and tonemapping are excluded in the same way UI elements are rendered separately from the game.

               1. Stevvo · 1790872687 · focus · HN ↗

                  That doesn&#x27;t make sense; every game has a depth buffer, but not every game has motion vectors.

                  1. ChocolateGod · 1790878033 · focus · HN ↗

                     Because the depth buffer isn&#x27;t always accurate or accessible.

                     WoW goes to lengths to hide its depth buffer.

Unofficial Hacker News client; not affiliated with Y Combinator.