Thread

OpenDLSS: A Vulkan Reimplementation of Nvidia's DLSS 5 Neural Rendering Network

277 points · 126 comments · sagacity

Lerc · 1790845274 · focus · HN ↗
I'm rather surprised that it doesn't take the z-buffer as an input. I would have thought that would have provided useful information, it's one of the more useful forms of contolnet.
1. avaer · 1790845516 · focus · HN ↗
  Even relatively small RGB -> depth models are pretty good. Which kind of implies depth is well encoded in the RGB, and adding depth would not really reduce entropy, while costing bandwidth.
2. rcarmo · 1790846155 · focus · HN ↗
  The official one seems to do, as well as other info from the engine (I think remember their mentioning LOD/UV map hints in one of the public demos, or articles, a few months back--or it might have been an Unreal Engine podcast)
  1. strangecasts · 1790849930 · focus · HN ↗
    The technical report suggests it does not use the depth buffer: <a href="https://research.nvidia.com/labs/adlr/files/DLSS5_Report.pdf" rel="nofollow">https://research.nvidia.com/labs/adlr/files/DLSS5_Report.pdf
    > The inference interface uses the engine-rendered RGB image as a dense, registered observation of visible scene appearance. It provides dense, pixel-aligned evidence for object support, occlusion boundaries, composition, and local material properties; engine motion vectors separately provide temporal correspondence.
    > Existing image generative models commonly rely on text embeddings, exemplar images, or spatial control fields such as depth, edges, segmentation, and pose [...] These conditions are effective for general-purpose generation and editing, but they do not uniquely determine the object identities, materials, visibility relationships, lighting decisions, and pixel-aligned detail contained in an engine-rendered frame. DLSS 5 is therefore conditioned on the rendered frame itself.
    mikepurvis · 1790866813 · focus · HN ↗
    Obviously they tried it multiple ways have brought receipts, but nonetheless it seems surprising that it wouldn't be of benefit to bring as much of that kind of metadata to the model as possible. You'd think depth and segmentation in particular would basically just be a straight shortcut without which the model spends its own time and effort re-deriving that stuff.
    I'd also be interested in how post-processing fits in with this. Like if you've got weather effects, film grain, tone mapping, etc, I would have thought the model would do better working on the image before those processes.
    strangecasts · 1790868484 · focus · HN ↗
    I think it has more to do with what kind of data they have access to at runtime - IIRC DLSS upscaling has only required the previous frames and motion vectors, so requiring depth buffers would mean it was no longer a "drop-in" replacement
    > I'd also be interested in how post-processing fits in with this.
    I think screenspace effects like film grain and tonemapping are excluded in the same way UI elements are rendered separately from the game.
    Stevvo · 1790872687 · focus · HN ↗
    That doesn't make sense; every game has a depth buffer, but not every game has motion vectors.
    ChocolateGod · 1790878033 · focus · HN ↗
    Because the depth buffer isn't always accurate or accessible.
    WoW goes to lengths to hide its depth buffer.
    strangecasts · 1790895232 · focus · HN ↗
    Oh, d'oh, no you are right, DLSS 2 on does pass the depth buffer to the model <a href="https://developer.download.nvidia.com/video/gputechconf/gtc/2020/presentations/s22698-dlss-image-reconstruction-for-real-time-rendering-with-deep-learning.pdf" rel="nofollow">https://developer.download.nvidia.com/video/gputechconf/gtc/... (but the neural rendering does not seem to rely on it?)
    jayd16 · 1790876495 · focus · HN ↗
    Hmm well the depth buffer only has accurate depth for opaque objects (and even that's not really true). Things like hair wouldn't be in there. Its not ground truth depth.
    Lerc · 1790914362 · focus · HN ↗
    It is data, It doesn't have to be ground truth Depth. In fact the difference between what optically appears to be at one depth and the depth map depth is in-itself information. You have described a mechanism that the model can use to identify that what it is supposed to be enhancing is more likely to be hair.
    awill88 · 1791012789 · focus · HN ↗
    I guess in this example since the model wasn’t trained on that, just like they said.
    If that data is some kind of missing key, go make your own and stop nitpicking someone who made a great point as to why they didn’t want that oh so precious data
    bcatanzaro · 1790894222 · focus · HN ↗
    Integration cost is very important to DLSS.
    More information is better, but more games is also better.
  2. cubefox · 1790884711 · focus · HN ↗
    The Nvidia video presentation on DLSS 5 says that the model was only trained with various G-buffers as input (including the depth buffer) but during inference, the model only uses the rendered frame. As well as the previous rendered frame reprojected via motion vectors, if I understand correctly, likely to improve temporal stability.
3. strangecasts · 1790846395 · focus · HN ↗
  I think this is mainly so it can use the existing hooks for DLSS upscaling without requiring changes to the renderer, AMD is working on a comparable method which uses adapter networks to slot normals and material properties from the renderer into the diffusion model: <a href="https://gpuopen.com/learn/temporally-stable-generative-illumination/" rel="nofollow">https://gpuopen.com/learn/temporally-stable-generative-illum...

Open on Hacker News to reply ↗

Unofficial Hacker News client; not affiliated with Y Combinator.