Not with any amount of temporal cohesion. This is using neural rendering that can stay consistent across arbitrary lights and their movement. But the authors chose to go proprietary, hopefully we’ll get some open alternatives in the near future.
This isn’t my work - just sharing the approach. I agree this should have been an open and reproducible publication. It won’t be long for this to become a normal feature in neural rendering engines though, there’s (IMO) no breakthrough novel idea in this pipeline.
I would like to implement this feature in software.
Now I cannot legally do so. Not even in free software.
We have to wait 20 years for this shit to expire. And by that point half of the people here will be retired and technology will have moved on, but this one important technique and feature will still be locked away.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
Or outside imagery: 3D printing, only took off after the patents expired. Which is now a huge industry. What could have been would that patent or with it expiring earlier?
Were you going to do in your basement all the computation necessary for the research? This is something that can't reasonably be accomplished by a hobbyist on their own. If it's going to be done, it's got to be financed by some larger entity.
Do you want that to be publicly-financed, or owned by some corporation with deep pockets?
I would do it illegally right now if I had the hardware resources. The article is 3 extremely vague pages but from the photo description it looks pretty simple.
You make a diffuse light only version of the image with one trained NN, estimate a depth map from it, take the delta from diffuse to original image and reverse raytrace it with Mitsuba to get the environment map, use differentiable rendering to optimize the environment map and make a new point light map.
You're left with a 3D environment where you can move the lights around and call Mitsuba again to get a physics based render. That's the whole innovation. I guess the secret sauce is in the optimization algorithm, but it could just be basic gradient descent.
That's pretty close to magick. Kudos!
Am I mistaken in thinking this was already doable with diffusion models?
Not with any amount of temporal cohesion. This is using neural rendering that can stay consistent across arbitrary lights and their movement. But the authors chose to go proprietary, hopefully we’ll get some open alternatives in the near future.
Neat! And the video was pretty interesting!
Wait ... patent pending? Research done with public money? Yuck! Downvoted immediately.
This isn’t my work - just sharing the approach. I agree this should have been an open and reproducible publication. It won’t be long for this to become a normal feature in neural rendering engines though, there’s (IMO) no breakthrough novel idea in this pipeline.
> patent pending
Ugh, more reason to want AI to automate research. No more patents.
This is going to be used to shake down teams that had no idea these folks happened to do this off in their own corner of the world.
Could you expand on that?
I would like to implement this feature in software.
Now I cannot legally do so. Not even in free software.
We have to wait 20 years for this shit to expire. And by that point half of the people here will be retired and technology will have moved on, but this one important technique and feature will still be locked away.
See also: gif, mp3, h.264/5, seam carving, the ribbon, etc.
Isn't the alternative that everything stays secret/proprietary, forever?
Or outside imagery: 3D printing, only took off after the patents expired. Which is now a huge industry. What could have been would that patent or with it expiring earlier?
Were you going to do in your basement all the computation necessary for the research? This is something that can't reasonably be accomplished by a hobbyist on their own. If it's going to be done, it's got to be financed by some larger entity.
Do you want that to be publicly-financed, or owned by some corporation with deep pockets?
I would do it illegally right now if I had the hardware resources. The article is 3 extremely vague pages but from the photo description it looks pretty simple.
You make a diffuse light only version of the image with one trained NN, estimate a depth map from it, take the delta from diffuse to original image and reverse raytrace it with Mitsuba to get the environment map, use differentiable rendering to optimize the environment map and make a new point light map.
You're left with a 3D environment where you can move the lights around and call Mitsuba again to get a physics based render. That's the whole innovation. I guess the secret sauce is in the optimization algorithm, but it could just be basic gradient descent.
> Ugh, more reason to want AI to automate research. No more patents.
Why would AI research be any less patented than non-AI research?
It can’t be copy writable.