It's incumbent upon you to show your work first, before demanding somebody else do it first. And if really helps if your math and understanding of bandwidth and latency isn't wrong, and even more if you can show you have some actual first hand experience in the field, like developing and shipping a working product.
I don't think your numbers add up either, and you're wildly underestimating the difficulty and complexity of the problem. Which real time multi player VR simulations have you shipped?
I'm happy to show you my work, an application called "Pantomime" that I developed and shipped 7 years ago, which runs smoothly over local WiFi, but terribly over long haul internet. My work that I'm showing you taught me that merely 2mm translational resolution and 1/6 degree rotational resolution would totally ruin the physics simulation and sense of immersion and realism:
https://www.youtube.com/watch?v=NEeFdDluEJ8
https://apps.apple.com/nl/app/pantomime-bug-farm/id969738938
https://apps.apple.com/nl/app/reality-construction-kit/id151...
Representing user input and object rotations and positions as precise floating point numbers is absolutely necessary when there is a physics simulation involved (see "good VR" prerequisite #3 below, "graphically and physically realistic worlds").
The "Butterfly Effect" explains why slight differences in initial conditions (and especially the high frame and precision with which the iPad can measure real time human input gestures, which can be meaningfully gentle and nuanced, like carefully balancing a spinning coin on an iPad, or gently pushing over a stone monolith -- see 2:00 in the video I linked to above) will cascade into enormous differences in later state.
https://en.wikipedia.org/wiki/Butterfly_effect
The person giving the Pantomime demo is my colleague David Levitt, who used to work with Chuck Blanchard, Thomas Zimmerman, and Jaron Lanier at VPL, and whose definition of the three goals of "good" VR I've written about on HN before, over 8 years ago:
https://news.ycombinator.com/item?id=8023893
DonHopkins on July 12, 2014 | parent | context | favorite | on: What I mean when I say "I think VR is bad news"
Here are three goals for "good VR" from David Levitt, with whom I work at Pantomime Corporation, and who used to work on VR at VPL:
http://pantomimecorp.com/pantomime-technology/virtual-realit...
When Pantomime co-founder Levitt was a research scientist and product manager with VPL Research, the inventors of virtual reality, they had three prerequisites for a VR system:
1) a way to reach in, in 3D
2) shared reality — support for multiple users and viewpoints
3) graphically and physically realistic worlds
VPL offered a DataGlove to provide 3D input, while its flagship Reality Built for Two VR product offered networked multi-person worlds. Expensive graphics computers and custom hardware brought the full 1992 price to $500,000, which only a few huge corporations could afford.
When Dr. Levitt joined VPL, thanks to an amazing infrastructure by lead VPL engineer Chuck Blanchard, he added realistic gravity, collisions, and throwing a ball into the VR system for physical realism.
But two decades later, the public and technologists have become so impatient that the new systems calling themselves VR have punted even on the core original criteria.
Head-mounted systems like the Oculus Rift offer no way to reach in. In demos, visitors twiddle a 1980s style game joystick. And users don’t natively network — in an Oculus demonstration you don’t see the other users in the VR world — not even the other players sitting alongside you in the demo.
http://pantomimecorp.com/2014/06/10/facebook-vp-to-pantomime...
David Levitt: "I work in Virtual Reality, and everyone’s wondering what you can say about your acquisition of Oculus VR. In particular, I’ve had demos of it: I could look around but I couldn’t reach in. Do you have solutions for that that you can talk about?"
Jay Parikh: “You can’t interact with anything. These are big, hard problems … what you do with your hands, because you can’t do anything with your hands — or it’s hard to be using a controller when you can’t see your hands and you have the goggles on — these are problems we have to solve in a good and seamless way.”
More on the VPL DataGlove:
http://www.britannica.com/EBchecked/topic/630181/virtual-rea...
[...]
> [My software] runs smoothly over local WiFi, but terribly over long haul internet.
I'm pretty sure that this is the core of your confusion. We're talking about software that handles ~3000 simultaneous participants. You're not going to find a userbase that large all running on a LAN (wired or not) outside of a convention hall or research facility. So, we're necessarily talking about networked software that users run on their (often abysmally designed) home networks and connect to other players over their (frequently terrible) Internet connections.
I expect that the tolerances you've specced out are absolutely correct for simulation of the local player. They may have even been reasonable for remote players on a LAN. However, those tolerances are overkill when you're designing a system that's designed for use by the general public on the greater Internet. This means that it _must_ do latency compensation for 100->500ms of potentially-highly-variable RTTs, deal with 1%+ packet loss, and also -if it has any even-vaguely-serious adversarial gaming components- have a server component that handles anti-cheat by way of player input validation.
There's certainly value in the high-fidelity VR software that your company wrote. However, the fact that it worked poorly over the Greater Internet suggests that you folks either didn't have experience with writing mass-market networked video games, or that your focus was exclusively on a high-fidelity simulator that would never have to paper over the messes that you get on the regular from the home Internet connections of your typical gamers. (As one example, because of the general shittiness and unpredictability of the Internet, it's fairly common practice in networked video games for all clients to run their physics simulation locally. Clients get notifications of events that will affect objects in the simulation, and then they slam those events into their local sim and play the results locally. For events that might matter to all players in the game, -say some physics object gets bounced around that might block pathing- the server will do a comparatively low-fi simulation of that and validate clients' attempts to move into or through the blocked area.)
> It's incumbent upon you to show your work first, before demanding somebody else do it first.
My math is right next to that demand!
> My work that I'm showing you taught me that merely 2mm translational resolution and 1/6 degree rotational resolution would totally ruin the physics simulation and sense of immersion and realism
That's a much much smaller workspace, so at that scale it would be a fraction of a millimeter. I don't see why the degrees aren't enough though; that tablet looks to me like there's a small amount of jitter but enough to make it not matter.
But if you want to 10x the precision of everything I said, that's only 11 more bytes.
> Representing user input and object rotations and positions as precise floating point numbers is absolutely necessary when there is a physics simulation involved (see "good VR" prerequisite #3 below, "graphically and physically realistic worlds").
I can understand that, but even before worrying about diminishing returns, you can only throw so many bits at a bad input:
> It was determined that the translational accuracy of the [Oculus Rift S] was 1.66 ± 0.74 mm for the head-mounted display and 4.36 ± 2.91 mm for the controller, and the rotational accuracy of the system was 0.34 ± 0.38° for the HMD and 1.13 ± 1.23° for the controller.
> Kreylos estimated the precision of Lighthouse tracking to be around RMS 1.5mm and the accuracy around RMS 1.9mm.
So you could fit the full precision of those devices into 36-40 bytes, and that's before we even consider that most people are only tracking 3 points, not 6. Even with some extra precision, I already tossed on some more bytes and it doesn't really matter in the end whether it's 14Mbps or 20Mbps.
> But two decades later, the public and technologists have become so impatient that the new systems calling themselves VR have punted even on the core original criteria.
> Head-mounted systems like the Oculus Rift offer no way to reach in. In demos, visitors twiddle a 1980s style game joystick. And users don’t natively network — in an Oculus demonstration you don’t see the other users in the VR world — not even the other players sitting alongside you in the demo.
That's a real shame, and worth keeping in mind, but I was talking about taking the kind of VR immersion that is currently widely deployed and adding more people.
I've learned so much from your disagreement. :) Thanks for sharing all of this information. Seriously