I made this, fun to see my project on the front page!
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
Is a basic webcam really able to read text at that angle for the “what is this word” example?
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
In that demo, the lightbulb sits on the desk, less than a meter away from the page. What feels implausible about this setup? This is kind of thing you can do with your smartphone today.
So is the demo real or is it smoke and mirrors? Could you have pointed at other words and have it work?
(I think I overestimated how much of an angled shot it was the first time I saw it, on top of how easy or hard turning your finger into “actually capture this text”… but you’re totally right that in principle a camera at a non agressive angle can capture that text)
(I like this overall concept a lot! Cooking is the right example, voice interfacing’s 96% use case is basically “my hands are dirty but I must use computing”)
It doesn't feel implausible, it is implausible, because your idea would require either a camera with good optics pointing in a specific direction, which means it's no longer a set-and-forget lightbulb, or some kind of 360 view more-or-less 3D camera to cover all the angles, which probably isn't going to have the resolution to read angled text at that distance.
At the very least it's going to have a range limit on that feature.
And there's a projector in there too.
In something smaller than a smart phone.
It's a great concept, but not a very workable one. Kitchens get steamy, tables can be huge, and management will want to use the cameras for user surveillance.
The author doesn't come out and say it, but I don't think there's anything inside that lightbulb in the pictures and videos.
The actual webcam and projector and Mac are not shown.
If you look at the map example, the shadows from the hands go almost straight down. The projector is not in the lightbulb, it is above the table out of frame. Probably the webcam is up there too.
Which of course greatly diminishes the "cool" factor.
I appreciate how you used the light bulb glow as a "prop light" (in stage lighting: a lamp the audience sees but doesn't actually illuminate the scene) to sell the effect in your demo video.
Seems simple, but it really brought it up. Many folks would've stopped after getting the projector working. I should've known, given my experience with projectors, and it fooled me too! :)
"A bad design with good presentation is doomed eventually. A good design with bad presentation is doomed immediately."
I met @heliographe a few years ago in a HN Tokyo (Slack) meeting. Happy to see you are still designing awesome things after your time at Apple. I look forward to seeing how your project(s) develop. PS: as soon as I saw the kitchen I knew this was Japan, and then I saw the hair and though —“hmm, I feel like I have seen this person before”— and it was you!!
Right. Tiny projectors have been done before.
Samsung had a smartphone with a projector back in 2014, but nobody wanted it.
Some startup called Moviphone has announced a new one for 2027.[1]
The use cases have not been compelling and fighting ambient lighting with a small projector is always tough.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door.
Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
It's a good thing that most inventors don't fall for the "something vaguely like this has been tried with inferior technology before, and it sucked" line of reasoning, or our world would look very different :)
Well, if you're a inventor, you look at the prior art to see if it's been done before. I hold six issued patents, two of which were very profitable. I had one patent rejected because I missed relevant prior art from 1948.
Then again, I feel like I know industrial designers who play with building on ideas that came before, reexamining whether those ideas might have a fresh form and a place in the world today.
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Legal counsel at multiple companies have always advised me to not do a patent search before writing up an invention. I guess often the lawyers do the search, and the inventors don’t.
That said, this project isn’t well summarized as merely “tiny projectors”. Many patents have been granted for systems that include parts that are also patented. Indeed, I can see recently granted patents that include digital projectors. And many products have good sales without being patented; patents are not the end-all of invention. So I feel like you’ve been much too quick to dismiss.
> The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
Imagine downvoting John effing Nagle. However my humble opinion is that sometimes revisiting, if not reinventing, the wheel is beneficial in light of accrued epistemic benefits (or for the fun of tinkering with a cutesy creation) . Not for patenting, of course. Besides, the bulb form-factor is not too different from cctv cam ceiling installations.
This is an awesome concept. How do you see this working across several rooms (if you've thought this far)? This would be great for when I have ideas in the shower.
The appeal of the lightbulb form factor is that you can mount it in a portable base to move it around the house; or in a fixed socket for broader room coverage.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
A lot of people have Alexa or Google devices in each of their rooms, this seems like a very logical step. Perhaps 2 generations from what is being currently released based on how things go. Maybe sooner!
Could you do beamforming with this, or would that get too expensive/awkward? At the very least, having more of these guys in the house (if you can get the tech cheap, ofc) would lead to better coverage potentially?
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
Its sad that search engines have become useless. There was another UX researcher who did something very much like this and then later got recruited to Google. As I recall he was using the cameras in Wii Nunchucks because they could do SLAM easily. And yet I cannot find his research, sigh.
This is the most impressive tech demo I have seen in years. Spatial computing just became real. I hope, I can buy something like this in a few years. Thanks you for inventing and sharing.
Nice. This reminds me how I felt watching the first pre-iPhone multitouch demos.
One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (using a short low-FPS video buffer?) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Alternately a low-latency speech analyzer could look for the word "here" and quickly snap a screenshot. Perhaps project a temporary dot on the surface, so the user knows they can put their hand down now?
Quite the impressive demo! There's huge potential here.
Another very small usability improvement might be a "follow where I'm pointing" vision algorithm. This way, you don't have to stretch your arm out so far.
Early multitouch in drafting table and wall-mount form factors (ie similar
to what's shown) suffered from arm fatigue after extended use, aka Gorilla Arm syndrome.
Perhaps, using vision, a Lightbulb Computer could greatly reduce that fatigue, even in short sessions? All by itself that would be a big deal.
Quite nice. But this imbues the house with a constant presence of the machine. A jinn which is always ready to grant your wish, but never leaves you alone, ever watching and listening. The fact that the jinn takes orders spoken in normal human language amplifies the effect quite a bit. I would have rather not have to feel such presence in my bathroom. Or bedroom. I would still be glad to be able to operate "smart" switches, and even find items. But I would rather explicitly unlock my phone, start an app, and scan the bookshelf, until a book is highlighted.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
For me, a lot would depend on who controls the information gathered by the machine. If it’s all purely local and under my control, I doubt I’d even feel it as a “presence” - I already have that today with various cameras in my house, and it doesn’t bother me because I know the data never leaves my local system.
If it’s all being sent to some big tech co, then it becomes extremely different.
Sadly the reality of consumer product development under current economic and legal frameworks is that we’re much more likely to get the latter than the former.
I've got something similar to this using cheap ONVIF security cameras and kasa smart bulbs. I have an nvidia card process the capture and for the smart speaker I use just a microphone and speaker hooked up to a normal computer. The architecture is spread across many machines and it's nowhere close to this realtime. Also the video capture is fairly lousy because the lighting and resolution and coverage isn't great. The cameras can change the lighting to try to get a better shot. (well it's qwen with toolcalls)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total
still researching them ... some of these look like garbage ... i'll come back to this thread in a few days when i find something decent. I don't want high nits or resolution, just want to be able to send pixels and not have it be noisy and get a response in under 500ms.
Sadly if it ever gets manufactured it will use cloud and become just a better version of Flock. Even cat in the photo looks with obvious displeasure realizing this. Concept itself is novel and amazing though.
It's still the same old concept of "magic surfaces", using screens, glasses, or projectors (all implementation details) to realize it is irrelevant.
Speaking of implementation details, projecting light on real-world surfaces absolutely sucks for this concept to gain adoption: issues like angular distortion, color accuracy, brightness, and energy efficiency are unsolvable. Don't waste time beating this dead horse (again).
Economy of scale currently favors the screen approach, but it's awkward to integrate multiple screens from different vendors to provide a seamless experience.
Both screens and projectors require multiple units to achieve the demo scenes. Screens would have to be mounted on much more surfaces of the home, while at least one projector is needed for each room. The cost adds up quickly.
That leaves us the only hope with smart glasses. Actually, smart glasses are also a type of projector——casting light not onto real-world surfaces, but directly into our eyes. It has the benefit of requiring just one unit for each viewer, and it could adapt to any kind of space, not limited to just surfaces. It has no inherent shortcomings of angular distortion/color accuracy/brightness. Its energy efficiency is the best and will keep improving. It just needs many more iterations to miniaturize so it can be just the size/weight of normal eye glasses to be unnoticed.
I love to see the human-centric vision! This reminds me of Lenovo and Microsoft's "table computer" experiments from ~15 years ago. I remember being really excited at the possibility of my coffee table also being a computer--shared experiences, board games (I think lenovo even sold dice and pawns that connected to it). I had visions of having it also connected to the TV/Projector for controlling media. That was even before I had my first "smartphone". It could have been really cool!
One of the many cancelled form-factors the various FAANGs were playing with was a projector-centric assistant that screwed into a standard lightbulb fitting. The idea was it would placed over your desk/table and project downwards onto the surface. I still think its a very cool concept, even though none of the experiments ever panned out well enough for a real product.
This is an amazing concept; I went looking for a purchase link before I realized it's just a concept and not actually a real product. I would absolutely put one of these in my house provided it was open software that didn't send video out of my network.
Author here; I was definitely inspired by my visits to Dynamicland many years ago, which Folk Computer is also inspired by.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
An interesting concept. I'm (probably) one of the few people that thought the Humane Pin had an interesting idea at its core, it was just executed poorly at the wrong time. A stationary version that leans into ambience solves a lot of the interaction and engineering issues.
I also like that it addresses my gripe with XR interfaces; you aren't actually sharing a digital object in space and it looks weird to everyone else.
I love it. I only skimmed through the post but if this project has/will have a "hey lightbulb, where did I put my keys/wallet/brain" function I'm buying/building it on the spot.
What does the lightbulb have to do with the projection? Obviously the lightbulb is not the camera... nor the projector (nothing clear is getting through a frosted white shell). Is the lightbulb just a prop and imagined form-factor?
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
I realize this is not the makers' focus, but I find myself less enthused with the human-computer interaction side and more anxious about how we can ensure the human maintains true ownership and agency over the device and its software.
I mean, merge one of these with a modern anti-consumer "Smart TV", and we're uncomfortably close to 1984's telescreen.
I totally agree. It feels like we’ve crossed a line where tech is overtly hostile to users because everyone is chasing recurring revenue and data mining is the only viable option
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I think the mission statement here is on to something. Helmets, obtrusive physical appendages etc are unappealing to me. The idea of using extant space or surfaces however, is extremely appealing. And while it might suggest lower quality today, as with most things, it would almost certainly improve over iterations and time.
Holographic computing, I think, will be the future. At least for interfaces.
I don't know about these things. To me they just turn everything into a screen.
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
The future of computing is certainly multimodal. What I see with beaming is being both hands-free and prep-free further enhances the spontaneity of life and helps fill in small gaps.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
yeah, I don't disagree, though you could think that a robot with a heat-paper printer or pen could go a long way.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
I don't need the family planner to be on the wall 24/7 though, it can be projected only when I'm at the room. Maybe even only when I'm close to the wall.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
Most of the videos are shot in real life, during summer daylight (only exception are the 2 person around a table ones, which are shot at night in a room with regular ceiling lighting).
And the projector I used is only 1100 ISO lumens; fancier models go way higher.
If you project mostly white content, it's remarkably readable + visible. Outdoors in direct sunlight would be another story, of course.
I made this, fun to see my project on the front page!
The main question I tend to get from hacker-types is "what are the tech specs?". The demos run on a Mac with custom software I built for projection mapping/rendering/etc. (the hand tracking uses Apple's built-in framework, which is very good), a small consumer 4k laser projector, and a basic webcam for vision.
But really, this is primarily a research/design prototype - I think it's much more important to figure out what the user experience should ideally feel like, if the interactions are even plausible & desirable in the first place; and then work backwards to the technology.
Is a basic webcam really able to read text at that angle for the “what is this word” example?
I wasn’t sure if any of this was real and that example looked extremely fake to me because I had a hard time believing the device being able to read text from that distance.
In that demo, the lightbulb sits on the desk, less than a meter away from the page. What feels implausible about this setup? This is kind of thing you can do with your smartphone today.
So is the demo real or is it smoke and mirrors? Could you have pointed at other words and have it work?
(I think I overestimated how much of an angled shot it was the first time I saw it, on top of how easy or hard turning your finger into “actually capture this text”… but you’re totally right that in principle a camera at a non agressive angle can capture that text)
(I like this overall concept a lot! Cooking is the right example, voice interfacing’s 96% use case is basically “my hands are dirty but I must use computing”)
It doesn't feel implausible, it is implausible, because your idea would require either a camera with good optics pointing in a specific direction, which means it's no longer a set-and-forget lightbulb, or some kind of 360 view more-or-less 3D camera to cover all the angles, which probably isn't going to have the resolution to read angled text at that distance.
At the very least it's going to have a range limit on that feature.
And there's a projector in there too.
In something smaller than a smart phone.
It's a great concept, but not a very workable one. Kitchens get steamy, tables can be huge, and management will want to use the cameras for user surveillance.
The author doesn't come out and say it, but I don't think there's anything inside that lightbulb in the pictures and videos.
The actual webcam and projector and Mac are not shown.
If you look at the map example, the shadows from the hands go almost straight down. The projector is not in the lightbulb, it is above the table out of frame. Probably the webcam is up there too.
Which of course greatly diminishes the "cool" factor.
I appreciate how you used the light bulb glow as a "prop light" (in stage lighting: a lamp the audience sees but doesn't actually illuminate the scene) to sell the effect in your demo video.
Seems simple, but it really brought it up. Many folks would've stopped after getting the projector working. I should've known, given my experience with projectors, and it fooled me too! :)
"A bad design with good presentation is doomed eventually. A good design with bad presentation is doomed immediately."
Inspiring design and presentation. Thanks.
I met @heliographe a few years ago in a HN Tokyo (Slack) meeting. Happy to see you are still designing awesome things after your time at Apple. I look forward to seeing how your project(s) develop. PS: as soon as I saw the kitchen I knew this was Japan, and then I saw the hair and though —“hmm, I feel like I have seen this person before”— and it was you!!
Haha cheers! I should join one of those meetups again. And yes, it's funny how recognizable Japanese interiors are.
Right. Tiny projectors have been done before. Samsung had a smartphone with a projector back in 2014, but nobody wanted it. Some startup called Moviphone has announced a new one for 2027.[1] The use cases have not been compelling and fighting ambient lighting with a small projector is always tough.
Possible use case: projecting menus, ads, and bills on restaurant tables. With a camera and enough automated layout to work around what's on the table.
It might sell to the people who buy refrigerators with displays on the door. Could be useful as a cooking training aid, like cooking videos but present where you need them.
The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
[1] https://moviphones.com/
It's a good thing that most inventors don't fall for the "something vaguely like this has been tried with inferior technology before, and it sucked" line of reasoning, or our world would look very different :)
Well, if you're a inventor, you look at the prior art to see if it's been done before. I hold six issued patents, two of which were very profitable. I had one patent rejected because I missed relevant prior art from 1948.
Here’s a WiFi lightbulb I made 20+ years ago. https://boingboing.net/2003/05/14/wifi-nightlight.html
Then again, I feel like I know industrial designers who play with building on ideas that came before, reexamining whether those ideas might have a fresh form and a place in the world today.
Who iterate and hone until they have something worth existing in today’s world—often doing do more publicly than corporations’ R&D departments—then let the patent technicalities be the well-paid lawyers’ problem…
Nothing truly new under the sun
Legal counsel at multiple companies have always advised me to not do a patent search before writing up an invention. I guess often the lawyers do the search, and the inventors don’t.
That said, this project isn’t well summarized as merely “tiny projectors”. Many patents have been granted for systems that include parts that are also patented. Indeed, I can see recently granted patents that include digital projectors. And many products have good sales without being patented; patents are not the end-all of invention. So I feel like you’ve been much too quick to dismiss.
> The light bulb form factor is cute but probably not that useful. Most of the light will be aimed in non-useful directions. Look into ultra short throw projectors.
gonna hard disagree! you aren't going to have 360° projector in there with wasted light going nowhere. you're going to have 1-4 steerable projectors. the "facade" as a globe bulb means you can place it at any direction you like depending on where the cord needs to go. for ceiling as well you don't want it to have a fixed throw direction. bulb makes perfect sense.
Imagine downvoting John effing Nagle. However my humble opinion is that sometimes revisiting, if not reinventing, the wheel is beneficial in light of accrued epistemic benefits (or for the fun of tinkering with a cutesy creation) . Not for patenting, of course. Besides, the bulb form-factor is not too different from cctv cam ceiling installations.
> Imagine downvoting John effing Nagle.
You’re not doing anyone any favors by replacing genuine feedback with sycophancy. That’s how you end up with Nobel disease and similar afflictions.
This is an awesome concept. How do you see this working across several rooms (if you've thought this far)? This would be great for when I have ideas in the shower.
The appeal of the lightbulb form factor is that you can mount it in a portable base to move it around the house; or in a fixed socket for broader room coverage.
It's not mentioned in the piece, but for the latter I don't see any technical obstacle to having several of these in different rooms in the house, or a couple in the same room to augment coverage.
A lot of people have Alexa or Google devices in each of their rooms, this seems like a very logical step. Perhaps 2 generations from what is being currently released based on how things go. Maybe sooner!
>a small consumer 4k laser projector
How many watts does the projector use for how many lumens? I would be curious if these numbers plus size would be a fundamental limit to the tech.
Could you do beamforming with this, or would that get too expensive/awkward? At the very least, having more of these guys in the house (if you can get the tech cheap, ofc) would lead to better coverage potentially?
The main issue is probably gonna be that the display sucks compared to actual displays :( but it looks cool though! and the whole VLM thing makes a lot of sense
Just want to say, what an incredible vision. Congratulations and thanks for sharing!
Its sad that search engines have become useless. There was another UX researcher who did something very much like this and then later got recruited to Google. As I recall he was using the cameras in Wii Nunchucks because they could do SLAM easily. And yet I cannot find his research, sigh.
Is it Johnny Chung Lee and his work on using the IR camera in the Wii controller?
http://johnnylee.net/projects/wii/
Thank you! Yes his interactive whiteboard where he projects a whiteboard on to a surface was the project I was reminded of.
This is the most impressive tech demo I have seen in years. Spatial computing just became real. I hope, I can buy something like this in a few years. Thanks you for inventing and sharing.
It may have been inspired by it, but I was struck how similar to the vision Ikea/IDEO set out way back in 2015 for the "future of the kitchen"
https://www.ideo.com/case-study/designing-the-future-kitchen
Here's hoping the induction cooling/cooking becomes prevalent too!
Nice. This reminds me how I felt watching the first pre-iPhone multitouch demos.
One small quality-of-life improvement that could be added: have the software timestamp when you say "here" and then (using a short low-FPS video buffer?) analyze the corresponding frame of video. This would eliminate the pause when you're pointing at something, waiting for the computer to respond.
Alternately a low-latency speech analyzer could look for the word "here" and quickly snap a screenshot. Perhaps project a temporary dot on the surface, so the user knows they can put their hand down now?
Quite the impressive demo! There's huge potential here.
Another very small usability improvement might be a "follow where I'm pointing" vision algorithm. This way, you don't have to stretch your arm out so far.
Early multitouch in drafting table and wall-mount form factors (ie similar to what's shown) suffered from arm fatigue after extended use, aka Gorilla Arm syndrome.
Perhaps, using vision, a Lightbulb Computer could greatly reduce that fatigue, even in short sessions? All by itself that would be a big deal.
Quite nice. But this imbues the house with a constant presence of the machine. A jinn which is always ready to grant your wish, but never leaves you alone, ever watching and listening. The fact that the jinn takes orders spoken in normal human language amplifies the effect quite a bit. I would have rather not have to feel such presence in my bathroom. Or bedroom. I would still be glad to be able to operate "smart" switches, and even find items. But I would rather explicitly unlock my phone, start an app, and scan the bookshelf, until a book is highlighted.
OTOH this would be a great thing for non-private spaces, that are watched by security cameras 24/7 anyway. A school, a library, a clinic, an office, a workshop, a warehouse, a parking garage, maybe even a museum.
Of course, trust and security are going to be major considerations. Maybe not for everyone though: people voluntarily enable various voice assistants, and even pay to order Amazon Echo.
For me, a lot would depend on who controls the information gathered by the machine. If it’s all purely local and under my control, I doubt I’d even feel it as a “presence” - I already have that today with various cameras in my house, and it doesn’t bother me because I know the data never leaves my local system.
If it’s all being sent to some big tech co, then it becomes extremely different.
Sadly the reality of consumer product development under current economic and legal frameworks is that we’re much more likely to get the latter than the former.
I've got something similar to this using cheap ONVIF security cameras and kasa smart bulbs. I have an nvidia card process the capture and for the smart speaker I use just a microphone and speaker hooked up to a normal computer. The architecture is spread across many machines and it's nowhere close to this realtime. Also the video capture is fairly lousy because the lighting and resolution and coverage isn't great. The cameras can change the lighting to try to get a better shot. (well it's qwen with toolcalls)
I also have (I think, haven't counted recently) 12 cameras that are all ballhead so they move around and look for me when I say something to the smart speaker.
But I didn't think of the projector ... there's ~$30 ones ... might be worth buying a few.
Getting this level of response, feedback and accuracy, that's the real magic. The latency is garbage. I'm just using cheap OTS hardware though.
Anyone can do this ... it's not that expensive. Smart bulbs are cheap, wifi cams are cheap, omnidirectional usb microphones are cheap and projectors are cheap too. I'm talking like ~$100 total
What $30 projector are you looking at?
still researching them ... some of these look like garbage ... i'll come back to this thread in a few days when i find something decent. I don't want high nits or resolution, just want to be able to send pixels and not have it be noisy and get a response in under 500ms.
Sadly if it ever gets manufactured it will use cloud and become just a better version of Flock. Even cat in the photo looks with obvious displeasure realizing this. Concept itself is novel and amazing though.
It's still the same old concept of "magic surfaces", using screens, glasses, or projectors (all implementation details) to realize it is irrelevant.
Speaking of implementation details, projecting light on real-world surfaces absolutely sucks for this concept to gain adoption: issues like angular distortion, color accuracy, brightness, and energy efficiency are unsolvable. Don't waste time beating this dead horse (again).
Economy of scale currently favors the screen approach, but it's awkward to integrate multiple screens from different vendors to provide a seamless experience.
Both screens and projectors require multiple units to achieve the demo scenes. Screens would have to be mounted on much more surfaces of the home, while at least one projector is needed for each room. The cost adds up quickly.
That leaves us the only hope with smart glasses. Actually, smart glasses are also a type of projector——casting light not onto real-world surfaces, but directly into our eyes. It has the benefit of requiring just one unit for each viewer, and it could adapt to any kind of space, not limited to just surfaces. It has no inherent shortcomings of angular distortion/color accuracy/brightness. Its energy efficiency is the best and will keep improving. It just needs many more iterations to miniaturize so it can be just the size/weight of normal eye glasses to be unnoticed.
Extremely cool. Amazon was at one point working on something similar to this concept. Feels much more plausibly useful with LLMs these days though. https://www.geekwire.com/2015/amazon-files-patent-to-transfo...
I love to see the human-centric vision! This reminds me of Lenovo and Microsoft's "table computer" experiments from ~15 years ago. I remember being really excited at the possibility of my coffee table also being a computer--shared experiences, board games (I think lenovo even sold dice and pawns that connected to it). I had visions of having it also connected to the TV/Projector for controlling media. That was even before I had my first "smartphone". It could have been really cool!
One of the many cancelled form-factors the various FAANGs were playing with was a projector-centric assistant that screwed into a standard lightbulb fitting. The idea was it would placed over your desk/table and project downwards onto the surface. I still think its a very cool concept, even though none of the experiments ever panned out well enough for a real product.
A lightbulb computer demo in which "the lightbulb" could easily be swapped with a rubber duck or an ikebana to the same effect.
The Liebulb Computer, that should be the actual title.
This is an amazing concept; I went looking for a purchase link before I realized it's just a concept and not actually a real product. I would absolutely put one of these in my house provided it was open software that didn't send video out of my network.
Reminds me of https://folk.computer/ , but facilitated more via voice interactions.
Author here; I was definitely inspired by my visits to Dynamicland many years ago, which Folk Computer is also inspired by.
One big difference in direction is that Dynamicland + Folk Computer use projector+computer vision as substrate to try and reinvent/rethink the interactive/dynamic medium, and what end-user programming could be (which is very cool).
In that way I am much less ambitious; this exploration is more about how there is an entire class of existing interactions (the ones showed in the piece) that we perform today on screens, that could be much more interesting + collaborative if overlaid in the real world. And I am also terribly uninspired by the eyewear-based computing vision all major companies have pushed lately.
An interesting concept. I'm (probably) one of the few people that thought the Humane Pin had an interesting idea at its core, it was just executed poorly at the wrong time. A stationary version that leans into ambience solves a lot of the interaction and engineering issues. I also like that it addresses my gripe with XR interfaces; you aren't actually sharing a digital object in space and it looks weird to everyone else.
“Lightbulb, where is Commander Riker?”
“Commander Riker is in his quarters.”
I love it. I only skimmed through the post but if this project has/will have a "hey lightbulb, where did I put my keys/wallet/brain" function I'm buying/building it on the spot.
What does the lightbulb have to do with the projection? Obviously the lightbulb is not the camera... nor the projector (nothing clear is getting through a frosted white shell). Is the lightbulb just a prop and imagined form-factor?
I feel like the demos are disingenuous because they suggest that the lightbulb is responsible for the entire thing.
Naively asked myself the same thing - how can the camera see through that clouded glass and the projector project through it?
Very cool interaction concept! Given how many lightbulbs are in a house, this gives a lot of possibilities!
I thought the lightbulb computer was the computer they had aboard the submarine Seaview.
Really some good thinking here. We need these awesome ideas to escape from the screens, which I call as blackholes of attention.
However, the light bulb concept is limited to indoor usecases - which is not bad - that's where people live most of the time.
The broader goal should be to get rid of dependency on screens, technologies and information. That's when AI can stop enslaving us with HITL etc.
I realize this is not the makers' focus, but I find myself less enthused with the human-computer interaction side and more anxious about how we can ensure the human maintains true ownership and agency over the device and its software.
I mean, merge one of these with a modern anti-consumer "Smart TV", and we're uncomfortably close to 1984's telescreen.
I totally agree. It feels like we’ve crossed a line where tech is overtly hostile to users because everyone is chasing recurring revenue and data mining is the only viable option
Author here, and yes - I am much more interested in what a local processing/open source version of this would entail, than a Meta/Amazon/etc. version.
So why isn’t this a thing yet? This is way more useful for me than VR headsets.
How about 3d laser projector
I wonder if something like this would work for whole-room projection (i.e. not just a projector on a swivel): https://dl.acm.org/doi/fullHtml/10.1145/3531073.3531126
Lovely. Mark Weiser would have loved this.
This is a very cool demo, but I think it really needs to be made much clearer that the "lightbulb" shown is seemingly a total prop.
From what I can tell the actual projector and webcam and computer are not shown.
In the comments below, the author seems to be going of his way to be clever and disingenuous about this.
Oh my god!!! I was working towards this exact thing for the last 7+ years. Albeit from a home IoT smart speaker angle. So glad to see this!!
looks fun to experiment with
I think screens are time tested and will have some enduring use for now, but I could see people liking this; I've wanted to try a projector screen out but there was a whole rabbit hole of choices that didn't seem to improve upon screens for me for now (and this takes things in a whole other direction with projection)
I would like to have this! Is this a real, working prototype or just some computer animation?
Author explains the demos are real but current setup bulky.
Where do they say that?
> All the videos shown here are from a real - yet bulky - prototype setup. No video editing or special effects, or AI generation.
Oh, i see. Thank you!
> Oh, i see. Thank you!
What they didn't say is that the bulb shown in all the images are a prop.
Very impressive; I would design the cover as cube/cylinder as an opaque glass/plastic case cover. A modern Edison-shaped bulb would be better?
I think the mission statement here is on to something. Helmets, obtrusive physical appendages etc are unappealing to me. The idea of using extant space or surfaces however, is extremely appealing. And while it might suggest lower quality today, as with most things, it would almost certainly improve over iterations and time.
Holographic computing, I think, will be the future. At least for interfaces.
This is one of the few “new” things in tech that have really excited me. Very clever idea, and I’d be happy to see it go further.
The smartphone succeeded partly because it consolidated things we previously carried separately: a phone, iPod, camera, GPS, and more.
Assuming I’d buy one of these things, transitioning between each use case makes it yet another thing to carry.
Wearable devices at least alleviate that problem.
That was really impressive. Roughly how much do you think a project like that would be valued at?
How does the lightbulb shaped projector work?
It’s just a design concept, the lightbulb projector doesn’t exist. The demo videos were made with a normal projector.
I guess the projector is one of the hardest and most expensive parts
I'd like a GPU storage heater please. We could create a distributed data centre!
https://en.wikipedia.org/wiki/Storage_heater
Cool
That looks good.
awesome
I don't know about these things. To me they just turn everything into a screen.
Sure, it's a little more hands free, and projecting around definitly is nicer than screens. But projecting has a lot of problems too. Shadows, projection surface, ambiant light, heat are difficult to handle, but to me the worst is that it's not persistant. You can't just project 24/7 your family planner on the wall.
I'd really rather have something that prints, or better draws.
You want a recipe => print it, don't beam it.
You want a map of kyoto => tiny robot walks around the page and draws it for you. And you can draw over it if you want.
And it doesn't have to be exclusive, you can do both beam and print. But I think protable printing / drawing has a lot of unexplored potential.
The future of computing is certainly multimodal. What I see with beaming is being both hands-free and prep-free further enhances the spontaneity of life and helps fill in small gaps.
Printing otoh requires hands and prep. That's greater effort. You wouldn't print a recipe if you were just a little uncertain about step 3, you would ask for some kind of just in time modality instead. Voice might be fine for short info but sometimes you'd prefer to have it up and there's beam.
yeah, I don't disagree, though you could think that a robot with a heat-paper printer or pen could go a long way.
But more importantly I ultimately think that one of the unstated goals is to reduce screen time. And setups like these are only one voice command away from beaming youtube videos all around your house.
I don't need the family planner to be on the wall 24/7 though, it can be projected only when I'm at the room. Maybe even only when I'm close to the wall.
If I printed each recipe I followed... gush that would be a lot of paper, and also I bet that paper would get messy very quickly.
For a map, losing the ability to zoom, mark points etc seems like a big loss. Sure you can draw on it (how do you erase your mistakes?), but if everything you draw just ends up lost when you discard the paper, that's not so ideal.
That’s all fair points.
I guess what disturbs me is that I’d rather have one screen in my pocket than tens of screens in my house.
Reminds me of "Sixth Sense" technology developed over a decade ago at MIT Media Lab by Pranav Mistry/Steve Mann - https://en.wikipedia.org/wiki/SixthSense
Youtube videos - https://www.youtube.com/results?search_query=sixth+sense+tec...
Projectors are cheap enough for this to work.
Projectors aren't cheap though? Especially if you want to use them in a lit room.
Most budget projectors are at least moderately visible in a lit room, and shortening the throw distance boosts brightness considerably
Now show how the projection will look like in real life during daylight.
Reminds me of the projection bracelet scam that Captain Disillusion debunked years ago https://youtu.be/Cw7g8ixbomU?t=75
Hi, I made this.
Most of the videos are shot in real life, during summer daylight (only exception are the 2 person around a table ones, which are shot at night in a room with regular ceiling lighting).
And the projector I used is only 1100 ISO lumens; fancier models go way higher.
If you project mostly white content, it's remarkably readable + visible. Outdoors in direct sunlight would be another story, of course.
I would draw the blinds