Webcam background reflection challenge

[jerome]

Hi guys,

This time, I won't publish any demo but just an idea.

Looking at the existing demos with some environment reflection (usually done with some cube map and a fixed texture), I wondered if this could be a little more dynamic.

When you look carefully at this example from the BJS web site : https://www.studio-ramble3d.com/MakingOf_En.html you can see that the reflected environment on the robot surface is something like a room with, probably, some pieces of furniture, a window, the sky, etc. Well, a fixed texture under the hood.

But wouldn't it be funnier if the robot could reflect my own real environment at the moment I'm watching the scene : my room, my windows and maybe my face looking at it ? Let's imagine that the texture used for the reflection would be caught from my webcam at some right time interval to keep a decent scene framerate and the feeling that the reflection is really live.

I have no idea how to do this... but maybe someone of you would like to meet the challenge. There's nothing to win except the eternal gratefulness from this community

[Wingnut]

Hi J!  Cool challenge.  Nothing to win?  Pfft.    Oh that's right, I'm against competing, anyway. 

Although I'm not recommending a product, I have a couple of these things... https://www.ebay.com/itm/Panorama-Remote-Control-Pan-Tilt-Mount-Adapter-For-Smart-Phone-Gopro-DSLR-Camera-/222761475157

Although this thing is RF control, the low-budget webcam I have Gorilla-taped to it... needs a USB cable connection.

If you COULD get this pan/tilt device to aim itself... using BJS camera alpha/beta values,  the camera cord could get coiled around the pan-pivot shaft (after a few pan-spins of the BJS cam).

So, you'd need an onboard-battery-operated camera, with wireless transmit (battery eater)... and your pan and tilt device must do 360+ degrees of pan and tilt.

The "rostrum thing" shown above, does a decent 360 degrees beta/tilt/pitch, but it will only do 355 degrees alpha/pan/yaw.

AND... when you look straight down (beta)... you'll see the base of the rostrum device... and not the floor of the room.  (or, if rostrum is mounted inverted like mine is, you cannot see the ceiling.  The base of the rostrum device... gets in the way of the view of the ceiling.

SO, there's really 5 problems...

1.  Getting a wireless IP network cam... to run a decent amount of time on self-contained batteries (and still not weigh a ton).
2.  Sending BJS camera angles to camera rostrum device... causing it to pan and tilt  (needs to be wireless).
3.  Pan and tilt limits, and view-blockages caused by base of rostrum device.
4.  Streaming live video from camera... into cubeMap textures or equirectangular texture for skydome/skybox.  (or Babylon.Layer - .isBackground = true);
5.  High prices on drugs and pain killers needed to think about this challenge. 

Thx for the URL, J.  Are you involved with this project, J?  (if it's ok for me to ask).

[yuccai]

Hi,

I've read something interesting about this but it uses multiples cameras.

https://www.semanticscholar.org/paper/Synthesis-of-Environment-Maps-for-Mixed-Reality-Walton-Thomas/14181a7fafde2bd17a3b9e45a1415b1fa508537b

Basically, it acquires the environment and reflect it on 3d object.

Maybe you can adapt this solution with only one low fov camera.

If you need more information, the full paper or more resources, just ask

[Pryme8]

I could prolly pull this off...

You would just need to get the cameras input bound as a texture Channel.

then convert that image to polar or cubic coordinate system, then project it onto the mesh.

Effectively cub map support, but with a couple extra steps.

[jerome]

no, I'm not in this project at all.  This is just an idea ... the ability to see the window on my left side, the door on my right side of the office and my face all reflecting on a surface while looking at it

[Wingnut]

I meant... are you involved in MakingOf_En site/people?  Know them/him/her/company?

I was confused about the wanted effect, earlier.  I didn't read Jerome's post very well.

I guess the cam would need to be across the room, aimed back at user, with FOV that is proper.

There's many camera systems that publish a still pic from the camera... to a web/file server... once per minute or so.

AssetsManager texture task could be sent to retrieve a fresh picture (same URL over and over)... once per minute.

It could be done, eh?  Periodically-acquired camera-grab reflection texture.  yuh yuh yuh... I think so.

[jerome]

that's  what I understood, Wingy  ... and I'm really not involved in this site.