BABYLON.Augmented Reality ...

Recommended Posts

@Deltakosh, I will. Docs, ok.

Look at that nice embedded tweet. Lol. 

I fav'd that link back then, [ and must muster artistic resolve to doc again : ].... so I will.


-getting closer on multiple fronts

r&d time ramping on: Babylon app about to go AR. (!~!)

Want to see what happens!!! ... : ) I'll eventually re-doc this thread with results of the experiment:


your awesome skybox is fully functional in Ionic App. So Nice. +1.

a few wrench turns and an engine drop in... should give AR, but -> it is a different (physical) camera access instance. Right?

So upcoming AR TEST, today...


A bunch of different  BABYLON Scenes on different Ionic pages:


First, need a landscape...



Hide skybox and set clear background color:

     // Set clearColor background to transparent.
    scene.clearColor = new BABYLON.Color4(0, 0, 0, 0);


Next AR engine drop in...

export class Ar1Page {
  BABYLON: any;
  ARjs: object;

  constructor(private camera: Camera) {
      this.BABYLON = BABYLON;
      this.ARjs = ARjs;

And init ARjs (after createScene() before engine.runRenderLoop):

ARjs.Babylon.init(engine, scene, scene.cameras[0], document.getElementById('ar1Canvas'), engine, BABYLON)


confirm (physical) camera not available. : )

Needs a native camera instance...? Or HTML5 permission?

Or - what is needed for HTML5 getUserMedia()? (maybe only simulator... TEST)

Share this post

Link to post
Share on other sites

Indeed. : ) Following native (physical) camera path...  returnRealCam ( :rolleyes: ) or declared... (looking). thx...

READING: https://forum.ionicframework.com/t/ionic-how-to-turn-on-camera/29703/4

- ionic plugin add cordova-plugin-camera

- "The problem with desktop is that you can't use any Cordova plugins (like the Camera plugin)."

- "typically a plugin is used to access the native Camera API, it is also possible to access the camera through plain HTML5:"

- " testing  app in the Android simulator not the browser, you can access the camera and use it in your local testing."


DOCLINK: https://ionicframework.com/docs/native/

- Ionic Native: https://github.com/ionic-team/ionic-native

import { Camera } from '@ionic-native/camera';

... something like all that... 

Yes, HTML5  not simulators? Looking...

getUserMedia()...?  ~dives in ,<feet-first>...

https://www.html5rocks.com/en/tutorials/getusermedia/intro/ (actual best link...)


One amazing use case for video capture is to render live input as a WebGL 

It is called Video Input to WebGL Texture...

"this texture is special, it is a video. So it need to be constantly updated." - Jerome... : )

STATUS: (HTML AR into simulator-jungle):
Stepped down to getUserMedia( ) catch (e) {e === 'PermissionsDeniedError'} . Ding! Few reasons for this...

Tried adding permissions to an .xml -> (we still use xmllol). no change. Maybe different permission... found docs.

(derp, restart ionic serve?)

STATUS: trying simulator with both HTML5 and Native access TEST.

Install Ionic Native

Install Ionic Native core & Camera,  and accessed camera object from Angular Provider...

Great - there are 4 ways to simulate, likes USB the best... DONE.

STATUS: Simulation, or USB debugging -> must I download the moon?

Yes. <downloads... the moon> : )

Simulator worked (eventually).

EPIC-AR-VIDEO-TEST-RESULT-FAIL ->  it took a picture. : )

There must be a video stream around there somewhere...

... if getUserMedia can't cut it.

In progress...

Share this post

Link to post
Share on other sites

Found it in the docs:


- "Cordova Html5Video Plugin solves ... HTML5 tags for local video ... which is not functional in Cordova for Android."

Another link, digging into HTML5Video Cordova

- permissions in xml file ... 

HTML5 Rocks: https://www.html5rocks.com/en/tutorials/getusermedia/intro/ camera stream piped into video src ... <cheers>

navigator.webkitGetUserMedia('video', function(stream){
  video.src = webkitURL.createObjectURL(stream);
}, function(error){
  console.log("Failed to get a stream due to", error);


- permissions in xml manifest

- cordova Html5Video Plugin

- native camera format "SECURE_VIDEO" -> assign to video.src.



As usual DeltaK has the answer in under 10 words.... took me 200 lines of technobabble to get there!

I found more great links...


Share this post

Link to post
Share on other sites

@brianzinn yep, I will doc latter side of journey. There are odds-and-ends issues...AR peculiarities arise!

The white ion-card displayed between <canvas> and <video> at every z-index (because ionic CSS something) - blocking video with white <always>. Finally the fix:

<ion-card style="background-color:rgba(0,0,0,0)"> , simple super-power++...


There remains small-problems in the matrix-crossover.

It needs a new-pattern, ... and someone will need to draw a diagram(ok). Stuff...

Thinking, it is a three-part entity: visual-engine(s), ar-engine, app-engine. So .... stuff.

The way it is currently organized is not the best way, so what is the best way?

- could go angular "provider", but that is so baked in, so I'm thinking oldjedi generic namespace module, in ES6 Export/Import?

Idk. But yeah, maybe next weekend! Anyway...


The coolest things to encounter are:

-  enumerateDevices: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices

- manifest permissions (thanks DK) :https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/Firefox_OS_apps/App_permissions#video-capture

- getUserMedia: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia


Something else...


this had some great information, counterpoint...  

... and finally the answer, with an inside look into the wild-west saloon (3/4ths down).


Share this post

Link to post
Share on other sites

Did someone tried jsartoolkit5 to implement it with babylonjs from scratch?

Here is a sample code in github: https://github.com/mkaouri/jsartoolkit5/blob/master/examples/babylonjs_from_scratch.html

Unfortunately, jsartoolkit5 supports only marker tracking objects such as patterns and bar code, and it doesn't support Natural Feature Tracking (NFT).

There're some students made NFT tracker with pure javascript, here's an abstract: 


Also, I'm attaching here their publication in pdf.

I wish someone can implement it and share the code of NFT tracker in pure javascript.


Share this post

Link to post
Share on other sites

@aFalcon Great :D

It's working with pure jsartoolkit5 and Babylonjs, without AR.js. 

Demonstration done on Google Chrome ver 62. For mobile devices, It should work on iOS Safari ver. 11 and Chrome for Android ver. 62

Way forward, I will make it with NFT detection (without markers).



Share this post

Link to post
Share on other sites

Very good news!

I have developed the Natural Feature Tracking (NFT) tracker example with BabylonJs and JsArtoolkit5. It works like a charm. ;)

Find it here: https://github.com/mkaouri/jsartoolkit5/blob/master/examples/nft_babylonjs.html

Clone the full project from here: https://github.com/mkaouri/jsartoolkit5.git



Share this post

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    No registered users viewing this page.