Jump to content

BABYLON.Augmented Reality ...


Recommended Posts

Hello, BabylonJS is fantasic. Thank you mr @Deltakosh and mr @davrous

Research: Augmented Reality making rapid advancements this year.

GIS AR, Multi-Marker AR, TableTop AR - 3D on flat surfaces... Happening fast.

Babylon is my 3D home. So I must ask:

          Is AR porting a priority for upcoming versions of BabylonJS. And if not, can it be?

I looked at wishlist and roadmap. Found milestone/5 with no mention of AR: https://github.com/BabylonJS/Babylon.js/milestone/5

Here is link with history I'd follow:


Seems to be lineage of successful AR examples:




For AR -> Babylon is most promising to me.

How would one go about such a feature?


Link to comment
Share on other sites

Ping, @jerome ... hello, : )

Apologies if this is bad timing, or if I misunderstand, or if I am completely off-course, not uncommon, etc. : )

 Considering Babylon AR integration study (attempt). Do you have tips?

Perhaps we chatted on twitter about level-of-difficulty - so curious about it. I'd like to learn AR better and show findings here.


Considering the complexities: cross-browser camera access, marker-tracking, ... ok that makes sense.

Thinking perhaps a good place to start might  be -> worldspace / local space / 3D translation over background video component? Is that it, would like to try...

For a basic hello world knot, are there other complications? (scaling, I wonder.) Any tips from the trenches? : )

Hoping to (try to) get down in there this next month.


All the best, Thanks. : )

Link to comment
Share on other sites



Learning comes from @jeromeetienne and @AndraConnect (follow twitter) +1. Hello && welcome. : )

For everyone who already knows, JE and AC are doing phenomenal work in AR.

Of the 3D rendering options for AR, BabylonJS would be an exceptional renderer - I think the best. However, I am biased by great results. 


Turns out -> there are two phenomenal "Jeromes".

Two phenomenal "Davids" we can handle! But now two phenomenal "Jeromes"!?! Does this happen? I only wish...

Jk. : ) Straight to the point.


Here-Is-What-I-Am-Learning about AR:

AR.js - is here -  https://github.com/jeromeetienne/AR.js thanks to @jerome_etienne

Reading through master +1. Great components under THREEx = {}.  the code is vanilla and beautiful..

The ReadMe (local) is the best starting point (https://github.com/jeromeetienne/AR.js/blob/master/README.md).

Too many links!!! A  good sign. : )



         From ReadMe: AR.js uses genius solution:

           -  "**emscripten and asm.js**! ... compile artoolkit c into javascript"

                      Thank you Brendan Eich. : ) Jerome is on it...

             - "Only thanks to all of them, i could do my part : Optimizing performance from 5fps on high-end phone, to 60fps on 2years old phone." 

            - ARToolkit, is also impressive: https://artoolkit.org/  wow - crashes Chrome Dev Tools over "asm". Oh Text search. Watch out for that!

                 mmm, look at those shaders!?!.

           - Innovations in; shadows, reflection, portals, multi-marker, table-top, & "Principles of AR" Look at that...  with principles. even. Like+1.

            - In addition to ARToolkit, it looks like two other engines may be used: aruco, tango. 


2. After a big  BLOB of minified ASM awaits this helpful hint:


                   " The ARController is the main object for doing AR marker detection with JSARToolKit. To use an ARController..."


Nicely done, and then...

var ARController = {};

... as a prototype : )

In a nutshell...


"ARController detects markers in  given image. The process() method dispatches marker detection events during run."

a good place to put a breakpoint? : ) Yes... ARController talks to artoolkit... : ) Basically accesses AR-API. Awesome. As expected...

obj.onSuccess = function() { } 

It calls back ... with a video? yep <video> saw that... new world. : )



Look: <a-frame><a-marker>export AND Threex base object full of tools. (interesting hoisting pattern) -> AND (using .position [gotcha] camers.matrix) maybe slip in BABYLON.Mesh.CreateKnot() ... at the last moment? Rendered to a second <canvas> <video> at a higher z-index... would it work? Could it work? -> Idk.

UPDATE -> It worked

UPDATE -> more of a "cannonball" than a "belly-flop".

UPDATE -> lucky is good.

Looks like: PresentationMatrix.... Transform Matrix.... Camera.matrix.

UPDATE -> It was BABYLON.Camera._computedViewMatrix and TransformMatrix calculation in an array to BABYLON.Matrix.FromArray().


Of course this is only a learning exercise in AR...

Because 1) The "Davids" will surely do something more phenomenal, and if not -> the"Jeromes" will. : )

And 2) I am only pedaling furiously on a tricycle, far, far... behind. 


If you are NOT a "Jerome" or "David",  like me, this thread sprinkles out a few breadcrumbs from a BABYLONJS + AR.js learning collision.

Hope it helps. 




Link to comment
Share on other sites

More AR Notes:

3) In UI Renderloop it looks like scene.visibility = true, onMarkerFound(); And position aligned by putting Matrix into camera.matrix.

Confirm: 3D is rendered in <canvas> above <video>

And scene.visibility flickers  on and off. 


Position Alignment and scaling, seems to be the big trick... Presentation Matrix, Tranformation Matrix, ... 


This is helpful: https://github.com/artoolkit/jsartoolkit5/blob/master/doc/introduction.html

And this: https://github.com/artoolkit/jsartoolkit5

And: https://threejs.org/docs/#api/math/Matrix4

... Yep, that describes problem .... 

Link to comment
Share on other sites

UPDATE: Found out why the freeze on "use asm"... in devtools.

It is because of word-find!  

Same thing happens in Sublime.

WARNING: Searching through the huge blob of ASM... can be slow!

SOLUTION: Search "around" it. Lol. Or separate file... : )

Also found out some other stuff.

   -> Into the ProjectionMatrix, init, float32Array(16) and the Matrix4 compatibility review.

Link to comment
Share on other sites

is webRTC supported by iOS Safari and Chrome now?  Last I checked, there was no support but I heard Apple was planning on it.  Bowser from Ericsson supports it so you can always test there until Apple catches up

Link to comment
Share on other sites

AR Deep Dive Day 3 -> success.

"Hello World" - BabylonJS knotBox rendering in AR.js (with three.js knotbox):


TransformMatrix placed into camera._computedViewMatrix

Code posted: http://www.3dwebscapes.com/demos/arbjs5/ 

Insert this second line: look at with Hiro marker.

            markerObject3D.matrix.getInverse( modelViewMatrix )  
            cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray()); //SPLICE-FINAL!  

There is still some offset bug. Perhaps default world rotation thing in artoolkit (I'll look).


@Deltakosh & @davrous & @jeromeetienne I hope this helps.

Key-Takeaways: The type float23Array(16) looks compatible at the Transformation Matrix.

Building from Array, can re-render in a single line of code. : )

There was some difference in Matrix Math -> around multiply (so yeah, consideration)

BABYLON.Math seem sufficient coverage with (https://doc.babylonjs.com/classes/2.5/matrix) so no need for gl-matrix.js? I'd think, at first. But then...

       Ah wait, dependancy-consideration... AR API, so ok, understand fully. That might get tricky. : )

       Babylon.Matrix lives here (https://github.com/BabylonJS/Babylon.js/blob/master/src/Math/babylon.math.ts)


Link to comment
Share on other sites

Thanks @Deltakosh ... thought about it. 


That was a really good question. So I rewrote the answers twice (quick)(thoughtful). : )

QUICK: I do not see changes for Babylon as a result of this test.

THOUGHTFUL: Here is (possible) minimum and maximum of the problem space.


At minimum: it is a single-line-code-splice (above) 

               - Theoretically all BJS could redirect through that line of code. Into the projection matrix...

               - Subject to robustness testing of more complex sets will show if anything breaks. I'll do that.

               - and fix the Projection bug (above) [possibly default axis mis-alignment from artoolkit].


At maximum: AR Integration is a fascinating landscape of vast latitude (it seems to me).

So maybe - thinking of AR in steps, stages, or phases might be helpful. For example; hello-world, multi-marker, flat-surface, gps, reflective, etc... 

1) #1 Big question of integration. Seems to be - who does the Matrix Math? This is the tricky delta I mentioned. I don't know the solution. It needs good ideas.

              Honestly, this was shown to me by @jeromeetienne. He mentions a big refactor for an API that reduces dependencies. Sensible. For gl-matrix.

              The question I have is: if BABYLON.Math is fully suitable (and can be extended quickly to match) - what then?

               Like I said a big question. Many options.  Probably it needs ideas from the group.


               My question, for that is: how could AR be added in as a package? And then pass Transformation Math to Babylon?

               AR - does come with a big house-guest - who eats a lot! JK, mostly.

                          The Artoolkit, (or any toolkit) is a big chunk of asm - which would be good in a separate file (package).

               Or, maybe it should be rewritten eventually? For example, there is barcode scanning code, so maybe it needs streamline, fruther optimization. Idk.

               A pure AR solution from ground up might be inevitable. And why tango integration may be cool. But that is only a guess.

              Hello Hololens ar-engine compiled to JS with enscripten? Idk.


               Strangely running both WebGL 3D renderers - at the same time - has not (yet) melted my laptop!

               This is the simple path through the woods or the mirage I mentioned.  I will look at robustness. : )

                As mentioned, emulating matrix math stymied at one spot around multiply and invert. Can still isolate - but not sure what that was, and it was late. 2am. : )


2) A number of important improvements likely remain for full AR implementation.

By that I mean all this: A) there is much latitude in continued improvements in AR for a while (...). B ) They do not appear to need BABYLON framework change. Probably package addition. And finally C) notion of fresh perspective of context, methodology, and optimization (below). Like building a flying vehicle - we have a ways to go yet. : )

Apologies for excessive detail (this is what I saw while looking):

              EXAMPLE 1: "Matrix Methodology" ->Placing the computed Transform Matrix into the camera._computedViewMatrix is ok. And possibly limited programmatically.

                          QUESTION: Are there ways to manipulate MatrixTransform with absolutePosition instead (making rotation scale independent)? Using Compose(), etc?

                                         In otherwords, leave camera matrix as is - and just compute mesh as a result of local space(ish). I'll run tests.. 

                                         No doubt, a deeper understanding of emulating by multi-marker may show ... something different. I'll try that.

             EXAMPLE 2: "UI stuff" User camera getUserMedia() requires https. Also it would be nice to know what symbol it is looking for, and a btn for camera shut off.

                                         All could be handled gracefully in standard way in the future. //TODO try  button, img, <span>... etc.

             EXAMPLE 3: There is barcode scanner code in the ar process loop, it seems, probably old example. Which I think is cool (barcode as anscestor - really?), but a performance optimization available? That is the notion of possibly a streamlined pure-ar context approach, from top down. But. Probably already happening. : )


 I will try those things and put results here. Thank you. 

Link to comment
Share on other sites

BJS in ARjs: small planetary system test -> success


Orbiting spheres.... background (skybox) plane. 

Everything seems ok. Needs fine-tuning ...  z and y axis seem inverted.

-> Twisting and turning the marker makes a big difference.

Single line of code:

cameraB._computedViewMatrix = new BABYLON.Matrix.FromArray(markerObject3D.matrix.toArray());


Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...