JackFalcon

BABYLON.Augmented Reality ...

Recommended Posts

@Deltakosh, I will. Docs, ok.

Look at that nice embedded tweet. Lol. 

I fav'd that link back then, [ and must muster artistic resolve to doc again : ].... so I will.

UPDATE:

-getting closer on multiple fronts

r&d time ramping on: Babylon app about to go AR. (!~!)

Want to see what happens!!! ... : ) I'll eventually re-doc this thread with results of the experiment:

image.thumb.png.1b322a2659ee235f9c2e2f4ae31bd931.png

your awesome skybox is fully functional in Ionic App. So Nice. +1.

a few wrench turns and an engine drop in... should give AR, but -> it is a different (physical) camera access instance. Right?

So upcoming AR TEST, today...

UPDATE:

A bunch of different  BABYLON Scenes on different Ionic pages:

image.thumb.png.37088304fbed3ecf7edc75a62f09c752.png

First, need a landscape...

image.png.039bbb8682237d558ac9910f6fd9334e.png

image.png.0c98e604b6ce18a9e8de9f6d11558c94.png

Hide skybox and set clear background color:

     // Set clearColor background to transparent.
    scene.clearColor = new BABYLON.Color4(0, 0, 0, 0);

image.png.11ac145ad5654ea0ad364828aac73ef5.png

Next AR engine drop in...

export class Ar1Page {
  BABYLON: any;
  ARjs: object;

  constructor(private camera: Camera) {
      this.BABYLON = BABYLON;
      this.ARjs = ARjs;
  }

And init ARjs (after createScene() before engine.runRenderLoop):

ARjs.Babylon.init(engine, scene, scene.cameras[0], document.getElementById('ar1Canvas'), engine, BABYLON)

image.png.e519dc03b7af4a46c00d945e50724c0a.png

confirm (physical) camera not available. : )

Needs a native camera instance...? Or HTML5 permission?

Or - what is needed for HTML5 getUserMedia()? (maybe only simulator... TEST)

Share this post


Link to post
Share on other sites

Indeed. : ) Following native (physical) camera path...  returnRealCam ( :rolleyes: ) or declared... (looking). thx...

READING: https://forum.ionicframework.com/t/ionic-how-to-turn-on-camera/29703/4

- ionic plugin add cordova-plugin-camera

- "The problem with desktop is that you can't use any Cordova plugins (like the Camera plugin)."

- "typically a plugin is used to access the native Camera API, it is also possible to access the camera through plain HTML5:"

- " testing  app in the Android simulator not the browser, you can access the camera and use it in your local testing."

 

DOCLINK: https://ionicframework.com/docs/native/

- Ionic Native: https://github.com/ionic-team/ionic-native

import { Camera } from '@ionic-native/camera';

... something like all that... 

Yes, HTML5  not simulators? Looking...

getUserMedia()...?  ~dives in ,<feet-first>...

https://www.html5rocks.com/en/tutorials/getusermedia/intro/ (actual best link...)

<in-progress>... 

One amazing use case for video capture is to render live input as a WebGL 

It is called Video Input to WebGL Texture...

"this texture is special, it is a video. So it need to be constantly updated." - Jerome... : )

STATUS: (HTML AR into simulator-jungle):
Stepped down to getUserMedia( ) catch (e) {e === 'PermissionsDeniedError'} . Ding! Few reasons for this...

Tried adding permissions to an .xml -> (we still use xmllol). no change. Maybe different permission... found docs.

(derp, restart ionic serve?)

STATUS: trying simulator with both HTML5 and Native access TEST.

Install Ionic Native

Install Ionic Native core & Camera,  and accessed camera object from Angular Provider...

Great - there are 4 ways to simulate, likes USB the best... DONE.

STATUS: Simulation, or USB debugging -> must I download the moon?

Yes. <downloads... the moon> : )

Simulator worked (eventually).

EPIC-AR-VIDEO-TEST-RESULT-FAIL ->  it took a picture. : )

There must be a video stream around there somewhere...

... if getUserMedia can't cut it.

In progress...

Share this post


Link to post
Share on other sites

Found it in the docs:

 https://www.npmjs.com/package/cordova-plugin-html5video

- "Cordova Html5Video Plugin solves ... HTML5 tags for local video ... which is not functional in Cordova for Android."

Another link, digging into HTML5Video Cordova

https://cordova.apache.org/docs/en/2.7.0/cordova/media/media.html
- permissions in xml file ... 

HTML5 Rocks: https://www.html5rocks.com/en/tutorials/getusermedia/intro/ camera stream piped into video src ... <cheers>

navigator.webkitGetUserMedia('video', function(stream){
  video.src = webkitURL.createObjectURL(stream);
}, function(error){
  console.log("Failed to get a stream due to", error);
});

TESTS:

- permissions in xml manifest

- cordova Html5Video Plugin

- native camera format "SECURE_VIDEO" -> assign to video.src.

...

UPDATE: 

As usual DeltaK has the answer in under 10 words.... took me 200 lines of technobabble to get there!

I found more great links...

 

Share this post


Link to post
Share on other sites

@brianzinn yep, I will doc latter side of journey. There are odds-and-ends issues...AR peculiarities arise!

The white ion-card displayed between <canvas> and <video> at every z-index (because ionic CSS something) - blocking video with white <always>. Finally the fix:

<ion-card style="background-color:rgba(0,0,0,0)"> , simple super-power++...

 

There remains small-problems in the matrix-crossover.

It needs a new-pattern, ... and someone will need to draw a diagram(ok). Stuff...

Thinking, it is a three-part entity: visual-engine(s), ar-engine, app-engine. So .... stuff.

The way it is currently organized is not the best way, so what is the best way?

- could go angular "provider", but that is so baked in, so I'm thinking oldjedi generic namespace module, in ES6 Export/Import?

Idk. But yeah, maybe next weekend! Anyway...

 

The coolest things to encounter are:

-  enumerateDevices: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/enumerateDevices

- manifest permissions (thanks DK) :https://developer.mozilla.org/en-US/docs/Archive/B2G_OS/Firefox_OS_apps/App_permissions#video-capture

- getUserMedia: https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

 

Something else...

https://blairmacintyre.me/2017/05/20/its-not-webar-yet/

this had some great information, counterpoint...  

... and finally the answer, with an inside look into the wild-west saloon (3/4ths down).

 

Share this post


Link to post
Share on other sites

Did someone tried jsartoolkit5 to implement it with babylonjs from scratch?

Here is a sample code in github: https://github.com/mkaouri/jsartoolkit5/blob/master/examples/babylonjs_from_scratch.html

Unfortunately, jsartoolkit5 supports only marker tracking objects such as patterns and bar code, and it doesn't support Natural Feature Tracking (NFT).

There're some students made NFT tracker with pure javascript, here's an abstract: 

https://jensgrubert.wordpress.com/research/natural-feature-tracking-in-javascript/

Also, I'm attaching here their publication in pdf.

I wish someone can implement it and share the code of NFT tracker in pure javascript.

vr2012-poster_v1.4.pdf

Share this post


Link to post
Share on other sites

@Mostafa Kaouri I like this very much... 

The implementation successes above were using jsartoolkit underneath AR.js, as I recall.... but looks you bring UPDATES... fantastic!

That code looks really good. Screenshot of it working? A lighter footprint would be ideal on mobile...

Share this post


Link to post
Share on other sites

@aFalcon Great :D

It's working with pure jsartoolkit5 and Babylonjs, without AR.js. 

Demonstration done on Google Chrome ver 62. For mobile devices, It should work on iOS Safari ver. 11 and Chrome for Android ver. 62

Way forward, I will make it with NFT detection (without markers).

 

BabylonJsAR.png

Share this post


Link to post
Share on other sites

Very good news!

I have developed the Natural Feature Tracking (NFT) tracker example with BabylonJs and JsArtoolkit5. It works like a charm. ;)

Find it here: https://github.com/mkaouri/jsartoolkit5/blob/master/examples/nft_babylonjs.html

Clone the full project from here: https://github.com/mkaouri/jsartoolkit5.git

Cheers...

nft_babylonjs_jsartoolkit.png

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.