The Leftover

Members
  • Content Count

    98
  • Joined

  • Last visited

  • Days Won

    2

Posts posted by The Leftover


  1. Gentlemen and ssaket, I apologize for the slow response.  I will try to remedy it with a short summary of what I do.

    The biggest factor for Illuminated City is that I don't need a continuous map.  The subject is always a bracketed geography.  Another factor is that the Lat/Lon must be precise, down to 20 meters or so.  I overlay police districts and crime incidents.  They need to line up.

    To do this, I create the biggest, fattest tile the browser will handle.  8K pixels squared roughly.  That's all the map is, a fat image.  (There are potential future enhancements including multiple tiles but I don't use that in production.)  Boring, huh?  

    There is some pre-processing.  The map is stored on the server (and injected into Babylon) as an SVG.  Several factors here:
      ~ Map size (from server to client) is always less than 2Mbytes
      ~ I can control styling

    There is no dynamic connection to an outside data source.  I get the maps custom tailored, per city, and organize Illuminated City styling in CSS and "survey" them for precise fit.  The styling is important to me.  The one you see now is called "Retro-minus".  I may want to do a "night" view.  I would only have to redefine some CSS and voila!  Night.

    The result looks good and is very controllable.  Adding a new city map is a straightforward recipe but takes a couple hours.

    Again, apologies for the slow response.  If I can aid your discussion (or learn from your experiences) I am now paying attention!

    https://blog.illuminated.city


  2. Hello coders and countrymen,

    I have continued the labor with Illuminated City in ways that are not nearly geeky enough for this forum.  I have, for example, spent hours with crime analysts for San Mateo and Santa Clara Counties.  (You know, stay close to the target user.)

    One request was the ability to take crime data from multiple geo-adjacent cities and put it onto one display.  Oddly enough, crime does not stop at administrative boundaries.  And cities are utterly incompetent at unifying data-anything.  Did that.

    Another request, only partially complete, is to be able to observe a "patrol waveform" and compare it to the "crime waveform".  The goal would be to help place patrols more effectively.  Patrol presence reduces crime.  It's a challenging problem; I will discuss another time.

    Finally starting to take specific steps to create buzz.  This might include creating consternation on the way -- this is crime after all.  My first blog post is 90% complete and I am itching to show it to someone.  Tag.  You're it.

    http://blog.illuminated.city/

    Maybe 85%.  Anyway, almost there.


  3. I see no error right now.  The Illuminated City certificate is valid and has not been changed for more than a week.  It is provided my GoDaddy and the site is hosted by Amazon.

    HOWEVER.  I get that error intermittently on all kinds of sites, myverizon.com, facebook.com, nytimes.com.  For the past six months, it would appear, rarely and briefly, and then disappear.  I assumed it had something to do with my crappy ISP or Russian hackers.  (NY Times had a DNS attack that lasted 18 hours.)


  4. 1 hour ago, jerome said:

    For now, it seems that passing big bunches of data to WASM is still the bottleneck

    Jerome, if I am belaboring the obvious, sorry.

    I only have one copy of the data structures.  It is shared between WebAssembly and Javascript.  If you look at the code (from my August 3 post above) you will see that I create the array as WebAssembly memory.  Then I create a view into that memory.  Javascript or WebAssembly both may address the array at that point.  As far as JavaScript is concerned, it is just a typed array.  This array is sized/created early in the session and tends to persist for quite a while.  In fact, sometimes I clear the array and start over; as opposed to allocating a new one.

    To drive the point home:  Sometimes I write the same function in JavaScript and WebAssembly.  More specifically, I have already written it in JavaScript and it is working.  When I develop the WebAssembly, I set it up so that I can call either "flavor" of the same function.  Given that I don't have a battery of test suites, this helps me spot unintended changes of behavior.

    Once I get things settled, smaller interactions with the array are usually done through JavaScript.  Larger ones are done with WebAssembly.

    After all of this, I am happy with the effort.  I did hit many dead ends but I also got several large tasks to run 3X faster.

    I hope this is useful.


  5. brianzinn, not sure I understand why that would be.  (I am not doubting it, though.  And please don't explain it to me.)

    I ran into a problem where I had to choose between WebAssembly and Web Workers unless I was willing to do large array copying.  They both have data sharing for large vectors but they wouldn't work together.  I chose to use WebAssembly and skip Web Workers for now.

    Life on the bleeding edge . . . 


  6. Jerome, delighted to read your report.  This FYI:

    I had done some profiling of Illuminated City and determined that speeding 'VertexData.ComputeNormals' would be a win for my application.  It is a good candidate in other ways, being about 200 lines of vector arithmetic (with two calls to Math.sqrt).  I had considered rewriting it in native WASM but quickly realized that I did not understand application well enough.

    Also, there were a bunch of boundary issues:  it deals with multiple arrays that - I think - were not typed.  Converting typed <--> untyped is gonna add a lot of overhead.  Also, the only way I know how to process multiple arrays in a single WebAssembly module is to put them in one array and use indexed addressing.  I do this for my own application but it is a hefty bunch of work.

    At the end, I did not.

    'VertexData.ComputeNormals' is an example of a function that I would optimize the hell out of in JavaScript first.  Good chance that is a win.

    Here is my only actual concrete suggestion, respectfully placed from someone who doesn't really understand your code:  Migrate to typed arrays for the obvious suspects, like uvs and normals and so on.


  7. The heartbreak of today was that the module I snipped this from runs faster in JavaScript.  In other areas, I have reaped a 3x speedup with WebAssembly.

    I believe that issue is that there is substantial overhead in entering and exiting WebAssembly.  You want the work it does while there to be large enough to make up the overhead and still give you a win.

    Gonna go soak my head . . . 


  8. Jerome, thanks.  I usually do something like this.  I create a module config object, and attach the memory there.  This hands in whatever settings are needed and the memory.  Now both JavaScript and WebAssembly can access the same typed array using 'hexlatticeMemoryView' and 'i32.load'/'i32.store'.

    I treat the WebAssembly module as a persistent "closure".  Some of them have four or five function entry points.

     JAVASCRIPT  JAVASCRIPT  JAVASCRIPT  JAVASCRIPT  JAVASCRIPT 
                hexlatticeImportObject = {
                  settings             : {
                    weekbreaksLength   : weekbreaks.length*4,
                    weekbreaksStart    : 16,
                    monthBreaksLength  : 0,
                    monthBreaksStart   : 16+weekbreaks.length 
                  },
                  imports              : {
                    log                : console.log
                  }
                }; 
              hexlatticeMemory       = new WebAssembly.Memory({initial:1});
              hexlatticeMemoryView   = new Uint32Array(hexlatticeMemory.buffer);
              hexlatticeImportObject.imports.mem = hexlatticeMemory;
     JAVASCRIPT  JAVASCRIPT  JAVASCRIPT  JAVASCRIPT  JAVASCRIPT 
     WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY 
    (module
      (import "imports" "log"      (func $log     (param i32)))
      (memory                      (import "imports"    "mem") 1)
      (global $weekbreaksstart     (import "settings"   "weekbreaksStart")     i32)
      (global $weekbreakslength    (import "settings"   "weekbreaksLength")    i32)
    
     WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY  WEBASSEMBLY 

     


  9. http://www.illuminated.city/mp4/DistrictZoom.mp4

    I do this.  It looks good.  Code is below.  At first glance it looks readable.  No warranty.

           function animateCameraToTargetAndPosition(camera,newPosX,newPosY,newPosZ,zoom,newAlpha,newBeta){
            var radiusAnimation      = new BABYLON.Animation("camRadius", "radius", 30, BABYLON.Animation.ANIMATIONTYPE_FLOAT, BABYLON.Animation.ANIMATIONLOOPMODE_CONSTANT),
                alphaAnimation       = new BABYLON.Animation("camAlpha", "alpha", 30, BABYLON.Animation.ANIMATIONTYPE_FLOAT, BABYLON.Animation.ANIMATIONLOOPMODE_CONSTANT),
                betaAnimation        = new BABYLON.Animation("camBeta", "beta", 30, BABYLON.Animation.ANIMATIONTYPE_FLOAT, BABYLON.Animation.ANIMATIONLOOPMODE_CONSTANT),
                targetAnimation      = new BABYLON.Animation("camTarget", "target", 30, BABYLON.Animation.ANIMATIONTYPE_VECTOR3, BABYLON.Animation.ANIMATIONLOOPMODE_CONSTANT),
                newTarget            = new BABYLON.Vector3(newPosX,(Ø.tyN(newPosY)?newPosY:ψ(0)),newPosZ),
                thenStopCA           = ()=>setTimeout(stopCA,150),
                newRadius            = Ø.PN(zoomLevel[zoom])?zoomLevel[zoom]:zoomLevel[0],alpha,beta,radiusKeys,alphaKeys,betaKeys,targetKeys,twoPI=pi+pi;
    
            startCA();
    
            newAlpha                 = Ø.tyN(newAlpha)?newAlpha:standardAlpha;
            newBeta                  = Ø.tyN(newBeta)?newBeta:standardBeta;
    
            alpha                    = (camera.alpha+twoPI) % twoPI;
            beta                     = (camera.beta+twoPI) % twoPI;
    
            targetKeys = [{
              frame    : 0,
              value    : camera.target
            }, {
              frame    : 100,
              value    : newTarget
            }];
            radiusKeys = [{
              frame    : 0,
              value    : camera.radius
            }, {
              frame    : 100,
              value    : newRadius
            }];
            alphaKeys  = [{
              frame    : 0,
              value    : alpha
            }, {
              frame    : 100,
              value    : newAlpha
            }];
            betaKeys   = [{
              frame    : 0,
              value    : beta
            }, {
              frame    : 100,
              value    : newBeta
            }];
    
            targetAnimation.setKeys(targetKeys);
            radiusAnimation.setKeys(radiusKeys);
            alphaAnimation.setKeys(alphaKeys);
            betaAnimation.setKeys(betaKeys);
            camera.animations.push(targetAnimation);
            camera.animations.push(radiusAnimation);
            camera.animations.push(alphaAnimation);
            camera.animations.push(betaAnimation);
            scene.beginAnimation(camera,0,100,false,5,thenStopCA);
          };

  10. Gentlemen, thank you for the links.  Let me some opinions base on three days of work.

    I started writing in straight WAT.   Because I have a genetic defect that causes me to do things the hard way.  However, it has caused me to learn a lot of things.

    WebAssembly is at the "MVP" stage as they call it.  One can only create a module with functions below that - two levels.  One can create a list of which functions may be exported.

    The MVP status shows:  I couldn't figure out how to make a module-global variable that was mutable; so I did a work-around.  One can share a typed array between JS and WA.  In WA, it is called "memory" but there may only be one of them.  I redesigned things a bit so all processing was applied against one array.  This could put a crimp in my style.

    Is it possible the "C" converter bypasses these functionality bottlenecks?  It seems a little unlikely; I think wat is the textual representation of wasm and they go hand-in-hand.  They do appear to be beavering away at this much as we are here.

    The integration makes it *NOT* an all or nothing kind of thing.  When the module is built, it can receive JS functions, notably console.log.  So I can log things to the console.  I could make other JS calls if I wanted.  Exported functions are just a function.  You can call it from JS.  (If you print them it says "native code", which gave me a kick.)

    In light of this, I am pushing forward with creating limited functions for the three or four places where Illuminated City sits for more than a second.  It requires some re-organization but I have the substantial advantage of being the only author.  I can also write these functions in JS.  That part is really neat; the array is one array and looks the same whether the manipulation was done by JS or WA.  This will be helpful for testing.


  11. I should not being doing this . . . but I have been experimenting with Web Assembly.  With so many hexagons and so many crime incidents, Illuminated City has come computationally intensive tasks.

    My question is, have y'all tried to use it for some functions (e.g. ComputeNormals)?  Did it perform well?  Did it seem worthwhile?


  12. A couple interesting things in this video:
    1) This is Los Angeles with 250K cells (probably 75K visible hex prisms in this demo).  When I optimize for SPS, the performance is really pretty amazing.
    2) You see the navigation inspired by Wingnut's suggestion to change focus.

    http://www.brianbutton.com/babylon/GeoSearch-Intro1.mp4

    I did clip out some waiting periods but the video is all 1X speed.


  13. When you have a hammer, problems look like nails.

    I would like users to be able to move the Arc Rotate Camera focal point to different spots around the city.  To help communicate that, I needed a "focal point marker" to show the current point on the map.  I picked an anchor.  (Anyone with a better idea is hereby given the mic.)  Constructing a 3D anchor from an SVG rendering of an anchor turned out to be essentially the same exercise as creating a letter.  So, in a repetition of history, I created a new font family called "WebGL Dings"

    https://www.babylonjs-playground.com/#PL752W#17


  14. Last week, I discovered that real-time aircraft flight data is available through an open API.  Two, in fact.  Already having a city, I decided show overflights.  I put in a five-minute delay to permit me to smooth the data but this is almost real-time.

    Video shows FedeX coming into SFO over San Francisco.  This is not very zoomy but it feels signficant.  I am not entirely sure what to do with it.
    http://www.brianbutton.com/babylon/MonoPlane.mp4

    Airplane model was built out of cardboard, dowels, plywood and rubber bands.  https://www.babylonjs-playground.com/#H3Z6J1#7