Boneless deformation/animation (Blender shape keys) exploratory

[JCPalmer]

Hello, decided that something talked about at the end of another thread was best addressed in its own thread.  As intro (gryff correct as needed) Blender allows for meshes to deformed without the use of bones.  This could used for animation.  Not having a skeleton seems easier to do than bones in Blender to me, and the results can be way beyond them as well.  Copied link from prior thread:

.

 

Question is:  Is there even any hope that this could be exportable from Blender to BabylonJS?  Gryff also provided a .blend, copied link Cloth Animation2 .Load it up in blender, and hit the play button, '>', in the bottom middle of the window.

 

What I have done is run this blend through a Babylon export script with extra stuff added to find & for now write extra lines into the log file which describes the data for shape keys.  FYI, the Blender doc I got my info from is at http://www.blender.org/documentation/blender_python_api_2_70_release/bpy.types.Key.html#bpy.types.Key

 

It seems that shape keys can be shared across meshes, so I added this code to reference them this way in the main loop & a ShapeKey class because it seems the data structure is recursive:

# Shape Keys from the main loopfor key in bpy.data.shape_keys:    ShapeKey(key, 1))  ...class ShapeKey:    def __init__(self, key, level):        TowerOfBabel.log('processing begun of shape :  ' + key.name, level)        if hasattr(key, 'eval_time'   ): TowerOfBabel.log('eval time:  '    + str(key.eval_time), level + 1)        if hasattr(key, 'slurph'      ): TowerOfBabel.log('slurph:  '       + str(key.slurph   ), level + 1)        if hasattr(key, 'use_relative'): TowerOfBabel.log('use relative:  ' + format_bool(key.use_relative), level + 1)        if hasattr(key, 'animation_data'):            TowerOfBabel.log('animation_data:', level + 1)            for fcurve in key.animation_data.action.fcurves:                TowerOfBabel.log('fcurve.data_path:' + fcurve.data_path, level + 2)                    # recursion, oh crap        if hasattr(key, 'key_blocks'):            for block in key.key_blocks:                ShapeKey(block, level + 1)

This produced this output (probably a dead end from this angle):

processing begun of shape :  Key	eval time:  0.0	slurph:  0	use relative:  true	animation_data:		fcurve.data_path:key_blocks["Draped"].value	processing begun of shape :  Basis	processing begun of shape :  Draped

In the processing of a mesh I added the following code if shape keys were associated:

# shape keys for mesh if object.data.shape_keys:    TowerOfBabel.log('Shape Key found in mesh:  ' + object.data.shape_keys.name, 2)    keyBlocks = object.data.shape_keys.key_blocks    for block in keyBlocks:        TowerOfBabel.log('Block Name:  ' + block.name + ', num vertices:' + str(len(block.data)), 3)        for data in block.data:            vertex = data.co            TowerOfBabel.log('x: ' + str(vertex.x) + ', y: '+ str(vertex.y) + ', z: ' + str(vertex.z), 4)

This produced a lot of output,  here is a shortened version of the log for the mesh with the shape key:

processing begun of mesh:  Cloth, using base class: BABYLON.Mesh	WARNING:  No materials have been assigned: 	num positions      :  1156	num normals        :  1156	num uvs            :  0	num uvs2           :  0	num colors         :  0	num indices        :  6534	Shape Key found in mesh:  Key		Block Name:  Basis, num vertices:1156			x: 1.0, y: -1.0, z: 0.0			x: -1.0, y: -1.0, z: 0.0			x: 1.0, y: 1.0, z: 0.0			x: -1.0, y: 1.0, z: 0.0			x: 0.8181818127632141, y: -1.0, z: 0.0			x: 0.6363636255264282, y: -1.0, z: 0.0			x: 0.45454543828964233, y: -1.0, z: 0.0			x: 0.27272725105285645, y: -1.0, z: 0.0			x: 0.09090906381607056, y: -1.0, z: 0.0			x: -0.09090910851955414, y: -1.0, z: 0.0			x: -0.27272728085517883, y: -1.0, z: 0.0			x: -0.4545454680919647, y: -1.0, z: 0.0			x: -0.6363636255264282, y: -1.0, z: 0.0			x: -0.8181818127632141, y: -1.0, z: 0.0			x: 0.8181818127632141, y: 1.0, z: 0.0			x: 0.6363636255264282, y: 1.0, z: 0.0			x: 0.45454543828964233, y: 1.0, z: 0.0			x: 0.27272725105285645, y: 1.0, z: 0.0			x: 0.09090906381607056, y: 1.0, z: 0.0			x: -0.09090910851955414, y: 1.0, z: 0.0			x: -0.27272728085517883, y: 1.0, z: 0.0			x: -0.4545454680919647, y: 1.0, z: 0.0			x: -0.6363636255264282, y: 1.0, z: 0.0			x: -0.8181818127632141, y: 1.0, z: 0.0			x: -1.0, y: 0.8181818127632141, z: 0.0			. . .			x: 0.8787878751754761, y: 0.7575757503509521, z: 0.0			x: 0.939393937587738, y: 0.6969696879386902, z: 0.0			x: 0.939393937587738, y: 0.7575757503509521, z: 0.0		Block Name:  Draped, num vertices:1156			x: 0.42931264638900757, y: -0.532781720161438, z: -0.8952973484992981			x: -0.4533596634864807, y: -0.5035873055458069, z: -0.9059669375419617			x: 0.5022068023681641, y: 0.5161137580871582, z: -0.9113012552261353			x: -0.5195051431655884, y: 0.592650294303894, z: -0.846235990524292			x: 0.315280556678772, y: -0.4323137700557709, z: -0.8073179721832275			x: 0.23072978854179382, y: -0.5339680314064026, z: -0.7025083899497986			x: 0.12728682160377502, y: -0.45900893211364746, z: -0.6225178241729736			x: 0.2509845793247223, y: -0.36435461044311523, z: -0.6379876732826233			x: 0.09559790790081024, y: -0.3883511424064636, z: -0.6364848017692566			x: -0.08387131989002228, y: -0.38822996616363525, z: -0.6374479532241821			x: -0.24025821685791016, y: -0.39399784803390503, z: -0.637752115726471			x: -0.25066906213760376, y: -0.34393423795700073, z: -0.22135230898857117			x: -0.2696780264377594, y: -0.3371814489364624, z: 0.011421998962759972			x: 0.43590784072875977, y: 0.1628814935684204, z: -0.566344141960144			x: 0.45525118708610535, y: 0.22023046016693115, z: -0.5630139708518982			x: 0.3902812600135803, y: 0.21390795707702637, z: -0.5098786950111389			x: 0.4428275227546692, y: 0.1841123402118683, z: -0.5045356154441833			x: 0.3944912552833557, y: 0.2104395627975464, z: -0.5703118443489075			x: 0.4410724639892578, y: 0.17216885089874268, z: -0.5640472173690796			x: 0.5403734445571899, y: 0.24221545457839966, z: -0.5275962352752686			x: 0.5631066560745239, y: 0.2916404604911804, z: -0.5530022382736206			x: 0.548436164855957, y: 0.2113366574048996, z: -0.5791581273078918			. . .			x: 0.5152558088302612, y: 0.40521472692489624, z: -0.6688541769981384			x: 0.5745273232460022, y: 0.3480875492095947, z: -0.6703621745109558			x: 0.5555965900421143, y: 0.3864023685455322, z: -0.7100901007652283

Now I recognize some of this code, but I do not know what is going on yet.  If someone can enlighten me on what I might try next, I would be grateful.

 

Jeff

[gryff]

Is there even any hope that this could be exportable from Blender to BabylonJS?

 

Jeff, with your programming skills and those of others around here, I think so

 

This produced a lot of output

 

Which actually told me something - I had not applied my scaling

 

Select the cloth/plane with right mouse click, then  hit the N key - a properties bar appears.  Go to the very top of the bar that appears - to Transform and look at the scale values. To apply the scale, with the cloth still selected go to the menu bar for that window and Object->Apply->Scale. The scale values change to one.

 

Now run your scripts again. Do the values for the keys change?

 

What you are seeing are the vertex co-ordinates for each vertex for the "Basis Key/Block" which is the undeformed mesh and from which all other keys are made. The "Draped Key/Block" has a set of the deformed vertex co-ordinates. What is missing in your log is the vertex number to show how they match up.

 

The animation (the timeline for the Shape Key Editor) has only one Shape Key (Draped) but you will see the contribution fraction for each key frame (far left of that window, move/scrub the green line along the timeline). So the animation is a blend between the "Basis Key" and the "Draped Key" How easy is that to do ? Does it have to be done for all vertex co-ordinates?

 

Looking at "the Blender doc I got my info from" I see "use_relative". What does that do? Give relative changes to the vertex co-ordinates? Only list co-ordinates that change?

 

My lipsynced head that you post a link to has 3 shape keys but  only vertices  around the mouth are changing - the rest don't change. So can only the changing vertices be used with the "Basis Key" or will all vertex co-ordinates for each shape key have to be stored somewhere?

 

Just some initial thoughts - will have more time next week when visitors are gone.

 

cheers, gryff

 

 

 

 

 

[JCPalmer]

Just a quick answer so far. Did apply the scale, and yes I do see that the basis has very similar values as the positions & exact same length, just have a mixed up order.  Need to fix the order so it matches.  Have done some plumbing work, on the Tower of Babel side  I am generating the shape key blocks as javascript functions:

Cloth.prototype.getBasis = function () {        return [3,0,-3,-3,0,-3,3,0,3,-3,0,3,2.4545,0,-3,1.9091,0,-3,1.3636,0 ...];};Cloth.prototype.getDraped = function () {        return [1.2879,-2.6859,-1.5983,-1.3601,-2.7179,-1.5108,1.5066,-2.7339,...];};

Also, changed the updateable to true for the position vertices & normals.  Finally, I have changed the superclass of Cloth to DeformableMesh, which is a hand built BABYLON.Mesh subclass:

var DeformableMesh = (function (_super) {	__extends(DeformableMesh, _super);	function DeformableMesh(name, scene){            _super.call(this, name, scene);            this.numOfDeformSteps = 100;	}        // optional callback to super-class, performed by constructor of Tower of Babel generated code        DeformableMesh.prototype.postInit = function () {            this._scene.registerBeforeRender(DeformableMesh.prototype.incrementallyDeform);        };                DeformableMesh.prototype.incrementallyDeform = function () {            console.log('In incrementallyDeform ');        };                DeformableMesh.prototype.getBasis = function () {            return [0];        };                DeformableMesh.prototype.getDraped = function () {            return [0];        };        	return DeformableMesh;})(BABYLON.Mesh);

Right now it does not do anything, but if I can get the order of keys to positions to match,  I can try to do a 100 step deformation by interpolating the positions by 1/100 serially.  This is essentially doing it with the CPU,  but in rapid prototyping this does not matter.  None of this would go to production.  Its just the first step in my mind.  The weekend is just too nice to waste anymore of. bye.

[JCPalmer]

Quick update:  The order in the Cloth.getBasis() function now matches the positions buffer exactly!  Turns out you cannot just put out shape keys, or vertex colors, etc in Blender native vertex order.  The Babylon order is tessfaces with in each similar material.  The similar material sort is so submeshes can easily be simple ranges for fragment shaders, I think.  Not sure why when you actually have no materials, would the Babylon order to be the same as native, but it is different.  You would think I would know that, but it did not occur to me at the time.

 

Toggling back to the Babylon side!

 

Fluff rabbit:  To start, it might just be terminology but to be more exact, shape keys are final values of all vertex positions when a deformation is complete.  How this is important is let's say you have a hand.  This hand has the shape keys of:  Open(Basis), closed, pointing, peace sign, "the finger", "the devil", & "thumbs up/down".  7 endpoints, or shape keys in total.  If I am wrong I will know very shortly.

 

There should be 7 * 7 = 49 direct deformations possible, e.g. going directly from pointing to peace sign.  Look-up table just does not seem accurate enough.  gryff said up above that the talking mouth only had 3 shape keys.  If you were going to do the hand with bones, not only would you have to put them in, but you would have to do 49 animations, where you would have to pose the fingers for each frame.

 

Second,  my linear piece of shit experiment is wrong in so many ways, and nobody should think otherwise.  First linear.  There is data I have not even found yet which you can tell blender how fast certain each phase of the deformation should occur.  It is not even about how fine the number of steps is.

 

Also your assumption is incorrect about all the processing being done at load time.  What registerBeforeRender() does is tell babylon to call my incrementallyDeform() each frame.  Yes, I could the work at load time, but if there maybe 49 possible deformations it gets not feasible quick.  Plus that's pushing a lot of data (vertices & normals) up to the gpu, every frame.

[JCPalmer]

Correction 7 * 7 = 49, but some of these would be like peace sign to peace sign.  Should be 7 * (7 - 1) = 42

[JCPalmer]

update, I'm through, it works!  Have just made a total mess though, took out the DeformableMesh intermediate class, shoving a lot of code to the web page, and manually changing the way setVerticesData() get called in the generated Cloth constructor - switched to "_super.prototype.setVerticesData.call(this,".  All to do with the fact that cloth.updateVerticesData() did not acknowledge _geometry existed.  Nothing was happening.  Have to step wise undo the mess, & will publish.  Good exercise for Tower of Babel extensibility though.  This is my first Javascript program (python too), & this subclassing illusion thing is tricky.

 

 

Also see an older thread close to this subject: http://www.html5gamedevs.com/topic/6441-mesh-morph-animation-in-babylonjs/?hl=setverticesdata#entry38399 .  I do not view this as morphing, as in my mind that might mean the # of veritices could change.  Terminology aside, my experiments will continue.  Their primary purpose is to find and validate the Blender shape-key data.  Good to know this was already being planned.  Hey, I can handle the Blender side, others may be interested in doing the vertex shader side.  I have GPGPU & OpenCL experience, but it would take me much longer than others.

[JCPalmer]

Well, here is the result of the first test.  It is all code, so you can just double-click inline.html, without putting it your local web-server.  This is an unaltered cloth1a.js that was generated.  Did change the base class custom property of the Cloth mesh back to BABYLON.Mesh in the .blend file (not included).  Still sent deformableMesh.js in the .zip & it is still included in .html, but it does not get used.  The DeformableMesh equivalent code is in the .html .

 

Not sure why _geometry is not found in the updateVerticesData() call, but it might have something to do with scene.registerBeforeRender() being passed a prototype.  Was not worth holding this up.  Any pointers appreciated. 

// optional callback to super-class, performed by constructor of Tower of Babel generated codeDeformableMesh.prototype.postInit = function () {    this.basis  = this.getBasis();    this.draped = this.getDraped();    this.nVertexCoords = this.basis.length;    this._scene.registerBeforeRender(DeformableMesh.prototype.incrementallyDeform);};        DeformableMesh.prototype.incrementallyDeform = function () {    var count = this.count;    var pctComplete = this.count / this.numOfDeformSteps;    if (pctComplete === 0 || pctComplete > 1) return;                var increment = [];    for (i = 0; i < this.nVertexCoords; i++){        increment.push(this.basis[i] + ((this.draped[i] - this.basis[i]) * pctComplete));    }    // THIS call does not find _geometry in _super    _super.prototype.updateVerticesData.call(this, BABYLON.VertexBuffer.PositionKind, increment, false, true);     this.count++;};

On to finding / exporting this "timing data" for non-linear deformations!

tablecloth2.zip

[JCPalmer]

gryff:

About the part above where only some of the veritices might be involved in a deformation:   I agree, if the affected vertices can be isolated on the TOB side (it's just CPU, on a development machine at that), we might be able send over just those end xyz positions, and the babylon index of each.

 

But let's say, someone made an entire body as a single mesh.  They could have shape keys for the mouth, hands, and the whole thing walking.  Isolation across all keys could yield pretty poor improvement.  What I was thinking was there could be shape-key groups, controlled by how the key is named in the form group-state: (Mouth-open, Mouth-closed), (Hand-pointing, Hand-peace sign,...).  Isolation would be limited, to all vertices which were not identical across all keys of a group. Key names made not case sensitive.

 

It could be enforced by ignoring all keys without a '-', and not a least 2 of the same group.  Remember my exporter has a log file, so the modeller would have a way to find out what the complaint is.

 

Literally, 'walking and chewing gum at the same time' might not be doable right now,  BUT I think it is very important that the data not get represented in such a way that would preclude it.  Well at least as far as to Javascript.  All the way to vertex buffer objects might not be crucial.

 

What do you or anyone think?

 

Jeff

[gryff]

But let's say, someone made an entire body as a single mesh.  They could have shape keys for the mouth, hands, and the whole thing walking.

 

 

A pretty common approach except for perhaps the eyes,teeth and any weapons that may be picked up.

 

The picture below illustrates how I approached creating the talking head. Before I created any shape keys I created "vertex groups" that defined collections of vertices. I do this so that if I want to tweak a shape key later, In the Blender Edit Mode, with all vertices deselected,I can select a vertex group and hit the Select button - then tweak the shape keys. I'm always using the same vertices.

 

Now in addition to creating your own vertex groups, when you parent a mesh to a rig/armature shape keys are automatically created for each bone in the rig - so if you have bones Hand.L and Hand.R then vertex groups will automatically be created with those names.

 

Maybe more food for thought?

 

cheers, gryff

[JCPalmer]

More indeed.  First, making life easier for yourself by creating vertex groups, sounds smart for model development, but is both optional and not a guaranteed to get the perfect info.  Isolating all affected by comparison might be slower, but time to building exports is not nearly as important as how fast it is in Babylon.

 

Need a little clarification when you said, parenting bones to an armature creates shape keys.  You meant vertex groups, not shape keys right?  I re-exported man1, with the latest exporter and got no shape keys.

 

I am starting to weight in my mind the trade-offs of different approaches.  The first:

--------------------

- CPU calculating positions verses in the vertex shader:

    - Doing in the CPU could easily make the system CPU bound, not good for WebGL that can only use 1 core.

    - Being able to isolate the vertices & possibly updating directly to Float32Array should help, but the whole mesh position buffer still needs to be copied to GPU each deformation.

    - CPU could also not be so bad, if it was not a constant/looped effect.

    - The notion of Instances is incompatible with calculating on the CPU for sure.

    - The vertex shader might be to do the positions, if it was the whole mesh not just parts.  Having an arbitrary number of shape keys or having them operate on less than the whole mesh might prove very difficult.  

    - Vertex shaders number and types of parameters are determined at program design time.  Might be able to have 2 optionally used parameters & data buffers just for the current shaping process.  Add one more for the step completed ratio.  Suppose you could even another of the indexes of the isolated vertices.

--------------------

- CPU calculating normal verses in the vertex shader:

   - Fortunately this should be calculatable on the vertex shader regardless of where the new positions were done.

   - If they are calc'd on the vertex shader, then they would not have to be transported up like the positions.

Ref:  http://www.opengl.org/wiki/Calculating_a_Surface_Normal

 

Sounds like CPU for positions & GPU for the normals might be the best way to go.  I was very set against using CPU, but limiting some of the calcs thru analysis by the exporter might be enough.  'Walking an chewing gum at the same time' seems unlikely to ever be achieved with calcing positions on GPU, as the number of parameters would have to arbitrary to handle multiple sets of shape keys simulataneously.  Not sure how many people would use it,  but it should not cost you when only one at a time is used.

[JCPalmer]

gryff,

I recently downloaded the .blend files from Blender support that Wingnut posted the link for.  There is like 330 of them .  Noticed one in the animation sub-directory that was very good (attached as .txt)

 

There is no armature, nor shape keys.  A lot of nodes.  What is doing this animation?  Would have been the perfect .blend to test "shape key groups" concept, if it had them.

driver-object-eyes.blend.txt

[gryff]

There is no armature, nor shape keys.

 

Jeff, it is a whole different way of doing animation using lattice modifiers (the cages around the eyeballs), empties (objects that do not have vertices - all those arrow only objects) and hooks.

 

It is not a methodology that I know a lot about. I've used lattices for modifying meshes - example with a human figure creating a lattice around the mid body area can allow you to change the waist thickness. The lattice has fewer vertices then the humanoid mesh and changing a lattice vertex or group of vertices will modify the more complex mesh by changing the all the vertices close to to the chosen lattice vertices.

 

The lattice can be used to create animations of the mesh as the lattice only modifies vertices with its boundary. So if you shape the vertices of the lattice from the box shape,  then pass an object mesh though the lattice,  then as the mesh moves through the lattice the mesh vertices get changed to reflect the shape of the lattice. (think of a snake wriggle animation or moving a tail).

 

What this example shows is rather than having a fixed lattice, the empties through a second modifier - a hook modifier - are animating the lattice which in turn is modifying the mesh inside it.

 

Given that there are two modifiers at work here, how this would work as an animation in .babylon/TOB I have no idea.

 

cheers, gryff

[JCPalmer]

Update, shape-key group analysis / building on the TOB side is done.  On the Javascript side, have decided on a 4 class system:

 

1. Automaton - Sub-class of BABYLON.Mesh (beforeRender() queries each ShapeKeyGroup for changes)
2. ShapeKeyGroup - Hold each shape key, & processes a queue of  Deformation objects
3. Deformation - Holds info of end state, milli duration, millis before, ratio of end state, Pace, any prerequisites
4. Pace - Interpolation object; completion milestone calculator; non-linear capable

Each of the classes is underway.  Pace is complete, though untested:

var BABYLON;/** @immutable, reusable  * completionRatios[] - values from (> 0 to 1.0), not required to increase from left to right, for 'hicup' effects * durationRatios  [] - values from (> 0 to 1.0), MUST increase from left to right *  * The last value of BOTH args must be 1.0. * Straight line interpolation between duration Ratio elements. */var Pace = (function () {    function Pace(completionRatios, durationRatios){        // argument validations        if (!completionRatios instanceof Array || !durationRatios instanceof Array) throw new UserException("ratios not arrays");        if (completionRatios.length !== durationRatios.length) throw new UserException("ratio arrays not of equal length");        this.steps = completionRatios.length;                if (this.steps === 0) throw new UserException("ratio arrays cannot be empty");                var cRatio, dRatio, prevD = -1;        for (i = 0; i < this.steps; i++){            cRatio = completionRatios[i];            dRatio = durationRatios  [i];            if (cRatio <= 0 || dRatio <= 0) throw new UserException("ratios must be > 0");            if (cRatio >  1 || dRatio >  1) throw new UserException("ratios must be <= 1");            if (prevD >= dRatio) throw new UserException("durationRatios must be in increasing order");            prevD = dRatio;        }        if (cRatio !== 1 || dRatio !== 1) throw new UserException("final ratios must be 1");                // public member assignment for all, since immutable        this.completionRatios = completionRatios;        this.durationRatios = durationRatios;                this.incremetalCompletionBetweenSteps = [completionRatios[0]]; // elements can be negative for 'hicups'        this.incremetalDurationBetweenSteps   = [durationRatios  [0]];        for(i = 1; i < this.steps; i++){            this.incremetalCompletionBetweenSteps.push(completionRatios[i] - completionRatios[i - 1]);            this.incremetalDurationBetweenSteps  .push(durationRatios  [i] - durationRatios  [i - 1]);        }               Object.freeze(this);  // make immutable    }        Pace.prototype.getCompletionMilestone = function (currentDurationRatio) {        // at start & running late cases, easier later to have broken out here        if (currentDurationRatio <= 0) return 0;        else if (currentDurationRatio >= 1) return 1;                var upperIdx = 0;  // ends up being index into durationRatios 1 greater than highest obtained        for (; upperIdx < this.steps; upperIdx++){            if (currentDurationRatio < this.durationRatios[upperIdx])                 break;        }        var baseCompletion = (upperIdx > 0) ? this.completionRatios[upperIdx - 1] : 0;                var interStepRatio = (this.durationRatios[upperIdx] - currentDurationRatio)/ this.incremetalDurationBetweenSteps[upperIdx];                return baseCompletion + (interStepRatio * this.incremetalCompletionBetweenSteps[upperIdx]);    };           Pace.LINEAR = new Pace([1.0], [1.0]);    return Pace;})();

Pace, puts as much effort it can into the constructor, so as little as possible is required in getCompletionMilestone().  There is also a static Pace.LINEAR prebuilt.  Getting closer.

 

Jeff