Huge performance hit after upgrading from 3.1.0-alpha3.5 to 3.1.0-alpha3.7

[thrice]

in 3.5 scene was rending at 60 fps,  now I'm stuck around 45 fps as soon as I upgraded to 3.7

Even worse, load time of the scene increased from 28ish seconds to 75 seconds.

My only hint is a cryptic webgl warning being thrown 

WebGL: INVALID_ENUM: getQuery: invalid parameter name

but even then, I doubt it's responsible for the 15 fps drop (maybe the initial loading increase, IDK)

Any ideas?

-- side issue

Additionally, I tried looking at drawCalls, as that was my KPI in the past for figuring out why frame rate was so low, and I am met with an unhelpful "drawCalls is deprecated. Please use SceneInstrumentation class" warning, and draw calls returning 0.

So I instantiate the instrumentation, and I still don't see draw calls. I would think having that metric easily discoverable is pretty important?

[MackeyK24]

First of all... I get this alot too, because every now and then I grab the latest PREVIEW release to make sure the toolkit is compatible with the UP AND COMING changes.

With that being said... IT IS A PREVIEW RELEASE... That means some guy (anybody) can really make changes to the codebase and submit a PR... It may compile and get built in to the latest preview .... That does NOT mean everything is all tested and working... If you want TESTED and WORKING code,  you gotta stick with the official releases.

Also... we are currently in ALPHA so there are gonna be some issues with just grabbing the latest release... you gotta ride it out, check back in a few days, grab again and try rebuild.

If you had a version of Babylon.js that is working ... STICK WITH THAT... Use ANOTHER project to test the viability of the current PREVIEW release.. If ok then update your main project babylon.js

BTW... We should be going to beta soon, so things will be a little more ironed out in the preview release

 

[adam]

Thanks for reporting.  It is important to know if there are drops in performance.

If you could give us some detail about your scene or reproduce the issue, that would be very helpful.

Edit:

Here is where you should now be getting drawCall info:

https://github.com/BabylonJS/Babylon.js/blob/0bd16a8b659d71f98c39adf3c1197232d8551830/src/Instrumentation/babylon.sceneInstrumentation.ts#L395

You can also use the debug layer to see the number of draw calls.

https://doc.babylonjs.com/how_to/debug_layer

[thrice]

4 hours ago, MackeyK24 said:

First of all... I get this alot too, because every now and then I grab the latest PREVIEW release to make sure the toolkit is compatible with the UP AND COMING changes.

With that being said... IT IS A PREVIEW RELEASE... That means some guy (anybody) can really make changes to the codebase and submit a PR... It may compile and get built in to the latest preview .... That does NOT mean everything is all tested and working... If you want TESTED and WORKING code,  you gotta stick with the official releases.

Also... we are currently in ALPHA so there are gonna be some issues with just grabbing the latest release... you gotta ride it out, check back in a few days, grab again and try rebuild.

If you had a version of Babylon.js that is working ... STICK WITH THAT... Use ANOTHER project to test the viability of the current PREVIEW release.. If ok then update your main project babylon.js

BTW... We should be going to beta soon, so things will be a little more ironed out in the preview release

 

I'm well aware, I'm simply trying to help out and narrow down what could have changed between versions to cause issues in my project locally.

[thrice]

2 hours ago, adam said:

Thanks for reporting.  It is important to know if there are drops in performance.

If you could give us some detail about your scene or reproduce the issue, that would be very helpful.

Edit:

Here is where you should now be getting drawCall info:

https://github.com/BabylonJS/Babylon.js/blob/0bd16a8b659d71f98c39adf3c1197232d8551830/src/Instrumentation/babylon.sceneInstrumentation.ts#L395

You can also use the debug layer to see the number of draw calls.

https://doc.babylonjs.com/how_to/debug_layer

Ah, thanks for the draw call info. I will look into it further.

[thrice]

Trying to load the debug layer is now throwing 

[MackeyK24]

1 hour ago, thrice said:

I'm well aware, I'm simply trying to help out and narrow down what could have changed between versions to cause issues in my project locally.

I didn't mean to offend, was just trying to clarify

 

[adam]

54 minutes ago, thrice said:

Trying to load the debug layer is now throwing 

Unfortunately, the inspector is currently not working well in IE or Edge.  

[thrice]

1 hour ago, adam said:

Unfortunately, the inspector is currently not working well in IE or Edge.  

I'm using chromium, app is packaged via electron. So, I can't switch to a diff browser to run it ATM to see if I get different results/debug.

I'm going to downgrade back to previous version for now.

[thrice]

1 hour ago, MackeyK24 said:

I didn't mean to offend, was just trying to clarify

 

No offense taken  

[GameMonetize]

Hello!

I cannot repro the performance issue. The load time increase is also utterly suspicious. I'm really interested to understand this because it is too much for a simple bug

Furthermore, I do not repro the getQuery issue. Can you try outside of Electron? 

Can you profile as well to help me understand where the issue is?

 

[thrice]

Hey Delta!

I'll try and dig into it again this weekend. Unfortunately I can't just run it without electron ATM because of the way I am loading assets (via the node process) - Ya I wasn't expecting you to be able to reproduce the getQuery issue was more hoping someone knew what it meant, or possibly what it could allude to in terms of the code itself.

I will get a profile if I can't figure out what is going on. Honestly the bigger issue is the fact that my go to tool for figuring out what is going on(simply, engine.drawCalls), has been removed, and I am unable to load any of the new instrumentation stuff currently, so I can't even see if my current draw calls matches bumped version draw calls.

I had a similar issue in the past IIRC, which I saw the drawCalls spiking after upgrading, so I was able to determine that a large number of the meshes that I load off screen and disable (for the sake of tooltips/hover states), were being loaded in at 0,0,0, which was being covered up, so basically the camera was trying to render hundreds more meshes than it should have been. - The reason I think it may be different this time is the strange webgl error, but IDK. Will try and debug it further when I have time.

[GameMonetize]

For pure drawcallm here is a hack:

this.scene.getEngine()._drawCalls will give you the value

[MackeyK24]

Yo @thrice the new 3.1 Beta looks good... Check it out if you haven't already

 

[agmcleod]

I've noticed my scene running slower too, even on 3.1.0 beta.

[MackeyK24]

52 minutes ago, agmcleod said:

I've noticed my scene running slower too, even on 3.1.0 beta.

Well some of my scenes have issues with the current 3.1 but performance is NOT one of them. They all run much faster than a few versions ago.

Maybe there was a bug in there for a HOT-SECOND. But my safari builds where running DOG SLOW like 5 - 15 FPS. But now the run at full 60FPS.

I am getting a bunch of uniforms blocking errors, but I am going to track those down and try post the error.

But all and all, compared to a few versions ago 3.1 beta is great so far... Will get better if we can get these little issues tracked down and sent to @Deltakosh 

I can run most (actually, all but one, my space shooter) scenes again at full 60FPS on my Mac OSX.

 

[GameMonetize]

@agmcleod can you help me identifying the bottleneck? Can you repro on the PG (using stable vs latest) ?

[MackeyK24]

I found one... This is the scene I made to test shadow baking... On These Later BabylonJS preview release it does seem to be more CPU intensive, especially if I leave the focus of the browser to another window or application, My WHOLE Computer slows down a bit and of course the browser games drop to about 10 - 20 FPS... If I click back and give browser focus... On Chrome and Firefox it speeds back up to 60 FPS... But again seems a bit CPU intensive than before.

There was a version of babylon 3.1 alpha that worked smoothly on all browser for that same scene... I used to show snapshots all at 60 FPS

Anyways... I put that scene on my server at: http://mackey.cloud/games/survive/

Yo @Deltakosh ... Can you please take look and see if your can tell whats up... It does list any errors in console... just real slow on safari and a bit CPU intensive on all others and it was not that way before (at least a few version ago, not sure the number)

Here is the raw project files: Export Output Zip in case you want to put on server and check out all the files.

 

[agmcleod]

@MackeyK24 @Deltakosh I may have been misunderstanding the core issue. I don't think FPS actually drops, but the scene itself takes longer. It's a physics based scene using cannon.js. The results seem to be the same, but it just runs longer for some reason.

[GameMonetize]

ok let me know if you find what is the issue

[GameMonetize]

@MackeyK24 I do not see something obvious on the perf side of your demo

Works pretty well here

Can you run a profile session on your device as well?

[MackeyK24]

50 minutes ago, Deltakosh said:

@MackeyK24 I do not see something obvious on the perf side of your demo

Works pretty well here

Can you run a profile session on your device as well?

I checked on windows too... Works pretty good.

It must just be with the Mac Version... Can you try it on Mac ???

[GameMonetize]

I do not have a Mac. I have several PC though and none has an issue. Did you have the same on firefox as well?

Can you run the profiler on your Mac?

[MackeyK24]

Yo @thrice ... Did you ever make any head way... I have noticed the same issues on a scene that I had where I use cannon physics for everything. Kinda real sluggish not to mention the collisions... Let me know if you make any headway here, I will do the same

 

[thrice]

@MackeyK24 sorry for slow response, just saw you tagged me. - I'm just finally digging back into this. - I can tell you though that I don't use cannon for anything, so not sure thats the culprit. Unless it somehow is autoloaded/started in newer release or something? (I do see when I yarn install the latest version that it adds cannon as a dependency by default).

So I just updated to latest release, and scene instrumentation now working for me (yay) - good news and bad:

GOOD news is the issue regarding the scene load time seems to be fixed, and it is loading about the same amount of time as my locked version branch.

BAD news is FPS is still sluggish. Upper 40/low 50s -

@Deltakosh

I've collected some output from various instrumentation calls, was hoping you could take a look and see if you see anything abnormal, because I'm not sure exactly what to look for. - Also I did some grouping and counting of other objects in the scene that in my limited experience, have seemed to affect performance the most (and I've optimized scene to do many, i.e. freezingWorldMatrix and disabling meshes that are not currently in use, of which I have many, because my project is a card game and I render a master list of all the relevant cards off screen for sake of hovers and what not)

 

activeMeshesEvaluationTimeCounter
{
  "_startMonitoringTime": 48000.035,
  "_min": 0,
  "_max": 125.875,
  "_average": 1.927268518518444,
  "_lastSecAverage": 1.3325480769225089,
  "_current": 2.6350000000020373,
  "_totalValueCount": 324,
  "_totalAccumulated": 624.4349999999758,
  "_lastSecAccumulated": 17.705000000009022,
  "_lastSecTime": 47866.435000000005,
  "_lastSecValueCount": 14
}

renderTargetsRenderTimeCounter
{
  "_startMonitoringTime": 85556.585,
  "_min": 0,
  "_max": 0,
  "_average": 0,
  "_lastSecAverage": 0,
  "_current": 0,
  "_totalValueCount": 660,
  "_totalAccumulated": 0,
  "_lastSecAccumulated": 0,
  "_lastSecTime": 0,
  "_lastSecValueCount": 660
}

frameTimeCounter
{
  "_startMonitoringTime": 47999.685000000005,
  "_min": 0,
  "_max": 2674.9200000000055,
  "_average": 28.738333333333202,
  "_lastSecAverage": 6.943367346938894,
  "_current": 6.405000000006112,
  "_totalValueCount": 162,
  "_totalAccumulated": 4655.609999999979,
  "_lastSecAccumulated": 229.41500000000087,
  "_lastSecTime": 47396.780000000006,
  "_lastSecValueCount": 31
}

engine.gpuFrameTimeCounter
{
  "_startMonitoringTime": 0,
  "_min": 0,
  "_max": 2800898000,
  "_average": 14517034.810126582,
  "_lastSecAverage": 4680863.636363637,
  "_current": 4774000,
  "_totalValueCount": 316,
  "_totalAccumulated": 4587383000,
  "_lastSecAccumulated": 45263000,
  "_lastSecTime": 85156.96,
  "_lastSecValueCount": 10
}

engine.shaderCompilationTimeCounter
{
  "_startMonitoringTime": 73101.945,
  "_min": 0,
  "_max": 60.06500000000233,
  "_average": 12.432499999999386,
  "_lastSecAverage": 15.997708333332412,
  "_current": 15.690000000002328,
  "_totalValueCount": 36,
  "_totalAccumulated": 447.5699999999779,
  "_lastSecAccumulated": 0,
  "_lastSecTime": 73117.63500000001,
  "_lastSecValueCount": 0
}

// these lodash methods if you're not familiar with the library, are just calling the properties/methods and grouping by the count of the results

_.groupBy(_.map(play_game.scene.babylon.meshes, 'isWorldMatrixFrozen'))
Object {false: Array[430], true: Array[1395]}

_.groupBy(_.invokeMap(play_game.scene.babylon.meshes, 'isEnabled'))
Object {true: Array[487], false: Array[1345]}

_.groupBy(_.map(play_game.scene.babylon.meshes, 'areNormalsFrozen'))
Object {false: Array[1263], undefined: Array[576]}

_.groupBy(_.map(_.compact(play_game.scene.babylon._activeMeshes.data), 'areNormalsFrozen'))
Object {false: Array[91], undefined: Array[143]}

play_game.scene.babylon._activeMeshes.data.length
256

_.filter(play_game.scene.babylon._activeMeshes.data, _.isUndefined).length
23
// (note: ^, is this a problem that there are 23 undefined values in _activeMeshes.data?)

_.groupBy(_.invokeMap(_.compact(play_game.scene.babylon._activeMeshes.data), 'getClassName'));
Object {Mesh: Array[91], InstancedMesh: Array[143]}

_.groupBy(_.invokeMap(_.compact(play_game.scene.babylon.meshes), 'getClassName'))
Object {Mesh: Array[1263], InstancedMesh: Array[576]}

play_game.scene.babylon.materials.length
690

Object {ShaderMaterial: Array[7], GradientMaterial: Array[1], StandardMaterial: Array[682], PBRMaterial: Array[2]}

_.groupBy(play_game.scene.babylon.materials, 'isFrozen')
Object {false: Array[692]}
// ^ this freeze / isFrozen on materials option, I've just recently seen in docs, is this new and or could this be related? Was it auto freezing in the past or something like that?