Drawing app with Pixi - some challenges

[awdng]

Dear Pixie Community,

i am developing a small drawing app with Pixi.js - So far the experience has been amazing, but we are facing a challenge:

In IOS browser (iPad, iPhone) the browser crashes if you draw a lot. This takes a while to happen and many users might not experience this but its reproducible. It looks like this is due to running out of memory (JS heap) because of the large number of Graphics objects. Each Graphics Object is basically a stroke, containing hundreds or thousands of points (50Hz input sampling rate). The number of Graphics objects can go into the thousands if you draw a lot, but the number of points for each stroke is far higher of course

What we have tried so far:

1. Reducing the number of Graphics objects to include multiple strokes. This improves performance (as there are less drawcalls i guess) but still does not solve the problem.

2. After a certain number of strokes, render the viewport to a texture and load it as a background and remove the Graphics objects up to that point. This worked quite well in the beginning, but we quickly realised that panning/zooming the viewport makes this impossible as some strokes that are not visible don't get rendered to the texture. Another problem is that if you zoom out while doing this process, you loose quality when zooming in again as texture is rendered from the zoomed out view. I wonder if it's feasible to render to a second canvas for this? Possible? How would it affect performance?

3. Reducing stroke fidelity (less points) using something like simplify.js - This could work, but then we loose support for pressure sensitivity, which is a major feature we wanted to keep as browsers support this now and its quite amazing to draw like that on an iPad in the browser.

One idea that is left: Find the bounding boxes for each stroke and remove it from the scene when not visible and store in localstorage, reloading it when it is in view again. This would not solve the problem when fully zoomed out though.

So we are running a bit out of ideas now, is there maybe something we are not thinking of? Or do we have to say goodbye to pressure sensitivity or Viewport zoom/panning? We are using a 4000x4000 "world" with a viewport that can be adjusted via touch gestures.

Interestingly, Android devices seem to fare much better before crashing possibly iOS has stricter memory restrictions?

Any suggestions welcome

[ivan.popelyshev]

Hello, and Welcome to the forums!

Yes, the problems you are mentioning all part of drawing app. Yes, people use graphics or rope to make strokes, the best case is mesh rope that adjusts width based on pen pressure.

1. Pixi graphics objects require a lot of mem, yes, because there are many duplicates of vertices inside. I'm working to solve that one for my app (not related to drawing) , so I cant just help you with it before I publish all my stuff  I can just tell its possible to reduce that load significantly and you can see that if you look in sources. 

2. The best storage in that case is tiled - many 256x256 or 512x512 renderTextures , when you zoom more than 2x you should generate mip levels accordingly. If you modify texture you have to re-generate them.

> I wonder if it's feasible to render to a second canvas for this? Possible? How would it affect performance?

It wont be easier than to use renderTexture's. Second context is a big problem on its own, if you dont have experience with that - dont try it in production.

3. Yes, you have to add more points when mouse is lagging and remove unnecessary points when they are too close.

The stuff you are asking was asked before by guys who were ready to pay for this. PixiJS team is not obliged, and while I can point you to possible solutions - you have to be specialist in drawing apps, not me. Yes, I can do all that stuff and get the result you want, but I arleady have too many things on me. I know that so far no one made real ideal system, you have all chances to do that.

>So we are running a bit out of ideas now, is there maybe something we are not thinking of

The best case is combination of all. many renderTextures, mip levels, graphics/ropemesh for strokes, lazy rendering of strokes to your cached renderTextures, e.t.c.

If you dont have , you can try to hire people from pixijs slack , I can send you an invite

Here's one of open-source projects of drawing apps: https://github.com/wonderunit/alchemancy

Here's the demo on how to make good strokes with special shader and not antialiasing: https://www.pixiplayground.com/#/edit/nyhhOobH-Sq1Vh93Q-6E_

Here's the old pixi-heaven v4 demo of rope that can change width, I have not port it to v5 yet, please express that you really need it if you want me to do it faster: https://gameofbombs.github.io/examples-heaven/#/mesh/rope-colored.js , https://github.com/gameofbombs/pixi-heaven/blob/master/srcOld/mesh/Rope.ts

Also I recommend to look what drawing app mrdoob implemented, I havent looked at it, yet.