Get backdrop in a shader in Pixi 5

[ZackMercury]

Seemingly simple task, yet so unclear and confusing in Pixi. 

After searching for a while, found this and that is exactly what I need

https://gameofbombs.github.io/examples-heaven/#/picture/displacement-backdrop.js

And that is for Pixi 4. 

Once I copied the code into pixi examples window and fixed the method names, I saw this (Attachment) 

const shaderVert =
    `
attribute vec2 aVertexPosition;
attribute vec2 aTextureCoord;

uniform mat3 projectionMatrix;

varying vec2 vTextureCoord;
varying vec2 vFilterCoord;

void main(void)
{
   gl_Position = vec4((projectionMatrix * vec3(aVertexPosition, 1.0)).xy, 0.0, 1.0);
   vTextureCoord = aTextureCoord;
}
`;

const shaderFrag =
    `
varying vec2 vTextureCoord;

uniform vec2 scale;

uniform sampler2D uSampler;
uniform sampler2D backdropSampler;

uniform vec4 filterArea;
uniform vec4 filterClamp;

void main(void)
{
  vec4 map =  texture2D(uSampler, vTextureCoord);

  map -= 0.5;
  map.xy *= scale / filterArea.xy;

  vec2 dis = clamp(vec2(vTextureCoord.x + map.x, vTextureCoord.y + map.y), filterClamp.xy, filterClamp.zw);
  gl_FragColor = texture2D(backdropSampler, dis);
}
`;

class DisplacementFilter extends PIXI.Filter {
    constructor(scale) {
        super(
            shaderVert,
            shaderFrag
        );

        this.uniforms.scale = {x: 1, y: 1};

        if (scale === null || scale === undefined) {
            scale = 20;
        }

        this.scale = new PIXI.Point(scale, scale);

        this.backdropUniformName = 'backdropSampler';
    }

    apply(filterManager, input, output) {
        this.uniforms.scale.x = this.scale.x;
        this.uniforms.scale.y = this.scale.y;

        // draw the filter...
        filterManager.applyFilter(this, input, output);

        this.clearColor = [0.5, 0.5, 0.5, 1.0];
    }
}


const app = new PIXI.Application(800, 600);
document.body.appendChild(app.view);

app.stage.interactive = true;
app.stage.filters = [new PIXI.filters.AlphaFilter()];
app.stage.filterArea = app.screen;

const container = new PIXI.Container();
app.stage.addChild(container);

const padding = 100;
const bounds = new PIXI.Rectangle(
    -padding,
    -padding,
    app.screen.width + padding * 2,
    app.screen.height + padding * 2
);
const maggots = [];

for (let i = 0; i < 20; i++) {
    const maggot = PIXI.Sprite.from('https://pixijs.io/examples/examples/assets/maggot.png');
    maggot.anchor.set(0.5);
    container.addChild(maggot);

    maggot.direction = Math.random() * Math.PI * 2;
    maggot.speed = 1;
    maggot.turnSpeed = Math.random() - 0.8;

    maggot.x = Math.random() * bounds.width;
    maggot.y = Math.random() * bounds.height;

    maggot.scale.set(1 + Math.random() * 0.3);
    maggot.original = new PIXI.Point();
    maggot.original.copyFrom(maggot.scale);
    maggots.push(maggot);

}

const displacementContainer = new PIXI.Container();
const displacementTexture = PIXI.Texture.from('https://pixijs.io/examples/examples/assets/pixi-filters/displace.png');
for (let i = -1; i <= 1; i += 2) {
    let sprite1 = new PIXI.Sprite(displacementTexture);
    sprite1.position.set(100*i, 0);
    sprite1.anchor.set(0.5);
    displacementContainer.addChild(sprite1);
}


app.stage.addChild(displacementContainer);

const displacementFilter = new DisplacementFilter();
displacementContainer.filters = [displacementFilter];
displacementFilter.scale.x = 110;
displacementFilter.scale.y = 110;
//displacementFilter.padding = 0;


const ringTexture = PIXI.Texture.from('https://pixijs.io/examples/examples/assets/pixi-filters/ring.png');
const rings = new PIXI.Container();
for (let i = -1; i <= 1; i += 2) {
    let sprite1 = new PIXI.Sprite(ringTexture);
    sprite1.position.set(100*i, 0);
    sprite1.anchor.set(0.5);
    rings.addChild(sprite1);
}
rings.visible = false;

app.stage.addChild(rings);

const bg = PIXI.Sprite.from('https://pixijs.io/examples/examples/assets/bg_grass.jpg');
bg.width = app.screen.width;
bg.height = app.screen.height;

bg.alpha = 1;

app.stage.addChildAt(bg, 0);

app.stage
    .on('mousemove', onPointerMove)
    .on('touchmove', onPointerMove);

function onPointerMove(eventData) {
    rings.visible = true;

    displacementContainer.position.set(eventData.data.global.x, eventData.data.global.y);
    rings.position.copyFrom(displacementContainer.position);
}

let count = 0;

app.ticker.add(function () {

    count += 0.05;

    for (let i = 0; i < maggots.length; i++) {
        const maggot = maggots[i];

        maggot.direction += maggot.turnSpeed * 0.01;
        maggot.x += Math.sin(maggot.direction) * maggot.speed;
        maggot.y += Math.cos(maggot.direction) * maggot.speed;

        maggot.rotation = -maggot.direction - Math.PI / 2;
        maggot.scale.x = maggot.original.x + Math.sin(count) * 0.2;

        // wrap the maggots around as the crawl
        if (maggot.x < bounds.x) {
            maggot.x += bounds.width;
        }
        else if (maggot.x > bounds.x + bounds.width) {
            maggot.x -= bounds.width;
        }

        if (maggot.y < bounds.y) {
            maggot.y += bounds.height;
        }
        else if (maggot.y > bounds.y + bounds.height) {
            maggot.y -= bounds.height;
        }
    }
});

Obviously, something isn't working as in Pixi 4.

[ivan.popelyshev]

> Seemingly simple task, yet so unclear and confusing in Pixi. 

OK, which webgl libraries/frameworks give you access to backdrop?

https://pixijs.io/examples/?v=v5.3.7#/plugin-picture/displacement.js - works for 5.3.7 release, doesnt work for 6.0-rc (dev) because pixi-picture wasnt modified to take into account changes in systems. Do you need me to update it?

btw, im going to update it anyway, because i found out that it should work even without extra filters if renderer is created with "transparent:true", and it has same format as rendertextures, RGBA

Edited January 23, 2021 by ivan.popelyshev

[ZackMercury]

Dunno about WebGL libraries, I come from Unity and in Unity it's as easy as described here http://tinkering.ee/unity/asset-unity-refractive-shader/

Use a GrabPass and you got the background in place. Same you can do in any game engine since it's an essential feature for creating VFX.

Hmm, last time I checked there was no Picture plugin in the examples folder.

Wait, so it's not pure Pixi? I didn't spot any additional class usages in the code, it extended PIXI.Filter, what exactly does pixi-picture do for this code to work?

Thanks for your assistance.

[ivan.popelyshev]

> Dunno about WebGL libraries, I come from Unity 

WebGL has less features than whatever unity has access on low-level, in webgl we have to make extra steps to get backbuffer. Also, PixiJS is the only lib that is exposing backbuffer in a plugin, there's also AwayJS but its only used internally there.

> Wait, so it's not pure Pixi?

Its here: https://github.com/pixijs/pixi-picture/tree/master/dist

Its a hack over filterSystem, and it wasnt approved to be merged in main pixi repo because by default we used "RGB" format for main backbuffer and it's not easy to inform users that certain feature works only if there's a filter on top of everything. I think its likely to be merged in v6.

 

[ZackMercury]

A slightly offtopic question here. Never had to deal with external plugins yet, so if I just specify this package in dependencies https://www.npmjs.com/package/pixi-picture , it's gonna overwrite all that's needed?

[ivan.popelyshev]

its not offtopic, its a real problem all users of webpack have it

Make PIXI provided, include it through <script src> , then you can include plugins in dependencies.

alternatively, make a bundle - special file that re-exports pixi and adds it to window before plugin if imported

export * as PIXI from 'pixi.js';
window.PIXI = PIXI;
require('pixi-picture');

you can change it to pixi.js-legacy if you need canvas2d

Why is it difficult? because people who made ES6 modules standards and those who made basic implementations screwed up whole ecosystem. Basically "hey guys here's new stuff all new coders will use, please use it and abandon all your old users in favor of new guys who know only react"

We only address it now, for pixi-v6, because we had to actually make our own tools to make everything work the right way

 

Edited January 23, 2021 by ivan.popelyshev

[ZackMercury]

Would that work if I prefer the PIXI.-less class syntax like this

[ZackMercury]

Oh, I kinda get it. We're assigning it to window so that pixi-picture has a reference to it, so the export * as PIXI method should preserve everything as is.

[ivan.popelyshev]

yeah. we are making experiments with pixi-tilemap to make hybrid approach that solves everything, pixi-picture will be one of next if we succeed

 

[ZackMercury]

Hmm, I can't seem to get it working.

Is it legit like this?

UPD:

Okay, seems to be working now, gotta check if the demo runs now.

Edited January 23, 2021 by ZackMercury

[ZackMercury]

Also FYI, you can send null instead of the vertex shader since your vertex shader is identical to the default one

[ZackMercury]

Nah, I can't get it working. Changed to JS, still throws me weird errors 

 

Edited January 23, 2021 by ZackMercury

[ZackMercury]

Doesn't work like this either

[ZackMercury]

Ok this works now!

 

[ZackMercury]

When I use typescript tho, it keeps throwing me that error and intellisense is not working

When I compile it works well.

Is there any work around?

Ok, so I figured a workaround. Set "allowJs" flag of tsconfig to true and use a JS file for the bundle. didn't work, it didn't recognize the module...

Edited January 24, 2021 by ZackMercury

[ivan.popelyshev]

There "workarounds":

1. wait for v6 and convertion of all modules.

2. throw out webpack a, rollup or whatever you use, for typescript use <reference types= ...>

[ZackMercury]

For that, I need to know how long I have to wait. Any planned deadline or release date?

[ivan.popelyshev]

maybe write bundle as ts file too, and do better re-export there?

[ZackMercury]

In that case, it gives me the mentioned above error

[ivan.popelyshev]

`export * as PIXI from "pixi.js"` doesnt work?

[ZackMercury]

It works if I compile, but Intellisense is giving me the same error as above, and I get no code autocompletion for PIXI classes and methods.

UPD: Oh wait, it compiles, but doesn't work! The only code that works both in JS and TS is 

import * as PIXI from "pixi.js";
export * from "pixi.js";
window.PIXI = PIXI;
require("pixi-picture");

 

Edited January 24, 2021 by ZackMercury

[ZackMercury]

I'm sorry for being excessively persistent, but I'd still like to know when to expect this issue resolved so I can be able to use this with TypeScript.

I need some promised date of deadline or a release date.

[ivan.popelyshev]

> I need some promised date of deadline or a release date.

Im sorry but I cant provide you any date. If you want - you can copy all pixi-picture sources to your project and add import/exports, that'll resolve it for sure.

This is biggest thing for pixi-v6, really. You want ot speed it up - go talk to @bigtimebuddy in pixijs slack , place a huge bounty on it. Team just doesnt have enough time working on their jobs to finish it with reliable date.

Edited February 4, 2021 by ivan.popelyshev