Dickensian

video alpha channel approaches

Recommended Posts

Hi,

thanks for your reply

i've tried the Chrome support for WebM with alpha which works but is chrome desktop only, and I've read previous posts that people have done using half the video frame as a luminosity mask with html5 canvas libraries. I've not explicitly looked at multiplexed streams, but I guess they'd be another option.

Even if we could encode a matte colour and mask on that with a defringe filter a single RGB video texture could work. The challenge with using two non-multiplexed sources for the texture and the matte is around maintaining the sync.

Should I be looking at a custom shader or do you recommend another approach to achieve this; I'd like to ultimately have an actor walk on canvas and point at the 3D model and talk about it for an educational demo I'm working on in my free time.

Dickensian

 

Share this post


Link to post
Share on other sites

Yes I guess the best option is to use a custom shader with a videoTexture. This way you can control how to apply the mask

Here is an article about customshaders in babylon.js: https://blogs.msdn.microsoft.com/eternalcoding/2014/04/17/what-do-you-mean-by-shaders-learn-how-to-create-shaders-with-babylon-js/

Share this post


Link to post
Share on other sites

Great link, thank you

i'll pursue this and post back when I know more.

BTW did you know that a lot of the tutorial links from Google search results are hitting 401's since the site changes, if an admin could set up some 301 redirects based on the logs it would be a great help to noobs like me.

Dickensian

Share this post


Link to post
Share on other sites

So, has anyone been able to recognize alpha channels (RGBA video) on supported video formats such as webm in babylon.js? I doubt it, but it's worth asking. Has anyone such as @Dickensian who started this topic built a shader to recognize a color range in RGB pixels and convert to transparent pixels? I beileve this might be the quickest route to a demo, but then I would love to have the support of alpha channels RGBA on video formats such as webm and forthcoming formats. Any ideas beyond the above?

OK @Wingnut you love a challenge.:D And just so you know, I've spent days trolling the web and found extensions and possible solutions, but nothing really worth noting here - as I'd really like to add this function distinctly to BJS.

Cheers,

DB

EDIT - What about using the alphaindex of a mesh? Is this a crazy approach?

Share this post


Link to post
Share on other sites

@dbawel I've not had time to look at this unfortunately, although the shader that @tranlong021988  linked to looks promising for integration into Babylon.js.

@dbawel I'm looking to make this work for a not-for-profit educational tool, if you'd like help with peer review on your code, please publish a link to your working repo and I'd be happy to contribute some effort to it.

Thanks,

Dickensian.

Share this post


Link to post
Share on other sites

@Dickensian -

We have a simultanious multiuser media design application we are alpha testing. Please message me personally for more info if you are interested in learning more. Perhaps you might have a group which can help test and us build a better app. It is currently going to Weta and to Lockheed Martin for testing. Video compositing is for a secondary feature, and not for first version release.

@adam - Please let me know what is NSFW about the link, as I've not found any issues with it thus far. If so, I should be more on my game.

DB

Share this post


Link to post
Share on other sites

A shader that can recognize a chroma key.  I'd look at the video shader and then see if you can pass a color and a threshold value to the shader then modify the shader to turn the pxls that it's displaying a alpha accordingly.

Share this post


Link to post
Share on other sites
<script id="fragmentShader" type="x-shader/x-fragment">
	uniform sampler2D texture;
	uniform vec3 color;
	varying vec2 vUv;
	void main()
	{
		vec3 tColor = texture2D( texture, vUv ).rgb;
		float a = (length(tColor - color) - 0.5) * 7.0;
		
		gl_FragColor = vec4(tColor, a);
	}
</script>

source: https://github.com/makc/makc.github.io/blob/master/three.js/chromakey/index.html 

@Pryme8 would you impliment the shader like this example from a three.js project or differently? Are you aware of any approaches to defringe the edges, where the greescreen cast can show through?

Share this post


Link to post
Share on other sites

Um yeah I can possible write something up for you tonight or tomorrow, I'll just modify Babylons video shader and upload the source for you, and yes I'll include a bounding box for the alpha but I'm not sure why you would need that.  Anyways I'll post updates once I have them unless someone beats me to it.

Share this post


Link to post
Share on other sites

we really need fix alpha blend problem 

we don't use that in BJS for now so maybe need some team start work that (i wanna join but i know less in this stuff )

gl.blendEquation( gl["FUNC_ADD"] )

gl.blendFunc( sourceBlendType  ,  destinationBlendType , blendEquation );

gl.enable(gl.BLEND);

gl.depthMask(false);

you can see parameter value from thire

https://msdn.microsoft.com/en-us/library/dn302371(v=vs.85).aspx

we just need list of alpha Object (for find gl Render Object ) and refresh that when we change camera position

http://delphic.me.uk/webglalpha.html

Demo : http://delphic.me.uk/Fury/demos/AlphaTest/

but about discard is good solution

other plan for that ( play video in 3d scene like lizer draw ) you just draw light color 

http://www.babylonjs-playground.com/#3KRGA#1

http://www.babylonjs-playground.com/#3KRGA#2

i dont know why sample 1 work fine but sample 2 is wrong ( alpha sorting problem )

anyway i think if you work with single face maybe you can use alpha without any problem (if you don't care for Low FPS )

http://www.babylonjs-playground.com/#3KRGA#3

http://www.babylonjs-playground.com/#3KRGA#4

 

 

 

Share this post


Link to post
Share on other sites

I can not find the babylon.js git on the videoShader

http://makc.github.io/three.js/chromakey/

yeah looks like using a chromatic is a valid approach.

also @NasimiAsl
I striped a bunch of stuff out of your example http://www.babylonjs-playground.com/#2BBTZV

how did you do the alpha calculations?  I was gonna edit the GLSL files but it looks like your able to do it with your shader builder include?
can you explain whats going on in your example on how your generating the basic alpha transparency.  I know exactly how I will do the chromatic key I just need to be able to see the videoShaders fragment and vertex scripts, its a really easy clamp calculation and the only area of concern is the dithering area between key and objects on the video.  Should not be to bad and I will add a threshold to adjust how fast that fades.

***UPDATE EDIT,
still cant find the video shader, but I have confirmed that discard is going to be the method of choice, just setting the alpha makes the texture colors dimmer.

If you look at the discard example on the CYOS and modify the line for the discard to read:

if (color.g > 0.45) {
        discard;
    }else{
        color.r = 1.0;
        color.g = 0.0;
        color.b = 0.0;
    }
you can see that its pretty clean, and would work for general purposes, and the color correction at the end could be tailored. now to just load a video in and have a variable for the color you want to discard and how much of it.

Share this post


Link to post
Share on other sites
1 hour ago, Pryme8 said:

ow your generating the basic alpha transparency

in Each Step of ShaderBuilder Commands I change the 'result' variable

'result' is your Fragment Color you Made be before and you most change that if you want change the color

for use Enable Alpha you just need Add '.Transparency()' step in your  commands

and for set Alpha parameter you just need write a line

 .InLine('result.w = 0.1;')  or InLine('result.w = sin(time*01);')

 

http://www.babylonjs-playground.com/#2BBTZV#1

for understanding InLine you just write your shader in source and in last make one vec4 and set that in result

http://www.babylonjs-playground.com/#2BBTZV#2

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...

  • Recently Browsing   0 members

    No registered users viewing this page.