Video Texture
Video texture is a new feature of mCL on DirectX 9.x. It allows you to show a video over an object surface while an object is moving. You can write a vertex and pixel shader with HLSL language to generate lighting or deformation effects over the video. And you can also implement algorithms of image processing in the pixel shaders for realtime video processing. An interesting application is to add MP3 background music as audio stream into the 3D animation.
To mix video/audio with 3D graphics you need only add graph object into a mCL shader.
- In the Shader class declare variable
variable graph
- In InitDeviceObjects method insert
set userid 0xACDCADCD
set filename [$app dialog file "*.mp3" $initfolder]
set graph [$app graph $filename $userid]
you may set fixed file name string without using the open file dialog box. The userid is a 32-bit integer.
- In the RestoreDeviceObjects method insert
$graph start
- In the InvalidateDeviceObjects method insert
$graph end
- In the DeleteDeviceObjects method insert
$graph release
Examples
You can find video texture samples under the folder shaders\dx9\video of the Ladybug Studio XP. The DirectX 9.0 or later is required. Following mCL code demonstrates a video over a plane. The plane rotates along the axis between the bottom-left and top-right vertices while the video is playback. The graph object is used to open, start, and close media. Entire program is 260 lines.
call mcl package
package require -exact mcl 2.5
video manager shader is a class which includes variables and methods.
class VideoManageShader {
variable app
variable ft
variable mat
variable vbPlane
variable vertice
variable texture
variable declPlane
variable mtime
variable tex
variable graph
In the constructor we create the application object and specify a vertice with position, diffuse, and tex1 components.
constructor {args} {
# hide control window
catch {wm withdraw .}
# create application object
set app [application]
# create vertice struct decl
set vertice [struct {float3 position dword diffuse float2 tex1}]
}
these methods create, run, and release applications
method Create {} {$app create $this}
method Run {} {$app run}
method Release {} {$app release; delete object $this}
getFile is an help method to find a folder path
method getFile {{type x} {fname ""}} {
set curpath [file join [pwd] [file dirname [info script]]]
switch -exact $type {
x {return
[file join $curpath .. .. .. media models x $fname]}
shader {return
[file join $curpath .. .. .. media shader dx9 $fname]}
texture {return
[file join $curpath .. .. .. media texture $fname]}
default {return
[file join $curpath .. .. .. media models x $fname]}
}
}
showHelp method display the help keys on the screen when you press F1
method showHelp {} {
if {[$app showHelp]} {
set cmdTitle "Keyboard Controls:"
set cmdOp "Play\nStep\nHelp\nPlay\nRestart\nFullscreen\nExit"
set cmdKey "Enter\nSpace\nF1\nF2\nF4\nF5\nEsc"
$ft draw 2 40 0xFFFFFFFF $cmdTitle
$ft draw 20 60 0xFFFFFFFF $cmdOp
$ft draw 210 60 0xFF00FFFF $cmdKey
$ft draw 2 0 0xff00ff00 [$app getFrameStats]
$ft draw 2 20 0xff00ff00 [$app getDeviceStats]
}
}
InitStartup method creates a font object and declare global matrix.
method InitStartup {} {
# create font object
set ft [$app font Vertana 10 0]
# declare global matrix
matrix mat(World)
matrix mat(View)
matrix mat(Proj)
}
In the FrameMove method, we set new position and update them in the vertex buffer.
The difference is the current time minus the start time. Then the new (x, y, z) position is modified with a sin/cos function. mask variables are used to colors.
method FrameMove {} {
# set new position
set curtime [$app getTickCount]
set difference [expr {mtime - curtime}]
set x [expr -cos(difference / 2000.0)]
set y [expr cos(difference / 2000.0)]
set z [expr sin(difference / 2000.0)]
set mask0 [expr int(255*((y+1.0)/2.0))]
set mask0_8 [expr int(mask0)<<8]
set mask0_16 [expr int(mask0_8)<<8]
set mask3 [expr int(255*((-y+1.0)/2.0))]
set mask3_8 [expr int(mask3)<<8]
set mask3_16 [expr int(mask3_8)<<8]
fTU and fTV are values for texture coordinates.
set fTU [$app getFTU]
set fTV [$app getFTV]
to update video buffer, we lock the vbPlane with four vertices. Then we get buffer header in the vertice, so we can update each components position, diffuse, and tex1 points by vertice. The [$vertice next 1] move $vertice to next position. Finally the unlock is required to release the lock of the video buffer.
$vbPlane lock 0 [expr 4*[$vertice size]]
$vertice init [$vbPlane getBuffer]
$vertice set position [list $x $y $z]
$vertice set diffuse [expr {0xff0000ff
| int(mask0_8) | int(mask0_16)}]
$vertice set tex1 [list 0.0 0.0]
$vertice next 1
$vertice set tex1 [list 0.0 $fTV]
$vertice next 1
$vertice set tex1 [list $fTU 0.0]
$vertice next 1
$vertice set position [list [expr -x] \
[expr -y] [expr -z]]
$vertice set diffuse [expr {0xff0000ff
| int(mask3_8) | int(mask3_16)}]
$vertice set tex1 [list $fTU $fTV]
$vbPlane unlock
}
In the Render method, the app object clears screen and begins a scene. Then the plane declaration is set and the video texture is set to 0 stream. The next step is to set texture stage states to deal with texture, diffuse and texture. [$vbPlane stream ...] and [$app drawPrimitive ...] render the plane. Finally showHelp may allow you to show the help text and [$app endScene] close the scene.
method Render {} {
$app clear {target zbuffer} 0x00000000 1.0 0
$app beginScene
# set video texture
$declPlane set
$app setVideoTexture 0
# texture stage state
$app setTextureStageState 0 alphaop modulate
$app setTextureStageState 0 alphaarg1 texture
$app setTextureStageState 0 alphaarg2 diffuse
$app setTextureStageState 0 colorarg1 texture
# draw plane
$vbPlane stream 0 0 [$vertice size]
$app drawPrimitive trianglestrip 0 2
# help
showHelp
$app endScene
}
InitDeviceObjects initlizes text object and creates graph. [$app dialog file filetype folder] launches a file open dialog with filetype and initial folder. You can select a video file from it. The 0xACDCACDC is the user-id of the graph. Graph is the set of media connection and controls.
method InitDeviceObjects {} {
$ft init
# create graph by open a video file
set graph [$app graph \\
[$app dialog file "*.*" D:\\media\\video] \\
0xACDCACDC]
create decl
set vDecl [list \\
[list 0 0 FLOAT3 DEFAULT POSITION 0] \\
[list 0 12 D3DCOLOR DEFAULT COLOR 0] \\
[list 0 16 FLOAT2 DEFAULT TEXCOORD 0]]
set declPlane [$app decl $vDecl]
set top-left, bottom-left, top-right, and bottom-right positions.
float3 pos(0) [list -1.0 1.0 0.0]
float3 pos(1) [list -1.0 -1.0 0.0]
float3 pos(2) [list 1.0 1.0 0.0]
float3 pos(3) [list 1.0 -1.0 0.0]
set color component
set color(0) 0xffffffff
set color(1) 0xff0000ff
set color(2) 0xffffffff
set color(3) 0xff0000ff
set low-lef, bottom-left, low-right, and bottom-right texture coordiantes.
float2 tex(0) [list 0.0 0.0]
float2 tex(1) [list 0.0 1.0]
float2 tex(2) [list 1.0 0.0]
float2 tex(3) [list 1.0 1.0]
create vertex buffer with four vertices and set its initial values.
set vbPlane [$app vbuffer 4 \\
[$vertice size] writeonly 0 managed]
# init vertex buffer
$vbPlane lock
$vertice init [$vbPlane getBuffer]
for {set i 0} {i < 4} {incr i} {
$vertice set position $pos($i)
$vertice set diffuse $color($i)
$vertice set tex1 $tex($i)
$vertice next 1
}
$vbPlane unlock
get init time
set mtime [$app getTickCount]
}
RestoreDeviceObjects initial the render state and sampler state.
method RestoreDeviceObjects {} {
$ft restore
set render state
$app setRenderState cullmode none
$app setRenderState lighting false
$app setRenderState alphablendenable true
$app setRenderState srcblend srcalpha
$app setRenderState destblend invsrcalpha
$app setRenderState alphatestenable true
$app setRenderState alpharef 0x10
$app setRenderState alphafunc greater
set sampler state
$app setSamplerState 0 addressu clamp
$app setSamplerState 0 addressv clamp
$app setSamplerState 0 magfilter linear
$app setSamplerState 0 minfilter linear
$app setSamplerState 0 mipfilter linear
set world transform
MatrixIdentity $mat(World)
$app setTransform world $mat(World)
set view transform
float3 vEyePt [list 1.0 1.0 -3.0]
float3 vLookatPt [list 0.0 0.0 0.0]
float3 vUpVec [list 0.0 1.0 0.0]
MatrixLookAtLH $mat(View) $vEyePt $vLookatPt $vUpVec
$app setTransform view $mat(View)
set projector transform
float fAspectRatio [expr {double([$app getBackBufferWidth])
/ double([$app getBackBufferHeight])}]
MatrixPerspectiveFovLH $mat(Proj) 45.0 $fAspectRatio 1.0 100.0
$app setTransform projection $mat(Proj)
graph start to launch the video or audio.
$graph start
}
InvalidateDeviceObjects invalidate the font and end the graph.
method InvalidateDeviceObjects {} {
$ft invalidate
$graph end
}
DeleteDeviceObjects deletes the text object, and releases plane declaration, vertex buffer, and graph.
method DeleteDeviceObjects {} {
$ft delete
$declPlane release
$vbPlane release
$graph release
}
FinalCleanup releases the text object ft.
method FinalCleanup {} {
$ft release
}
}
main procedure creates a shader object. Then the shader object creates a windows, runs into a message loop, and finally releases all when shader exits the loop.
proc main {} {
set shader [VideoManageShader #auto]
$shader Create
$shader Run
$shader Release
}
main execution
main
exit
|
Copyright ©2004-2015 Neatware. All Rights Reserved.