[Bf-blender-cvs] [eea8941] master: BGE: DeckLink card support for video capture and streaming.

Benoit Bolsee noreply at git.blender.org
Sat Jun 11 22:27:30 CEST 2016


Commit: eea89417f4a3e2221e25c9c3ae6a601a61a05c22
Author: Benoit Bolsee
Date:   Fri Jun 10 10:09:26 2016 +0200
Branches: master
https://developer.blender.org/rBeea89417f4a3e2221e25c9c3ae6a601a61a05c22

BGE: DeckLink card support for video capture and streaming.

   You can capture and stream video in the BGE using the DeckLink video
   cards from Black Magic Design. You need a card and Desktop Video software
   version 10.4 or above to use these features in the BGE.
   Many thanks to Nuno Estanquiero who tested the patch extensively
   on a variety of Decklink products, it wouldn't have been possible without
   his help.
   You can find a brief summary of the decklink features here: https://wiki.blender.org/index.php/Dev:Source/GameEngine/Decklink
   The full API details and samples are in the Python API documentation.

bge.texture.VideoDeckLink(format, capture=0):

   Use this object to capture a video stream. the format argument describes
   the video and pixel formats and the capture argument the card number.
   This object can be used as a source for bge.texture.Texture so that the frame
   is sent to the GPU, or by itself using the new refresh method to get the video
   frame in a buffer.
   The frames are usually not in RGB but in YUV format (8bit or 10bit); they
   require a shader to extract the RGB components in the GPU. Details and sample
   shaders in the documentation.
   3D video capture is supported: the frames are double height with left and right
   eyes in top-bottom order. The 'eye' uniform (see setUniformEyef) can be used to
   sample the 3D frame when the BGE is also in stereo mode. This allows to composite
   a 3D video stream with a 3D scene and render it in stereo.
   In Windows, and if you have a nVidia Quadro GPU, you can benefit of an additional
   performance boost by using 'GPUDirect': a method to send a video frame to the GPU
   without going through the OGL driver. The 'pinned memory' OGL extension is also
   supported (only on high-end AMD GPU) with the same effect.

bge.texture.DeckLink(cardIdx=0, format=""):

   Use this object to send video frame to a DeckLink card. Only the immediate mode
   is supported, the scheduled mode is not implemented.
   This object is similar to bge.texture.Texture: you need to attach a image source
   and call refresh() to compute and send the frame to the card.
   This object is best suited for video keying: a video stream (not captured) flows
   through the card and the frame you send to the card are displayed above it (the
   card does the compositing automatically based on the alpha channel).
   At the time of this commit, 3D video keying is supported in the BGE but not in the
   DeckLink card due to a color space issue.

===================================================================

M	CMakeLists.txt
M	build_files/cmake/macros.cmake
A	doc/python_api/examples/bge.texture.2.py
M	doc/python_api/rst/bge.texture.rst
M	intern/CMakeLists.txt
A	intern/decklink/CMakeLists.txt
A	intern/decklink/DeckLinkAPI.cpp
A	intern/decklink/DeckLinkAPI.h
A	intern/decklink/linux/DeckLinkAPI.h
A	intern/decklink/linux/DeckLinkAPIConfiguration.h
A	intern/decklink/linux/DeckLinkAPIDeckControl.h
A	intern/decklink/linux/DeckLinkAPIDiscovery.h
A	intern/decklink/linux/DeckLinkAPIDispatch.cpp
A	intern/decklink/linux/DeckLinkAPIModes.h
A	intern/decklink/linux/DeckLinkAPITypes.h
A	intern/decklink/linux/DeckLinkAPIVersion.h
A	intern/decklink/linux/LinuxCOM.h
A	intern/decklink/win/DeckLinkAPI_h.h
A	intern/decklink/win/DeckLinkAPI_i.c
A	intern/gpudirect/CMakeLists.txt
A	intern/gpudirect/dvpapi.cpp
A	intern/gpudirect/dvpapi.h
M	source/blenderplayer/CMakeLists.txt
M	source/gameengine/VideoTexture/CMakeLists.txt
M	source/gameengine/VideoTexture/Common.h
A	source/gameengine/VideoTexture/DeckLink.cpp
A	source/gameengine/VideoTexture/DeckLink.h
M	source/gameengine/VideoTexture/Exception.cpp
A	source/gameengine/VideoTexture/VideoDeckLink.cpp
A	source/gameengine/VideoTexture/VideoDeckLink.h
M	source/gameengine/VideoTexture/blendVideoTex.cpp

===================================================================

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 15ee834..e7a0f7a 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -229,6 +229,7 @@ option(WITH_BULLET        "Enable Bullet (Physics Engine)" ON)
 option(WITH_SYSTEM_BULLET "Use the systems bullet library (currently unsupported due to missing features in upstream!)" )
 mark_as_advanced(WITH_SYSTEM_BULLET)
 option(WITH_GAMEENGINE    "Enable Game Engine" ${_init_GAMEENGINE})
+option(WITH_GAMEENGINE_DECKLINK "Support BlackMagicDesign DeckLink cards in the Game Engine" ON)
 option(WITH_PLAYER        "Build Player" OFF)
 option(WITH_OPENCOLORIO   "Enable OpenColorIO color management" ${_init_OPENCOLORIO})
 
diff --git a/build_files/cmake/macros.cmake b/build_files/cmake/macros.cmake
index 3aa938b..d29f086 100644
--- a/build_files/cmake/macros.cmake
+++ b/build_files/cmake/macros.cmake
@@ -685,6 +685,14 @@ function(SETUP_BLENDER_SORTED_LIBS)
 		list_insert_after(BLENDER_SORTED_LIBS "ge_logic_ngnetwork" "extern_bullet")
 	endif()
 
+	if(WITH_GAMEENGINE_DECKLINK)
+		list(APPEND BLENDER_SORTED_LIBS bf_intern_decklink)
+	endif()
+
+	if(WIN32)
+		list(APPEND BLENDER_SORTED_LIBS bf_intern_gpudirect)
+	endif()
+
 	if(WITH_OPENSUBDIV)
 		list(APPEND BLENDER_SORTED_LIBS bf_intern_opensubdiv)
 	endif()
diff --git a/doc/python_api/examples/bge.texture.2.py b/doc/python_api/examples/bge.texture.2.py
new file mode 100644
index 0000000..9661900
--- /dev/null
+++ b/doc/python_api/examples/bge.texture.2.py
@@ -0,0 +1,237 @@
+"""
+Video Capture with DeckLink
++++++++++++++++++++++++++++
+Video frames captured with DeckLink cards have pixel formats that are generally not directly
+usable by OpenGL, they must be processed by a shader. The three shaders presented here should
+cover all common video capture cases.
+
+This file reflects the current video transfer method implemented in the Decklink module:
+whenever possible the video images are transferred as float texture because this is more
+compatible with GPUs. Of course, only the pixel formats that have a correspondant GL format
+can be transferred as float. Look for fg_shaders in this file for an exhaustive list.
+
+Other pixel formats will be transferred as 32 bits integer red-channel texture but this
+won't work with certain GPU (Intel GMA); the corresponding shaders are not shown here.
+However, it should not be necessary to use any of them as the list below covers all practical
+cases of video capture with all types of Decklink product.
+
+In other words, only use one of the pixel format below and you will be fine. Note that depending
+on the video stream, only certain pixel formats will be allowed (others will throw an exception).
+For example, to capture a PAL video stream, you must use one of the YUV formats.
+
+To find which pixel format is suitable for a particular video stream, use the 'Media Express'
+utility that comes with the Decklink software : if you see the video in the 'Log and Capture'
+Window, you have selected the right pixel format and you can use the same in Blender.
+
+Notes: * these shaders only decode the RGB channel and set the alpha channel to a fixed
+value (look for color.a = ). It's up to you to add postprocessing to the color.
+       * these shaders are compatible with 2D and 3D video stream
+"""
+import bge
+from bge import logic
+from bge import texture as vt
+
+# The default vertex shader, because we need one
+#
+VertexShader = """
+#version 130
+   void main()
+   {
+      gl_Position = gl_ModelViewProjectionMatrix * gl_Vertex;
+      gl_TexCoord[0] = gl_MultiTexCoord0;
+   }
+
+"""
+
+# For use with RGB video stream: the pixel is directly usable
+#
+FragmentShader_R10l = """
+    #version 130
+    uniform sampler2D tex;
+    // stereo = 1.0 if 2D image, =0.5 if 3D (left eye below, right eye above)
+    uniform float stereo;
+    // eye = 0.0 for the left eye, 0.5 for the right eye
+    uniform float eye;
+
+    void main(void)
+    {
+        vec4 color;
+        float tx, ty;
+        tx = gl_TexCoord[0].x;
+        ty = eye+gl_TexCoord[0].y*stereo;
+        color = texture(tex, vec2(tx,ty));
+        color.a = 0.7;
+        gl_FragColor = color;
+    }
+"""
+
+# For use with YUV video stream
+#
+FragmentShader_2vuy = """
+    #version 130
+    uniform sampler2D tex;
+    // stereo = 1.0 if 2D image, =0.5 if 3D (left eye below, right eye above)
+    uniform float stereo;
+    // eye = 0.0 for the left eye, 0.5 for the right eye
+    uniform float eye;
+
+    void main(void)
+    {
+        vec4 color;
+        float tx, ty, width, Y, Cb, Cr;
+        int px;
+        tx = gl_TexCoord[0].x;
+        ty = eye+gl_TexCoord[0].y*stereo;
+        width = float(textureSize(tex, 0).x);
+        color = texture(tex, vec2(tx, ty));
+        px = int(floor(fract(tx*width)*2.0));
+        switch (px) {
+        case 0:
+            Y = color.g;
+            break;
+        case 1:
+            Y = color.a;
+            break;
+        }
+        Y = (Y - 0.0625) * 1.168949772;
+        Cb = (color.b - 0.0625) * 1.142857143 - 0.5;
+        Cr = (color.r - 0.0625) * 1.142857143 - 0.5;
+        color.r = Y + 1.5748 * Cr;
+        color.g = Y - 0.1873 * Cb - 0.4681 * Cr;
+        color.b = Y + 1.8556 * Cb;
+        color.a = 0.7;
+        gl_FragColor = color;
+    }
+"""
+
+# For use with high resolution YUV
+#
+FragmentShader_v210 = """
+    #version 130
+    uniform sampler2D tex;
+    // stereo = 1.0 if 2D image, =0.5 if 3D (left eye below, right eye above)
+    uniform float stereo;
+    // eye = 0.0 for the left eye, 0.5 for the right eye
+    uniform float eye;
+
+    void main(void)
+    {
+        vec4 color, color1, color2, color3;
+        int px;
+        float tx, ty, width, sx, dx, bx, Y, Cb, Cr;
+        tx = gl_TexCoord[0].x;
+        ty = eye+gl_TexCoord[0].y*stereo;
+        width = float(textureSize(tex, 0).x);
+        // to sample macro pixels (6 pixels in 4 words)
+        sx = tx*width*0.25+0.01;
+        // index of display pixel in the macro pixel 0..5
+        px = int(floor(fract(sx)*6.0));
+        // increment as we sample the macro pixel
+        dx = 1.0/width;
+        // base x coord of macro pixel
+        bx = (floor(sx)+0.01)*dx*4.0;
+        color = texture(tex, vec2(bx, ty));
+        color1 = texture(tex, vec2(bx+dx, ty));
+        color2 = texture(tex, vec2(bx+dx*2.0, ty));
+        color3 = texture(tex, vec2(bx+dx*3.0, ty));
+        switch (px) {
+        case 0:
+        case 1:
+            Cb = color.b;
+            Cr = color.r;
+            break;
+        case 2:
+        case 3:
+            Cb = color1.g;
+            Cr = color2.b;
+            break;
+        default:
+            Cb = color2.r;
+            Cr = color3.g;
+            break;
+        }
+        switch (px) {
+        case 0:
+            Y = color.g;
+            break;
+        case 1:
+            Y = color1.b;
+            break;
+        case 2:
+            Y = color1.r;
+            break;
+        case 3:
+            Y = color2.g;
+            break;
+        case 4:
+            Y = color3.b;
+            break;
+        default:
+            Y = color3.r;
+            break;
+        }
+        Y = (Y - 0.0625) * 1.168949772;
+        Cb = (Cb - 0.0625) * 1.142857143 - 0.5;
+        Cr = (Cr - 0.0625) * 1.142857143 - 0.5;
+        color.r = Y + 1.5748 * Cr;
+        color.g = Y - 0.1873 * Cb - 0.4681 * Cr;
+        color.b = Y + 1.8556 * Cb;
+        color.a = 0.7;
+        gl_FragColor = color;
+    }
+"""
+
+# The exhausitve list of pixel formats that are transferred as float texture
+# Only use those for greater efficiency and compatiblity.
+#
+fg_shaders = {
+    '2vuy'       :FragmentShader_2vuy,
+    '8BitYUV'    :FragmentShader_2vuy,
+    'v210'       :FragmentShader_v210,
+    '10BitYUV'   :FragmentShader_v210,
+    '8BitBGRA'   :FragmentShader_R10l,
+    'BGRA'       :FragmentShader_R10l,
+    '8BitARGB'   :FragmentShader_R10l,
+    '10BitRGBXLE':FragmentShader_R10l,
+    'R10l'       :FragmentShader_R10l
+    }
+
+
+#
+# Helper function to attach a pixel shader to the material that receives the video frame.
+#
+
+def config_video(obj, format, pixel, is3D=False, mat=0, card=0):
+    if pixel not in fg_shaders:
+        raise('Unsuported shader')
+    shader = obj.meshes[0].materials[mat].getShader()
+    if shader is not None and not shader.isValid():
+        shader.setSource(VertexShader, fg_shaders[pixel], True)
+        shader.setSampler('tex', 0)
+        shader.setUniformEyef("eye")
+        shader.setUniform1f("stereo", 0.5 if is3D else 1.0)
+    tex = vt.Texture(obj, mat)
+    tex.source = vt.VideoDeckLink(format + "/" + pixel + ("/3D" if is3D else ""), card)
+    print("frame rate: ", tex.source.framerate)
+    tex.source.play()
+    obj["video"] = tex
+
+#
+# Attach this function to an object that has a material with texture
+# and call it once to initialize the object
+#
+def init(cont):
+    # config_video(cont.owner, 'HD720p5994', '8BitBGRA')
+    # config_video(cont.owner, 'HD720p5994', '8BitYUV')
+    # config_video(cont.owner, 'pal ', '10BitYUV')
+    config_video(cont.owner, 'pal ', '8BitYUV')
+
+
+#
+# To be called on every frame
+#
+def play(cont):
+    obj = cont.owner
+    video = obj.get("video")
+    if video is not None:
+        video.refresh(True)
diff --git a/doc/python_api/rst/bge.texture.rst b/doc/python_api/rst/bge.texture.rst
index e226d2f..9d6166b 100644
--- a/doc/python_api/rst/bge.texture.rst
+++ b/doc/python_api/rst/bge.texture.rst
@@ -49,8 +49,15 @@ When the texture object is deleted, the new texture is deleted and the old textu
    
 .. literalinclude:: ../examples/bge.texture.1.py
    :lines: 8-
-   
-   
+
+
+.. include:: ../examples/bge.texture.2.py
+   :start-line: 1
+   :end-line: 6
+
+.. literalinclude:: ../examples/bge.texture.2.py
+   :lines: 8-
+
 *************
 Video classes
 *************
@@ -189,7 +196,6 @@ Video classes
       :return: see `FFmpeg Video and Image Status`_.
       :rtype: int
 
-
 *************
 Image classes
 *************
@@ -256,7 +262,7 @@ Image classes
    .. method:: refresh(buffer=None, format="RGBA")
 
       Refresh image, get its status and optionally copy the frame to an external buffer.
-      
+
       :arg buffer: An option

@@ Diff output truncated at 10240 characters. @@




More information about the Bf-blender-cvs mailing list