DirectShow transform

[jj2007]

Anybody around for whom the text below is not Chinese?

What I really need is a pointer to the bitmap before the frame is rendered to the DC. Ideally I'd like to use Gdi+ to rotate the image. My code runs fine with most video formats, the only problem is that some need to be rotated...

How to add a filter in a graph
Actually you have several options. Yes you anyway need to add your filter to the graph manually as suggested by Michel.

Then you can:

    Instead of IGraphBuilder:: RenderFile call IGraphBuilder:: AddSourceFilter to only auto-add first filter for the sourec file. After that use IGraphBuilder::Connect (or ICaptureGraphBuilder:: RenderStream) where you can provide your filter (filter's pin) interface to engage it into rendering explicitly
    You can still IGraphBuilder:: RenderFile but before Run'ning break the connection using IFilterGraph:: Disconnect and re-connect pins through your filter at the point of interest

Before running the graph it is available to you for any reconstruction. Once you are finished with inserting your filter, Run the graph.

[HSE]

It's not Chinese JJ!!!

(Perhaps some extraterrestrial translation of chinese)

[qWord]

Anybody around for whom the text below is not Chinese?

Translation: ,,If you already have a filter that transform the frames, it must be inserted before the Video Renderer Filter in the graph of filters."

You might take a look at the tool "GraphEdit" that comes with the Windows SDKs.

[jj2007]

Translation: ,,If you already have a filter that transform the frames, it must be inserted before the Video Renderer Filter in the graph of filters."

Good, now I have it also in Japanese. Any volunteers for English?

What I really need is a pointer to the bitmap before the frame is rendered to the DC. Ideally I'd like to use Gdi+ to rotate the image.

[Siekmanski]

Did you try "GetCurrentBuffer" from the ISampleGrabber interface? Then you can get a pointer to the pixel data.
No promises I will succeed, but I'll give it a try this week and maybe come up with something in English

[jj2007]

Thanks a lot, Marinus. I had stumbled over the word ISampleGrabber in one of the many posts I read, but I am still far from understanding the logic. So with ISampleGrabber::GetCurrentBuffer, I hope there is no need to register a DLL with GUID and all that crap....? Looking forward to your example. Take it easy, the next days I am busy anyway with family ;-)

P.S.: Pixel data sounds good in principle. What I wonder, though, if the graphics card can do the rotation faster, e.g. with Gdi+ Image.RotateFlip; that's why I mentioned the DC.

[Siekmanski]

No dll's needed, we can get direct access via COM interfaces. It was a pain in the ass to find and collect them all.
I found the interfaces in these C++ include files: strmif.h control.h and uuids.h
Don't know how the DirectShow filters work internally, but if we can get the pointer to the pixel buffer we can handle the rotations and flipping ourselves, by copying the pixels with SSE AVX to a DirectX texture or to a GDIplus bitmap buffer.
I managed to get the pixel buffer pointer from capture devices, so I think and hope it can be done. we will see....

It will be a nice routine for my multimedia library.

[Siekmanski]

After reading hours on MSDN to get a grasp how things work under the hood of DirectShow filters and pin connections etc.
I put something together that works in a very direct way to get the encoded raw bitmap data from video movies.
Made an example using Direct3D9 in 2D mode which is IMHO by far the best and fastest way of manipulating bitmaps.
It offers much more possibilities than GDI or GDI+ and is mega fast in rendering, alpha blending etc.

I'm not very well known with the DirectShow api so, if any of you guys have better ideas let us know.

But of course you can use this routine in any api you like.

Example how to use it without Direct3D9,

Code Select Expand

.const

VIDEO_INFO struct
    AvgTimePerFrame     dq ?
    biBitCount          dd ?
    biWidth             dd ?
    biHeight            dd ?
    pBitMapBuffer       dd ?
    BitMapBufferSize    dd ?
VIDEO_INFO ends

include dx9macros.inc ; here are the macros used in the "sampleGrabber" code, look in the Includes folder.

.data?

VideoInfo_1         VIDEO_INFO <?>

.data
Video_file db "Test.avi"

    include DSVideoGrabber.asm

.code
    invoke  CoInitialize, NULL ; put this line somewhere at the start of your code.
                               ; this is needed for the COM interfaces to work correctly.


    invoke  RtlZeroMemory,offset VideoInfo_1,sizeof VideoInfo_1
    invoke  VideoGrabber,offset Video_file,offset VideoInfo_1 ; start the VideoGrabber.
    cmp     eax,S_OK
    jne     no_video_found

    coinvoke g_pMediaControl,IMediaControl,Run ; start the movie.

; Now poll +/- 60 times a second the VideoGrabber buffer.

PollVideoData proc
    mov     ecx,pVideoInfo
    coinvoke g_pSampleGrabber,ISampleGrabber,GetCurrentBuffer,addr VideoInfo_1.BitMapBufferSize,VideoInfo_1.pBitMapBuffer
    cmp     eax,S_OK    ; is there a new VideoFrame available?
    je      GetVideoFrame
    ret
GetVideoFrame:
; copy the data to your own destination buffer
    mov     esi,VideoInfo_1.pBitMapBuffer   ; pointer to the raw BitMapBuffer from the SampleGrabber
    mov     edi,MyBitMapBuffer   ; pointer to your destination BitMapBuffer
    mov     ecx,VideoInfo_1.BitMapBufferSize

; place your copy routine here.
    ret
PollVideoData endp

    invoke  ReleaseVideoObjects,offset VideoInfo_1 ; when done...

    invoke  CoUninitialize         



The main routine,

Code Select Expand

align 4
VideoGrabber proc uses esi edi pVideoName:DWORD,pVideoInfo:DWORD

    invoke  ReleaseVideoObjects,pVideoInfo
    call    ReleaseVideoImageObjects        ; remove this line if you're not using D3D9

    ; create the Filter Graph Manager   
    invoke  CoCreateInstance,GUID_ADDR(CLSID_FilterGraph),NULL,CLSCTX_INPROC_SERVER,GUID_ADDR(IID_IGraphBuilder),addr g_pGraphBuilder
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; create the IMediaControl interface
    coinvoke g_pGraphBuilder,IGraphBuilder,QueryInterface,GUID_ADDR(IID_IMediaControl),addr g_pMediaControl
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Create an instance of the Sample Grabber filter
    invoke  CoCreateInstance,GUID_ADDR(CLSID_SampleGrabber),NULL,CLSCTX_INPROC_SERVER,GUID_ADDR(IID_IBaseFilter),addr g_pGrabberFilter
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Add it to the filter graph
    coinvoke g_pGraphBuilder,IGraphBuilder,AddFilter,g_pGrabberFilter,NULL
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Query the Sample Grabber filter for the ISampleGrabber interface
    coinvoke g_pGrabberFilter,ISampleGrabber,QueryInterface,GUID_ADDR(IID_ISampleGrabber),addr g_pSampleGrabber
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; When you first create the Sample Grabber, it has no preferred media type.
    ; This means you can connect to almost any filter in the graph, but you would have no control over the type of data that it received.
    ; Before building the rest of the graph, therefore, you must set a media type for the Sample Grabber, by calling the ISampleGrabber.SetMediaType method.

    ; When the Sample Grabber connects, it will compare this media type against the media type offered by the other filter.
    ; The only fields that it checks are the major type, subtype, and format type. For any of these, the value GUID_NULL means "accept any value."
    ; Most of the time, you will want to set the major type and subtype. For example, the following code specifies uncompressed 32-bit RGB video:

    invoke  CoTaskMemAlloc,sizeof AM_MEDIA_TYPE ; allocate memory for the AM_MEDIA_TYPE structure
    mov     pVideoMediaType,eax
   
    invoke  RtlZeroMemory,pVideoMediaType,sizeof AM_MEDIA_TYPE
    mov     edi,pVideoMediaType
    invoke  RtlMoveMemory,addr [edi].AM_MEDIA_TYPE.majortype,GUID_ADDR(MEDIATYPE_Video),16
    invoke  RtlMoveMemory,addr [edi].AM_MEDIA_TYPE.subtype,GUID_ADDR(MEDIASUBTYPE_RGB32),16
    invoke  RtlMoveMemory,addr [edi].AM_MEDIA_TYPE.formattype,GUID_ADDR(FORMAT_VideoInfo),16
    ;  Regardless of what type you set, the Sample Grabber Filter rejects any video types with top-down orientation (negative biHeight).
    coinvoke g_pSampleGrabber,ISampleGrabber,SetMediaType,pVideoMediaType
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Now you can build the rest of the filter graph. Because the Sample Grabber will only connect using the media type you have specified,
    ; this lets you take advantage of the Filter Graph Manager's Intelligent Connect mechanisms when you build the graph.

    ; For example, if you specified uncompressed video, you can connect a source filter to the Sample Grabber,
    ; and the Filter Graph Manager will automatically add the file parser and the decoder.

    invoke  MultiByteToWideChar,CP_ACP,0,pVideoName,-1,offset szLstring,MAX_PATH
    coinvoke g_pGraphBuilder,IGraphBuilder,AddSourceFilter,offset szLstring,offset szLSource,addr g_pSourceFilter
    cmp     eax,S_OK
    je      asf_ok
    cmp     eax,VFW_E_CANNOT_LOAD_SOURCE_FILTER
    jne     enf
    invoke  MessageBox,0,TEXT_("The source filter for this file could not be loaded."),TEXT_("AddSourceFilter Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
enf:
    cmp     eax,VFW_E_NOT_FOUND
    jne     euft
    invoke  MessageBox,0,TEXT_("File or object not found."),TEXT_("AddSourceFilter Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
euft:
    cmp     eax,VFW_E_UNKNOWN_FILE_TYPE
    jne     asfe
    invoke  MessageBox,0,TEXT_("The media type of this file was not recognized."),TEXT_("AddSourceFilter Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
asfe:
    invoke  wsprintf,addr szString_buffer,TEXT_("AddSourceFilter Error: %0x"),eax
    invoke  MessageBox,0,addr szString_buffer,TEXT_("AddSourceFilter Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
asf_ok:

    ; The following example uses a ConnectFilters helper function.
    ; It tries to find an output pin of the SourceFilter and an input pin of the GrabberFilter, if succeeded the 2 filters are connected.
    ; It can happen that there are more streams in a filter such as the audio stream. In this example only the raw Video is encoded.
    ; IGraphBuilder.Connect will return VFW_S_PARTIAL_RENDER (040242h), which is not an error because there is no audio filter added.

    invoke  ConnectTwoFilters,g_pSourceFilter,g_pGrabberFilter
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; The Sample Grabber is a transform filter, so the output pin must be connected to another filter.
    ; Often, you may simply want to discard the samples after you are done with them.
    ; In that case, connect the Sample Grabber to the Null Renderer Filter, which discards the data that it receives.

    ; This is exactly what jj2007 and I (Siekmanski) want, direct access to the raw encoded bitmap data and nothing more.
   
    ; Query the NullRendererFilter
    invoke  CoCreateInstance,GUID_ADDR(CLSID_NullRenderer),NULL,CLSCTX_INPROC_SERVER,GUID_ADDR(IID_IBaseFilter),addr g_pNullRendererFilter
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Add it to the filter graph
    coinvoke g_pGraphBuilder,IGraphBuilder,AddFilter,g_pNullRendererFilter,NULL
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    ; Connect the SampleGrabber to the NullRendererFilter
    invoke  ConnectTwoFilters,g_pGrabberFilter,g_pNullRendererFilter
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    coinvoke g_pSampleGrabber,ISampleGrabber,SetOneShot,FALSE       ; After the first sample, the Sample Grabber continues to process samples.
    cmp     eax,S_OK
    jne     VideoGrabber_Error

    coinvoke g_pSampleGrabber,ISampleGrabber,SetBufferSamples,TRUE  ; This method does not return a value.

    invoke  CoTaskMemAlloc,sizeof VIDEOINFOHEADER
    mov     esi,pVideoMediaType
    mov     dword ptr [esi].AM_MEDIA_TYPE.pbFormat,eax              ; insert allocated VIDEOINFOHEADER structure memory, to receive the BITMAPINFOHEADER info

    coinvoke g_pSampleGrabber,ISampleGrabber,GetConnectedMediaType,pVideoMediaType ; get the bitmap info from the SampleGrabber
    cmp     eax,S_OK
    je      gcmt_ok
    cmp     eax,VFW_E_NOT_CONNECTED
    je      not_connected
    invoke  wsprintf,addr szString_buffer,TEXT_("GetConnectedMediaType Error: %0x"),eax
    invoke  MessageBox,0,addr szString_buffer,TEXT_("GetConnectedMediaType Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
not_connected:
    invoke  MessageBox,0,TEXT_("The filter is not connected."),TEXT_("GetConnectedMediaType Error."),MB_ICONERROR
    jmp     VideoGrabber_Error
gcmt_ok:

    mov     edi,pVideoInfo                                          ; copy the bitmap info to the VIDEO_INFO structure
    mov     esi,pVideoMediaType
    movsd   xmm0,qword ptr [esi].VIDEOINFOHEADER.AvgTimePerFrame
    movsd   qword ptr [edi].VIDEO_INFO.AvgTimePerFrame,xmm0
    mov     esi,dword ptr [esi].AM_MEDIA_TYPE.pbFormat
    lea     esi,[esi].VIDEOINFOHEADER.bmiHeader
    movzx   eax,word ptr [esi].BITMAPINFOHEADER.biBitCount
    mov     dword ptr [edi].VIDEO_INFO.biBitCount,eax
    mov     eax,dword ptr [esi].BITMAPINFOHEADER.biWidth
    mov     dword ptr [edi].VIDEO_INFO.biWidth,eax
    mov     eax,dword ptr [esi].BITMAPINFOHEADER.biHeight
    mov     dword ptr [edi].VIDEO_INFO.biHeight,eax

    invoke  CoTaskMemFree,dword ptr [esi].AM_MEDIA_TYPE.pbFormat    ; free the VIDEOINFOHEADER memory from the AM_MEDIA_TYPE structure memory
    mov     dword ptr [esi].AM_MEDIA_TYPE.pbFormat,NULL
    invoke  CoTaskMemFree,pVideoMediaType                           ; free the AM_MEDIA_TYPE structure memory
    mov     pVideoMediaType,NULL
       
    ; Pausing the filter graph cues the graph for immediate rendering when the graph is next run.
    ; While the graph is paused, filters process data but do not render it.
    ; Data is pushed through the graph and processed by transform filters as far as buffering permits, but renderer filters do not render the data.
    ; However, video renderers display a static poster frame of the first sample.

    coinvoke g_pMediaControl,IMediaControl,Pause

    ; "IMediaControl.Pause" returns:
    ; S_FALSE if the graph paused successfully, but some filters have not completed the state transition.
    ; S_OK if all filters in the graph completed the transition to a paused state.

    ; So, we have to wait for the filters have completed the state transition.
    ; Because "IMediaEventEx.WaitForCompletion" can stall forever, we wait for "ISampleGrabber.GetCurrentBuffer" is ready, and gives back the BitMapBufferSize.

    mov     edi,pVideoInfo
WaitForCompletion: 
    invoke Sleep,10
    coinvoke g_pSampleGrabber,ISampleGrabber,GetCurrentBuffer,addr [edi].VIDEO_INFO.BitMapBufferSize,NULL
    cmp     dword ptr [edi].VIDEO_INFO.BitMapBufferSize,0
    jz      WaitForCompletion
    cmp     eax,S_OK
    jne     WaitForCompletion

    invoke  CoTaskMemAlloc,dword ptr [edi].VIDEO_INFO.BitMapBufferSize ; Now we know the size of the BitMapBufferSize, allocate the memory for the SampleGrabber.
    mov     dword ptr [edi].VIDEO_INFO.pBitMapBuffer,eax

    ; coinvoke g_pMediaControl,IMediaControl,Run
    ; "IMediaControl.Run" returns:
    ; S_FALSE if the graph is preparing to run, but some filters have not completed the transition to a running state.
    ; S_OK if all filters in the graph completed the transition to a running state.

    ; So, we don't check the return message, it will run when the graph completed the transition to a running state.
    mov     eax,S_OK
    ret

VideoGrabber_Error:
    call    ReleaseVideoObjects
    mov     eax,S_FALSE
    ret
VideoGrabber endp


EDIT: new source code, (see Reply #24)

Placed the AM_MEDIA_TYPE structure and the VIDEOINFOHEADER structure in memory allocated by CoTaskMemAlloc as suggested by Microsoft.
Replaced the sse2 SIMD vertex fill routine by a simpler routine. ( my wife's laptop choked in the previous one )
Removed the SetSamplerState,0,D3DRS_DITHERENABLE,TRUE. ( Not all video-cards have it implemented and black out the texture )
Added some error handling.

You have to install additional DirectShow Video Codecs if certain video formats refuse to work on your PC.

http://www.codecguide.com/download_kl.htm ( See jj2007's Reply #25 for handling this. )

Marinus

[jj2007]

Great work, Marinus :t

The exe works fine, it opens with a delay of about 4 seconds.
Building succeeds if I use crt_sprintf, but the new exe choked once shortly before CoTaskMemFree. Since then, it builds fine and behaves well ::)

As soon as I have some spare time, I will have a closer look. Thanks a lot for investigating this :icon14:

[Siekmanski]

Thanks, the delay is because it streams a movie from the internet. 
If you load it as a file it runs immediately.
The sprintf behaviour, I may be using an older msvcrt.lib ?

It was fun and time spent well.  :t

but the new exe choked once shortly before CoTaskMemFree

VIDEO_INFO.pBitMapBuffer was uninitialized.

Zeroing the VIDEO_INFO structure was needed before calling VideoGrabber.

    invoke  RtlZeroMemory,offset VideoInfo_1,sizeof VideoInfo_1
    invoke  VideoGrabber,TEXT_("https://archive.org/download/Apollo_9__Three_To_Make_Ready/Apollo_9_Three_To_Make_Ready_512kb.mp4"),offset VideoInfo_1

Uploaded the new sources (Reply #7), also includes the older msvcrt.lib I used.

[jj2007]

Thanks, the delay is because it streams a movie from the internet. 

I've downloaded the movie (30 seconds - Internet is slow here), it plays fine with my player, but DShowGrabber.exe stops at the image below :(

[Siekmanski]

 :(
I've also downloaded the movie and loaded it as a file, it runs immediately.
I don't have a clue, it runs fine here.

Try changing the window dimensions in DShowGrabber.Asm to:

Screen_Width        dd  320
Screen_Height       dd  240

[jj2007]

My best guess so far is that the download is too slow, and that DShow cancels at a certain point.
When modifying the path, the exe starts immediately but no video; changing w+h has no effect except that the window is smaller.

[Siekmanski]

Does this one work?
It loads a small avi from file.

[aw27]

Thanks, the delay is because it streams a movie from the internet. 
If you load it as a file it runs immediately.
The sprintf behaviour, I may be using an older msvcrt.lib ?

It was fun and time spent well.  :t

but the new exe choked once shortly before CoTaskMemFree

VIDEO_INFO.pBitMapBuffer was uninitialized.

Zeroing the VIDEO_INFO structure was needed before calling VideoGrabber.

    invoke  RtlZeroMemory,offset VideoInfo_1,sizeof VideoInfo_1
    invoke  VideoGrabber,TEXT_("https://archive.org/download/Apollo_9__Three_To_Make_Ready/Apollo_9_Three_To_Make_Ready_512kb.mp4"),offset VideoInfo_1

Uploaded the new sources (Reply #7), also includes the older msvcrt.lib I used.

I am having a problem also (trying your DShowGrabber2.zip).
I changed the mp4 to a smaller one downloaded from my site but did not help.

I could track the error to here:

Code Select Expand


coinvoke g_pSampleGrabber,ISampleGrabber,GetConnectedMediaType,offset VideoMediaType ; get the bitmap info from the SampleGrabber
    cmp     eax,S_OK
    jne     VideoGrabber_Error

where eax is 80040209