Video Processing Procedures¶

The following pseudo code shows the video processing procedure:

MFXVideoVPP_QueryIOSurf(session, &init_param, response);
allocate_pool_of_surfaces(in_pool, response[0].NumFrameSuggested);
allocate_pool_of_surfaces(out_pool, response[1].NumFrameSuggested);
MFXVideoVPP_Init(session, &init_param);
mfxFrameSurface1 *in=find_unlocked_surface_and_fill_content(in_pool);
mfxFrameSurface1 *out=find_unlocked_surface_from_the_pool(out_pool);
for (;;) {
   sts=MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
   if (sts==MFX_ERR_MORE_SURFACE || sts==MFX_ERR_NONE) {
      MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
      process_output_frame(out);
      out=find_unlocked_surface_from_the_pool(out_pool);
   }
   if (sts==MFX_ERR_MORE_DATA && in==NULL)
      break;
   if (sts==MFX_ERR_NONE || sts==MFX_ERR_MORE_DATA) {
      in=find_unlocked_surface_from_the_pool(in_pool);
      fill_content_for_video_processing(in);
      if (end_of_stream())
         in=NULL;
   }
}
MFXVideoVPP_Close(session);
free_pool_of_surfaces(in_pool);
free_pool_of_surfaces(out_pool);

Note the following key points about the example:

The application uses the MFXVideoVPP_QueryIOSurf() function to obtain the number of frame surfaces needed for input and output. The application must allocate two frame surface pools: one for the input and one for the output.
The video processing function MFXVideoVPP_RunFrameVPPAsync() is asynchronous. The application must use the MFXVideoCORE_SyncOperation() function to synchronize in order to make the output result ready.
The body of the video processing procedure covers the following three scenarios:
- If the number of frames consumed at input is equal to the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_NONE when an output is ready. The application must process the output frame after synchronization, as the MFXVideoVPP_RunFrameVPPAsync() function is asynchronous. The application must provide a NULL input at the end of the sequence to drain any remaining frames.
- If the number of frames consumed at input is more than the number of frames generated at output, VPP returns mfxStatus::MFX_ERR_MORE_DATA for additional input until an output is ready. When the output is ready, VPP returns mfxStatus::MFX_ERR_NONE. The application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.
- If the number of frames consumed at input is less than the number of frames generated at output, VPP returns either mfxStatus::MFX_ERR_MORE_SURFACE (when more than one output is ready), or mfxStatus::MFX_ERR_NONE (when one output is ready and VPP expects new input). In both cases, the application must process the output frame after synchronization and provide a NULL input at the end of the sequence to drain any remaining frames.

Configuration¶

oneVPL configures the video processing pipeline operation based on the difference between the input and output formats, specified in the mfxVideoParam structure. The following list shows several examples:

When the input color format is YUY2 and the output color format is NV12, oneVPL enables color conversion from YUY2 to NV12.
When the input is interleaved and the output is progressive, oneVPL enables deinterlacing.
When the input is single field and the output is interlaced or progressive, oneVPL enables field weaving, optionally with deinterlacing.
When the input is interlaced and the output is single field, oneVPL enables field splitting.

In addition to specifying the input and output formats, the application can provide hints to fine-tune the video processing pipeline operation. The application can disable filters in the pipeline by using the mfxExtVPPDoNotUse structure, enable filters by using the mfxExtVPPDoUse structure, and configure filters by using dedicated configuration structures. See the Configurable VPP Filters table for a complete list of configurable video processing filters, their IDs, and configuration structures. See the ExtendedBufferID enumerator for more details.

oneVPL ensures that all filters necessary to convert the input format to the output format are included in the pipeline. oneVPL may skip some optional filters even if they are explicitly requested by the application, for example due to limitations of the underlying hardware. To notify the application about skipped optional filters, oneVPL returns the mfxStatus::MFX_WRN_FILTER_SKIPPED warning. The application can retrieve the list of active filters by attaching the mfxExtVPPDoUse structure to the mfxVideoParam structure and calling the MFXVideoVPP_GetVideoParam() function. The application must allocate enough memory for the filter list.

See the Configurable VPP Filters table for a full list of configurable filters.

Configurable VPP Filters¶
Filter ID	Configuration Structure
`MFX_EXTBUFF_VPP_DENOISE`	`mfxExtVPPDenoise`
`MFX_EXTBUFF_VPP_DENOISE2`	`mfxExtVPPDenoise2`
`MFX_EXTBUFF_VPP_MCTF`	`mfxExtVppMctf`
`MFX_EXTBUFF_VPP_DETAIL`	`mfxExtVPPDetail`
`MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION`	`mfxExtVPPFrameRateConversion`
`MFX_EXTBUFF_VPP_IMAGE_STABILIZATION`	`mfxExtVPPImageStab`
`MFX_EXTBUFF_VPP_PROCAMP`	`mfxExtVPPProcAmp`
`MFX_EXTBUFF_VPP_FIELD_PROCESSING`	`mfxExtVPPFieldProcessing`

The following example shows video processing configuration:

/* enable image stabilization filter with default settings */
mfxExtVPPDoUse du;
mfxU32 al=MFX_EXTBUFF_VPP_IMAGE_STABILIZATION;

du.Header.BufferId=MFX_EXTBUFF_VPP_DOUSE;
du.Header.BufferSz=sizeof(mfxExtVPPDoUse);
du.NumAlg=1;
du.AlgList=&al;

/* configure the mfxVideoParam structure */
mfxVideoParam conf;
mfxExtBuffer *eb=(mfxExtBuffer *)&du;

memset(&conf,0,sizeof(conf));
conf.IOPattern=MFX_IOPATTERN_IN_SYSTEM_MEMORY | MFX_IOPATTERN_OUT_SYSTEM_MEMORY;
conf.NumExtParam=1;
conf.ExtParam=&eb;

conf.vpp.In.FourCC=MFX_FOURCC_YV12;
conf.vpp.Out.FourCC=MFX_FOURCC_NV12;
conf.vpp.In.Width=conf.vpp.Out.Width=1920;
conf.vpp.In.Height=conf.vpp.Out.Height=1088;

/* video processing initialization */
MFXVideoVPP_Init(session, &conf);

Region of Interest¶

During video processing operations, the application can specify a region of interest for each frame as shown in the following figure:

Specifying a region of interest guides the resizing function to achieve special effects, such as resizing from 16:9 to 4:3, while keeping the aspect ratio intact. Use the CropX, CropY, CropW, and CropH parameters in the mfxVideoParam structure to specify a region of interest.

The VPP Region of Interest Operations table shows examples of VPP operations applied to a region of interest.

VPP Region of Interest Operations¶
Operation	VPP Input Width X Height	VPP Input CropX, CropY, CropW, CropH	VPP Output Width X Height	VPP Output CropX, CropY, CropW, CropH
Cropping	720 x 480	16, 16, 688, 448	720 x 480	16, 16, 688, 448
Resizing	720 x 480	0, 0, 720, 480	1440 x 960	0, 0, 1440, 960
Horizontal stretching	720 x 480	0, 0, 720, 480	640 x 480	0, 0, 640, 480
16:9 4:3 with letter boxing at the top and bottom	1920 x 1088	0, 0, 1920, 1088	720 x 480	0, 36, 720, 408
4:3 16:9 with pillar boxing at the left and right	720 x 480	0, 0, 720, 480	1920 x 1088	144, 0, 1632, 1088

Multi-view Video Processing¶

oneVPL video processing supports processing multiple views. For video processing initialization, the application needs to attach the mfxExtMVCSeqDesc structure to the mfxVideoParam structure and call the MFXVideoVPP_Init() function. The function saves the view identifiers. During video processing, oneVPL processes each view individually. oneVPL refers to the FrameID field of the mfxFrameInfo structure to configure each view according to its processing pipeline. If the video processing source frame is not the output from the oneVPL MVC decoder, then the application needs to fill the the FrameID field before calling the MFXVideoVPP_RunFrameVPPAsync() function. This is shown in the following pseudo code:

mfxExtBuffer *eb;
mfxExtMVCSeqDesc  seq_desc;
mfxVideoParam init_param;

init_param.ExtParam = &eb;
init_param.NumExtParam=1;
eb=(mfxExtBuffer *)&seq_desc;

/* init VPP */
MFXVideoVPP_Init(session, &init_param);

/* perform processing */
for (;;) {
    MFXVideoVPP_RunFrameVPPAsync(session,in,out,NULL,&syncp);
    MFXVideoCORE_SyncOperation(session,syncp,INFINITE);
}

/* close VPP */
MFXVideoVPP_Close(session);

Encoding Procedures Transcoding Procedures

oneAPI Specification 1.1-provisional-rev-1 documentation

Video Processing Procedures¶

Configuration¶

Region of Interest¶

Multi-view Video Processing¶