Pluggable video codec for RTC 1.5 stack
RTC 1.5 (CE 6.0 R 2 release) supports Point to Point Video calling. However, the stack does not ship any built-in video codecs. Instead it supports a pluggable video codec architecture through which, one can add their own video codecs.
Below are the steps one needs to take to plug-in a video codec into the RTC stack.
1. Implement your video codec in form of a DShow filter. Here is a MSDN link that explains How to write a DShow filter.
2. On the device, set the following reg keys before RTC is initialized:
HKEY_LOCAL_MACHINE\Comm\RTC\Video\Codecs\<key set#>
*Note: <key set#> can be any name (e.g. 1,2,3 or codec name itself)
*Note: For latest and up to date pluggable video codec reg keys listed below, please visit the MSDN link here .
Name
Type
Description
EncoderFilter
REG_SZ
Specifies the COM CLSID of the encoding IBaseFilter.
DecoderFilter
REG_SZ
Specifies the COM CLSID of the decoding IBaseFilter.
DecoderMediaSubtype
REG_SZ
Specifies the media subtype GUID of the format of the incoming data. Examples of incoming data format include H263 and H264.
CodecType
REG_SZ
Specifies the codec type. Always set the data to Filter. There is no default value.
CodecRank
REG_DWORD
Sets the relative priority of the codec. The data is an integer between 0 and 100. The highest priority is 0 and the lowest is 100.
CodecName
REG_SZ
Specifies the name of the codec sent in the Session Description Protocol (SDP) payload. For example, the codec name could be H263-1998. The name must contain fewer than 32 characters.
PayloadType
REG_DWORD
Specifies the RTP payload type associated with the codec. For example …
ClockRate
REG_DWORD
Indicates the clock rate of the codec as it appears in the a=rtpmap line of the SDP . For many codecs the correct data is 90,000.
Note: If a codec needs to be represented by multiple payload types (as some codec do), then a separate key set needs to be added for each payload type. RTC supports plugging in a maximum of 7 such key sets (7 separate payload types).
3. If your codec supports custom codec properties, which needs to be negotiated through SDP using FMTP lines, then the codec can put its FMTP string in the following reg key:
HKEY_LOCAL_MACHINE\Comm\RTC\Video\Codecs\<key set#>
Name
Type
Description
SDPFmtp
REG_SZ
Indicates the format specific parameters FMTP of the codec as it appears in the SDP a=fmtp line of the SDP. If SDPFmtp is not set, no FMTP lines will be sent.
This data indicates only the parameters themselves. SDP automatically fills in the following text:a=fmtp:<RTP Payload Type #>
RTC will read the reg key during initialization, and will add it to the outgoing SDP as is, as FMTP properties of the codec. If no reg key is specified, no FMTP lines will get published in SDP for the codec.
4. RTC also supports passing of incoming FMTP property string to the plugged in codec, so that the codec can make a decision if it supports those properties or not. The pluggable codec needs to implement the following interface: IRTCSDPValidator to accept incoming FMTP strings from the SDP. The following reg keys needs to be set for the codec, to let RTC know of the interface:
HKEY_LOCAL_MACHINE\Comm\RTC\Video\SdpValidators
Name
Type
Description
<Codec Name>
REG_SZ
Specifies the CLSID that implements this object.
e.g.
Name
Description
H263-1998
{D018239F-4022-4c15-BC87-03CCADC1A4E3}
Note: An IRTCSDPValidator registry value is not required. If a codec does not implement this registry key, the codec accepts all SDP.
5. Implement interfaces that allow adding and removing custom RTP headers.
RTC uses RTP as the media transport protocol for video. RTP allows adding codec specific headers for each RTP payload. Since this header is codec specific, RTC provides a mechanism so that pluggable codecs can add/remove these custom RTP headers during the encode/decode process.
Implement the interface IRTCPluggableVideoEncoder to add custom RTP header and the interface IRTCPluggableVideoDecoder to remove custom RTP headers. Publish these interfaces in the reg key:
HKEY_LOCAL_MACHINE\Comm\RTC\Video\Codecs\<key set#>
Name
Type
Description
EncoderPlugin
REG_SZ
Specifies the COM CLSID of the IRTCPluggableVideoEncoder interface. Set H.263 to {A7D8A6EF-575B-4750-A727-04234A70D9DE} to use the Microsoft provided implementation of this interface. Set H.264 to {92A200F8-0981-11DC-813E-B77255D89593} to use the Microsoft provided implementation of this interface.
DecoderPlugin
REG_SZ
Specifies the COM CLSID of the IRTCPluggableVideoDecoder interface. Set H.263 to {0CE193DF-27F0-48c7-92F7-75CA7A5965E6} to use the Microsoft provided implementation of this interface. Set H.264 to {86546C00-0981-11DC-88BE-AE7255D89593} to use the Microsoft provided implementation of this interface.
*Note: The Microsoft implementation of the above interfaces for H263 and H264 video codecs can be found at CE OS installation location: public\fp_voip\oak\codecs\H263Parse and public\fp_voip\oak\codecs\H264Parse respectively.
Now that we know how to write a pluggable video codec and add it to RTC, let’s see how RTC will use the codec and all other interfaces by the codec module:
1. As soon as RTCClient object is initialized, RTC goes through the all reg keys defined above to check for any pluggable video codecs. If the reg keys are set, RTC will check, verify and load the codec for future use.
2. When a user makes an outgoing video using RTC, then:
a. All pluggable video codecs that are successfully loaded by RTC, will get published in the outgoing SDP.
b. If the codec has set the SDPFmtp reg key to specify its FMTP properties, then RTC will add that string in the outgoing SDP for the respective video codec.
3. When an incoming video call is received by RTC, then:
a. For each video codec published in the incoming SDP, RTC will try to match it with its plugged in codecs. If any of the codec name matches, then RTC checks to see if the codec implements IRTCSDPValidator interface. If it does, then RTC will call the CheckAcceptableFmtp function of the interface with the incoming FMTP string, to let the codec check if it can support those properties. If the function returns S_OK, RTC considers that codec is capable of handling the incoming FMTP properties, and it proceeds with the call as usual. Any other return value is considered a failure. If there is no codec match, RTC will reject the call.
b. If the codec does not implement IRTCSDPValidator, then RTC assumes that the codec is capable of handling any incoming FMTP property, and will proceed with the call as usual (generally by triggering an event of an incoming video call)
4. Once the call has been established:
a. During the encode process, RTC will call the appropriate filter APIs to encode the raw video frames. After that, RTC will call the IRTCPluggableVideoEncoder interface API to let the codec generate custom RTP header for the encoded frame. Please see the MSDN page for more details on the interface.
b. During the decode process, RTC will first call the IRTCPluggableVideoDecoder interface API to let the interface extract the encoded frame from RTP payload. Once an entire encoded frame is extracted, RTC will then call the filter APIs to decode the raw video frames. Please see the MSDN page for more details on the interface.