WebRTC Remote Streams before Decoder: Extract Timestamp

In this article, we will guide you through the process of modifying incoming remote video streams before displaying them, using the powerful LiveSwitch SDK. LiveSwitch's SDK offers various points in the video stream's lifecycle where you can apply transformations, and in this article, we will focus on doing transformation at the earliest stage for incoming video. Any transformation can be done, but we'll walk you through extracting timestamps from video frames as an example transformation here.

To begin, we will first need to get access to the video stream.

var remoteMedia = new fm.liveswitch.RemoteMedia();

remoteMedia.setAudioMuted(true);

remoteMedia.setVideoMuted(false);

// get remote video stream object

let videoStream = new fm.liveswitch.VideoStream(remoteMedia);

// wait for the connection to open

await connection.open();

// get the stream

var stream = remoteMedia._internal._videoMediaStream;

// get the video track

const videoTrack = stream.getVideoTracks()[0];

The stream represents a constant flowing source of data from the User Media devices. Our next step is to get the video track and create our new video processor resources.

// create a processor

const trackProcessor = new MediaStreamTrackProcessor({ track: videoTrack });

// create a generator

const trackGenerator = new MediaStreamTrackGenerator({ kind: "video" });

The track processor is given access to the local media’s video track and is responsible for creating the pipeline that will allow us to transform and recreate a new stream. The TrackGenerator is responsible for taking tracks one at a time and inserting them into a stream.

Next, let's define the logic for our transformer, which you can adjust according to your specific requirements. The transformer will operate on a single frame at a time.

// define a transformer (this will be the logic run on each video frame)

const transformer = new TransformStream({

    async transform(videoFrame, controller) {

    // in this example we just want to examine the timestamp on the video frame

    console.log(videoFrame.timestamp);

    // close the frame once we are done with it

    videoFrame.close();

  }

});

Now that we have defined our transformation logic and have our pipeline resources defined, we need to create our processing pipeline.

//define the transformation pipeline

trackProcessor.readable

.pipeThrough(transformer)

.pipeTo(trackGenerator.writable);

Using our processor, we'll pass the tracks through the transformer and then forward the transformed frames to our track generator. We'll create a new MediaStream object that utilizes the stream from our generator and use this new stream in our LocalMedia object.

// get the new stream coming off the transformation

const streamAfter = new MediaStream([trackGenerator]);

// create a video tag for the DOM

const video = document.createElement("video");

// set the video source to the transformed stream

video.srcObject = streamAfter;

// add the video element to DOM

player.appendChild(video);

We now have our fully transformed remote video track! We should now add this to our layout so we can view the remote video from others in our meeting.

Check out the complete example and experiment with additional transformations in our CodePen example. You can also try applying the same transformation discussed in our previous blog post on transforming local media.

It's important to note that while this approach allows you to apply a wide range of transformations to remote video, you should be mindful of the additional processing load it imposes on the local device. Test your transformations' performance on the oldest devices you intend to support. A transformation that runs smoothly on an iPhone 15 Pro might cause an older iPhone SE to overheat and shut down in minutes.

Stay tuned for additional articles on transforming your media using the power of the LiveSwitch SDK.

Need assistance in architecting the perfect WebRTC application? Let our team help out! Get in touch with us today!