WebRTC Insertable Streams After Decoder: Extract Timestamp

In this blog, we will show you another way you can modify incoming remote video streams. I will not be mutating the video frames in this example; instead, I will be logging the frame timestamps from the incoming video streams. You can replace the logging logic with your frame mutation logic.

To begin, let’s consider what we will be doing. Since we have custom logic we want to apply to each downstream connection, we want to create a function that takes a remote connection and applies our logic to it.

let openDownstream = async (remoteConnectionInfo, layoutManager, channel) => {

The first step is to establish our remote video object.

// create a remote media object to represent the remote feed

 var remoteMedia = new fm.liveswitch.RemoteMedia();

 remoteMedia.setAudioMuted(true);

 remoteMedia.setVideoMuted(false);

Following that, we need to define a stream object that will represent the decoded remote feed.

Once we have these two objects, we can define our connection object to represent the downstream connection.

// create a video stream object from the remote media object

 let videoStream = new fm.liveswitch.VideoStream(remoteMedia);

 // establish the downstream connection and pass in your object to ensure they get connected to the remote feed

 let connection = channel.createSfuDownstreamConnection(

  remoteConnectionInfo,

  videoStream

 );

Next, we need to add some handlers to make sure we address the different negative states that our connection could enter into.

// add a handler to address changes to the downstream connection

 connection.addOnStateChange(function (connection) {

     // is the connection is closing or is failing to be established, remove the remote video feed from the layout and cleanup the objects

     if (
     connection.getState() == fm.liveswitch.ConnectionState.Closing ||
     connection.getState() == fm.liveswitch.ConnectionState.Failing
   ) {

         layoutManager.removeRemoteView(remoteMedia.getId());

         remoteMedia.destroy();

       // if the connection has failed try to reopen it

  } else if (connection.getState() == fm.liveswitch.ConnectionState.Failed) {

         openDownstream(remoteConnectionInfo, layoutManager);

  }

 });

With all of our connection logic in place, we can now open the downstream connection.

// open the downstream connection

 await connection.open();

It is now time to prepare the media object and the pipeline that will allow us to mutate the incoming feed.

We start by gaining access to the internal remote media’s stream and then select the first video track from the stream.

// get access to the internal video stream from the remote media object

 var stream = remoteMedia._internal._videoMediaStream;

 // get access to the video track, in this case we only have 1 so we pull the first from the array

 const videoTrack = stream.getVideoTracks()[0];

We can now create our custom media pipeline. We need a trackProcessor to manage the pipeline, a trackGenerator to create our new stream after the logic has been created, and a transformer that contains our media mutation logic.

// track processor allows us to define our custom media pipeline

 const trackProcessor = new MediaStreamTrackProcessor({ track: videoTrack });

 // track generator is responsible for rebuilding a new stream

 const trackGenerator = new MediaStreamTrackGenerator({ kind: "video" });

 // transform stream holds the custom logic that can be used to mutate or read the incoming remote video frames

 const transformer = new TransformStream({

     // define our transformation logic

     async transform(videoFrame, controller) {

         // in this example we will just output the video timestamp and not actually mutate the video

         console.log(videoFrame.timestamp);

         // enqueue the video frame once we are done working on it

    controller.enqueue(videoFrame);

  }

 });

Reminder, your logic should go inside the transform function here and should be modifying the video one frame at a time.

Finally, we’ll string these objects together and output the video stream to the layout.

// string the components of the media pipeline together

 trackProcessor.readable

  .pipeThrough(transformer)

  .pipeTo(trackGenerator.writable);

 // define our media stream that will come out of the generator

 const streamAfter = new MediaStream([trackGenerator]);

 // create a new HTML Element to host the video stream

 const video = document.createElement("video");

 // the the mutated stream as the video elements source

 video.srcObject = streamAfter;
// add a listener to trigger the video to play
video.onloadedmetadata = function (e) {
  video.play();
}
// add the video element to the layout 
player.appendChild(video); 

We have now completed our downstream connection handler. To recap, we instantiated our media objects, attached them to our downstream connection, pulled the internal video track off of the remote media object, and applied a transformer to generate a new video track one frame at a time.

It is now time to connect to a channel and trigger our custom remote connection logic.  We will use a button press at the starting action of this example.

// add a click handler for the start button

button.addEventListener("click", async () => {

If you have been following our blog series (WebRTC Streams before Encoder and WebRTC Remote Streams before Decoder), the next few steps will be familiar to you. We need to create our client object, create a claim for the channel we would like to join, create a client token, and register our client using the token.

// define the LiveSwitch Layout Manager helper object

   let layoutManager = new fm.liveswitch.DomLayoutManager(player);

   // create our client object to define our user

   let client = new fm.liveswitch.Client(gateway, app);

   // create a claim to join the channel of our choosing

   let claims = [];

   let claim = new fm.liveswitch.ChannelClaim(channelId);

   claims.push(claim);

   // generate a token that allows us to register a client to a channel

   let token = fm.liveswitch.Token.generateClientRegisterToken(

  client,

  claims,

  sharedSecret

   );

   // register our client using the token

   let channels = await client.register(token);

   // fetch the channel object we will be using

   let channel = channels[0];

The last step we need to do is add a handler that listens for when a new remote connection is opened and triggers our custom logic on that connection.

// add a handler for when a new remote connection is opened on the channel

   channel.addOnRemoteUpstreamConnectionOpen(function (remoteConnectionInfo) {

       // trigger our custom downstream connection logic

       openDownstream(remoteConnectionInfo, layoutManager, channel);

   });

We can also add another handler for the connections that were established before we joined the channel. It is a good practice to add this type of logic to your application since oftentimes you will not control the order in which users will join a session.

// handle remote connections that have already been registered on the channel before you joined

for (

       // loop through the remote connection info objects already registered

       var _i = 0, _a = channel.getRemoteUpstreamConnectionInfos();

  _i < _a.length;

  _i++

   ) {

  // fetch the current connection info object in our loop

       var remoteConnectionInfo = _a[_i];

       // trigger the custom downstream logic for each remote connection
  openDownstream(remoteConnectionInfo, layoutManager, channel);

   }

Congratulations! You now know how to modify any incoming remote video feed after it has already been decoded. Please note that this will increase the amount of processing being done on the receiving client’s device, and depending on the number of participants in a channel, could result in the device slowing down. As with any device-side processing, please test your logic across a number of devices and scenarios to ensure a great user experience.

If you want to see these changes live, please take a look at the CodePen.  You can also try this in your own application after signing up for our free 30-day trial.

Need assistance in architecting the perfect WebRTC application? Let our team help out! Get in touch with us today!