In our previous article we discussed the different aspects that contribute to the overall time delays in networking communications. In this article we will focus on how to reduce them and how to cope with them. We will discuss the different solutions for streaming applications.
Why are time delays a problem for video streaming:
As you can imagine, too much delay between a person to person stream does not help in communicating. Furthermore for some applications we want to provide the user that is streaming a video with feedback. If the time delay is too big the user cannot use this feedback. So lets look at some numbers.
For a 1-on-1 stream acceptable time delays are around 150 ms one way which gives us a round trip time delay of 300 ms. So for streaming purposes where a 1 to 1 call between 2 people is made the total delay should be below 300 ms.
Acceptable time delays change when we want to provide feedback to the user. For example if a user is controlling something that he sees in the video and he/she needs feedback on his input. This aspect is called input lag, and user performance/satisfaction starts to decrease at around 50 ms round trip delay and decreases drastically at above 100 ms (See: Input lag: How important is it?)
So for our streaming applications where we want to provide feedback to the user we want our one way time delay to stay under 50 ms preferably.
Another issue that can arise from video streaming is that too much packet loss will cause the stream to be corrupted and thus the video data cannot be used anymore. This is caused by the decoding of the incoming video data that cannot be performed correctly and thus resulting in an unreadable stream or in a distorted image (like the image above).
We will now look at how we can reduce these delays for streaming applications (with user feedback). We will provide you with some DIY examples using your own PC with GStreamer.
Setup Gstreamer DIY environment
To run the different streaming test scenario’s in this article you will need to PC’s connected to the same network. One PC will be our source (it will need to have a webcam) The other PC will be our sink and will playback the video. You can also perform most tests using the same PC as the source and Sink PC, if you don’t have 2 devices available. The latency effects will just be a little less noticeable.
On both PC perform the following commands to install GStreamer:
brew install gstreamer gst-plugins-good gst-plugins-bad gst-plugins-ugly gst-libav
Once GStreamer is installed you can test it by sending the video from your webcam on the source PC to the sink PC. Make sure you open a local network port of your choosing on the firewall of your sink PC and get the IP address of the sink PC (see sources down below for more information). Then run the following:
Source PC:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480' ! jpegenc ! rtpjpegpay ! udpsink host=<sink-pc-ip> port=<chosen port>
Sink PC:
gst-launch-1.0 udpsrc port=5555 ! 'application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000' ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
You should see a window appearing after some time displaying the video from your source PC on your sink PC. You might experience some delay in the video stream but this depends mostly on the encoding/decoding abilities of your hardware. We will discuss this more in details further on.
Now that we have our first stream up and running we will monitor how all the different aspects of time delay discussed in the previous article impact our stream.
Processing
First of all we must mention that video processing (like encoding and decoding) will generate time delays as well, unrelated to the networking delays. These delays are cause by the processing power required to encode and decode the video stream. It is difficult to send raw video over the internet as the bandwidth requirements are often too large. The delays caused by the processing of the video is related to the codecs that are used and to the video encoding/decoding processing power of the device that performs this process.
We will not consider this delay in this video and thus will keep the same video processing algorithms throughout our tests.
Communication link / Routing
The first aspect of our networking time delay while streaming a video that we are going to identify is the delay caused by the communication link and the routing of the data. In our previous article we saw that the transmission link and the amount of nodes in the network influences the time delay.
So to reduce the delay we want to optimise the link between the source and the sink. Ideally we would like a link with a high transmission speed (wired over wireless) and the least amount of nodes between them (as direct as possible)
So lets do some tests and see how this affects the delay in our stream:
Local network wireless connection over UDP:
To test the delay on a network connection with very few hops use the source and sink PC commands from the setup. Normally the delay you observe in the video (for example when you clap your hands in front of the camera) is minimal.
Source PC:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480' ! jpegenc ! rtpjpegpay ! udpsink host=<sink-pc-ip> port=<chosen port>
Sink PC:
gst-launch-1.0 udpsrc port=5555 ! 'application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000' ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
4G Internet connection over UDP:
Now we are going to increase the amount of hops between our source and our sink. To do this connect the source PC to a 4G mobile network hotspot using your phone. On the sink side we will have to enable port forwarding on our home router. Forward the chosen port (5555 in my example) to the ip address of your sink PC for UDP and on the source PC for TCP communications (see sources down below for more info). Also make sure this port is open on the firewall of your sink PC. Then check what your internet ip address is (whatsmyip.org).
Change the destination ip address on the source PC to your internet ip address.
Source PC:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480' ! jpegenc ! rtpjpegpay ! udpsink host=<internet-ip> port=<chosen port>
Sink PC:
gst-launch-1.0 udpsrc port=5555 ! 'application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000' ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
If everything is configured correctly you should see the video stream on the sink PC, and you should notice that the delay has increased.
To conclude for an optimal communication link with low latency we would prefer a wired connection over a WiFi or 4G connection. However, with the introduction of the 5G network, the wireless communication delays caused by the transmission link are greatly reduced (And no, corona infections are not increased as a side effect).
To further improve network delays for streaming try to create the most direct link possible between the source and the sink, in other words, create a direct route between them where the least amount of hops is required.
Packet loss
Now lets tackle the delays caused by packet loss! In our previous article we saw that packet loss is caused by the quality of the transmission link and the capacity of the transmission link.
To reduce these delays preferably use a wired transmission link as these links will have much less errors and thus much less packet losses. Second try to keep the bandwidth used below the capacity of the link to prevent bandwidth congestion and consequently packets being dropped.
Furthermore if some aspects prevent the use of a high quality, high bandwidth ,transmission link, one can consider to use the TCP protocol over the UDP protocol. The TCP protocol checks if packets have arrived at their destination and resents them if not. Thus completely reducing the packet loss to zero. However, as you can imagine, if there is a lot of packet loss in the network, using TCP over UDP communications will drastically increase the time delay as the TCP protocol constantly needs to resend the packets. So a careful consideration needs to be made between the transmission link quality and bandwidth, the bandwidth used by the stream and the protocol that is chosen.
Lets see how these different aspects influence the overall time delay of our stream so that you can test the different configurations on your own and make an informed decision.
4G connection with low bandwidth congestion over UDP
Source PC:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480,framerate=30/1' ! jpegenc ! rtpjpegpay ! udpsink host=<internet-ip> port=<chosen port>
Sink PC:
gst-launch-1.0 udpsrc port=5555 ! 'application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, framerate=(fraction)30/1' ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
4G connection with low bandwidth congestion over TCP (The source PC must have the port forwarding enabled on the chosen port now)
Source PC:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480,framerate=30/1' ! jpegenc ! tcpserversink host=0.0.0.0 port=<chosen port>
Sink PC:
gst-launch-1.0 tcpclientsrc host=<internet-ip> port=<chosen-port> ! 'image/jpeg,width=640,height=480,framerate=30/1' ! jpegdec ! videoconvert ! autovideosink
Now perform the same tests with the UDP and the TCP streaming, but now we will simulate a congested bandwidth to see the consequences. On both PC’s run the following commands alongside the GStreamer commands. Open another port to perform a bandwidth test with IPerf that will congest the bandwidth.
On the PC with the new port forwarded:
iperf -s -u -p <new-chosen-port> -i 1
On the other PC:
iperf -c <internet-ip> -p <new-chosen-port> -u -b 300M
You probably will need to play a bit with the -b setting of the IPerf command, it sets the bandwidth that will be used, chose a value that is close to your maximum bandwidth.
What we can see is that for an un-congested bandwidth, the TCP stream is slightly more delayed, but this is barely noticeable. It gets interesting when we start congesting the bandwidth. For the TCP connection, the stream start to get more delayed the more the bandwidth is congested, up to a point where the stream is frozen as the TCP communication tries to resend all the lost packets. However, the UDP connection is more capable of handling a highly congested bandwidth. As the UDP protocol does not care about packet loss, it just continues sending data, with as a result that the video is a bit more delayed, and freezes shortly when there is too much packet loss for the decoder to correctly decode the stream.
To conclude here are some scenarios to consider:
- If bandwidth capacity is sufficient and the transmission link is of high quality, packet loss will be minimal, so UDP would be our best choice for a low latency stream.
- If bandwidth capacity is sufficient but the transmission link is of low quality, packet loss will be considerable, TCP will be your choice here to prevent corrupted data.
- If bandwidth capacity is insufficient but the transmission link is of high quality, packet loss will be considerable but using TCP will cause high delays. Try using a UDP connection and decreasing the used bandwidth on the link and the bandwidth used by the stream.
- If both bandwidth capacity is insufficient and the transmission link is of low quality, packet loss will be great. Try decreasing the bandwidth usage of the stream and other services on the link and use a TCP connection to prevent further packet loss, and thus prevent a corrupted stream.
Jitter
The last aspect we are gonna tackle in our stream communications is Jitter. As previously seen Jitter is caused by the delays in our network being variable. When jitter increases, the chances of ending up with a corrupted stream also increases.
There are 3 methods to solve jitter, the first is to decrease the overall traffic on our transmission link. Aside from decreasing packet loss as previously mentioned the variation in the delay will also decrease.
An other solution is to add a buffer on the sink side of the stream that synchronises the packets arriving and thus removing jitter. And the last solution is to use a TCP stream, the TCP protocol, besides handling packet loss, will also synchronise the stream to remove jitter. However, as you can imagine when we stock some stream data before using it, this will increase the time delay of our stream. A good consideration has to be made between the jitter remaining in the transmission link after optimising the traffic and between the length of the buffer used.
Lets try some practical configurations to see the impacts this has on our stream.
We are going to add a buffer to our UDP stream. To compare the differences launch a bandwidth congested UDP stream over the internet as in the previous section.
And then we are going to add a buffer to the sink PC streamer command, the video will have less freezes but will have more delay.
Source PC stream with buffer:
gst-launch-1.0 autovideosrc ! 'video/x-raw,width=640,height=480,framerate=30/1' ! jpegenc ! rtpjpegpay ! udpsink host=<internet-ip> port=<chosen port>
Sink PC stream with buffer:
gst-launch-1.0 udpsrc port=5555 ! 'application/x-rtp, media=(string)video, payload=(int)26, clock-rate=(int)90000, framerate=(fraction)30/1' ! rtpjitterbuffer ! rtpjpegdepay ! jpegdec ! videoconvert ! autovideosink
To conclude, the first step to tackle this challenge is to measure the jitter in the transmission link and identify if this is too high for a stable stream. Then try to resolve most of the jitter, you do not need to account for all delay variations, by adding an as small as possible buffer on the sink side.
It can be challenging to find the correct set of configurations to minimise the delays caused by the network for our stream. Optimising the transmission link is always a good idea before diving in to the methods and configuration options to solve the networking delays aspects.
Now it is up to you to set up the perfect smooth fast and reliable stream!
Thank you for reading!
Sources
IP address lookup: https://www.howtogeek.com/117371/how-to-find-your-computers-private-public-ip-addresses/ (29–01–2021)
Port forwarding: https://www.noip.com/support/knowledgebase/general-port-forwarding-guide/ (18–02–2021)
iPerf: https://iperf.fr/
Input lag: How important is it? https://www.cnet.com/news/input-lag-how-important-is-it/
GStreamer: https://gstreamer.freedesktop.org/ (18–02–2021)