Did you ever found yourself in a situation where you are baffled by the time it takes for your data to travel? Aren’t radio signals, be it by wire or over the air, supposed to travel with the speed of light?
In this article we will dive under the hood of networking links to find out what is causing our data to be delayed. And you’ll be able to try measuring some delays in your network yourself.
Aspects
Delays in networks are caused by a variety of factors in a communication network. The most obvious one is simply the distance between the source and the destination. But our data does not travel directly from a source to a destination. It will pass by a lot of stations that will contribute to the delays. Some data may be lost on the way or the order of the data may be scrambled.
We will attempt to explain the different aspects of a network that will contribute to delays. You as reader can try to see some real life numbers to measure this delay.
Throughout this article I will mention two network communication protocols, UDP and TCP. Without going into the details of these protocols, it is good to know their main difference:
UDP: User Datagram Protocol is a protocol that sends data packets with a minimalist header and without any checks. Data packets are sent and that’s it.
TCP: Transmission Control Protocol is a protocol that sends packets and checks if they have arrived. If not it will resend the packet. This protocol has more overhead (more info in the header) to perform this control.
Latency
Latency in network communication refers to the time it takes for a packet to be send from one end of a communication link to the other. It is composed of 4 different aspects that each contribute to the overall delay.
Propagation delay is the time between the transmission of a data packet to the reception of this data packet. This time t [s] depends on the distance d [m] between the sender and the receiver and the speed v [m/s] of the transmission link: t = d/v
Transmission delay is the time it takes for a data packet, that is processed and ready to be sent, to be transmitted to the outgoing link. This time t [s] depends on the size of the packet S [bytes] and the transmission capacity (Bandwidth) C [bytes/s] of the outgoing link: t = S/C
Queuing delay is the time a data packet has to wait in the buffer of a transmission link before it is processed and sent. This time is dependent on the amount of incoming traffic, the capacity of the transmission link and the type of transmission protocol (UDP/TCP).
Processing delay is the time it takes for the transmission link to process the data packet. This time depends on the processing speed of the transmission link.
Adding all these time delays together amounts to the total networking time delay. Also called network latency. However, there is a lot more magic happening that will cause more delays in networks.
Routing
Now that we know what different parts of a network link cause delays we can start to understand where the networking delays come from. And start to look at some numbers.
If we would have a network that would only have one link our delays would be minimal and barely measurable. However in current network infrastructures a data packet does not travel over one link but actually hops over a lot of different transmission nodes. Each of these nodes will have their own set of propagation, transmission, queuing and processing delays. So you can imagine that if for one node our delay would be a couple of milliseconds, a transmission from your home laptop in France to a server in the United States and back, hopping over many nodes, would cause significant delays. So lets measure this.
To get an insight in the delays in your network from home you can simply use the ping command in your terminal. The commands in this article will work in terminals on Linux or Mac machines. For windows machines you can use powershell but the commands might differ a bit.
Try it for yourself.
First we will measure the round trip delay from your terminal to your local network interface, its address is localhost or 127.0.0.1:
ping -c 5 localhost PING localhost (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.067 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.058 ms 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.020 ms 64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.057 ms 64 bytes from localhost (127.0.0.1): icmp_seq=5 ttl=64 time=0.058 ms --- localhost ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4073ms rtt min/avg/max/mdev = 0.020/0.052/0.067/0.016 ms
You can see that the round trip time delays are very minimal. Now we can go a little further and see the delays between our machine and our local network router. You will see that as the distance increases, and the number of transmission nodes increase, the time delay will increase as well. Find your router’s IP address, it is often 192.168.0.1 or 192.168.1.1 (you can check here to find your IP addresses https://www.howtogeek.com/117371/how-to-find-your-computers-private-public-ip-addresses/):
ping -c 5 192.168.0.1 PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data. 64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=3.58 ms 64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=3.74 ms 64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=2.16 ms 64 bytes from 192.168.0.1: icmp_seq=4 ttl=64 time=1.91 ms 64 bytes from 192.168.0.1: icmp_seq=5 ttl=64 time=2.48 ms --- 192.168.1.254 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 1.912/2.773/3.736/0.746 ms
To see even longer delays try to ping an address that is much further away and requires more hops. For example google.com, or try addresses that you know are very far away:
ping -c 5 google.com PING google.com(par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e)) 56 data bytes 64 bytes from par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e): icmp_seq=1 ttl=120 time=3.09 ms 64 bytes from par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e): icmp_seq=2 ttl=120 time=5.48 ms 64 bytes from par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e): icmp_seq=3 ttl=120 time=3.17 ms 64 bytes from par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e): icmp_seq=4 ttl=120 time=4.96 ms 64 bytes from par21s03-in-x0e.1e100.net (2a00:1450:4007:810::200e): icmp_seq=5 ttl=120 time=5.77 ms --- google.com ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 3.091/4.492/5.769/1.142 ms
In addition of measuring the delay between our machine and some destination on the internet, we can also see all the different steps that our data packet takes over the network. You can try the following from your local machine, first we check the route to our local router, then we can check the route to a website on the internet. You will see that only one hop is necessary to get to your local router. That is why the delay is lower, here the delay will be mostly due to the wireless connection if you are on WiFi. But many more hops are required to get to a website on the internet.
traceroute 192.168.0.1 traceroute to 192.168.0.1 (192.168.0.1), 30 hops max, 60 byte packets 1 _gateway (192.168.0.1) 4.309 ms 4.277 ms 4.265 ms traceroute google.com traceroute to google.com (216.58.201.238), 30 hops max, 60 byte packets 1 _gateway (192.168.0.1) 2.342 ms 2.251 ms 2.203 ms 2 78.192.XX.XXX (78.192.XX.XXX) 7.211 ms 7.165 ms 7.121 ms 3 78.255.XX.XXX (78.255.XX.XXX) 4.567 ms 4.523 ms 5.998 ms 4 mil75-49m-1-v902.intf.nro.proxad.net (78.254.253.33) 4.736 ms 4.674 ms 5.002 ms 5 rhi75-49m-1-v904.intf.nro.proxad.net (78.254.252.134) 4.558 ms 4.896 ms 4.852 ms 6 fes75-ncs540-1-be3.intf.nro.proxad.net (78.254.242.206) 4.826 ms 2.897 ms 2.813 ms 7 p13-9k-3-be2003.intf.nro.proxad.net (78.254.242.45) 2.724 ms 4.220 ms 4.140 ms 8 194.149.165.209 (194.149.165.209) 2.895 ms 2.850 ms 2.805 ms 9 * * * 10 * * * 11 be2151.agr21.par04.atlas.cogentco.com (154.54.61.34) 2.542 ms 3.688 ms 3.030 ms 12 tata.par04.atlas.cogentco.com (130.117.15.70) 11.159 ms 11.111 ms 10.887 ms 13 72.14.212.77 (72.14.212.77) 3.838 ms 3.789 ms 3.744 ms 14 108.170.244.161 (108.170.244.161) 3.764 ms 3.696 ms 108.170.244.225 (108.170.244.225) 4.880 ms 15 216.239.48.27 (216.239.48.27) 3.543 ms 3.277 ms 3.198 ms 16 par10s33-in-f14.1e100.net (216.58.201.238) 3.074 ms 3.032 ms 2.984 ms
To go even further in our discovery of networking delays we will investigate two more aspects that contribute to the total delay. These aspects need to be considered and are often solved in data communication implementations. But to compensate for these aspects, the queuing and processing delays in our data transmission nodes will increase even more.
Packet Loss
Sometimes when we send data across a network it gets lost while travelling. This is called packet loss, when a packet that is send never arrives at its destination. Packet loss occurs for 2 different reasons.
The first cause for packet losses are errors in the transmission of data. This occurs more often on wireless transmission links than wired links. But can also occur in transmission switches due to processing errors.
The second cause for packet loss is called bandwidth congestion. This happens when the amount of data that is transferred over a transmission node is greater than the capacity of this node. The buffer will increase and when a certain threshold is reached the transmission node will start dropping data packets.
When the amount of packet loss increases the risk for data corruption increases as well. This is especially harmful in transmissions where data that is being sent has to be split in a lot of packets due to its size. For example when streaming video. When too much packets are lost the receiving process cannot get all the data required to perform the decoding of the video stream. Thus the video data will be corrupted and lost.
Lets see this packet loss in action. We can use the ping command for this and increase the interval between each ping packet and decrease the time we wait for a response. You will need super user rights for this command.
Be aware that you will need to find a route that is unable to cope with your request to see packet loss. Google will cope easily (they are designed for very high traffic) so try another website.
I tried github.com. Be aware that if you ping the same website a large amount of times, they will not be happy and your IP address could get banned. If you do not see any packet loss, then your connection is too stable and not enough traffic is occurring on your network for bandwidth congestion. What you can try to do is start a high quality video stream on YouTube and rerun the ping command. If you are on WiFi you can try to put your PC that is connected to WiFi far away from the router.
PS: On macOS replace the comma’s with dots for the -i and -w values in the command below.
sudo ping -c 1000 -i 0,01 -W 0,02 github.com ... ... --- github.com ping statistics --- 1000 packets transmitted, 996 received, 0,4% packet loss, time 14249ms rtt min/avg/max/mdev = 16.164/17.806/29.675/1.289 ms, pipe 3
Packet loss can be prevented by using a TCP transmission instead of a UDP transmission. A TCP link will detect packets that are lost and re-transmit them. However, re-transmitting lost data packets will increase the networking delays. Our sending node will have to wait on confirmation for the destination node. We will dive more into the details of this aspect in the next article.
Jitter
Jitter, also called packet delay variation, is a direct consequence of delays in the network, especially variable delays. The previously mentioned aspects of data transmission delay are not constant in a network. They depend on a lot of factors. So as you can imagine the delay between the sending of a data packet and its arrival is variable.
With this variable delay in mind we can quickly see that when we send a set of data packets in a particular order, the variable delays will cause the data packets to arrive in a different order (see image below). This in term will have the same effect as packet loss. Even if data is not lost it is still corrupted as the different data packets can not be put together in the correct order.
To measure jitter on your own you will need two different machines. We will perform a network test with iPerf to measure the jitter between our 2 machines (you can also see bandwidth performance and packet loss with this command, to install IPerf, see https://iperf.fr/iperf-download.php).
Please try to use these commands on two machines who are connected on the same network. With machine 1 named the server that has its firewall temporarily turned off. Otherwise port forwarding and firewall rules must be configured to make this work.
On machine 1, called the server:
iperf -s -u -i 1 ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 208 KByte (default) ------------------------------------------------------------ [ 3] local 192.168.1.44 port 5001 connected with 192.168.1.13 port 52192 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 11.5 MBytes 96.5 Mbits/sec 1.715 ms 5/ 8209 (0.061%) [ 3] 1.0- 2.0 sec 13.6 MBytes 114 Mbits/sec 0.166 ms 2/ 9681 (0.021%) [ 3] 2.0- 3.0 sec 12.5 MBytes 105 Mbits/sec 0.229 ms 2/ 8900 (0.022%) [ 3] 3.0- 4.0 sec 12.2 MBytes 102 Mbits/sec 1.366 ms 0/ 8680 (0%) [ 3] 4.0- 5.0 sec 12.8 MBytes 107 Mbits/sec 0.336 ms 54/ 9180 (0.59%) [ 3] 5.0- 6.0 sec 12.4 MBytes 104 Mbits/sec 0.230 ms 45/ 8909 (0.51%) [ 3] 6.0- 7.0 sec 12.5 MBytes 105 Mbits/sec 0.184 ms 0/ 8920 (0%) [ 3] 7.0- 8.0 sec 12.5 MBytes 105 Mbits/sec 0.282 ms 0/ 8928 (0%) [ 3] 8.0- 9.0 sec 12.5 MBytes 105 Mbits/sec 0.492 ms 0/ 8912 (0%) [ 3] 0.0-10.0 sec 125 MBytes 105 Mbits/sec 0.104 ms 108/89166 (0.12%)
On machine 2, called the client (find the IP address of your machine 1, the server):
iperf -c 192.168.1.44 -u -b 100M ------------------------------------------------------------ Client connecting to 192.168.1.44, UDP port 5001 Sending 1470 byte datagrams, IPG target: 112.15 us (kalman adjust) UDP buffer size: 9.00 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.1.13 port 52192 connected with 192.168.1.44 port 5001 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 4] 0.0-10.0 sec 125 MBytes 105 Mbits/sec [ 4] Sent 89166 datagrams [ 4] Server Report: [ 4] 0.0-10.0 sec 125 MBytes 105 Mbits/sec 0.104 ms 108/89166 (0.12%)
Jitter can be solved by creating a buffer of data packets on our arrival node that will reorder them based on their sender timestamps. However, this will increase the queuing delay. And thus increase the networking delays even more. More about this subject in our next article.
What to do about it?
Now that we understand what elements of a network cause delay we can start to look at the different solutions to minimize it.
In our next article we will look into why we need to minimize it for certain communications, and how we can achieve a minimal delay in our network communications. We will dive into solving network delays for video streaming applications. And we will provide you with some practical examples that you can test on your own.
Thank you for reading!
Sources
Delays in Computer Network: https://www.geeksforgeeks.org/delays-in-computer-network/ (29–01–2021)
IP address lookup: https://www.howtogeek.com/117371/how-to-find-your-computers-private-public-ip-addresses/ (29–01–2021)
iPerf: https://iperf.fr/