In Search of Automotive Ethernet TSN Solutions: Results From Our Hands-On Lab

Ethernet has been deemed to be a promising technology by the IT world over the last few decades. Its speed and capacity have improved to meet the growing needs of consumers. However, the same can’t be said of its use in the automotive industry. The automotive industry uses technologies like the CAN bus and other protocols that provide unique benefits and help vehicles run well. For example, the CAN bus is cost-effective and simple to design, even though it’s not as fast as Ethernet. Its proven success as a technology makes it stand out in its field.

So, why do we need automotive Ethernet? It has special features that sets it apart from other technologies. Automotive Ethernet gives vehicles a big advantage by enabling vehicles to tap into a wide variety of bespoke open-source software solutions. It lets vehicles combine multiple services and signals while following specific protocols. This makes it easier to add different features and create a broad range of custom services.

Table 1. A comparison of available in-vehicle protocols
Note: We modified and enhanced this table created by ChatGPT.

Notably, when Ethernet is used in vehicles, a key observation is that the original design excludes important safety features that are deemed nonessential for everyday vehicle use. In contrast to the CAN bus, where messages are sorted based on their priority level to ensure that significant messages are sent in a timely manner, Ethernet is not configured to prioritize important messages that might cause safety issues, so much so that even the transmission of critical messages experiences long delays. For example, when a driver steps on the brakes while there’s an incoming phone call, traditional Ethernet might send the message too late because the network is busy. As a result, Time-Sensitive Networking (TSN) was suggested — initially meant for operational technology (OT) — to deal with this and other, similar concerns. TSN was designed to make sure that critical messages are delivered on time, and automotive Ethernet has at times proven to be a better fit for critical scenarios.

Table 2. Automotive Ethernet–related IEEE TSN standards (adapted from TSN Task Group)

Rather than merely talking about TSN standards, we stepped our research up a notch by making a small, low-cost TSN network laboratory. Our purpose with the lab is to assist people in creating similar labs and comprehending the TSN network’s capabilities. It entails more than just reading the standards, as we also established a small-scale testing environment and provided an option for integrating MACsec (IEEE 802.1AE) and TSN together to improve TSN security.

We sought to learn more about TSN standards by looking at software and hardware implementations beyond layer 2 or the data link network layer. In automotive Ethernet, layer 1 or the physical layer typically uses 10BASE-T1S or 100/1000BASE-T1, which is not a traditional RJ45 but Single-Pair Ethernet (SPE). In this research, we used RJ45 instead of SPE but we still achieved the same testing results because our goal was to test TSN, which is built on and goes beyond layer 2 or the data link layer.

We focused on IEEE 802.1AS and IEEE 802.1Qbv because they’re commonly found in larger-scale practical implementations.

Designing a TSN laboratory

Resources

After careful study and consideration, we collected the necessary parts for our project. We obtained two Intel network i201 cards and one SK-KR260-G-ED development board, in addition to adapting an old Dell 7040 PC to serve as a TSN node along with the KR260 board. The total cost of these materials was US$640.

Table 3. The price list of the purchased items for the TSN laboratory

Hardware

We selected the KR260 board as our hardware solution for TSN, given its integrated field-programmable gate arrays (FPGAs) that effectively manage Precision Time Protocol (PTP) and Time-Aware Shaper (TAS) functionalities, while the Dell 7040 PC serves as our software solution for TSN. To enable TSN features, we used the i201 network card as the baseline requirement for constructing a TSN network on the PC. We also took advantage of Linux’s qdisc Time-Aware Priority Shaper (TAPRIO) and  Earliest TxTime First (ETF) mechanisms. However, we must acknowledge a known issue: TAPRIO and ETF cannot operate simultaneously with PTP. To address this, we acquired an additional i201 card to create an alternative channel dedicated to PTP functionality.

Figure 1. The KR260 development board

Figure 2. The hardware connectivity diagram

Software

In terms of software design, we decided not to follow the KR260 Network Time Shaper Function (802.1Qbv demo) as a guide. Using the tsn-talker-listener isn’t a common method when working with existing software solutions. It can be quite challenging to tweak the software to use a low-level socket library supporting UDP or TCP, so we chose an approach similar to what’s used in current software solutions instead. By setting up a VLAN interface and connecting it to different IP subnets, we ensured compatibility and made software development easier. This approach is beneficial for various software environments like LXC (Linux containers) and hypervisors, as it simplifies the process of connecting specific network interfaces and providing necessary resources. Additionally, it allows for the establishment of system security policies to maintain strong security protection.

Configuration

A quick overview of the 802.1Qbv standard is in order to help us better understand the following configuration steps for a smoother implementation process. In simple terms, the 802.1Qbv standard — Time-Aware Shaper (TAS) — efficiently organizes network traffic by identifying different streams using methods like VLAN’s Priority Code Point (PCP) or Differentiated Services Code Point (DSCP). This helps in categorizing and prioritizing data based on its quality-of-service (QoS) requirements. The transmission of data is then synchronized with predefined time slots, ensuring smooth and reliable communication.

Figure 3. An example of an IEEE 802.1Qbv schedule

In our hands-on lab, we used the 802.1Q VLAN’s PCP field to make things work smoothly. Every packet needed a VLAN header. When we set the PCP value of the VLAN to a specific number, the packet would be sent according to a preset schedule. To make this happen, we created a special VLAN interface with its own ID in Linux. By assigning a particular PCP value to it, we ensured that data was transferred through this specific VLAN and followed the desired timing. This allowed us to control and manage the flow of data effectively in our lab environment.

Table 4. The network configuration

According to the standard, we set up two different priority levels in TSN: Scheduled Traffic (ST) and Best Effort (BE), each with its own cycle time. In theory, the transmission is split into two parts, making it look like there is quality of service. But the way these priority levels act is different, showing how TSN can manage network traffic based on how important they are.

We captured packets from both solutions during data transmission and made a line chart showing traffic changes over time and with 802.1Q VLAN IDs. It shows how TAS separates traffic based on our configuration using VLAN ID and PCP field. At first, the chart seems to say the software solution is better than its hardware counterpart, but it’s important to remember that the chart doesn’t show everything, as there are more factors and measurements to consider besides those indicated in the chart.

Figure 4. A timeline visualization of 802.1Q VLAN ID

Also, when setting up both solutions, we encountered some tricky problems, like packets being resent over and over when a TCP connection was disconnected. At first, we thought it was a one-time issue, but it kept happening, pointing to a bigger problem. We haven’t completely solved it yet, but we have managed to lessen the issue by using a specific firewall rule to stop the strange behavior and avoid more complications. (Given the focus of this blog post, a detailed discussion about troubleshooting is best done in a separate post as a likely follow-up to our research findings here.)

Benchmark

We did a lot of tests with iperf3, covering many different situations using both TCP and UDP. Our main goal was to see how the network performed at its highest capacity. Notably, UDP does not need confirmation when sending packets, while TCP does. Because of this difference, we foresee different results in real-world situations.

In our test scenario, we simultaneously launched two iperf3 servers on the target machine. As the stress test begins, two iperf3 clients initiate connections to the target machine via different IPs on the VLAN interface. This setup ensures that ST and BE traffic are sent concurrently, effectively putting the client’s TAS to the test and demonstrating its capabilities in managing network traffic.

We have simplified the iperf3 commands as follows:

• On the target machine:
  • iperf3 -s -p [listen port]
• On the client machine:
  • TCP - iperf3 -c [target machine IP] -p [target port]
  • UDP - iperf3 -c [target machine IP] -p [target port] -u -b 1G

The following are the results of our tests.

Figure 5. UDP from KR260 to PC within a TSN
Average ST: Speed 60.6 MB/s (508.9 Mbps), Error Rate: 0%, Jitter: 0.027 ms
Average BE: Speed 28.4 MB/s (240.6 Mbps), Error Rate: 0%, Jitter: 0.055 ms

Figure 6. TCP from KR260 to PC within a TSN
Average ST: Speed 57.6 MB/s (497.2 Mbps), Error Times: 64
Average BE: Speed 31.7 MB/s (226.2 Mbps), Error Times: 9

Figure 7. UDP from PC to KR260 within a TSN
Average ST: Speed 74.0 MB/s (621.3 Mbps), Error Rate: 0.01%, Jitter: 0.066 ms (Excluded problematic traffic, 0.019 ms)
Average BE: Speed 29.6 MB/s (249.0 Mbps), Error Rate: 10%, Jitter: 0.065 ms (Excluded problematic traffic, 0.053 ms)

Figure 8. TCP from PC to KR260 within a TSN
Average ST: Speed 64.9 MB/s (544.2 Mbps), Error: 74 times
Average BE: Speed 31.7 MB/s (266.2 Mbps), Error: 9 times

Based on our findings, we’ve come to these conclusions:

• ST has better jitter performance than BE traffic in both hardware and software solutions.
• The hardware solution using the KR260 board has almost stable network traffic on both UDP and TCP, but it’s slightly slower than the software solution. On the other hand, the software solution has some difficulties when pushed to the limit, causing errors and a 10% packet loss rate in UDP testing.
• Even in just one test scenario, the software solution has unexpected drops in performance, suggesting potential issues.

In summary, after taking all factors into account, it becomes clear that the hardware solution is the more favorable option, which aligns with our initial expectations.

MACsec

Using plain text for network traffic isn’t always the safest choice. Although we know about alternatives like IPsec or TLS for securing Ethernet communications, it’s essential to explore MACsec as well. Our goal is to use this technology to protect TSN networks at the layer 2 or L2 level, giving an additional layer of security for data transmission.

In a nutshell, MACsec — Media Access Control (MAC) Security aka IEEE 802.1AE — functions at the L2 network layer and predominantly uses AES-128-GCM for encrypting user data by default (meaning the encryption algorithm can be altered). After encryption, an additional header, 802.1AE SecTAG, is inserted at the beginning of the payload, and an integrity check value (ICV) is appended at the end, as shown in Figure 9. Furthermore, MACsec is specifically engineered to provide protection against various security threats, including man-in-the-middle, playback, and wiretapping attacks.

Figure 9. 802.1AE (MACsec) encryption

Problem

We observed that in line with the standards, when MACsec works with TSN, encrypting the VLAN header makes the PCP field unreadable, which causes problems in integrating TAS smoothly. However, there’s a method, called Clear Tag Mode or ClearTag Feature, that avoids encrypting the VLAN header, keeping it readable. This way, TAS can understand the PCP field and take the necessary actions. However, this technology isn’t available on the Linux platform yet. (We will explore other solutions as we continue to dig deeper into MACsec in future research.)

In theory, network traffic packets should put the MACsec header first, followed by the VLAN header. This results in MACsec encrypting and rendering the VLAN header unreadable. But since Clear Tag Mode, which would have kept the VLAN header readable, isn’t available, we can try another option. We can switch the header order and set up the Linux virtual interface to put the VLAN header before the MACsec header. We found this method, called WAN-Mode, on the Nokia website. This clever change lets TSN work well and keeps the needed encryption and protection from MACsec, resulting in a smooth and effective solution that works perfectly.

Figure 10. 802.1AE and 802.1Q (WAN-Mode)

Figure 11. The captured 802.1AE and 802.1Q BE packet

AES

For sure, AES needs a lot of CPU power for both encryption and decryption processes. Since the two processes are alike and use the same amount of power, we tested only encryption. Before benchmarking performance is done, it’s important to understand the speed limitations of the process for each of the hardware component we have chosen. A straightforward way to achieve this is by using OpenSSL with this command:

• openssl speed -elapsed -evp aes-128-gcm

Since MACsec’s usual algorithm is AES-128-GCM, we also tested this specific algorithm to find out the topmost speed possible for CPU encryption. The results show an important fact: One CPU core of the KR260 platform can encrypt about 30 MB/s (240 Mbps) only. This means that it could slow things down in later tests if there’s no AES hardware offload.

Table 5. Testing AES efficiency for each CPU core

Benchmark

We used iperf3 again to do stress tests on both platforms. The following are our test results.

Figure 12. UDP from KR260 to PC within a TSN and MACsec
Average ST: Speed 20.1 MB/s (168.7 Mbps), Error Rate: 0%, Jitter: 0.084 ms
Average BE: Speed 20.8 MB/s (174.7 Mbps), Error Rate: 0%, Jitter: 0.088 ms

Figure 13. TCP from KR260 to PC within a TSN and MACsec
Average ST: Speed 24.2 MB/s (202.7 Mbps), Error Times: 0
Average BE: Speed 24.7 MB/s (206.1 Mbps), Error Times: 6

Figure 14. UDP from PC to KR260 within a TSN and MACsec
Average ST: Speed 13.8 MB/s (116.3 Mbps), Error Rate: 84%, Jitter: 0.015 ms
Average BE: Speed 12.6 MB/s (106.6 Mbps), Error Rate: 82%, Jitter: 0.029 ms

Figure 15. TCP from PC to KR260 within a TSN and MACsec
Average ST: Speed 13.8 MB/s (116.3 Mbps), Error: 3,400 times
Average BE: Speed 12.6 MB/s (106.2 Mbps), Error: 2,082 times

As we probed further, we found a major issue: Using MACsec with TSN is troublesome without a powerful CPU or hardware support. We therefore conclude that:

• KR260 has a hard time keeping up with TSN speed. It works steadily and smoothly, but the speed limit is only around the AES benchmark.
• The performance test results show that MACsec encryption can use multiple CPUs when sending traffic, but it uses only one CPU when receiving traffic. This is why KR260’s ST and BE can reach a speed of 400 Mbps together when sending, but the receiving speed dips by almost half to about 220 Mbps.
• The software solution still shows a different latency between ST and BE, but it’s limited by KR260. This also causes big errors during transmission. While the software can send traffic fast, KR260 can’t decrypt and receive it quickly enough.

Even though combining the use of MACsec and that of TSN can be tough without the right hardware or setup, it’s worth bearing in mind that MACsec is great at protecting network traffic. Using MACsec to achieve the best performance and strong security at the same time in different situations makes it a worthwhile option.

Conclusion

We started looking into simpler hardware and software solutions for TSN networks. Through this research journey, we realized that the solution isn’t perfect, as we first thought it would be. As in any engineering challenge, finding a good solution needs time to implement the needed tweaks and improvements; there are no quick fixes to this situation. However, this process showed us the potential of TSN networks for a better vision of software-defined vehicles (SDVs), paving the way for better efficiency and more innovation in the automotive industry. As we keep exploring the details of TSN, we pursue our commitment to tackle challenges and finding new ways to make the most of this technology.