# Performance Measurements of Linux, DanOS, VYOS, VPP, and Linux XDP at 100GE

Tests still being performed - VPP Still as yet untested

#### Results: Table below represent Millions of Packets Per Second (MPPS) send for forwarding via the router software vs packetloss of the end destination of expected packets. *Note:* *- droptest % represents the % of loss of the legitimate packets* *- some higher rates not tested once significant loss was demonstrated at lower levels* *- no optimisations performed on these routers unless otherwise noted below* ***Single traffic flow Test, 50 firewall policies** \- Single destination IP, single protocol, same src/dst port \- 50 Firewall Policies \- % indicates forwarding packetloss - VYOSXDP is not running any firewall policies as firewall not supported \- LinXDP running custom firewall + 802.1Q*
MPPS0.450.751.534.567.5910121518
Danos0.0%0.0%0.0%0.0%0.1%0.1%13.7%31.1%40.3%
LinXDP0.0%0.0%0.0%0.5%28.2%46.2%57.3%65.1%70.1%74.4%80.2%
VYOS21.7%53.1%63.9%88.3%92.2%94.1%
VYOSXDP \*0.0%0.0%0.0%2.7%35.5%51.7%61.7%68.1%73.0%76.3%81.5%
***GRE Test, multiple flows, GRE encapsulations received on other end, fragmentation off** \- Random Destination IPs within single /24, multiple src/dst ports, all packets GRE encap \- 50 Firewall policies \- % indicates forwarding packetloss - LinXDP running custom firewall / GRE tunnel code + 802.1Q*
MPPS0.450.751.534.567.5910121518
Danos0.0%0.0%0.0%0.0%10.8%34.3%47.8%56.7%63.1%
Linux0.0%0.0%0.0%34.7%55.7%66.5%73.5%77.7%
LinXDP0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%2.2%10.1%30.9%42.3%
VYOS0.0%0.0%0.0%7.2%36.7%51.3%
***900K Route Test, multiple flows** \- Random Destination IPs within single /24, multiple src/dst ports \- 50 Firewall Policies \- % indicates forwarding packetloss \- LinXDP running custom firewall + 802.1Q \- VYOSXDP is not running any firewall policies as firewall not supported \- 900k routes loaded into routing table*
MPPS34.567.59101215182030
Danos0%0.0%0.0%0.0%0.0%0.0%0.0%0.1%15.1%27.0%49.2%
LinXDP0%0.0%0.0%0.0%0.1%12.7%24.9%30.0%43.4%59.1%65.1%
VYOS0%9.2%28.9%43.2%
VYOSXDP\*\*0%0.0%0.0%0.0%0.0%0.1%21.0%28.9%42.6%48.8%64.5%
***DDoS Drop Test (50% traffic dropped)** \- Random Destination IPs within single /24, UDP traffic, multiple src/dst ports \- 50 Firewall Policies \- % indicates forwarding packetloss of packets that were not supposed to be dropped \- LinuxXDP running custom firewall + 802.1Q \- VYOSXDP not tested because it has no firewall capability \- 900k routes loaded into routing table*
MPPS34.567.59101215182030
Danos0.0%0.0%0.0%0.0%0.0%0.0%0.0%0.0%10.6%22.5%47.2%
LinuxXDP0.0%0.0%0.0%0.0%0.0%0.0%0.2%30.4%37.2%45.1%65.6%
VYOS0.0%8.7%26.7%39.5%
#### Test Environment & pktgen tool Network Card: mlx5\_core 0000:3b:00.1: firmware version: 16.27.6120 mlx5\_core 0000:3b:00.1: 126.016 Gb/s available PCIe bandwidth, limited by 8.0 GT/s PCIe x16 link at 0000:3a:00.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link) We wont go into building pktgen as there's plenty of doco out there on this. Just for reference purposes on how we ran pktgen: ```shell LD_LIBRARY_PATH=/usr/local/lib64/ /root/pktgen-dpdk/usr/local/bin/pktgen -l 2,4,6 -n 2 -a 3b:00.1 -d librte_net_mlx5.so -- -p 0x1 -P -m "[4:6].0" ``` **Traffic generated:** Static destination MAC (The test Target) pktgen: ``` set 0 rate 10   (This is % of 100GE, adjusted accordingly at 1% = 1.5MPPS)

set 0 size 64

set 0 count 50000000

set 0 proto udp

set 0 dst ip 10.22.23.102

set 0 src ip 10.22.22.101/24

set 0 dst mac XX:XX:XX:XX:d1:7b

set 0 src mac XX:XX:XX:XX:36:75

set 0 type ipv4 ``` **Single flow:** Single target IP address UDP traffic, 64 bytes per packet, same src/dst ports No firewall policies **Single flow, 5 firewall policies:** Single target IP address UDP traffic, 64 bytes per packet, same src/dst ports 5 firewall policies **Single flow, 50 firewall policies:** Single target IP address UDP traffic, 64 bytes per packet, same src/dst ports 50 firewall policies **Single flow, GRE tunnel:** Single target IP address UDP traffic, 64 bytes per packet, same src/dst ports 50 firewall policies GRE encap traffic and forward to static destination **Multiple flow:** 254 destination IP addresses (multi-flow) UDP traffic, 64 bytes per packet, random src ports (multi-flow) 50 firewall policies pktgen: ```shell range 0 src port 53 53 1000 1

range 0 dst ip 10.22.23.1 10.22.23.1 10.22.23.254 0.0.0.1

range 0 src ip 10.22.22.101 10.22.22.101 10.22.22.101 0.0.0.0

range 0 src mac XX:XX:XX:XX:36:75 XX:XX:XX:XX:36:75 XX:XX:XX:XX:36:75 00:00:00:00:00:00

range 0 dst mac XX:XX:XX:XX:d1:7b XX:XX:XX:XX:d1:7b XX:XX:XX:XX:d1:7b 00:00:00:00:00:00

enable 0 range ``` **900K Route Test:** 254 destination IP addresses UDP traffic, 64 bytes per packet, random src/dst ports 50 firewall policies 900K loaded route table **Drop Test:** 1 destination IP addresses, multiple ports (multi-flow) UDP traffic, 64 bytes per packet, random src/dst ports 50 firewall policies, default deny 900K loaded route table Half test traffic configured to be dropped #### DanOS Version: 2105 Built on: Fri Jun 11 11:58:32 UTC 2021 HW Model: PowerEdge R440 CPU: Intel Xeon Silver 4210R CPU @ 2.4Ghz Routing Configuration: set protocols static arp 10.22.22.101 hwaddr 'XX:XX:XX:XX:XX:XX' set protocols static arp 10.22.22.101 interface dp0p59s0f1 set protocols static route 10.22.23.0/24 next-hop 10.22.22.101 Firewall policies: set security ip-packet-filter group ipv4 ip-version ipv4 set security ip-packet-filter group ipv4 rule 1 action drop set security ip-packet-filter group ipv4 rule 1 match source ipv4 host 1.1.1.1 set security ip-packet-filter group ipv4 rule 2 action drop set security ip-packet-filter group ipv4 rule 2 match source ipv4 host 1.1.2.1 set security ip-packet-filter group ipv4 rule 3 action drop set security ip-packet-filter group ipv4 rule 3 match source ipv4 host 1.1.3.1 ...etc... set security ip-packet-filter interface dp0p59s0f1 in ipv4 For GRE test: set interfaces tunnel tun0 address 10.90.4.102/24 set interfaces tunnel tun0 encapsulation gre set interfaces tunnel tun0 local-ip 10.22.22.102 set interfaces tunnel tun0 remote-ip 10.22.22.101 set protocols static route 10.22.23.0/24 next-hop 10.90.4.101 #### VyOS $ show system cpu CPU Vendor: GenuineIntel Model: Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz Total CPUs: 1,3,5,7,9,11,13,15,17,19 Sockets: 2 Cores: 10 Threads: 1 Current MHz: 1000.128 $ show system memory Total: 62.54 GB Free: 61.17 GB Used: 1.38 GB $ show version Version: VyOS 1.4-rolling-202203150317 Release train: sagitta Built by: autobuild@vyos.net Built on: Tue 15 Mar 2022 03:17 UTC Build UUID: 9da98191-be0b-42e1-937a-97fb016b22ac Build commit ID: f2655e2ae72e8c Architecture: x86\_64 Boot via: installed image System type: bare metal Hardware vendor: Dell Inc. Hardware model: PowerEdge R440 Hardware S/N: XXXXXXXXX Hardware UUID: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Copyright: VyOS maintainers and contributors Routing Configuration: set protocols static arp 10.22.22.101 hwaddr 'XX:XX:XX:XX:XX:XX' set protocols static route 10.22.23.0/24 next-hop 10.22.22.101 Firewall Policies: set firewall name TEST\_IN rule 1 action 'drop' set firewall name TEST\_IN rule 1 destination address 1.1.1.1 set firewall name TEST\_IN rule 2 action 'drop' set firewall name TEST\_IN rule 2 destination address 1.1.2.1 set firewall name TEST\_IN rule 3 action 'drop' set firewall name TEST\_IN rule 3 destination address 1.1.3.1 set firewall name TEST\_IN rule 4 action 'drop' set firewall name TEST\_IN rule 4 destination address 1.1.4.1 set firewall name TEST\_IN rule 5 action 'drop' set firewall name TEST\_IN rule 5 destination address 1.1.5.1 ..... etc ..... set interfaces ethernet eth2 firewall in name TEST\_IN For GRE test: set interfaces tunnel tun0 address '10.90.4.102/24' set interfaces tunnel tun0 encapsulation 'gre' set interfaces tunnel tun0 remote '10.22.22.101' set interfaces tunnel tun0 source-address '10.22.22.102' set protocols static route 10.22.23.0/24 next-hop 10.90.4.101