I've run into a one-way transmission issue with FreeBSD and Virtual Functions on an Intel 10GbE X520. So far we've tested:
- Ubuntu 12.04 LTS
- CentOS 6.5
- pfSense 2.1 / FreeBSD 8.3
- pfSense 2.2 / FreeBSD 10.0 (alpha)
- FreeBSD 8.3-STABLE
All are instances are running on top of KVM installed on Ubuntu 14, hosted on Intel Grizzly Pass OEM hardware with the above-mentioned card installed. The card's two 10GbE ports are connected back-to-back.
In our test case, all instances above have a VF inside VLAN 100 and are bridged via the 82599 controller's internal L2 'switch.' The Ubuntu and CentOS instances can reach each other on their VF/VLAN100 interfaces, but any FreeBSD instance spun up cannot. tcpdump on a CentOS instance and a FreeBSD instance show the FreeBSD instance send an ARP request, the CentOS instance receive it and respond, but it never reaches the FreeBSD instance. Since this is not an issue on Ubuntu or CentOS, and has persisted across two versions of FreeBSD and two Intel VF driver versions, I have to assume it's unique to FreeBSD. The examples below are between FreeBSD 8.3-STABLE and CentOS 6.5:
freebsd1# tcpdump -i ix0 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ix0, link-type EN10MB (Ethernet), capture size 96 bytes 21:24:44.137249 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:45.144856 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:46.154881 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:47.164913 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:48.174898 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:49.184903 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:50.194905 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:51.204917 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:52.214924 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:53.224947 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:54.234961 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:55.245032 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 21:24:56.255034 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 ^C 13 packets captured 13 packets received by filter 0 packets dropped by kernel
[root@centos1 ~]# tcpdump -i eth1 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 65535 bytes 17:24:44.274997 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:44.275053 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:45.282609 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:45.282651 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:46.292589 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:46.292602 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:47.302563 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:47.302576 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:48.312541 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:48.312553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:49.322596 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:49.322609 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:50.332542 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:50.332553 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:51.342602 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:51.342614 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:52.352554 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:52.352566 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:53.362614 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:53.362627 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:54.372599 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:54.372611 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 17:24:55.382681 ARP, Request who-has 192.168.100.101 tell 192.168.100.253, length 28 17:24:55.382802 ARP, Reply 192.168.100.101 is-at 52:54:00:db:57:b2 (oui Unknown), length 28 ^C 24 packets captured 24 packets received by filter 0 packets dropped by kernel
freebsd1# uname -a FreeBSD freebsd1 8.3-RELEASE FreeBSD 8.3-RELEASE #0: Mon Apr 9 21:23:18 UTC 2012 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
freebsd1# dmesg | grep Virtual CPU: QEMU Virtual CPU version 2.0.0 (2793.29-MHz K8-class CPU) ix0: <Intel(R) PRO/10GbE Virtual Function Network Driver, Version - 1.1.2> mem 0xfebf0000-0xfebf3fff,0xfebf4000-0xfebf7fff at device 5.0 on pci0
freebsd1# ifconfig ix0 ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=401bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,VLAN_HWTSO> ether 52:54:00:44:99:6c inet6 fe80::5054:ff:fe44:996c%ix0 prefixlen 64 scopeid 0x3 inet 192.168.100.253 netmask 0xffffff00 broadcast 192.168.100.255 nd6 options=3<PERFORMNUD,ACCEPT_RTADV> media: Ethernet autoselect status: active
Our issue is identical in description to this post. However, there is no "Simulated MSI Support" option in our BIOS and I believe that particular option was unique to his board based on Intel's BIOS release notes. Another individual ran into a similar problem with FreeBSD 10.0.
Note that pfSense runs v1.1.4 of the Intel VF driver. If you look here, Intel provides a link to e1000 FreeBSD VF drivers. However, no VF-specific drivers are mentioned for ixgbe. There is _vf.h and _vf.c source code in the FreeBSD tree, but we have been unable to compile newer drivers and get them to load.
I'm currently at a loss. I can provide whatever additional information may be helpful; I'm not sure what else is useful at this point. I've dumped some additional info below.
On the host, SR-IOV and IOMMU are enabled. The 10GbE card is on PCI bus 81:00:
root@ubuntu:~# dmesg | grep Intel | grep ixgbe [ 8.690398] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 3.15.1-k [ 8.690398] ixgbe: Copyright (c) 1999-2013 Intel Corporation. [ 8.950982] ixgbe 0000:81:00.0: Intel(R) 10 Gigabit Network Connection [ 9.211614] ixgbe 0000:81:00.1: Intel(R) 10 Gigabit Network Connection
VFs are enabled at the host level:
root@ubuntu:~# lspci -s 81: 81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01) 81:10.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:10.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:11.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:12.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.1 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.2 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.3 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.4 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.5 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.6 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01) 81:13.7 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
The virtual functions with several instances spun up. VF3 is centos1, VF6 is freebsd1.
root@ubuntu:~# ip link show dev eth4 5: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 90:e2:ba:47:2c:30 brd ff:ff:ff:ff:ff:ff vf 0 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 1 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 2 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 3 MAC 52:54:00:db:57:b2, vlan 100, spoof checking on, link-state auto vf 4 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 5 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 6 MAC 52:54:00:44:99:6c, vlan 100, spoof checking on, link-state auto vf 7 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 8 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 9 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 10 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 11 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 12 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 13 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 14 MAC 00:00:00:00:00:00, spoof checking on, link-state auto vf 15 MAC 00:00:00:00:00:00, spoof checking on, link-state auto
KVM is also pleased:
root@ubuntu:~# virsh nodedev-list | grep 81 pci_0000_81_00_0 pci_0000_81_00_1 pci_0000_81_10_0 pci_0000_81_10_1 pci_0000_81_10_2 pci_0000_81_10_3 pci_0000_81_10_4 pci_0000_81_10_5 pci_0000_81_10_6 pci_0000_81_10_7 pci_0000_81_11_0 pci_0000_81_11_1 pci_0000_81_11_2 pci_0000_81_11_3 pci_0000_81_11_4 pci_0000_81_11_5 pci_0000_81_11_6 pci_0000_81_11_7 pci_0000_81_12_0 pci_0000_81_12_1 pci_0000_81_12_2 pci_0000_81_12_3 pci_0000_81_12_4 pci_0000_81_12_5 pci_0000_81_12_6 pci_0000_81_12_7 pci_0000_81_13_0 pci_0000_81_13_1 pci_0000_81_13_2 pci_0000_81_13_3 pci_0000_81_13_4 pci_0000_81_13_5 pci_0000_81_13_6 pci_0000_81_13_7
The FreeBSD instance in question is on pci_0000_81_11_4, which maps to VF6 above.
KVM XML for the freebsd1's interface:
<interface type='hostdev' managed='yes'> <mac address='52:54:00:44:99:6c'/> <source> <address type='pci' domain='0x0000' bus='0x81' slot='0x11' function='0x4'/> </source> <vlan> <tag id='100'/> </vlan> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface>
FYI - we allow KVM to assign the MAC, etc. We only supply the bus, slot, and function numbers when adding a VF-based interface to the instance.
Thanks to anyone who reads and responds. I could really use some help. I've made similar posts on the FreeBSD and pfSense forums, as well as in both /r/freebsd and /r/pfsense on reddit. gonzopancho on reddit directed me to this link, which has a number of patches written by Ryan Stone. It seems like this may be a known issue. At this point we're just looking for confirmation that we aren't crazy. I've felt like this for a week.
Thanks!