I'm working with a piece of hardware that generates a data stream packaged in UDP packets. These are sent out on a dedicated 40Gb Ethernet link to another piece of receiver hardware. There are no hubs or switches involved, just a single sender and a single receiver.
We have recently disconnected the receiver hardware and plugged that end into a commodity Linux workstation in order to receive the data stream with software. I can crank up the sender hardware and with Wireshark running on the receiving workstation I can see exactly the data I am supposed to see. However, if I crank up a separate application on the receiver that does UDP recvfrom(), it can't see the UDP packets that Wireshark can see, it just gets recvfrom() timeouts. I bind to the exact same IP address and port that Wireshark reports as the destination for the packets. The IP address is a 192.168 private subnet, and the port is in the 4000's, so it is "reserved" but not "well-known". Setting the IP address to INADDR_ANY doesn't make any difference. Whether I have Wireshark running when I try to do the UDP receive doesn't make any difference.
The software that is trying to receive works if the sender is another software application (admittedly sending to some other NIC on the workstation). I check the return status for every call (socket(), setsockopt(), bind(), recvfrom()) and they all claim success.
The NIC that is receiving the data is a Mellanox ConnectX-4 (I believe) that is set to 40Gb Ethernet mode. The MTU is set to 9000, although the packets in the data stream are much smaller (100-200 bytes).
What conditions would make a packet stream visible to Wireshark but not to other application software? Might there be a NIC setting that is not right? Could there be flags in the Ethernet/IP/UDP headers that are questionable and don't make it up the stack? Where should I look next?
As noted in the comments above, the fundamental problem was that the sending hardware was using a destination MAC address that was not the one for the receiving NIC. Turns out it, as well as the IP address, were hard-coded in the sending hardware. And the hardware was not capable of using ARP to resolve. Discovering this involved cross-checking the output of "ip addr show" with what Wireshark was reporting. Fortunately it was possible to program a new MAC address for the sender to use. The lesson here, if any, is assume nothing, check everything.