Today I was asked by a customer: “By taking all of our storage traffic off of our Fibre Channel network and putting it onto our IP network, won’t that cause major network congestion?”
Quick answer is no if you implement some form of traffic “Isolation”. There are many ways to achieve isolation of vSAN traffic. All of them are identical to how you would isolate NAS or iSCSI storage traffic.
vSAN traffic can be isolated using dedicate physical switches. But most customers implementations leverage existing core switches that support all other IP traffic. In this case, for vSAN, the strong recommendation is to create a dedicated VLAN for the vSAN traffic.
On the host side, you can dedicate physical NIC’s for vSAN traffic. Some customers do this but others find that shared physical 10GbE NICs (2 for redundancy) on hosts provides enough bandwidth. When doing this, it is recommend to implement vSphere Distributed Switches and then configure vSphere NIOC to allocate bandwidth shares for different vSphere traffic. Typically vSAN should be allocated 50% of the shares.
There is an updated networking design guide for more details on all of this here:
Live Virtually 
Great topic Peter.