Clicky

Citrix XenServer Slow Network Performance

14 Flares Twitter 0 Facebook 9 Google+ 5 Pin It Share 0 LinkedIn 0 Email -- Buffer 0 14 Flares ×

Citrix XenServer 5.6 Console 300x178 Citrix XenServer Slow Network Performance

People new to the virtualization space could get very frustrated when they find out that their Hypervisor of choice preforms much worse than a bare-metal installation. Haven’t the vendors and independent analysis like Project Virtual Reality Check stated that there is no longer any excuses for not running Citrix XenApp on a Hypervisor. The latest numbers shows that the performance are almost the same on XenServer, VMware and Hyper-V.

That being said, to get these high performances requires that you have done your homework and implemented all the Best Practices. Unfortunately there is way to many bugs out there and I’m not even going to start…but I have almost begged Citrix to pay more attention to their own Support Forum in this post.

In this post I’m going to share some important tunings related to poor network performance on Citrix XenServer 5.x family.

Disable Task Offload :

For Windows 2003 VM’s you will need to create a registry key to DisableTaskOffload – REG_DWORD = 1 under HKLMSYSTEMCurrentControlSetServicesTcpipParameter. The easiest way is to create a Group Policy to push this registry key.

DisableTaskOffload2 300x93 Citrix XenServer Slow Network Performance

Disable CheckSum Offload :

The second fix you should try if you still have poor network performance is to Disable Checksum on the XenServer interfaces, both the Virtual (VIF) and the Physical (PIF). Be aware that you don’t need to restart the XenServer or the VM’s. This script will do this automatically on all you network interfaces in your XenServer Pool.

[sourcecode language=”plain”]
echo Setting checksum off on VIFs
VIFLIST=`xe vif-list | grep "uuid ( RO) " | awk ‘{print $5}’`
for VIF in $VIFLIST
do
echo Setting ethtool-tx=off and ethtool-rx=off on $VIF
xe vif-param-set uuid=$VIF other-config:ethtool-tx="off"
xe vif-param-set uuid=$VIF other-config:ethtool-rx="off"
done
echo Setting checksum off on PIFs
PIFLIST=`xe pif-list | grep "uuid ( RO) " | awk ‘{print $5}’`
for PIF in $PIFLIST
do
echo Setting ethtool-tx=off and ethtool-rx=off on $PIF
xe pif-param-set uuid=$PIF other-config:ethtool-tx="off"
xe pif-param-set uuid=$PIF other-config:ethtool-rx="off"
done
[/sourcecode]

Note :

Please note that this script is provided “as is” and that using this is at your own risk. Please feel free to leave comments below regarding bugs & suggested improvements in future releases.

 

Like what you've read?

If so join over 20.000 people and receive FREE Citrix XenApp
and XenDesktop Training, blog updates and information on
upcoming webinars. Click the red button below!
 Citrix XenServer Slow Network Performance

64 Responses to Citrix XenServer Slow Network Performance

  1. Jeremy says:

    Regarding the disable task offload key –
    What is the actual effect of that key? By which I mean, if I set it via GPO and it affects a non-virtualized server, will I be doing damage to anything?

    • admin says:

      Hi Jeremy,

      I would recommend testing this on VM’s you have problems with. The best would be GPO Preferences – Item-Level Targeting to e.g. check if OS = Windows 2003 and Citrix Tools is installed.

      The Citrix Support Forum thread.

      Eric

  2. evert van maanen says:

    if you use provisioning service in combination with Xen servers also check this key.
    HKEY_LOCAL_MACHINESystemCurrentControlSetServicesxenevtchnSetFlags 30000 hex.
    This effectively disables TSO and DMA on the Xen Tools.
    http://support.citrix.com/article/CTX125157

    • admin says:

      Thanks Evert,

      According to the KB this applies to all VM’s running on XenServer 5.0 and 5.5

      Eric

  3. Jeremy says:

    Thanks Eric!

  4. Spear says:

    How do you execute the mentioned script ? In a .sh file ?

    Thanks

    • Trond Eirik Håvarstein says:

      Hi Spear,

      You could just copy & paste it into the XenServer Console or do SSH to the XenServer.

  5. Roberto says:

    Hi guys thank you for the TIPs

    can you kindly share your experience on this:
    http://forums.citrix.com/message.jspa?messageID=1482685#1482685

    Thank you if you will

    Roberto

    • Trond Eirik Haavarstein says:

      Hi Roberto,

      I would recommend you forwarding the Citrix Forum Thread to @CitrixSupport on Twitter

  6. Luiz says:

    Trond,

    I´ve got some Windows 2k3 and Windows 2k8 VMs running on the Xenserver 5.5 update 2 and Xenserver 5.6.
    These settings apply to Xenserver 5.5 update 2?
    I did network tests with iperf and had good results, like this:
    [ ID] Interval Transfer Bandwidth
    [1860] 0.0-10.0 sec 113 MBytes 94.6 Mbits/sec.

    Thanks,

    • Trond Eirik Haavarstein says:

      Hi Luiz,

      This apply to XenServer 5.x and Provisioning Services VM’s running Windows 2003 x32. If you don’t experience slow network performance I would recommend you staying with your current configuration.

  7. Ray Hall says:

    If I have a pool of 9 XenServers, do I just run the checksum script/commands on 1 XenServer or all 9?

    • Trond Eirik Haavarstein says:

      Hi Ray,

      As long as all the XS servers are in the same pool you’ll only need to run it from one XenServer console. Be aware that we are setting checksum off on both physical and virtual interfaces, which means you’ll need to run the script each time you add a new VM. The best solution would be to create a script that does this automatically.

  8. Francois says:

    I have been using XenServer for a little while now, first on 5.5 and now 5.6. I have always had an issue where uploading file from a workstation to a Server 2003 VM was fine, but downloading from that VM was slow. If, instead, I consoled into the VM and “pushed” the file to the workstation it worked speedily.

    I tried the registry fix and now I’m getting full bandwidth in both directions. Thanks!

  9. Hanno says:

    xenapp plugin freezes when i access it on a pvs xen desktop image

    I have 6 xen servers on 6 servers in a cisco enclosure and a 1 gig connection

    what can i do

    • Trond Eirik Haavarstein says:

      Hi Hanno,

      Need some more information here on XD version, Plugin etc. The Citrix Online or Offline plugin is freezing inside of the XD when you try to do a ICA in ICA session?

  10. Thomas says:

    Hi
    Regarding the script.
    I just spoke to Citrix and the say that you have to reboot the servers before the changes will work.
    So now I’m a little confused.

    • Trond Eirik Haavarstein says:

      Hi Thomas, probably depends on who you are talking to at Citrix. Anyway it have worked for me without reboot. Just keep in mind that you have to run this script every time you add a new VM, because the script is also setting this on the VIF adapter (Virtual).

  11. Rodolfo says:

    Hello,

    This post, save my life!!

    :)

  12. Scott says:

    Hi,

    Can you elaborate on the Network Slowness that this is meant to address? In my situation I have two XenServer hosts running v5.6.0-31188p with each running 2 P2V converted VM’s running XenApp 5. Both Hosts are connected to a SAN via a separate Gb LAN. For some reason when more than 2 users land on the same server at the same time (give or take a minute) it can take as long as 10 minutes for them to get logged in. But if a single user logs on it can be a fast as 35 seconds. However I don’t have this problem on my Physical XenApp servers.

    Any thoughts would be appreciated.

  13. Lee says:

    Eirik –

    I have an issue where I have 915Blades connected through a 10GB network. Running benchmarks on my vm’s I noticed that my disk I/O is really bad. I get about 20MB/sec average. I have vm’s on EQLogic and NetApp and still get bad disk I/O. I’m wondering if implementing the Checksum Offload would help resolve my problem. Have you ever encountered what I’ve just written? Would like your feedback.

    • Hi Lee, I have not experienced this. Please try a iperf against physical and virtual host to see if there’s amy difference. cmd : iperf -c ipadr -f M -t 30 -i 1. Download iperf here : https://nocweboldcst.ucf.edu/files/iperf.exe

    • David says:

      Hi Lee,

      Did you ever find a resolution to this based on Eirik’s recommendation? We have a pool of M915’s each with 4x 10Gb connections (2x multipathed to our SAN and 2x active/passive network for VMs) and are experiencing the exact same issue as you with VMs on XenServer 6.0.2. Disk performance even when system is mostly idle (i.e.only a couple VMs per host) is well below 30Mb on a VM. Boot times especially are slow.

      Thanx,
      David

  14. Yves says:

    Hi Erik

    I have a similar problem as scott:

    I have 3 Hosts running xenserver 6

    If one VM is on the same host as the fileserver I get a performance problem from that vm connecting to the fileserver, slow IO!
    If I migrate the VM to one of the other two Hosts the problem disappears, but I can’t find the bottleneck…

    Thanks in advance
    Yves

  15. Eric says:

    I have 3 xenservers running 5.5 update 2. I have 6 2003 R2 x86 vms running xenapp 4.5. I have enabled the disabletaskoffload a while back per citrix recommendation> I cant not find the documentation of what my issue was i just have a vague memory. Lately we have been having extreme slowdowns. Like 30 seconds to click start type of stuff. Other times its less significant. All theses vms boot from an equallogic san. We can not find any hints of performance issues on the SAN or iscsi switches. I am really at a loss. Does this sound like it could be related to the issues your address with the script? This has been beating me up for a few months now. its coming to a head where it is causing so many productivity issues i am considering rebuilding a physical farm. It often happens to a few servers at a time. the vms usually do not report unusual CPU or memory useage during the slowness. I am at a loss.

    • Hi, I would recommend trying the script and/or doing a re-installation of XS6. Remember to update XS tools if going the XS6 route.

      • Eric says:

        XS6 requires disks which I would have to redo my setup as I’m using embedded Dell xenserver.

        I have strong suspicion this fixed it or at least improved it. Dell Equal logic did extensive troubleshooting and monitoring. The SAN and Force10 switches were not loaded very much at all.

        Your a wizard if this fix holds up over the next week. THanks !!!!

  16. peter says:

    Hi, my network problem ocured after i switched from 5.6 to 6.0.2 only on a SBS2003 VM. switching between bridge and openvswitch mode didnt help, but disabling Task Offload solved it. thanks

  17. Viola Matts says:

    I had experienced these problems before and referred to eGInnovations. They were real good.

  18. ipeng says:

    Are you sure we dont need to do a reboot? After running the script the checksumming still says it is on.

    [root@xenserver-2 ~]# ethtool -k vif26.0
    Offload parameters for vif26.0:
    rx-checksumming: on
    tx-checksumming: on
    scatter-gather: on
    tcp-segmentation-offload: on
    udp-fragmentation-offload: off
    generic-segmentation-offload: on
    generic-receive-offload: off
    large-receive-offload: off
    [root@xenserver-2 ~]#

  19. Mike says:

    Trond – thanks for the great post and included script. This seems to focus on tx/rx and other posts I’ve read on the forums & elsewhere seem to concentrate on the other offload settings. What of turning off generic-segmentation-offload, and the other offload settings on all pif/vifs on the XenServer?
    Thanks

    • Hi Mike, hope business is going great for you, see you in Anaheim 2013? Did some troubleshooting a couple of weeks ago regarding slow performance on HP BL 465C G7 blades, having an open ticket with Citrix support. When doing research if found this awesome post discussing what your asking about. http://rackerua.com/2012/07/xenserver-network-tuning/

      • Mike says:

        Hi Trond, yes, see you in Anaheim this year! Business is good so far, thanks. Hope the same is with you. I think I have seen this post, it’s one of the best ones I’ve seen. I just wish I had a little more understanding of how each of these items worked in depth and what their impact is, but I guess it depends a lot on hardware. Seems like the best course is to simply disable them all in most cases.
        Thanks again for your work and blog posts.
        Cheers

  20. Ashley Poxon says:

    Hi,

    Quite new to scripting. How do I get your script into a XenServer console to be able to run it? Also just read the website you linked to in your last post and using ethtool -k i can see that i still have rx and tx checksum on. Need to get checksum off for both PIF and VIF as we have some slow networking issues on the XenServers and VMs.

  21. Ashley Poxon says:

    OK, perfect. If I run the script on one XenServer, will it do the VIF and PIF on all the VMs in the pool as well?

  22. Ashley Poxon says:

    I meant the VIF for the all the VMs in the pool

  23. Ashley Poxon says:

    Is it recommended to reboot each host and VM once this script has been completed? I have had a few intermittent network problems today still

  24. deepti says:

    What is the fix for XenApp6.5 provisoned servers on XenServer 5.6?

  25. ravi says:

    I have a performance issue with accessing application on the Xenapp server from Window 7 workstation. Xenapp server – presentation server for windows 4.5.8 running on VM ware server. The win7 station takes 5 minutes to login into applications. We tried lot of workstation settings without any luck. The MS office 10 applications are very slow.(5 mins for word loading)

    • Hi Ravi, have you tried the different NIC drivers on VMware? E100 vs the other ones?

      • RAvi says:

        Eric,

        we didn’t. Other applications are running fine in the VM environment.
        The performance issue varies workstation to workstation within the same subnet. We are comparing the workstation and no luck with identifying affending processes.

  26. SURESH BABU. S says:

    In our office we are in the process of upgrading the Autocad version from 2009 to 2011. During Installation the server (VM) and the Host both completely hangs off while changing the installation medias (out of total 4 medias, only the first media is running good). The full host is to be powered off and powered on, when Installation starts from the beginning and as usual the hanging problem persists while changing the media.

    When the media has been tested and verified separately in another machine it was found ok and also works well.

    Version = Xenserver 5.6 (Xentools also installed)
    VM = Windows 2008 R2, Alloted Ram 14 GB

    Kindly guide us on the same.

  27. Jens Herrmann says:

    Hi Eirik, we recently upgraded our xenservers to xenserver 6.2, now the Network Performance Problems found their way back to our vm s. When we run the script to disable rx tx and gso -offloading just nothing habbened. When we set the taskoffloading to off with ethtool -k it seems to be off, but when we restart our Servers ethtool -K tells us that the taskoffloading is on again. Is there another way to disable the offloading configurations. We use xenserver 6.2 xenapp 6.5 and xendesktop 7. Thank you for your Help and your Website.

  28. Ashley Poxon says:

    Hi there, do you know if this affects XS 6.0.2 as well? Cant seem to find much about this

  29. Andy Lee says:

    Hi Eirik

    We are running 3 XS 5.6 sp2 hosts in a pool with two bonded nics for LAN and two for iSCSI. We have two Windows server 2003 VM’s that take 8 hours to backup 200gb of data to tape or disk. We identified very slow transfer to / from the 2003 VM’s whereas to 2008 VM’s on the same hosts the transfer was 10 times faster. After running ethtool tcp checksumming is on. Will running your script fix my issues ? What if something goes wrong how can I undo the changes ?

Leave a reply

Name: Email:
14 Flares Twitter 0 Facebook 9 Google+ 5 Pin It Share 0 LinkedIn 0 Email -- Buffer 0 14 Flares ×