Citrix XenServer Slow Network Performance

13 Shares

People new to the virtualization space could get very frustrated when they find out that their Hypervisor of choice preforms much worse than a bare-metal installation. Haven’t the vendors and independent analysis like Project Virtual Reality Check stated that there is no longer any excuses for not running Citrix XenApp on a Hypervisor. The latest numbers shows that the performance are almost the same on XenServer, VMware and Hyper-V.

That being said, to get these high performances requires that you have done your homework and implemented all the Best Practices. Unfortunately there is way to many bugs out there and I’m not even going to start…but I have almost begged Citrix to pay more attention to their own Support Forum in this post.

In this post I’m going to share some important tunings related to poor network performance on Citrix XenServer 5.x family.

Disable Task Offload :

For Windows 2003 VM’s you will need to create a registry key to DisableTaskOffload – REG_DWORD = 1 under HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameter. The easiest way is to create a Group Policy to push this registry key.

Disable CheckSum Offload :

The second fix you should try if you still have poor network performance is to Disable Checksum on the XenServer interfaces, both the Virtual (VIF) and the Physical (PIF). Be aware that you don’t need to restart the XenServer or the VM’s. This script will do this automatically on all you network interfaces in your XenServer Pool.

Note :

Please note that this script is provided “as is” and that using this is at your own risk. Please feel free to leave comments below regarding bugs & suggested improvements in future releases.

 

13 Shares

Automation Framework Community Edition

The fastest way to build your lab environment.

Virtual Expo

Friday 30th of September 2022

68 thoughts on “Citrix XenServer Slow Network Performance”

  1. Regarding the disable task offload key –
    What is the actual effect of that key? By which I mean, if I set it via GPO and it affects a non-virtualized server, will I be doing damage to anything?

    Reply
    • Hi Jeremy,

      I would recommend testing this on VM’s you have problems with. The best would be GPO Preferences – Item-Level Targeting to e.g. check if OS = Windows 2003 and Citrix Tools is installed.

      The Citrix Support Forum thread.

      Eric

      Reply
    • Thanks Evert,

      According to the KB this applies to all VM’s running on XenServer 5.0 and 5.5

      Eric

      Reply
    • Hi Spear,

      You could just copy & paste it into the XenServer Console or do SSH to the XenServer.

      Reply
    • Hi Roberto,

      I would recommend you forwarding the Citrix Forum Thread to @CitrixSupport on Twitter

      Reply
  2. Trond,

    I´ve got some Windows 2k3 and Windows 2k8 VMs running on the Xenserver 5.5 update 2 and Xenserver 5.6.
    These settings apply to Xenserver 5.5 update 2?
    I did network tests with iperf and had good results, like this:
    [ ID] Interval Transfer Bandwidth
    [1860] 0.0-10.0 sec 113 MBytes 94.6 Mbits/sec.

    Thanks,

    Reply
    • Hi Luiz,

      This apply to XenServer 5.x and Provisioning Services VM’s running Windows 2003 x32. If you don’t experience slow network performance I would recommend you staying with your current configuration.

      Reply
  3. If I have a pool of 9 XenServers, do I just run the checksum script/commands on 1 XenServer or all 9?

    Reply
    • Hi Ray,

      As long as all the XS servers are in the same pool you’ll only need to run it from one XenServer console. Be aware that we are setting checksum off on both physical and virtual interfaces, which means you’ll need to run the script each time you add a new VM. The best solution would be to create a script that does this automatically.

      Reply
  4. I have been using XenServer for a little while now, first on 5.5 and now 5.6. I have always had an issue where uploading file from a workstation to a Server 2003 VM was fine, but downloading from that VM was slow. If, instead, I consoled into the VM and “pushed” the file to the workstation it worked speedily.

    I tried the registry fix and now I’m getting full bandwidth in both directions. Thanks!

    Reply
  5. xenapp plugin freezes when i access it on a pvs xen desktop image

    I have 6 xen servers on 6 servers in a cisco enclosure and a 1 gig connection

    what can i do

    Reply
    • Hi Hanno,

      Need some more information here on XD version, Plugin etc. The Citrix Online or Offline plugin is freezing inside of the XD when you try to do a ICA in ICA session?

      Reply
  6. Hi
    Regarding the script.
    I just spoke to Citrix and the say that you have to reboot the servers before the changes will work.
    So now I’m a little confused.

    Reply
    • Hi Thomas, probably depends on who you are talking to at Citrix. Anyway it have worked for me without reboot. Just keep in mind that you have to run this script every time you add a new VM, because the script is also setting this on the VIF adapter (Virtual).

      Reply
  7. Hi,

    Can you elaborate on the Network Slowness that this is meant to address? In my situation I have two XenServer hosts running v5.6.0-31188p with each running 2 P2V converted VM’s running XenApp 5. Both Hosts are connected to a SAN via a separate Gb LAN. For some reason when more than 2 users land on the same server at the same time (give or take a minute) it can take as long as 10 minutes for them to get logged in. But if a single user logs on it can be a fast as 35 seconds. However I don’t have this problem on my Physical XenApp servers.

    Any thoughts would be appreciated.

    Reply
        • Eric,

          Thank you so much for the assitance you provided in your email.

          Once I was able to test and verify that my network was functiioning as expected, I was able to key in on one of your suggestions pertaining to the vCPU assignment. I had never read or came across anything mentioning this very important detail until you contacted me.

          As I stated in my original posting, I had converted a Physical Xenapp server to a Virtual Machine (4) and other than removing the left-over drivers, in my mind the options that remained (memory config and CPU’s etc) should have provided a snappy workload that either met or exceeded the physical server’s performance. What I exerienced was less than stellar. But after scaling back the vCPU’s to 2 (from 4), all of my login issues were gone and they’ve been fine since. I was even able to scale back the memory consumption to save even more resources on my hosts. Truth be told, my Xenapp VM’s are snappier than my Physical Xenapp servers.

          So thank you for your help and keep up the great work you’re doing here.

          Reply
  8. Eirik –

    I have an issue where I have 915Blades connected through a 10GB network. Running benchmarks on my vm’s I noticed that my disk I/O is really bad. I get about 20MB/sec average. I have vm’s on EQLogic and NetApp and still get bad disk I/O. I’m wondering if implementing the Checksum Offload would help resolve my problem. Have you ever encountered what I’ve just written? Would like your feedback.

    Reply
    • Hi Lee,

      Did you ever find a resolution to this based on Eirik’s recommendation? We have a pool of M915’s each with 4x 10Gb connections (2x multipathed to our SAN and 2x active/passive network for VMs) and are experiencing the exact same issue as you with VMs on XenServer 6.0.2. Disk performance even when system is mostly idle (i.e.only a couple VMs per host) is well below 30Mb on a VM. Boot times especially are slow.

      Thanx,
      David

      Reply
  9. Hi Erik

    I have a similar problem as scott:

    I have 3 Hosts running xenserver 6

    If one VM is on the same host as the fileserver I get a performance problem from that vm connecting to the fileserver, slow IO!
    If I migrate the VM to one of the other two Hosts the problem disappears, but I can’t find the bottleneck…

    Thanks in advance
    Yves

    Reply
  10. I have 3 xenservers running 5.5 update 2. I have 6 2003 R2 x86 vms running xenapp 4.5. I have enabled the disabletaskoffload a while back per citrix recommendation> I cant not find the documentation of what my issue was i just have a vague memory. Lately we have been having extreme slowdowns. Like 30 seconds to click start type of stuff. Other times its less significant. All theses vms boot from an equallogic san. We can not find any hints of performance issues on the SAN or iscsi switches. I am really at a loss. Does this sound like it could be related to the issues your address with the script? This has been beating me up for a few months now. its coming to a head where it is causing so many productivity issues i am considering rebuilding a physical farm. It often happens to a few servers at a time. the vms usually do not report unusual CPU or memory useage during the slowness. I am at a loss.

    Reply
      • XS6 requires disks which I would have to redo my setup as I’m using embedded Dell xenserver.

        I have strong suspicion this fixed it or at least improved it. Dell Equal logic did extensive troubleshooting and monitoring. The SAN and Force10 switches were not loaded very much at all.

        Your a wizard if this fix holds up over the next week. THanks !!!!

        Reply
  11. Hi, my network problem ocured after i switched from 5.6 to 6.0.2 only on a SBS2003 VM. switching between bridge and openvswitch mode didnt help, but disabling Task Offload solved it. thanks

    Reply
  12. Are you sure we dont need to do a reboot? After running the script the checksumming still says it is on.

    [root@xenserver-2 ~]# ethtool -k vif26.0
    Offload parameters for vif26.0:
    rx-checksumming: on
    tx-checksumming: on
    scatter-gather: on
    tcp-segmentation-offload: on
    udp-fragmentation-offload: off
    generic-segmentation-offload: on
    generic-receive-offload: off
    large-receive-offload: off
    [root@xenserver-2 ~]#

    Reply
  13. Trond – thanks for the great post and included script. This seems to focus on tx/rx and other posts I’ve read on the forums & elsewhere seem to concentrate on the other offload settings. What of turning off generic-segmentation-offload, and the other offload settings on all pif/vifs on the XenServer?
    Thanks

    Reply
      • Hi Trond, yes, see you in Anaheim this year! Business is good so far, thanks. Hope the same is with you. I think I have seen this post, it’s one of the best ones I’ve seen. I just wish I had a little more understanding of how each of these items worked in depth and what their impact is, but I guess it depends a lot on hardware. Seems like the best course is to simply disable them all in most cases.
        Thanks again for your work and blog posts.
        Cheers

        Reply
  14. Hi,

    Quite new to scripting. How do I get your script into a XenServer console to be able to run it? Also just read the website you linked to in your last post and using ethtool -k i can see that i still have rx and tx checksum on. Need to get checksum off for both PIF and VIF as we have some slow networking issues on the XenServers and VMs.

    Reply
  15. OK, perfect. If I run the script on one XenServer, will it do the VIF and PIF on all the VMs in the pool as well?

    Reply
  16. Is it recommended to reboot each host and VM once this script has been completed? I have had a few intermittent network problems today still

    Reply
  17. I have a performance issue with accessing application on the Xenapp server from Window 7 workstation. Xenapp server – presentation server for windows 4.5.8 running on VM ware server. The win7 station takes 5 minutes to login into applications. We tried lot of workstation settings without any luck. The MS office 10 applications are very slow.(5 mins for word loading)

    Reply
      • Eric,

        we didn’t. Other applications are running fine in the VM environment.
        The performance issue varies workstation to workstation within the same subnet. We are comparing the workstation and no luck with identifying affending processes.

        Reply
  18. In our office we are in the process of upgrading the Autocad version from 2009 to 2011. During Installation the server (VM) and the Host both completely hangs off while changing the installation medias (out of total 4 medias, only the first media is running good). The full host is to be powered off and powered on, when Installation starts from the beginning and as usual the hanging problem persists while changing the media.

    When the media has been tested and verified separately in another machine it was found ok and also works well.

    Version = Xenserver 5.6 (Xentools also installed)
    VM = Windows 2008 R2, Alloted Ram 14 GB

    Kindly guide us on the same.

    Reply
  19. Hi Eirik, we recently upgraded our xenservers to xenserver 6.2, now the Network Performance Problems found their way back to our vm s. When we run the script to disable rx tx and gso -offloading just nothing habbened. When we set the taskoffloading to off with ethtool -k it seems to be off, but when we restart our Servers ethtool -K tells us that the taskoffloading is on again. Is there another way to disable the offloading configurations. We use xenserver 6.2 xenapp 6.5 and xendesktop 7. Thank you for your Help and your Website.

    Reply
  20. Hi there, do you know if this affects XS 6.0.2 as well? Cant seem to find much about this

    Reply
  21. Hi Eirik

    We are running 3 XS 5.6 sp2 hosts in a pool with two bonded nics for LAN and two for iSCSI. We have two Windows server 2003 VM’s that take 8 hours to backup 200gb of data to tape or disk. We identified very slow transfer to / from the 2003 VM’s whereas to 2008 VM’s on the same hosts the transfer was 10 times faster. After running ethtool tcp checksumming is on. Will running your script fix my issues ? What if something goes wrong how can I undo the changes ?

    Reply
  22. Hi Eirkik,

    We receiving slow citrix performance, installed citrix receiver and all of them from Win7 laptops, but much faster when using desktop machine, and it happens mostly during work time, checked both configuration laptop and desktop same, any expectation?

    Reply
  23. Hi, What I’m doing wrong, because I get error:
    line 2: VIFLIST: command not found
    line 10: PIFLIST: command not found

    My script looks like this:

    echo Settings checsum off on VIFs
    VIFLIST = xe vif-list | grep “uuid ( RO) ” | awk ‘{print $5}’
    for VIF in $VIFLIST
    do
    echo Setting ethtool-tx=off and ethtool-rx=off on $VIF
    xe vif-param-set uuid=$VIF other-config:ethtool-tx=”off”
    xe vif-param-set uuid=$VIF other-config:ethtool-rx=”off”
    done
    echo Setting checksum off on PIFs
    PIFLIST = xe pif-list | grep “uuid ( RO) ” | awk ‘{printg $5}’
    for PIF in $PIFLIST
    do
    echo Setting ethtool-tx=off and ethtool-rx=off on $PIF
    xe pif-param-set uuid=$PIF other-config:ethtool-tx=”off”
    xe pif-param-set uuid=$PIF other-config:ethtool-rx=”off”
    done

    Reply

Leave a Comment