People new to the virtualization space could get very frustrated when they find out that their Hypervisor of choice preforms much worse than a bare-metal installation. Haven’t the vendors and independent analysis like Project Virtual Reality Check stated that there is no longer any excuses for not running Citrix XenApp on a Hypervisor. The latest numbers shows that the performance are almost the same on XenServer, VMware and Hyper-V.
That being said, to get these high performances requires that you have done your homework and implemented all the Best Practices. Unfortunately there is way to many bugs out there and I’m not even going to start…but I have almost begged Citrix to pay more attention to their own Support Forum in this post.
In this post I’m going to share some important tunings related to poor network performance on Citrix XenServer 5.x family.
Disable Task Offload :
For Windows 2003 VM’s you will need to create a registry key to DisableTaskOffload – REG_DWORD = 1 under HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameter. The easiest way is to create a Group Policy to push this registry key.
Disable CheckSum Offload :
The second fix you should try if you still have poor network performance is to Disable Checksum on the XenServer interfaces, both the Virtual (VIF) and the Physical (PIF). Be aware that you don’t need to restart the XenServer or the VM’s. This script will do this automatically on all you network interfaces in your XenServer Pool.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
echo Setting checksum off on VIFs VIFLIST=`xe vif-list | grep "uuid ( RO) " | awk '{print $5}'` for VIF in $VIFLIST do echo Setting ethtool-tx=off and ethtool-rx=off on $VIF xe vif-param-set uuid=$VIF other-config:ethtool-tx="off" xe vif-param-set uuid=$VIF other-config:ethtool-rx="off" done echo Setting checksum off on PIFs PIFLIST=`xe pif-list | grep "uuid ( RO) " | awk '{print $5}'` for PIF in $PIFLIST do echo Setting ethtool-tx=off and ethtool-rx=off on $PIF xe pif-param-set uuid=$PIF other-config:ethtool-tx="off" xe pif-param-set uuid=$PIF other-config:ethtool-rx="off" done |
Note :
Please note that this script is provided “as is” and that using this is at your own risk. Please feel free to leave comments below regarding bugs & suggested improvements in future releases.
Regarding the disable task offload key –
What is the actual effect of that key? By which I mean, if I set it via GPO and it affects a non-virtualized server, will I be doing damage to anything?
Hi Jeremy,
I would recommend testing this on VM’s you have problems with. The best would be GPO Preferences – Item-Level Targeting to e.g. check if OS = Windows 2003 and Citrix Tools is installed.
The Citrix Support Forum thread.
Eric
if you use provisioning service in combination with Xen servers also check this key.
HKEY_LOCAL_MACHINESystemCurrentControlSetServicesxenevtchnSetFlags 30000 hex.
This effectively disables TSO and DMA on the Xen Tools.
http://support.citrix.com/article/CTX125157
Thanks Evert,
According to the KB this applies to all VM’s running on XenServer 5.0 and 5.5
Eric
Thanks Eric!
How do you execute the mentioned script ? In a .sh file ?
Thanks
Hi Spear,
You could just copy & paste it into the XenServer Console or do SSH to the XenServer.
Hi guys thank you for the TIPs
can you kindly share your experience on this:
http://forums.citrix.com/message.jspa?messageID=1482685#1482685
Thank you if you will
Roberto
Hi Roberto,
I would recommend you forwarding the Citrix Forum Thread to @CitrixSupport on Twitter
Trond,
I´ve got some Windows 2k3 and Windows 2k8 VMs running on the Xenserver 5.5 update 2 and Xenserver 5.6.
These settings apply to Xenserver 5.5 update 2?
I did network tests with iperf and had good results, like this:
[ ID] Interval Transfer Bandwidth
[1860] 0.0-10.0 sec 113 MBytes 94.6 Mbits/sec.
Thanks,
Hi Luiz,
This apply to XenServer 5.x and Provisioning Services VM’s running Windows 2003 x32. If you don’t experience slow network performance I would recommend you staying with your current configuration.
If I have a pool of 9 XenServers, do I just run the checksum script/commands on 1 XenServer or all 9?
Hi Ray,
As long as all the XS servers are in the same pool you’ll only need to run it from one XenServer console. Be aware that we are setting checksum off on both physical and virtual interfaces, which means you’ll need to run the script each time you add a new VM. The best solution would be to create a script that does this automatically.
I have been using XenServer for a little while now, first on 5.5 and now 5.6. I have always had an issue where uploading file from a workstation to a Server 2003 VM was fine, but downloading from that VM was slow. If, instead, I consoled into the VM and “pushed” the file to the workstation it worked speedily.
I tried the registry fix and now I’m getting full bandwidth in both directions. Thanks!
Awesome Francois
xenapp plugin freezes when i access it on a pvs xen desktop image
I have 6 xen servers on 6 servers in a cisco enclosure and a 1 gig connection
what can i do
Hi Hanno,
Need some more information here on XD version, Plugin etc. The Citrix Online or Offline plugin is freezing inside of the XD when you try to do a ICA in ICA session?
Hi
Regarding the script.
I just spoke to Citrix and the say that you have to reboot the servers before the changes will work.
So now I’m a little confused.
Hi Thomas, probably depends on who you are talking to at Citrix. Anyway it have worked for me without reboot. Just keep in mind that you have to run this script every time you add a new VM, because the script is also setting this on the VIF adapter (Virtual).
Hello,
This post, save my life!!
🙂
Hi,
Can you elaborate on the Network Slowness that this is meant to address? In my situation I have two XenServer hosts running v5.6.0-31188p with each running 2 P2V converted VM’s running XenApp 5. Both Hosts are connected to a SAN via a separate Gb LAN. For some reason when more than 2 users land on the same server at the same time (give or take a minute) it can take as long as 10 minutes for them to get logged in. But if a single user logs on it can be a fast as 35 seconds. However I don’t have this problem on my Physical XenApp servers.
Any thoughts would be appreciated.
Hi Scott, please check the email I sent you
Trond, I’m sorry but I did not get your email. I’ve checked my AntiSpam system and didn’t see anything come in from you. You sent it to [email protected]?
Thanks,
Eric,
Thank you so much for the assitance you provided in your email.
Once I was able to test and verify that my network was functiioning as expected, I was able to key in on one of your suggestions pertaining to the vCPU assignment. I had never read or came across anything mentioning this very important detail until you contacted me.
As I stated in my original posting, I had converted a Physical Xenapp server to a Virtual Machine (4) and other than removing the left-over drivers, in my mind the options that remained (memory config and CPU’s etc) should have provided a snappy workload that either met or exceeded the physical server’s performance. What I exerienced was less than stellar. But after scaling back the vCPU’s to 2 (from 4), all of my login issues were gone and they’ve been fine since. I was even able to scale back the memory consumption to save even more resources on my hosts. Truth be told, my Xenapp VM’s are snappier than my Physical Xenapp servers.
So thank you for your help and keep up the great work you’re doing here.
Glad I could help you solve this Scott
Eirik –
I have an issue where I have 915Blades connected through a 10GB network. Running benchmarks on my vm’s I noticed that my disk I/O is really bad. I get about 20MB/sec average. I have vm’s on EQLogic and NetApp and still get bad disk I/O. I’m wondering if implementing the Checksum Offload would help resolve my problem. Have you ever encountered what I’ve just written? Would like your feedback.
Hi Lee, I have not experienced this. Please try a iperf against physical and virtual host to see if there’s amy difference. cmd : iperf -c ipadr -f M -t 30 -i 1. Download iperf here : https://nocweboldcst.ucf.edu/files/iperf.exe
Hi Lee,
Did you ever find a resolution to this based on Eirik’s recommendation? We have a pool of M915’s each with 4x 10Gb connections (2x multipathed to our SAN and 2x active/passive network for VMs) and are experiencing the exact same issue as you with VMs on XenServer 6.0.2. Disk performance even when system is mostly idle (i.e.only a couple VMs per host) is well below 30Mb on a VM. Boot times especially are slow.
Thanx,
David
Hi Erik
I have a similar problem as scott:
I have 3 Hosts running xenserver 6
If one VM is on the same host as the fileserver I get a performance problem from that vm connecting to the fileserver, slow IO!
If I migrate the VM to one of the other two Hosts the problem disappears, but I can’t find the bottleneck…
Thanks in advance
Yves
Hi Yves. Are these VM’s converted from physical to virtual? For XS6 there’s also exists a lot of hotfixes for different network cards and iSCSI devices.
Hello,
I have the exact same as Yves decribes – network speed (tested by simple file copying) in and out of VMs on any host is good, but network speed BETWEEN VMs on the same host is very poor.
Was this resolved, and if so how?
Many thanks
Adrian
Hi Adrian. Have you tried running this script on the XenServer host?
..not yet. Should I run them first?
Yes you should 🙂
I have 3 xenservers running 5.5 update 2. I have 6 2003 R2 x86 vms running xenapp 4.5. I have enabled the disabletaskoffload a while back per citrix recommendation> I cant not find the documentation of what my issue was i just have a vague memory. Lately we have been having extreme slowdowns. Like 30 seconds to click start type of stuff. Other times its less significant. All theses vms boot from an equallogic san. We can not find any hints of performance issues on the SAN or iscsi switches. I am really at a loss. Does this sound like it could be related to the issues your address with the script? This has been beating me up for a few months now. its coming to a head where it is causing so many productivity issues i am considering rebuilding a physical farm. It often happens to a few servers at a time. the vms usually do not report unusual CPU or memory useage during the slowness. I am at a loss.
Hi, I would recommend trying the script and/or doing a re-installation of XS6. Remember to update XS tools if going the XS6 route.
XS6 requires disks which I would have to redo my setup as I’m using embedded Dell xenserver.
I have strong suspicion this fixed it or at least improved it. Dell Equal logic did extensive troubleshooting and monitoring. The SAN and Force10 switches were not loaded very much at all.
Your a wizard if this fix holds up over the next week. THanks !!!!
Hi, my network problem ocured after i switched from 5.6 to 6.0.2 only on a SBS2003 VM. switching between bridge and openvswitch mode didnt help, but disabling Task Offload solved it. thanks
Thanks for your comment Peter, sure it will be helpful to others
I had experienced these problems before and referred to eGInnovations. They were real good.
Are you sure we dont need to do a reboot? After running the script the checksumming still says it is on.
[root@xenserver-2 ~]# ethtool -k vif26.0
Offload parameters for vif26.0:
rx-checksumming: on
tx-checksumming: on
scatter-gather: on
tcp-segmentation-offload: on
udp-fragmentation-offload: off
generic-segmentation-offload: on
generic-receive-offload: off
large-receive-offload: off
[root@xenserver-2 ~]#
Hi, I always recommend doing a reboot to be sure, but I have seen on some cases this is unnecessary.
Trond – thanks for the great post and included script. This seems to focus on tx/rx and other posts I’ve read on the forums & elsewhere seem to concentrate on the other offload settings. What of turning off generic-segmentation-offload, and the other offload settings on all pif/vifs on the XenServer?
Thanks
Hi Mike, hope business is going great for you, see you in Anaheim 2013? Did some troubleshooting a couple of weeks ago regarding slow performance on HP BL 465C G7 blades, having an open ticket with Citrix support. When doing research if found this awesome post discussing what your asking about. http://rackerua.com/2012/07/xenserver-network-tuning/
Hi Trond, yes, see you in Anaheim this year! Business is good so far, thanks. Hope the same is with you. I think I have seen this post, it’s one of the best ones I’ve seen. I just wish I had a little more understanding of how each of these items worked in depth and what their impact is, but I guess it depends a lot on hardware. Seems like the best course is to simply disable them all in most cases.
Thanks again for your work and blog posts.
Cheers
Hi,
Quite new to scripting. How do I get your script into a XenServer console to be able to run it? Also just read the website you linked to in your last post and using ethtool -k i can see that i still have rx and tx checksum on. Need to get checksum off for both PIF and VIF as we have some slow networking issues on the XenServers and VMs.
Hi Ashley, you just copy and paste the script into the XenServer console via SSH (Putty) or XenCenter.
OK, perfect. If I run the script on one XenServer, will it do the VIF and PIF on all the VMs in the pool as well?
I meant the VIF for the all the VMs in the pool
You’ll need to run it on every XS host + each time you add a new VM. You should also check out this new blog post on how to troubleshoot slow performance with iPerf – https://www.xenappblog.com/2012/citrix-xenserver-network-performance/
Is it recommended to reboot each host and VM once this script has been completed? I have had a few intermittent network problems today still
What is the fix for XenApp6.5 provisoned servers on XenServer 5.6?
Please provide some more info
I have a performance issue with accessing application on the Xenapp server from Window 7 workstation. Xenapp server – presentation server for windows 4.5.8 running on VM ware server. The win7 station takes 5 minutes to login into applications. We tried lot of workstation settings without any luck. The MS office 10 applications are very slow.(5 mins for word loading)
Hi Ravi, have you tried the different NIC drivers on VMware? E100 vs the other ones?
Eric,
we didn’t. Other applications are running fine in the VM environment.
The performance issue varies workstation to workstation within the same subnet. We are comparing the workstation and no luck with identifying affending processes.
In our office we are in the process of upgrading the Autocad version from 2009 to 2011. During Installation the server (VM) and the Host both completely hangs off while changing the installation medias (out of total 4 medias, only the first media is running good). The full host is to be powered off and powered on, when Installation starts from the beginning and as usual the hanging problem persists while changing the media.
When the media has been tested and verified separately in another machine it was found ok and also works well.
Version = Xenserver 5.6 (Xentools also installed)
VM = Windows 2008 R2, Alloted Ram 14 GB
Kindly guide us on the same.
Instead of installing the software from an ISO mounted to the XenServer I would try installing it from a file share.
Hi Eirik, we recently upgraded our xenservers to xenserver 6.2, now the Network Performance Problems found their way back to our vm s. When we run the script to disable rx tx and gso -offloading just nothing habbened. When we set the taskoffloading to off with ethtool -k it seems to be off, but when we restart our Servers ethtool -K tells us that the taskoffloading is on again. Is there another way to disable the offloading configurations. We use xenserver 6.2 xenapp 6.5 and xendesktop 7. Thank you for your Help and your Website.
Hi Jens, have not played around with XS6.2 – I’ve and many more have switch focus over to Hyper-V with SCVMM integration.
Hi there, do you know if this affects XS 6.0.2 as well? Cant seem to find much about this
Affect Windows 2003 VM’s, use the registry key above. Please test with iPerf to check if the performance is good or bad.
Hi Eirik
We are running 3 XS 5.6 sp2 hosts in a pool with two bonded nics for LAN and two for iSCSI. We have two Windows server 2003 VM’s that take 8 hours to backup 200gb of data to tape or disk. We identified very slow transfer to / from the 2003 VM’s whereas to 2008 VM’s on the same hosts the transfer was 10 times faster. After running ethtool tcp checksumming is on. Will running your script fix my issues ? What if something goes wrong how can I undo the changes ?
Hi Andy, should fix your problem, this is an issue with W2K3 VM’s. If that doesn’t work just change the script from off to on and rerun.
Hi Eirkik,
We receiving slow citrix performance, installed citrix receiver and all of them from Win7 laptops, but much faster when using desktop machine, and it happens mostly during work time, checked both configuration laptop and desktop same, any expectation?
Very strange, have you checked the power configuration? So you’re saying performance from Desktop is faster than laptop against a Citrix solution running on XenServer?
Hi, What I’m doing wrong, because I get error:
line 2: VIFLIST: command not found
line 10: PIFLIST: command not found
My script looks like this:
echo Settings checsum off on VIFs
VIFLIST = xe vif-list | grep “uuid ( RO) ” | awk ‘{print $5}’
for VIF in $VIFLIST
do
echo Setting ethtool-tx=off and ethtool-rx=off on $VIF
xe vif-param-set uuid=$VIF other-config:ethtool-tx=”off”
xe vif-param-set uuid=$VIF other-config:ethtool-rx=”off”
done
echo Setting checksum off on PIFs
PIFLIST = xe pif-list | grep “uuid ( RO) ” | awk ‘{printg $5}’
for PIF in $PIFLIST
do
echo Setting ethtool-tx=off and ethtool-rx=off on $PIF
xe pif-param-set uuid=$PIF other-config:ethtool-tx=”off”
xe pif-param-set uuid=$PIF other-config:ethtool-rx=”off”
done
Hi, currently don’t have XenServer running but will check during December.