«

»

Mar 26 2010

How to troubleshoot PCoIP performance

Recently I have been doing some performance troubleshooting on PCoIP display protocol. Well, actually not much, as PCoIP work out-of-the-box for most VMware View deployments without any intervention.

The protocol has a kind of a healing and adaptive nature. If you are not familiar with the progressive build and adaptive nature of PCoIP display protocol read this first.

Now that you are familiarized let’s see how to initiate a troubleshooting process.

A common complaint is poor performance, causing mouse lag and screen delays. Most often this type of problem is diagnosed as lack of network bandwidth for multimedia (MMR) or high display resolution with multi-monitor support. However, to find out exactly what it’s happening it’s necessary to observe the PCoIP Server log files and the network. I’ll leave the network for the networkers.

There are 2 log files that should provide enough information to start the troubleshooting. They are located in the virtual desktop at ‘c:\Documents and Settings\All Users\Application Data\VMware\VDM\logs’.

UPDATE (06/12/2010) – In Windows 7 the path has changed to ‘c:\program data\application data\VMware\VDM\logs’.

pcoip_server.txt – All transactions related to encoding, virtual channels, image management, bandwidth, etc…

pcoip_agent.txt – All client side transactions, such as connectivity, handshake etc..

In this post I’ll talk about the pcoip_server log file. The log initiate with a number of information collected from the environment. Attention to the following:

03/25/2010, 09:35:12.107> LVL:0 RC:   0        MGMT_ENV :cTERA_MGMT_CFG::Registry setting parameter pcoip.max_link_rate  =   512

VMware View unsupported Admin templates for AD GPO allows you to configure the maximum bandwidth consumption per PCoIP session. Make sure this is not in use or set to a reasonable number.

The extract below demonstrate a very constrained network with PCoIP dynamically adjusting the maximum transmission to 64Kb/s. The log also provide information about the round trip time, variance, decode rate and RTO. The higher the RTO the better is the performance. The VGMAC also identifies that there is packet loss of 0.09% from the client to the virtual desktop.

14:21:00.822> LVL:2 RC: 0 MGMT_PCOIP_DATA :Tx thread info: round trip time (ms) = 6, variance = 1, rto = 107
14:21:17.321> LVL:2 RC: 0 MGMT_IMG :log: cur_s 1 max_s 30 tbl 5 bwc 0.05 bwt 0.45 fps 2.03 fl_ps 30.33
14:21:17.321> LVL:2 RC: 0 MGMT_IMG :log: chg pix: 34688, chg pix not motion: 34688
14:21:17.321> LVL:2 RC: 0 MGMT_IMG :log: delta bits encoded: 970136, delta build bits encoded: 581248.
14:21:17.321> LVL:2 RC: 0 MGMT_IMG :log: enc bits/pixel – 27.97, enc bits/sec – 32329.12, enc MPix/sec – 0.00, decode rate est (MBit/sec) – 1.51
14:21:25.946> LVL:2 RC: 0 MGMT_PCOIP_DATA :Tx thread info: bw limit = 64, plateau = 62.7, avg tx = 5.5, avg rx = 2.4 (KBytes/s)
14:21:25.946> LVL:1 RC: 0 VGMAC :Stat frms: R=000000/000000/334586 T=014734/298585/136743 (A/I/O) Loss=0.00%/0.09% (R/T)

A scenario like that could be found in a sliced network with QoS prioritising all known traffic, and leaving minimal bandwidth to be shared across all unknown traffic. The solution could be as simple as adding prioritisation to PCoIP at the QoS. However, every case is different so engage your network team.

Now, below you will see a extract with better network throughput. However, because of packet loss identified (Loss=0.45%/0.21% (R/T)) due to reduced available bandwidth at that given moment PCoIP dynamically reduced the bandwidth consumption to 176Kb/s.

14:16:17.430> LVL:2 RC: 0 MGMT_PCOIP_DATA :Tx thread info: bw limit = 234, plateau = 235.0, avg tx = 194.7, avg rx = 3.4 (KBytes/s)
14:16:17.430> LVL:1 RC: 0 VGMAC :Stat frms: R=000000/000000/027895 T=011578/041311/010192 (A/I/O)
Loss=0.45%/0.21% (R/T)
14:16:17.805> LVL:1 RC: 0 MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438

Using the same technique PCoIP will dynamically increase the bandwidth if it is required by the protocol and if it is available on the network.

The sequence below demonstrate that there were two decrease moments but in between those moments the bandwidth has been automatically increased. The gradual increase of bandwidth consumption is not shown in the log file.

14:16:17.805> LVL:1 RC: 0 MGMT_PCOIP_DATA :BW: Decrease (loss) old = 234.9982 new = 176.8438
14:16:18.383> LVL:2 RC: 0 MGMT_IMG :rcv nak 2 seq_id 184 disp 0 fsp 5 f_ref 33
14:16:18.383> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: NAK for fsp 5 seq 33. (ref id=82) Ack ref available, recode from input
14:16:18.415> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: IPLP: recode from input: fsp=5 refvld=1 seq=34
14:16:18.524> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: Encoder clearing recode for fsp 5 seq 34 (ref id=11)
14:16:21.305> LVL:2 RC: 0 MGMT_IMG :rcv nak 2 seq_id 222 disp 0 fsp 4 f_ref 41
14:16:21.305> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: NAK for fsp 4 seq 41. (ref id=82) Ack ref available, recode from input
14:16:21.430> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: IPLP: recode from input: fsp=4 refvld=1 seq=42
14:16:21.587> LVL:2 RC: 0 MGMT_IMG :SW_HOST_IPC: Encoder clearing recode for fsp 4 seq 42 (ref id=61)
14:16:22.524> LVL:1 RC: 0 MGMT_PCOIP_DATA :BW: Decrease (loss) old = 207.2631 new = 143.0061

I hope this give you an introduction of how to troubleshoot PCoIP performance.

I’ll come back with more information soon.

14 comments

3 pings

Skip to comment form

  1. Adam

    PCoIP is certainly a great display protocol. However in some scenarios of slow remote connections (like over certain WANs) there may be issues where PCoIP doesn’t function quite as well. In those cases, you can complement the VMware View deployment with Ericom Blaze, a software-based RDP acceleration and compression product that provides improved performance over WANs. Besides delivering higher frame rates and reducing screen freezes and choppiness, Blaze accelerates RDP performance by up to 10-25 times, while significantly reducing network bandwidth consumption over low-bandwidth/high latency connections.

    You can use VMware View with PCoIP for your LAN and fast WAN users, and at the same time use VMware View with Blaze over RDP for your slow WAN users. This combined solution can provide enhanced performance in both types of environments, letting you get the best out of VMware View for your users.

    Read more about Blaze and download a free evaluation at:
    http://www.ericom.com/ericom_blaze.asp?URL_ID=708

    Adam

  2. JJ

    Hi,

    Mi test on a simetric 784 kpbs line, are not satisfied. PcoIP are heavy for this small line and i can´t change the line or buy an accelerator that costs more than the line. Is correct try to find alternatives to vmware like XenDesktop?

    Kind regards.

  3. Andre Leibovici

    @JJ
    784 Kbps is bandwidth enough for at least 4 to 5 high quality PCoIP sessions. Off course I am assuming not video is being used. What could you retrieve from your log files?

  4. Clayton Said

    Hi All,

    We’ve been experiencing PCoIP WAN issues and I was directed to this article. This is most interesting! Unfortunately I cannot locate the pcoip_agent.txt and pcoip_server.txt files. Do I need to enable logging?

    Thanks in advance.

  5. Andre Leibovici

    @Clayton,
    I have seen this problem before when a old version of VMware View agent is installed in your master image. Uninstall, reboot, Install the latest release of VMware View Agent in your master image. Take a new snapshot and recompose your Desktop or Pool. That should fix the problem. Let me know the result.

  6. Clayton Said

    @Andre,

    Thanks – installing the latest version of the View Agent resolved the issue.

  7. liyifei

    hi,
    How to enable/disable logs in view 4.5?

  8. Andre Leibovici

    Hi Lilyfei, there is a registry entry to enable/disable logginfg. However logging is enabled by Default. If you don’t see the log files I would recomend you to uninstall/reinstall VMware View agent on you ParentVM and re-deploy the pool. I have seen this issue before and I believe it’s caused by the in-place agent upgrade.

  9. Andrew

    If you are setting the max bandwidth for PCoIP sessions via group policy, this setting is in kilobits per second.

    When you review the pcoip_server log, the bw limit value is converted to kilobytes per second. We had set GPO for 512 kilobits and bw limit reported as 64. Changed GPO to 1024Kb and bw limit changed to 128.

  10. Johan Nieuwhoff

    Hi,
    Based on my results i must conclude that higher RTO values are related to less round trip times.
    For example: roundtrip times between 200 and 560 results in RTO between 600 and 1300
    and roundtrip times between 15 and 30 results in RTO values between 117 and 140.

  11. Andre Leibovici

    @Johan Nieuwhoff
    RTO is just a combination a few different metrics. Higher RTO means something is not behaving well and you should be looking at each individual metric, such as latency, roudtrip, jitter, bandwidth, loss etc..

    Andre

  12. Johan Nieuwhoff

    Johan Nieuwhoff :Hi,Based on my results i must conclude that higher RTO values are related to less round trip times.For example: roundtrip times between 200 and 560 results in RTO between 600 and 1300and roundtrip times between 15 and 30 results in RTO values between 117 and 140.

    I switches my conclusions as I see now. (good morning)

  13. Subversive

    Hi, I am in the testing phase of my View deployment, and we just cannot get good video playback. I have been reading through many of the pages on your site (very good stuff, btw, thanks!) and my RTO values are between 100 and 115, which should indicate pretty good performance. However, video playback is consistently choppy, although sometimes better than others. It’s very frustrating. Any other ideas on what I could look at?

  14. Andy

    what do you have your frame rate set to in the GPO?

  1. latoga labs | Virtualization Round Up 20100608

    […] Andre Leibovici provides some great tips on How to troubleshoot PCoIP performance […]

  2. The Breaking Point » Blog Archive » PCOIP: Taming The Beast On The WAN

    […] is congested and will throttle its bandwidth. Your can see this is action with a great article from http://myvirtualcloud.net/?p=751. However, the protocol is still greedy and I don’t think it throttles the bandwidth quick enough. […]

  3. Virtualized all the way to my desktop | Splunk Blogs

    […] Interpreting PCoIP logs is explained by Andre in this article and troubleshooting PCoIP performance is found here […]

Leave a Reply