Debugging Citrix XenDesktop & XenApp

Debugging Citrix XenDesktop &XenAppJamie Baker Manager, US Escalation TeamKapil RamlalSr. Software Maintenance EngineerMay, 2010

• Troubleshooting Theory

• Brief Architectural Review• XenApp and XenDesktop• Common Components

• Common Problem Types

• Debugging Tools and Techniques

Citrix Confidential - Do Not Distribute

Agenda

Troubleshooting Theory


Good

Facts of LifeBad

Compare them Both

• Good• Know how the system is supposed to work

• Determine how to collect the data you need

• Bad• Collect data from the bad system

• Compare the Good system with the Bad

Troubleshooting

Brief Architectural Review

Putting It All Together


SAN

XenServer

XenApp

PVS

Active Directory with roaming profiles

DesktopDelivery Controller

Virtual Machines

Authenticate

Find “best” virtual desktop

Start VM

PXE-boot VM and stream OS

Register

Connect using ICA

Acquire license and determine settings

Log in

Apply profile Deliver apps

Full range of authentication methods supported through web interface technology

Full support for SmartAccess and ICA session policies

The Desktop Delivery Controller


WIDDC Servers

VDAs(VMs, Blade PCs)

AD

ICA

Web Service

Web

Ser

vice

LDAP

VM Host(XenServer, Hyper-V, VMware)

LDAP

Licensing

User

• Desktop Delivery Controller (Broker) Farm

• Uses core XA technology

• IMA, licensing, WI, ...

• Delivers and controls access to virtual desktops

• VDA technology agnostic

• User authentication / single sign on

• VM power management

• ICA policy decision

The Desktop Delivery Controller

WIDDC Servers

VDAs(VMs, Blade PCs)

AD

ICA

Web Service

Web

Ser

vice

LDAP

VM Host(XenServer, Hyper-V, VMware)

LDAP

Licensing

User

• Worker / controller design• Few controllers / many VDAs

• DDC scales to 1000s of VDAs

• DDC not in connection path

• Dependent on Active Directory

• User authentication

• Communication security

• Controller discovery

The Desktop Delivery Controller Core Architecture

DDC ServerDDC Server

Licensing

LicenseServer

Workstation(VM or blade PC)




VDA(VM or blade PC)

VDA(VM or blade PC)

CGP SvcCGP Svc

PortICA DriversPortICA Drivers

PortICA Svc

PortICA Svc

DomainController

LDAP

DataStore

ADO

Pool Manager

Pool Manager

VM HostWeb Svc

XML Service

XML Service

HTTP

WI80, 443

DDCServers

IMA

IMAIMA

2512

DCOM,WCF

AMC2514, 8000,

DCOM

IMA

PSC

2513

MFCOM / IMAProxyMFCOM / IMAProxy WCF

8080

Controller Service

Controller Service

Desktop ServiceDesktop Service

Virtual Desktop Agent


• “Virtual Desktop Agent”• Collection of services, drivers, ...

• “PortICA”: ICA connectivity

• “Desktop Service”: web service interface communicating with DDCs

• How does it relate to XenApp?• Majority of ICA code is shared

• Does not use Terminal Services

• Major changes in: WinLogon integration, session management, USB support

XenDesktop and Active Directory


• XenDesktop relies on AD for1. Authentication of end users and admins

2. Mutual authentication of DDCs and VDAs

3. Encryption of network traffic

4. Discovery and authorization of DDCs by VDAs

• Each DDC farm can have an OU• Only used for purpose 4.

• May (but need not) contain computer accounts

• Need not be configured at root OU level

• Alternative Discovery Method

• Configure DDC identity in VDA registry (see CTX118976)

Common Components


• The same ICA client can be used to access both XenDesktop and XenApp

• Citrix Web Interface can also be used for both

• Active Directory

• XML

• IMA

• DDC/ZDC (Although roles are a bit different)

Common Problem Types

XenDesktop XenApp


•VDA Registration Failure

•VDA Connectivity / Reconnecting failure

•Hyper visor Issues

•Service hangs (DDC/Poolmgr)

•.NET Global Assembly Cache Exceptions

•CPU

•Memory

• Data Store

•Server Connectivity / Reconnecting failure

•Load Balancing

•Black Hole

•Service hangs (IMA/ZDC)

•CPU

•Memory

Debugging VDA Registration

DebuggingDebugging VDA Registration


• Use XDPing to check for time sync issues

• Check port connectivity (Telnet, XDPing, CtxPrtChk)

• Check resultant set of policy for AD inconsistencies

• Check Event Viewer

• Capture remote CDF trace using CDFControl (CTX111961)

DEMO: Capturing a remote CDF trace with CDFControlDEMO: Capturing a remote CDF trace with CDFControl

Debugging VDA Connectivity

Debugging VDA Connectivity


• Leverage XDPing (CTX123278) to rule out common causes

• Capture remote CDF trace

• Check for display driver switching issues

• Ensure no WDDM display driver is being used

• Try reducing screen resolution and color depth

The Global Assembly Cache

• Stores assemblies specifically designated to be shared by several applications

• Citrix XenDesktop VDA and DDC services use .NET, and rely on the GAC for shared assemblies


The Global Assembly Cache (GAC)

DebuggingThe Global Assembly Cache (GAC)


• When?•.NET components fail to start or encounters an exception, such as:

• How?•Enable Fusion Logging (Registry Based Setting)

• The GAC Utility•Comes with Microsoft Visual Studio IDE•Can be used to reinstall GAC components

FileNotFoundExceptionFileNotFoundException BadImageFormatExceptionBadImageFormatException FileLoadExceptionFileLoadException

The Black Hole Problem (XenApp)


The XenApp “Black Hole” Problem

• Connections are routed to the Least Loaded Server in the Farm

• An underlying problems exists on the Least Loaded server

• Least Loaded Server still responds to IMA heartbeat pings

• ZDC gets the pings and routes to the broken server, causing a "Black Hole” effect

• What are the 3 most important XenApp servers, from a connecting user's standpoint?

• How to quickly validate the health of these 3 servers?•How to monitor Farm Health using MedEvac (CTX119899)•Runs tests against:• Terminal Services

• RPC

• XML

• Least Loaded Server


The XenApp “Black Hole” Problem


What about XenDesktop?

DEMO: Xnapshot (Sneak Peek)DEMO: Xnapshot (Sneak Peek)

I got a problem

I need information A

Here it isWhen can you fix it?

I need information B

I need information X

A reasonable customerA typical customer support engineer

“He does not know what he is doing”

http://www.shutterstock.com/subscribe.mhtml

I got a problem

Have you changed anything ?

Hmmm. Not really

A capable customerA typical customer support engineer

“I better not tell him what I did”


Xnapshot tool makes it easy

Next generation customer Take-it-easy support engineer

“Smart – he knows what he is doing”We got all your information and we know what were changed.




Capturing Post-Mortem Memory Dumps

User and Kernel SpaceCapturing Post-Mortem Memory Dumps

• Windows uses 2 levels of protection to restrict access to areas of memory

• System memory is divided into 2 spaces:• User Space• Kernel Space

• Applications run in User Space

• Operating System code and Drivers run in Kernel Space

User Dump CaptureCapturing Post-Mortem Memory Dumps

• Setup a default post-mortem debugger to catch crashing applications

• How to Set the NT Symbolic Debugger as a Default Windows Postmortem Debugger (CTX105888)

• Check for the managed debugger under:• HKLM\Software\Microsoft\.NETFramework• Value: DbgManagedDebugger

System Dump CaptureCapturing Post-Mortem Memory Dumps

• Configure in Startup and Recovery

settings

• Ensure pagefile can store dump

• See MS Article cc976050 for registry

settings

System Dump CaptureCapturing Post-Mortem Memory Dumps


• Dedicated Dump Drive (Windows 7+)• Location:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\CrashControlName: DedicatedDumpFileType: REG_SZValue: Dump path, such as D:\dedicateddumpfile.sys

• How to Recover Windows Kernel Level Dump Files from Provisioned Target CTX123642

System Dump Capture ToolsCapturing Post-Mortem Memory Dumps


• The NMI switch (MS KB927069)

• Keyboard Initiated (MS FF545499)

• SystemDump 3.1 for 32-bit and 64-bit platforms (CTX111072)

API Hooking

• Try capturing a CDF trace on the particular component for deeper insight•Example: MF_Hook_SCardHook

• Try excluding the application from the hook to see how it behaves

• If it runs fine based on testing, then leave it excluded


To Hook or not to Hook

CTX107825 – HOW TO DISABLE CITRIX HOOKS ON A PER APP BASISCTX107825 – HOW TO DISABLE CITRIX HOOKS ON A PER APP BASIS

XenDesktop & XenApp Core Services

XenDesktop XenApp


DDC:

•Pool Manager Service (CdsPoolMgr.exe)

•XML Service

•IMA

•Desktop Delivery Service (CDSController.exe)

VDA:

•Workstaton Agent

•PortICA

•CtxSvcHost

•XML

ZDC:

•IMA

•XML Service

MEMBER XA SERVER:

•IMA

•XML

TechEdge Survey, Video Postings & PPTs

• The TechEdge survey will be emailed out to end-user customers

• If you complete the survey, you will be entered to win a $250 Amazon gift card. The winner will be announced June 1st.

• View TechEdge videos & PPTs on the Knowledge Center by Monday, May17th http://support.citrix.com/techedge2010

http://support.citrix.com/techedge2010

Documents

Debugging Citrix XenDesktop & XenApp