Troubleshooting 101 for HorizonLinus Bourque, VMware, IncJoseph Cooper, VMware, Inc

EUC8243R

#EUC8243R

CONFIDENTIAL 2

• This presentation may contain product features that are currently under development.

• This overview of new technology represents no commitment from VMware to deliver these features in any generally available product.

• Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind.

• Technical feasibility and market demand will affect final delivery.

• Pricing and packaging for any new technologies or features discussed or presented have not been determined.

Disclaimer

CONFIDENTIAL 3

You Are Here

1 Introduction to Troubleshooting

2 Troubleshooting Tools

3 The Top Ten

4 Where To From Here

CONFIDENTIAL 4

Learner Objectives

Use a structured approach to solve configuration and operational problems.

Apply troubleshooting methodology to logically diagnose faults and improve troubleshooting efficiency.

Document all steps taken to resolve the problem.

1 2 3

By the end of this session, you should be able to meet the following objectives:

CONFIDENTIAL 5

Definition of a System Problem

System problems arise from various sources:

Configuration Issues

Resource Contention

Network Attacks

Software Bugs or

Interoperability

Hardware Failures or

Compatibility

A system problem is a fault in a system, or one of its components, that negatively affects the services needed for normal production.

CONFIDENTIAL 6

VMware Severity Definitions and Response Targets

Severity Definition Mission-Critical Response

Critical 1 Production server or other mission-critical systems are down and no workaround is immediately available. 30 minutes or less, 24x7

Major 2 Major functionality is severely impaired. 2 business hours, 12x7

Minor 3 Partial, noncritical loss of functionality of the software. 4 business hours, 12x5

Cosmetic 4 General usage questions. 8 business hours, 12x5

For a full list of VMware’s response times and a description of problem severity, go tohttps://www.vmware.com/support/policies/severity.

CONFIDENTIAL 7

VMware Compatibility Guide

CONFIDENTIAL 8

VMware Product Interoperability

CONFIDENTIAL 9

Solution and Database Interoperability

CONFIDENTIAL 10

Effects of a System Problem

These problems can affect certain aspects of a system:

• Perceived effects or symptoms are generally exposed and reported. Symptoms of a system problem often appear to be the problem itself.

• Your must look at all of the symptoms of a system problem to determine the root cause.Usability Accuracy

Reliability Performance

CONFIDENTIAL 11

Adopting a Troubleshooting Process

Troubleshooting is a systematic approach to identifying the root cause of a problem and defining steps to resolve the problem.The troubleshooting process consists of several tasks:

Defining the problem.

Identifying the likely cause of the problem.

Resolving the problem.

• Identifying symptoms.• Gathering

supplemental information.

• Identifying possible causes.

• Determine the root cause.

• Identifying possible solutions.

• Implement the best solution.

Document the problem, the steps taken, and the solution.

CONFIDENTIAL 12

Collecting Symptoms of a Problems• Collecting symptoms and isolating the problem is the first step in troubleshooting.

• A single root cause often presents itself as several symptoms that users report.

• Differentiating between symptoms and the root cause of a problem is imperative.

Poor virtual desktop performance

Unable to provision or recompose VMware View desktops

Horizon Client has a black screen

Horizon Client has a black screen

Non-optimized desktop image

Failed to set the computer’s domain account password

PCoIP ports blocked

vRAM shortage on the View virtual machine

Root CauseSymptom

CONFIDENTIAL 13

Gathering Additional Information

Can the problem be reproduced?Provide a repeatable means to recreate the problem and a way to validate that the problem was resolved.

What is the scope?Does the problem affect only one object or multiple objects?

1

2

3

4

Was the system working before?If so, what changed in the enviroment or configuration?

Is the problem a known problem?Consult references such as release notes and VMware knowledge base articles to determine whether the problem is documented.

Ask questions to gather additional information to define the problem:

CONFIDENTIAL 14

Benchmarks

Real program

Microbenchmark

Component benchmark

Synthetic benchmarking

I/O, database benchmarks

VMware vRealize® Operations™

Various ways to quantify the problem:

CONFIDENTIAL 15

Viewing and Interpreting Diagnostic Information

View diagnostic messages in the GUI or in log files.

Example: No Desktop Available

Interpret the diagnostic messages to find the root cause.

CONFIDENTIAL 16

Identifying Possible Causes and Taking an Appropriate Approach

• A structured approach to troubleshooting enables you to determine the root cause quickly and effectively.

• Based on the problem’s characteristics, take one of the following troubleshooting approaches:

Bottom-up(Most General)

Approachthe cause by halves

Hardware(CPU, memory,

network, storage)

Virtual Desktop

ESXi Host

Horizon

Top-down(Most Specific)

Investigate cause top down

Investigate cause bottom up

Approach the cause by halves

Horizon Client

CONFIDENTIAL 17

Determining the Root Cause

To determine the root cause, test your environment and eliminate possible causes.

Example: No Desktop Available

Pool status. Lack of network connection to VMware ESXi™ host.

Customization error. Incorrect domain credentials.

Not enough host resources (CPU, memory, storage) are available.

Possible Causes

Physical resources are inaccessible.

Hardware(CPU, memory,

network, storage)

Virtual Desktop

ESXi Host

Horizon

Horizon Client

CONFIDENTIAL 18

Resolving the Problem

After identifying the root cause, assess the impact of the problem on operations:• High impact: Resolve immediately• Medium impact: Resolve when

possible• Low Impact: Resolve during next

maintenance window.

3

21 Identify possible solutions and their impact on the VMware Horizon enviroment:• Short-term solution: Workaround• Long-term solution: Reconfiguration• Impact analysis: Assess the impact

of the solution on operations

Resolve the problem by implementing the most effective solution.

4 Document the problem and solution.

CONFIDENTIAL 19

Example: Determining the Root Cause: Black Screen

Success?

Only when using PCoIP?

Using the correct video driver?

Success?

Yes

No

Start here:

Enough video memory for the resolutions

Success?

Verify that the agents were installed in the correct order

Success?

No

No

No

Yes

Yes

Yes

Yes

If you can, test possible causes by using a repeatable flow to determine the root cause. Document all the steps that you take.

Test next possible cause

Furtherinvestigationnecessary

Root causeidentified

Root causeidentified

Root causeidentified

CONFIDENTIAL 20

Example: Resolving the ProblemIn this example, assume that the root cause was not enough video memory for the resolutions for the user’s dual-monitor setup.

Assess the impact of the problem on operations:• Probably high impact: The problem affects any desktop that is

using dual monitors.

Identify possible solutions to resolve the problem:• Short-term solution: Change the vRAM memory for the user’s VM.• Long-term solution: Change the vRAM memory on the master image, and change

vRAM memory for all guests.• Test the solution for unintended consequences

Implementing the solution should require minimal downtime.3

2

1

Common Troubleshooting Tools (Manual and automated)

21

CONFIDENTIAL 22

Methods to Run Commands

VMware vSphere® Management Assistant :

vSphere ESXi Shell, which includes:

• esxcli commands

• A set of esxcfg-* commands: The esxcfg-* commands are deprecated in favor of the newer esxcli command-line utility.

• A set of other troubleshooting commands.

• With the installed VMware vSphere® Command-Line Interface package, an administrator can carry out configuration and troubleshooting tasks.

You can obtain command-line access on an ESXi host in several ways:

CONFIDENTIAL 23

Accessing vSphere ESXi ShellYou can access vSphere ESXi Shell in the following ways:

• Enable the vSphere ESXi Shell service, either in the DCUI or in vSphere Web Client.

• Access vSphere ESXi Shell from the DCUI by pressing Alt-F1.• Disable the vSphere ESXi Shell service when not using it.

Local access through the Direct Console User Interface (DCUI):

• Enable the SSH service, either in the DCUI or VMware vSphere®

Web Client. • Use an SSH client, such as PuTTY, to access vSphere ESXi Shell.• Disable the SSH service when not using it.

Remote access through SSH:

CONFIDENTIAL 26

vSphere Management Assistant Components

• Enables you to run common system administration commands against ESXi hosts, such as:

• esxcli• vmware-cmd• vicfg-* commands

• Requires credential connection options to a server

• Automates authentication to the VMware vCenter Server® system or ESXi host targets

• Relieves the user from having to continually add login credentials to every command that is executed

• Facilitates unattended scripted operations

• Entering this command changes the command prompt, which indicates the target server designation.

vSphere CLI command set: vi-fastpass authentication component:

The vifptarget command sets the target server for vSphere CLI commands:

vSphere Management Assistant is a virtual appliance that includes components for running vSphere commands:

CONFIDENTIAL 27

ESXCLI Commands

• esxcli vm namespace

• esxcli system namespace

• esxcli device namespace

• esxcli graphics namespace

• esxcli sched namespace

• esxcli vsan namespace

• esxcli esxcli namespace

• esxcli esxcli command list for a full listing

• esxcli namespace

• esxcli fcoe namespace

• esxcli hardware namespace

• esxcli iscsi namespace

• esxcli license namespace

• esxcli network namespace

• esxcli software namespace

• esxcli storage namespace

The esxcli command offers options in the following namespaces:

CONFIDENTIAL 32

vicfg-* Commands

vicfg-*

• You can use vicfg-* commands to manage your storage, network, and host configuration.

• For example, you can run the vicfg-vmknic -l command to display the IP information of your VMkernel interfaces.

CONFIDENTIAL 33

vmware-cmd Command on vSphere Management Assistant

The vmware-cmd command is used for configuring virtual machines and gathering information about them.The vmware-cmd command can be run on vSphere Management Assistant (not available on the local or remote vSphere ESXi Shell).

CONFIDENTIAL 36

Common Windows Commands

• ipconfig[-options] target_name

• ping [-options] target_name

• tracert[-options] target_name

• pathping [-options] target_name

• nslookup[-options] host

• gpupdate[-options] target_name

• gpresult[-options]

• rsop

• Set

You can use the following commands, among others:

CONFIDENTIAL 37

VDI Benchmarking

The following tools are used to measure the scalability and performance of the virtual

desktop infrastructure:

• VMware View Planner

• Login VSI

CONFIDENTIAL 38

Logging…

CONFIDENTIAL 40

Location of vCenter Server Logs

• The vCenter Server log files are on the vCenter Server system:– Location for vCenter Server on Windows 2008/2012:

• %ALLUSERSPROFILE%\VMWare\vCenterServer\logs

– Location for VMware vCenter® Server Appliance™:• /var/log/vmware/

• Subdirectories exist for vCenter Server components, such as vCenter Service, VMware vCenter® Single Sign-On, vCenter Inventory Service, and vSphere Web Client.

CONFIDENTIAL 41

vCenter Server Logs Useful for TroubleshootingThe vpxd.log file is the main log file for vCenter Server. Most vCenter Server actions are captured in vpxd.log.

Log File Purpose

vpxd.log The main vCenter Server log, consisting of all VMware vSphere® Client™ and VMware vSphere® Web Services SDK connections, internal tasks and events, and communication with the vCenter Server agent (vpxa) on managed ESXi hosts

vpxd-profiler.logprofiler.logscoreboard.log

Profiled metrics for operations performed in vCenter Server

cim-diag.logvws.log

Common Information Model (CIM) monitoring information, including communication between vCenter Server and the managed host’s CIM interface

CONFIDENTIAL 42

Location of ESXi Host Logs

The ESXi host log files are on the ESXi host at /var/log.

CONFIDENTIAL 43

ESXi Host Logs Useful for TroubleshootingESXi hosts write to multiple log files, depending on which action is being performed.

Log File Purposehostd.log Host management service logs

syslog.log Management service initialization, watchdogs, scheduled tasks, and DCUI use

vmkernel.logCore VMkernel logs, including device discovery, storage and networking device and driver events, and virtual machine startups

vmkwarning.log A summary of warning and alert log messages excerpted from the VMkernel logs

vmksummary.logA summary of ESXi host startup and shutdown, and an hourly heartbeat with uptime, number of virtual machines running, and service resource consumption

CONFIDENTIAL 44

Viewing Log Files in vSphere Web Client • You can use vSphere Web Client to view and search log files on vCenter Server

systems and ESXi hosts.

• The log browser is on the Monitor tab for vCenter Server systems and ESXi hosts.

CONFIDENTIAL 49

Using the vm-support Command to Export Data

1. Run vm-support at the command line.• By default, the command does not require arguments.• The command creates a ZIP file that includes the host

name and the time stamp in the filename:– Example: esx-esxi01.vclass.local-2013-01-23--14.09.tgz

2. Use the tar command to view the contents of the file.• Example: tar –tzf /var/tmp/filename

You can use the vm-support command utility at the local or remote vSphere ESXi Shell to collect and compress the following types of data:

• Log files• System status• Configuration files

To collect and compress useful troubleshooting data:

CONFIDENTIAL 50

Log Locations

Horizon Security Servers Use support tool installed on Connection Server or launch support.bat from C:\Program Files\VMware View\Server\DCT

View ComposerC:\Windows\Temp\vmware-viewcomposer-ga-new.log on the linked-clone desktop

Horizon ClientC:\Users\%username%\AppData\Local\VMware\VDM\logs

View Connection Server InstancesUse support tool installed on Connection Server or launch support.bat from C:\Program Files\VMware View\Server\DCT

View Composer Server

Horizon AgentC:\ProgramData\VMware\VDM\logs

Active Directory Use the Event Viewer to view AD-related logs.

CONFIDENTIAL 51

Enabling Verbose Logging for View Composer

On the desktop through Regedit

You should only enable View Composer verbose logging for a short period of time due to size considerations. On the View Composer server through a file

2

1You can enable verbose logging in the following ways:

CONFIDENTIAL 52

Collecting Windows Horizon Client Log Files

Log collection on Windows devices is set to basic logging by default.

You can enable advanced log collecting in Horizon Client for Windows by running the support.bat script.

CONFIDENTIAL 53

Collecting Horizon Client for Mac OS X Log Files

Horizon Client for Mac OS X does not have a script to collect logs. You must manually collect these files for diagnostic and troubleshooting purposes or use the VMware Labs Fling.

1

2

3

In View 5 and later, the Horizon Client logs on a Mac can be found in ~/Library/Logs/VMware Horizon View Client.

You can download Horizon Collector for Mac VMware Fling at https://labs.vmware.com/flings/horizon-collector-for-mac.

CONFIDENTIAL 54

Collecting iOS Horizon Client Log Files

You can solve most Horizon Client for iOS problems by resetting the desktop or reinstalling the app.

For more information about enabling and collecting iOS logs and files, see Using VMware Horizon Client for iOS at https://www.vmware.com/pdf/horizon-view/horizon-client-ios-document.pdf.

iOS clients do not have a script to collect logs. You must manually enable and collect these files for diagnostic and troubleshooting purposes.

1

2

3

CONFIDENTIAL 55

Collecting Android Horizon Client Log Files

Log collection on Android devices is disabled by default because it will affect the performance of the device. Disable log collection after you finish debugging because log collection affects the performance of the device.

For more information about enabling and collecting iOS logsand files, see Using VMware Horizon Client for Android at https://www.vmware.com/pdf/horizon-view/horizon-client-android-document.pdf.

Use the options in Horizon Client for Android to enable log collection.

1

2

3

CONFIDENTIAL 56

vRealize Log Insight

Machine-generated log data is typically massive in scale, difficult to manage, and overwhelming.

vRealize Log Insight provides a single location to collect, store, and analyze logs at scale.

vRealize Log Insight is a log management solution for physical, virtual, and cloud environments.

vRealize Log Insight helps monitor events and metrics. It analyzes root causes and performs intelligent grouping, which helps you troubleshoot efficiently.

vRealize Log Insight provides faster analytical queries and aggregation than traditional tools, especially on larger data sets. It adds structure to all types of unstructured log data, so administrators can troubleshoot quickly.

CONFIDENTIAL 57

Searching and Filtering Log Events

You can search and filter log events by specifying the keywords, time range, field operations, and so on.

• Event types grouping uses machine learning to group similar events, making root-cause analysis and troubleshooting faster and easier.

• You can view events in context and analyze event trends to uncover anomalies.

• You can also save, export, rename, share, or delete a query.

CONFIDENTIAL 58

Analyzing Logs with the Interactive Analytics Charts

• You can select different chart types to graphically analyze log events.

• You can modify the aggregation and grouping of query results to correlate events and make the chart meaningful for troubleshooting.

The charts provide visual representation of data and enable you to perform visual analysis on your query results.

CONFIDENTIAL 59

Dynamic Field Extraction

When facing a large environment with many log events, you can locate the data fields that are important to you and extract and save them.

Log field extraction turns unstructured data into structured data that is meaningful for troubleshooters.

CONFIDENTIAL 60

Troubleshooting Using Customized Dashboards

• vRealize Log Insight dashboards are collections of chart, field, table, and query list widgets.

• You can customize dashboards by adding, modifying, and deleting them.

• For example, you can save a filtered query to your custom dashboard by creating a query list widget.

CONFIDENTIAL 61

Monitoring Log Events and Sending Alerts

You can run specific alert queries at scheduled intervals. When the query exceeds the preconfigured threshold, alerts can be sent to your email

or to VMware vRealize® Operations Manager™ for Horizon.

The Top 10!Per VMware Global Support Services, July 2016

CONFIDENTIAL 64

Issue #1: Blackscreen• Most well known issue and usually isolated to PCoIP protocol.

• Remember that PCoIP is made up of two parts to the connection (this is true for most remote protocols):– A TCP connection for authentication, machine selection, and other management activities.

The FQDN of the connection server that is entered in Horizon Client is used to establish the connection.

– A connection to the machine

Resources (Variety of KBs written on the subject):• Misconfiguration of connection server settings. (KB 1028332)• Video RAM shortage on the View virtual machine. (KB 1027899)• Incorrect video driver version installed on the View virtual machine. (KB 1029706)• Disclaimers or login banners are being used. (KB 1016961)• Agents were not installed in the correct order. (KB 1018158)• Network ports are not open. (KB 1026498)

1

CONFIDENTIAL 65

• Order, order, order. The installation order of the Horizon Agents and other VMware products can result in misconfigurations as can forgetting to select the right options (e.g., Linked Clones vs Instant Clones, USB redirection, etc.)

Resources:

• Troubleshooting Agent Unreachable status in VMware Horizon View (2083535)

• Installation order of End User Computing Agents for User Environment Manager (UEM) and App Volumes (2118048)

Issue #2: Misconfigured Agent

2

CONFIDENTIAL 66

Issue #3: SSL Certs• Process is simple:

– Get Certification from CA and export– Import and change friendly name

•Remember:– Friendly name is vdm (not View, not Horizon, not Billy-Bob…)– Key length: 2048 or greater (not 2047, not 1024, not thrice...)

PFX/PKCS#12 when importing into Windows cert store

Resources• Using Microsoft Certreq to generate singed SSL certificates in VMware Horizon View

(2032400)• Scenarios for Setting up SSL Certificates for View (Horizon View 6.0)

3

CONFIDENTIAL 67

• Remember this caveat: “Just because you can doesn’t mean you should!!”

• PCoIP Settings that could cause problems:– Floor minimum– Bandwidth maximum– Frame rate– Enable/disable build-to-lossless (version dependent)– And many more!

• Test! Test! Test!

Issue #4: Network Configurations/Misconfigured GPOs

4

CONFIDENTIAL 68

Issue #5: Upgrading/upgrade Order

5

• Upgrading can be tricky. Always remember: backup, backup, backup.

• Plan your upgrade.

• Horizon then vSphere.

Resources:

1. Installation order of EUC agents for UEM/AppVolumes: http://kb.vmware.com/kb/2118048

2. Update sequence for vSphere 6.0 and compatible VMware products: http://kb.vmware.com/kb/2109760

CONFIDENTIAL 69

Issue #6: Forgetting to Install nVIDIA Pieces (VIB and/or Guest Driver)

• Just installing the card isn’t enough. You need to install the ESXi VIB and Guest OS Driver to take full advantage of those cards.

Resources to help:

1. NVIDIA GRID™ VGPU™ DEPLOYMENT GUIDE FOR VMWARE HORIZON 6.1: https://www.vmware.com/files/pdf/products/horizon/grid-vgpu-deployment-guide.pdf

2. Installing and configuring the NVIDIA VIB on ESXi: http://kb.vmware.com/kb/2033434

6

CONFIDENTIAL 70

• End user experience impacts you.

• A happy user means a happy desktop admin. (Moar Warcraft; less reboots!)

Resources:

1. Fling: https://labs.vmware.com/flings/vmware-os-optimization-tool

2. Optimization Guide: http://www.vmware.com/techpapers/2010/optimization-guide-for-windows-7-and-windows-8-vir-10157.html

Issue #7: Unoptimized Guest OS

7

CONFIDENTIAL 71

• Tasks between Connection Servers and Composer get hung and nothing provisions. Requires a View Environment (all connection servers down and then powered on one at a time) reboot to get going.

Resources:

1. Restart order of the View environment to clear ADLDS (ADAM) synchronization in View 4.5-6.1: https://kb.vmware.com/kb/2068381

Issue #8: Hung VCcache Issues

8

CONFIDENTIAL 72

• Sometimes VMs go missing because they are being managed outside of Horizon. When it’s removed as part of the activities of the vSphere Administrator, Horizon is often unaware that this and thus labels it as missing.

Resources:• Manually deleting linked clones or stale virtual desktop entries from the View Composer

database in VMware View Manager and VMware Horizon View: https://kb.vmware.com/kb/2015112

• Removing invalid linked clone entries automatically using the ViewDBChk tool in VMware Horizon View 5.3 and later versions: https://kb.vmware.om/kb/2118050

Issue #9: Missing/Orphaned or Stuck in Deleing VM

9

CONFIDENTIAL 73

• Customization failures can be frustrating. It’s important to make sure you account for each of the below, otherwise no provisioning will occur.

Resources:

• VMXNET 3 adapter on Parent Image for linked clones misconfigured (MS KB2550978 & VMware KB: https://kb.vmware.com/kb/2007319)

• not enough ports on vSwitch/dvPortGroup

• typos on passwords.

Issue #10: Customization Failures

10

Where to from Here?

74

CONFIDENTIAL 75

Learn MoreVisit us at VMworld• Education & Certification Lounge: Mandalay Bay, VMvillage• Certification Exam Center: Mandalay Bay, North Convention Center, Mariner’s A&B

Visit our website• VMware training and certification: www.vmware.com/education

CONFIDENTIAL 76

Q&A

For questionsLinus at lbourque@vmware.com OR @linusbourque (Twitter)

Joe at cooperj@vmware.com OR @joeicooper (Twitter)

?

Troubleshooting 101 for HorizonLinus Bourque, VMware, IncJoseph Cooper, VMware, Inc

EUC8243R

#EUC8243R