124
www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. [email protected] Shane Johns Senior Software Engineer Novell, Inc. [email protected]

Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. [email protected] Shane Johns

Embed Size (px)

Citation preview

Page 1: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

www.novell.com

Avoiding the Top iChain® Technical Support Issues

Avoiding the Top iChain® Technical Support Issues

Neil CashellTechnical Support EngineerNovell, [email protected]

Shane JohnsSenior Software EngineerNovell, [email protected]

Page 2: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Vision…one NetA world where networks of all types—corporate and public, intranets, extranets, and the Internet—work together as one Net and securely connect employees, customers, suppliers, and partners across organizational boundaries

MissionTo solve complex business and technical challenges with Net business solutions that enable people, processes, and systems to work together and our customers to profit from the opportunities of a networked world

Page 3: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns
Page 4: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Presentation Outline

• iChain® configuration files

• iChain troubleshooting tools• iChain components

Interfaces• Inputs and outputs• Flow of information

Troubleshooting steps Common issues Case study

Page 5: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Configuration Files

Page 6: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Configuration/Info Files

• iChain Proxy Server Configuration

• CURRENT.NAS• TCPIP.CFG• OAC.PROPERTIES/TRACERMEDIA.PROPERTIES• Custom login/logout pages• APPSTART.NCF and TUNE.NCF

Troubleshooting• CONSOLE.LOG• TRACE.TXT• CAPTERR.LOG and CAPTOUT.LOG• DEBUG00X.LOG/DEBUG.LOG • Proxy and aclcheck log files

Page 7: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Configuration/Info Files (cont.)

• iChain eDirectory™ LDAP Server LDIF file showing schema objects/attributes

• ICE or LDAP browser can export this to file• FormFill profile

• iChain Authentication Server Debug output for authentication method

• ‘Radius debug on’ captured to console log (radius)• DSTRACE.LOG with +LDAP/TIME enabled (LDAP

authentication)

Page 8: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Configuration/Info Files (cont.)

• Network layout Firewalls L4 switches DMZ

Page 9: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools

Page 10: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools

• ConsoleOne®

LDAP Group Object ISO object attributes

• Protected resource mode and OLAC parameters• Password management setup

RuleObject attributes (Rule TAB) Rules applying to users (User TAB)

• ICE (Server and client-based) Export configuration to file

Page 11: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools (cont.)

• LDAP browser http://www.iit.edu/~gawojar/ldap/ Easily export configuration to file Confirm iChain objects and attribute values are

valid

• LSEARCH.NLM from LDAP client SDK LDAP bind done for every request http://developer.novell.com/ndk/cldap.htm

Page 12: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools (cont.)

• ICS GUI Home->Health status for details of services running Monitor TAB gives services and stats information

• Services running• Disk space info, CPU utilization, cache hit ratio

Access ACLCHECK and Proxy logs via MONITOR TAB

• ICS Java console Proxy authentication and aclcheck profiles exists

Page 13: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools (cont.)

• Proxycfg debug screen LDAP profile errors

• TCPCON Connectivity specific tool (ICMP, TCP issues) Active TCP listeners

• Logs from authentication servers DSTRACE.NLM for LDAP (view DS trace traffic

for object/attribute resolution) ‘Radius debug ON’ trace from Radius server

Page 14: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Generic iChain Troubleshooting Tools (cont.)

• Network layout information Firewalls/L4 may pose Connectivity/State

problems

• LAN analyzer Trace traffic between proxy and auth server Trace traffic between browser and proxy server Trace traffic between proxy and origin server

Page 15: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Components“Proxy Authentication”

Page 16: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Interfaces

•Inputs and outputs

•Flow of information

Page 17: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Interfaces

• PROXY.NLM Calls authentication callback methods

• LDAP (requires LDAP, LDAPSDK), mutual, Radius (Radchk)

• TCPIP.NLM Connection into proxy ports

• PROXYCFG.NLM Stores profile information + Error reporting tool

• NILE/PKI Certificate management

Page 18: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Flow Control

Proxy processes incoming requests on Port 80 (default)• Check if authentication required

– Cookie exists - yes => process cookie (see next page)– No => need to identify user

» Compare URL with ISO protected resource defined and return mode if match found

» If mode is NOT public, authenticate connection (next page)

Page 19: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns
Page 20: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Flow Control

• Subsequent requests check for cookie in header

Verify checksum ok Verify source IP address match Forward request to origin server

Page 21: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Troubleshooting Tools

Page 22: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Troubleshooting Tools

• Proxy Console -> iAgent console

Page 23: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Troubleshooting Tools (cont.)

• Internet browser Useful for importing certificates Netscape browser setup with NULL encryption

– Enabled via Security TAB -> Navigator -> Configure SSL v3 and disable everything except for ‘No encryption with an MD5 MAC’

Internet Explorer debug WININET.DLL – Ability to decode SSL traffic

• Proxy debug logs Requires a debug installation of iChain

Page 24: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Troubleshooting Steps

Page 25: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Troubleshooting Steps

• Verify configuration (basic) ISO PR attributes set for authentication (mode) Proxy authentication profile configured LDAP server allows clear text passwords IP address/Port combination for authentication

server up via PING SSL Certificate assigned to proxy server

Page 26: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Initialization Problems

• “Proxy Failed to Get ISO Object From Proxy Server” or “Invalid authentication information” error in Proxycfg

Ping <ldap_srvr_addr:port> from ICS Java console Get authentication LDAP returns valid parameters

• Very LDAP request/responses (DSTRACE) for 81/85 errors Verify LDAP TCP connections exist in the established state in

TCPCON->Protocols Information->TCP Connections Check interpacket delay times between LDAP

request/responses• LDAP Server overloaded and may require addition of threads

– On NetWare® (display configuration: LDAP DISPLAY CONFIG)» LDAP MAXIMUM THREADS= changes the threads default

– On Unix» Daemon parameter (check man pages)

Page 27: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Initialization Problems (cont.)

• If LDAP over SSL enabled, try without SSL and verify if certificate-related problem

• Check for service errors in health screen of ICS GUI

Service failure error detected

Page 28: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Access granted to users that should NOT have access

ISO protected resource mode (public mode setup)

Page 29: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page not displayed Failure at this level would indicate an SSL/PKI

issue• Look closely at the SSL diagnostic screens on the

iChain Proxy server and try and check for SSL handshake errors

• Trace client to proxy connection and verify, after the first redirect, – That you see cert chains being transferred– That the ICS box doesn’t have time set in the future (Non

US)

Page 30: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page not displayed Failure at this level would indicate an SSL/PKI

issue• Trace proxy and CRL server (if CDP attribute for CRLs

enabled) and verify CRL downloaded– Time issues could occur here too. Look for two entries

that look like 010309154821Z—this translates to a year of 01, a month of 03, a day of 09, a time of 15:48 and 21 seconds—The first date listed is the creation date of the CRL, the second date is effectively the expiry

• Try using another browser type to see if the problem is unique to one type of browser

• Try and generate another certificate with small key size and see if the SSL handshake succeeds

Page 31: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (Certificate Timing Issue)

Page 32: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page not displayed Verify if login page customized (java scripts)

• Revert to original and retest• Check with multiple browsers to see if issue exists

Verify is authentication over HTTP works fine• Confirmation of SSL certificate issue

– ICS box has newer timestamp– Old certificate expired– CRL communication invalid– Corrupt certificates

Page 33: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page displayed but authentication fails Verify the authentication profile settings Verify the authentication server is active via PING Verify that login page hasn’t been customized Verify that no intermediate device stripping cookies Verify browser is sending the correct credentials when

POSTing information to the iChain Proxy server• No encryption on browser required• Check authentication server logs (DSTRACE, Radius) to see

if user being validated

Page 34: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page displayed but authentication fails

Problem with customized pages• No LDAP request sent to authentication server• Login page missing required attributes• Attributes correct but out of order• Browser failures

Page 35: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page displayed but authentication fails Verify accelerator name and cookie domain (IE

issue)• Case sensitivity

Verify that browser accepts and gets cookies• ‘Warn me before accepting cookies’ on Netscape->Edit-

>Preferences->Advanced• ‘Allow cookies that are stored on your computer’ in IE-

>Tools->Internet Options->Security->Custom Level• Verify cookie sending valid (Opera TID #10063326)

Verify if all authentication profiles have problems• e.g., Try authenticating based on email address in LDAP

Page 36: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Login page displayed but authentication fails

Verify whether or not it is possible to login to the directory using the users credentials• Password management servlet enabled

– Case sensitive java servlet

Verify if user authentication information available in Proxy Console’s iAgent screen

Page 37: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• LDAP problems LDAP profile has valid BIND username/password

• Must have Read (not just browse!) rights to DS no LDAP request sent in trace

• Stale LDAP handles at firewall/L4 switch• Max. LDAP handles reached and active

– 30 handles allocated—LDAP error 81 if all handles in use LDAP Server slow to respond to requests (need more

threads)– On NetWare display configuration: LDAP DISPLAY CONFIG)

» LDAP MAXIMUM THREADS= changes the threads default– On UNIX

» Daemon parameter (check man pages)

Page 38: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Authentication Problems (cont.)

• Radius problems Radius profile has valid radius secret with DAS

object Radius server listening on UDP port 1812/1645 Radius server has a valid DAS profile setup

• Radius client is valid ICS address Radius debug commands show no errors LAN trace shows successful RADIUS response

• Timeout issues

Page 39: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Proxy Case StudyHTTP 403 Forbidden error:

“Your browser must support cookies.”

Page 40: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

403 Forbidden Error

• iChain 2.0 setup to accelerate secured PR Browser hits Proxy and prompted to authenticate After entering credentials, gets above 403 error

• Disabled aclcheck (restricted PR) but 403 errors still sent

• Verified LDAP traffic generated• Enabled browser option to prompt when accepting

cookies– Cookies were being set

• checked Proxy Console->IAgent screen • Checked PROXYCFG/Proxy Console screens for errors

Page 41: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

403 Forbidden Error (cont.)

• Analyze network layout Suspect L4 switch

• Moved browser to bypass L4 switch and no error– Took good set of traces

• Put browser back to original position– Took good set of traces– Trace showed that the original requests for page went to

one ICS server, and next request to another ICS server; L4 switch was redirecting requests

Page 42: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

403 Forbidden Error (cont.)

Page 43: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

403 Forbidden Error (cont.)

• Enabled IP hashing option on L4 switch Forces a map of incoming client session to

destination IP address Note that enabling session broker in this

scenario will fail because the SB kicks in after a successful authentication has taken place

Page 44: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Components“Session Broker”

Page 45: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker (SB) Interfaces

•Inputs and outputs

•Flow of information

Page 46: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Interfaces

• PROXY.NLM Stores session broker profile information Calls SB code during authentication phase

• Winsock modules Winsock APIs used for connectivity between ICS

and SB servers

• SB.NLM SB server listening on TCP 5001 on both primary

and secondary

Page 47: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Interfaces (cont.)

• LDAPSDK.NLM Generate LDAP request for ISO SB attributes

• iChainPrimarySessionIPAddress• iChainSecondarySessionIPAddress• iChainMasterProxyIPAddress

Page 48: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Flow Control

Page 49: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Flow Control (cont.)

• Initialization—LDAP request sent to ISO object to extract SB attributes

• Proxy authentication phase iagent locates entry in database

• yes => allow request through• no => ICS server sends message to primary SB server

SB primary server locates entry in database• YES => allow request through• NO => force authentication

Page 50: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Flow Control (cont.)

• When user successfully authenticated to ICS server, primary SB updated with

• Authentication profile type• Authorization basic HTTP header• Username• Cookie domain

• Primary SB server returns a hash key for subsequent requests

Page 51: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker (SB) specific Troubleshooting Tools

Page 52: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SB Troubleshooting Tools

• TCPCON Procotol Information -> TCP -> TCP Connections

• TCP port 5001 listening

• Unencrypted SessionBroker sessions createnullsessionbrokerkey when generating SB

key Allows legible trace information to be obtained

• SB command line parameters -n => no encryption -d => verbose information

Page 53: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SB Troubleshooting Tools (cont.)

• Session broker debug screen

Page 54: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Troubleshooting Steps

Page 55: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Troubleshooting Steps

• Verify configuration (basic) sessionbroker keys exist and installed Set authentication sessionbrokerenabled SB.NLM loaded with no errors

• ISO attributes found Authentication with no SB works fine Third party L4 switches in network layout

Page 56: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Initialization Problems

• “Unable to initialize the Session Broker” Regenerate keys and verify ok

• SESSION.DAT file exists on floppy Memory errors on ICS server (NBMALERT) Verify TCP connections 5100 listening in

TCPCON->Protocols->TCP Connections• Check the SB debug screen for read or write errors

– recv() failed: error <errno>

Page 57: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Problems

• SB Authentication issues Multiple ICS servers in SB domain must have

authentication profile with same name• Shared data on TCP 5001

Connectivity issues between ICS and SB servers• No set/get traffic completed

L4 switches redirecting authentication traffic between ICS boxes

Page 58: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SessionBroker Case Study

Slow login when SB-enabled

Page 59: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Case Study: Slow Login When SB-Enabled

• Problem scenario Friday: iChain 2.0 setup with SB enabled—all ok Monday: Users complain of slow logins (15

mins)• Credentials valid but delay getting Web page to show

• Network layout 2 Proxy servers in parallel Browsers pointing to secondary SB (SB-S)

server Primary SB server not running services

Page 60: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

SB Case Study—Network Layout

Page 61: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Case Study: Slow Login When SB-Enabled

• Verified Different workstations gave problem Different browsers (IE, Netscape) showed same

issue Cookie prompt enabled showed we received cookie iAgent console screen showed User authenticated

with correct information• => authenticated to local iagent database

Ping to port 5001 on SB-P failed

• Took traces…

Page 62: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns
Page 63: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Case Study: Slow Login When SB-Enabled

• Solution Re-connect SB-P to the network SB-S was processing authentication requests

and trying to update the primary• Request sent to SB-P with user’s authentication

information• Response with hash key never arrives• Request resent 12 times with increasing

retransmission timeouts => waited ~20 mins for TCP RST to occur

Page 64: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Components“ACLCHECK”

Page 65: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Interfaces

•Inputs and outputs

•Flow of information

Page 66: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Interfaces

• PROXY.NLM Stores profile information Calls authorization code after authentication

• ACLCHECK.NLM Process URL requests for matches with rules Generates LDAP queries into eDirectory

• eDirectory Repository for configuration info Repository for rule objects and protected

resources

Page 67: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Flow Control

PROXY: verifies the PR mode is secured, the user is authenticated and URL not /RegNewUser/ or /servlet/DocumentServlet/—If true call ACLCHECK• Pass authenticated user, and the URL being accessed

ACLCHECK• Checks hash table for hit

– Match found => return allow; else

• Gets RO DN from user container object attribute (brdsrvRule attribute) via LDAP – LDAP config info taken from ACLCHECK authentication profile

• Read rules from the RO– Get URL and apply to settings

Page 68: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Flow Control (cont.)

• Compare URL in rule Match found => allow; else

• Find the RO for the users containers community (if /M enabled)– Get and process rules for each community and apply them to URL;

if no match found• Find the RO for the users groups, users group’s communities,

user itself and finally the communities the user belongs to Check for each of them and first one to allow will allow the

access and other rules will not be checked If none matches, then access for this user is “deny”

• At any stage where a match is found, check exceptions for a block

Page 69: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Specific Troubleshooting Tools

Page 70: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Troubleshooting Tools

• ACLCHECK logs Console.log output with /D1 enabled (debug ==

/D4)• No output => no aclcheck

• LSEARCH LDAP client from SDK Does a bind for every request

• DSTRACE.NLM View DS trace traffic for object/attribute

resolution

Page 71: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Troubleshooting Steps

Page 72: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Troubleshooting Steps

• Verify configuration (basic) ISO PR mode set for authorization (secured

only) NDS Rule Objects applied correctly ACLCHECK profile configured LDAP server allows clear text passwords LDAP mappings exists for attributes

Page 73: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Initialization Problems

Check for “ACL: ACLCHECK Failed to Get ISO Object From Proxy Server” error on system console• ‘Get authentication aclcheck’ returns valid LDAP

parameters• ping <ldap_srvr_addr:port> from ICS Java console• Verify lsearch command works• Verify TCP LDAP connections exist in the ‘established’

state in TCPCON->Protocols->TCP Connections• Very LDAP incoming/outgoing requests on LDAP server

– DSTRACE +LDAP, +TIME enabled– Check LAN trace for LDAP errors 81, or 85

Page 74: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Rule Processing Problems

• Users granted access that should NOT have access

ISO protected resource mode (public/restricted) Stale cache entry User a member of group, community that has

access User accessing /servlet/DocumentServlet/ or

/RegNewUser/ URLs ACLCHECK /D1 shows rule granting access

Page 75: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Rule Processing Problems (cont.)

• 403 forbidden errors ISO protected Resource granted for full path Rule Object exists granting user rights to URL

• Verify rule objects in DS• Verify user member of group, organization unit or

community with rights Check if rule exception blocks access ACLCHECK /M loaded for iChain 1.5

compatibility

Page 76: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Rule ProcessingProblems (cont.)

• 403 forbidden errors Check for stale cache entries

• Refresh ACLCHECK cache through GUI• Load ACLCHECK /F <refresh_time>

Memory issues (cannot update hash table) Radius server failing to return the FDN

• Error "Status : 403 Forbidden. Description : User Name Mismatch."

Page 77: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Rule Processing Problems (cont.)

• LDAP problems LDAP profile has valid BIND username/password Stale LDAP handles

• Lsearch application works• L4/firewall switch resetting ‘valid’ sessions• Max. LDAP handles reached (use /C<no_of_handles>)

Debug ACLCHECK /D4 errors Slow LDAP response due to overload—inc.

threads– On NetWare—LDAP MAXIMUM THREADS=– On UNIX—Daemon parameter (check man pages)

Page 78: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Case Study

403 Forbidden Error:

“Organizational policies prohibit access to this page”

Page 79: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Case Study—403 Errors

• iChain 2.0 setup for authentication/authorization FW-1 firewall exists between Proxy and LDAP servers All working fine

• Following morning users reporting 403 errors after authentication

• Verified No changes to setup (DS timestamps, current.nas)

• LDAP authentication profile existed, eDirectory objects unchanged

Ping to LDAP server successful

Page 80: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Case Study—403 Errors (cont.)

• Verified LSEARCH worked DSTRACE (+LDAP) showed no incoming LDAP

requests TCPCON showed no established LDAP sessions LAN trace showed outgoing request with TCP RSTs

responses from L4 switch ACLCHECK /D4 showed LDAP error 81 returned

• Occurs when no LDAP handles available to make request Everything works with no firewall between LDAP

and Proxy servers

Page 81: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ACLCHECK Case Study—403 Errors (cont.)

• Problem: FW-1 firewall timing out idle connections after 60 minutes

ACLCHECK LDAP handles were all stale

• Solved the problem by Disabling the idle_timeout timer on firewall, or Applying new ACLCHECK from IC20FP1.EXE

• added logic to detect and handle LDAP 81/85 errors

Page 82: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Components“Object Level Access Control”

Page 83: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Interfaces

•Inputs and outputs

•Flow of information

Page 84: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Interfaces

• PROXY.NLM• OACINT.NLM

shim to java application

• OACJAVA.NCF ldap, oac jar files jnet, jcert, jsse if SSL-enabled

• PROXYCFG.NLM

Page 85: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Flow Control

• Browser tries to accesses URL thru proxy Proxy authenticated and authorizes (if enabled)

• Proxy calls OACINT• OACINT talks to OACJAVA to retrieve values

OACJava generates LDAP requests and caches response

• OACJAVA sends response to Proxy Proxy checks if ICHAIN_UID and or ICHAIN_PWD is used

• Yes => replace values in authorization header• No => write query string and authorization header and forward

to origin server

Page 86: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Tools

Page 87: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Tools

• Sys:\Trace.txt file tracermedia.properties settings Note performance degradation due to swing

• Proxycfg debug screen LDAP profile errors reported here

• E.g., readiChainStringAttributebyLDAP failed

• Java -showxxx<threadID> output• Third party LDAP providers• Decoding Servlets from authentication Server

CD

Page 88: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps

Page 89: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps

• Verify configuration (basic) LDAP server allows clear text passwords LDAP mappings exists for attributes ACLCHECK profile configured Forward authentication information to web

server Debug OAC switches enabled

Page 90: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps (cont.)

• Common OACINT errors reported• No attributes returned for user cn=ncashell,o=novell,

resource my_web_server• ConnectToOAC failed: could not connect to OAC server:

Error xx• SendMessageToOAC failed: could not connect to OAC server

Tests• Increase java app mem size (java -Xms64m -Xmx128m)• Increase number of worker threads• Check ticks count (<270) for requests in OACINT

– LDAP server performance issue (increase LDAP threads)

• Try different LDAP provider• Check state of sockets, threads, memory with JAVA -SHOW

Page 91: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps (cont.)

• Common LDAP related errors reported• “Unable to connect to any ldap server to read ISO

information”• “Could not locate any LDAP profile”• “Failed to connect to any of %d LDAPservers”

Tests• ACLCHECK profile information valid• OACINT debug output

– tracerfilter.properties—change DEBUG 0 to 5– tracermedia.properties—log info to text file

Page 92: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps (cont.)

• Common OACJAVA errors• java.net.ConnectException (invalid port)• illegalMonitorState (out of worker threads)• java.lang.NumberFormatException (1.5 oac.properties)

Tests• iChainProtectedResource ISO attribute valid• oac.properties tuning issue• Provider issue• JVM issue (JAVA -SHOW)• LDAP server issue

– Performance - LDAP interpacket delay time– Resolution - DSTRACE errors (+LDAP, +TIME)

Page 93: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Troubleshooting Steps (cont.)

• Verify parameters seen with servlets Check that correct request/response

combination seen in oacjava debug screen• Check LDAP server for valid attributes (ldap browser,

dstrace)• Check LDAP server connectivity issues (L4 switch)• Check trace from ICS to LDAP and origin server for TCP

issues

Page 94: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Case StudyDuplicate Parameter Passed

Page 95: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Case Study

• Backend Web application authenticated user based on LDAP CN

OLAC setup to return users CN

• Users accessing application after authenticating to iChain received login error

• Verified• OACINT and OACJAVA initialized correctly• Problem not load/performance related• Servlets return valid credentials

Page 96: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Problem User Had Following Profile

Page 97: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

ISO OLAC Parameters

Page 98: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns
Page 99: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

OLAC Case Study

• ‘Other Name’ field in eDirectory is returned as a CN object via LDAP

• Application parsed last CN returned which was the user ‘Other Name’ rather than CN

Modified application to accept first CN in string

Page 100: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

iChain Components“FormFill”

Page 101: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Interfaces

•Inputs and outputs

•Flow of information

Page 102: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Interfaces

• PROXY.NLM FilterFramework (FF) model

• SSO.NLM Interface into Proxy FilterFrameWork via

callbacks

• eDirectory ISO object attributes User attributes (Novell SecretStore®)

Page 103: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Interfaces (cont.)

• LDAPSDK.NLM Pull formfill parameters from ISO object

• SSCLD.NLM SecretStore LDAP client

• NILE/PKI Certificate management if secure LDAP-enabled

Page 104: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Flow Control

• Initialization requires Generation of LDAP pool of handles

• Using authentication profile for LDAP Use LDAP to read FormFill ISO attributes

• Reading of FormFill profile• SecretStore enabled

• Proxy processing Request passed to filter framework code at

various stages where SSO filter created

Page 105: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Flow Control

Page 106: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Flow Control (cont.)

• SSO Processing• Verify POST HTTP method (no support for GET)• Find URL policy that matches the given URL

– INITIAL: Parse POST data» Get and remember list of attributes from form» Check if "don't remember this form" action in profile» Write out modified user data (LDAP request or local cache)» Forward data to origin server

– SUBSEQUENT: Get user data from LDAP» Get actions to be performed» build redirect request to browser with form attributes

Page 107: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Troubleshooting Tools

Page 108: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Troubleshooting Tools

• LDAP Browser/ConsoleOne®

Confirm ISO FormFill attribute (profile, SecretStore) User “iChainFormFillCrib” attribute

• ‘FFichain refresh rule’ server console command• iChain server console screens for SecretStore

SSL stack and server screens • Use to check the state of the LDAP SSL sessions

handshake

• LAN traces Most useful troubleshooting tool

Page 109: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Troubleshooting Tools (cont.)

• Proxy System Console -> SSO screen (debug build only)

Page 110: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Troubleshooting Steps

Page 111: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Troubleshooting Steps

• Verify configuration (basic) LDAP server allows clear text passwords Proxy authentication profile configured and correct Ping IP address/Port combination for LDAP server ISO attributes set for formfill (profile, SSO) SSL Certificate imported to proxy server (SS only) Login form includes java script?

• Only support HTML forms in current release HTML page must POST credentials (no GET support)

Page 112: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Common FormFill Problems

• Non-SecretStore problems FormFill profile matching HTML information Remove POST/ from FormFill profile to only fill Simplify profile to one variable if possible

• Use test profile written to confirm (available from support) Verify iChainFormFillCrib attribute created Verify DSTRACE +LDAP setting show valid

responses Verify LAN trace

• Confirm redirects and LDAP communication Apply debug SSO.NLM and view debug screen

Page 113: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Common FormFill Problems (cont.)

• SecretStore problems Verify all works fine without SecretStore Verify LDAP over SSL authenticates fine

• Import trusted root• Timestamp issues with certificates

Delete user iChainFormFillCrib attribute Enable DSTRACE logs with +LDAP, +TIME

Page 114: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

FormFill Case StudyAuthentication Failure to Web

Application

Page 115: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application

• Problem: Back-end application, using FormFill feature to authenticate, continuously prompting user to enter credentials for external users

Form Fill POSTing NULLs for external users; worked fine for internal users

• Network layout BM Server proxy’ing internal users to iChain Gauntlet firewall proxy’ing external users to

iChain

Page 116: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application (cont.)

Page 117: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application (cont.)

• Troubleshooting Removed SecretStore setup—also failed Removed POST/ entry from Profile—showed

blanks Looked at DSTRACE +LDAP info from LDAP

server• Updating entries correctly

Got a trace of working/non working scenarios• Saw that the POST header and data split thru gauntlet

Page 118: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application (cont.)

Page 119: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application (cont.)

Page 120: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Authentication Failure to Web Application (cont.)

• SSO.NLM expected POST header and data to be in the same packet

Didn’t find POST data so assumed and wrote NULL• iChainFormFillCrib attribute existed but without data

• New SSO.NLM in IC20FP3.EXE fixes problem

Page 121: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Miscellaneous Issues

Page 122: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Miscellaneous iChain Issues

• Troubleshooting iChain installation issues—10068257

• Troubleshooting Mutual authentication issues— 10066648

• Custom rewriter issues—10066908• External rewriter issues—10068222

Page 123: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns

Summary

• Proxy interfaces Inputs and outputs from all dependent modules Flow of information through iChain

• Proxy troubleshooting tools More than enough

• Proxy troubleshooting steps Follow flow and identify broken interface

Page 124: Www.novell.com Avoiding the Top iChain ® Technical Support Issues Neil Cashell Technical Support Engineer Novell, Inc. ncashell@novell.com Shane Johns