EPICS IOC Diagnostics Tools

Jeff Hill

Overview

EPICS built-in diagnostic tools solutions to specific problems

Background

vxWorks shell allows any function to be called

vxWorks programs frequently provide special functions for diagnostic purposes

syntax is (almost) the same as calling a C function– parenthesis may be omitted

strings must always be in quotes

List Names of Function Blocks (Records) Installed listing all records

– dbl

listing all records of a particular type– dbl (“<record type>”)

list all records with names matching a wildcard pattern– dbgrep

(“*portionOfCommiteeDesignedName*”)

Examining and Writing Function Blocks (Records) Examining a function block (record)

– dbpr “<record name>”, <interest level>

Examining a field within a function block (record)– dbgf (“<recordname>.<field name>”)

Writing a Field– dbpf (“<record name>.<field name>”, “<new

value>”

Tracing Function Block (Record) Processing set trace processing field, “TPRO”, to true

– dbpf “<recod name>.TPRO”,”1”– generates a message whenever the function

block (record) is processed

Setting a Function Block (Record) Processing Break Point set function block (record) processing break

point – dbb ( “<record name>”)

delete function block (record) processing break point – dbd ( “<record name>”)

Single Stepping Function Block (Record) Processing single step lock set after hitting break point

– dbs or dbs (“<record name>”)

continue after hitting a break point– dbc or dbc (“<record name>”)

list records that have breakpoints installed– dbstat

Hardware Diagnostics

Examine the state of device drivers– dbior “<driver name>”, <interest level>

List all interrupt vectors currently in use– veclist – currently does not work on Intel architecture

Channel Access Communications Status Channel Access server status

– casr <interest level>

Channel Access client status– ca_channel_status <task id of client>

List Channel Access event subscriptions– dbel <record name>

Status of database links to other IOCs– dbcar <record name>

Diagnosing Specific Problems

clients don’t connect unexpected exception

Clients Don’t Connect

is the IOC’s CPU saturated?– type “spy” / “spyStop” to see if the server is

starved for CPU– CA server will starve first– Common CPU consumption culprits

• sequencer, driver, record processing load

is the network functioning?– from workstation type “ping -s <IOC name>”

No Response to “ping”

response at IOC’s console port? Missing route?

– no route to off-subnet clients by default– type “routeShow”

No Response to “ping”

creative thin wire (BNC) Ethernet networking?– any network where the connectors fit must be

ok - not!

check for mBuf starvation with “mbufShow”

No Response at IOC’s Console Port interrupt or DMA related problem are

common culprits– exception at interrupt level

• typically caused by device driver problems

– unacknowledged VME interrupts• frequently this is a missing jumper on the VME

back plane

– VME bus master has not released the bus

Clients Don’t Connect

client and server not on the same IP subnet– if not, use EPICS_CA_ADDR_LIST

environment variable

network interface improperly configured?– IP subnet masks of interfaces must match

• on vxWorks type: “ifShow”

• on UNIX type: “ifconfig -a” or “netstat -i”

• on windows type: “ipconfig”

Unexpected Exception

the task which generated the exception will be suspended– type “i” to list tasks and find out which one

determine if there is stack over flow– checkStack

determine if the system is low on memory– memShow

• strange output indicates corruption of pool

Unexpected Exception

before rebooting obtain a stack trace– type: “tt (0x<task id>)

• frequently this is all that is needed

• strange output indicates stack corruption

send a bug report– certain bugs occur only in rare or intermittent

circumstances

Observations

in vxWorks any C function can be used as a diagnostic

if you experience trouble– try to avoid rebooting without saving a stack

trace of suspended tasks