SAP Start - Application Server Infrastructure

Getting Started Newsletters Store

Search the Community

Welcome, Guest Login Register

Added by Koju Iijima, last edited by Koju Iijima on Sep 17, 2014

Application Server Infrastructure / Application Server Infrastructure / SAP Startup Framework

SAP does not start

SAP system does not start.

We have received many incident reports (aka message) whose titles are SAP system does not start.

For SAP support engineers, this title has broad meanings. Since SAP system is based on the client server technology and consists of many processes or threads, there are many places to be checkedin order to understand what the exact problem you are facing.

What do you mean by SAP system does not start?

By reading the description of the incident report further, the reporters describe the situation like: .. the users cannot logon after restart.., .. the SUM tool says startup failure.., .. my message serverdidnt come up.., ..I cannot see OS processes.., ..I started my own startup scripts but nothing happened.., and so forth.

Some of those descriptions are still quite vague, and some are quite to the point of the root cause.

First of all, please check this question:

Are Message Server, a dispatcher and work processes up and running?

Yes: -> this means that the startup of SAP system was finished successfully. It can be said that the problem is the logon itself, and not SAP system startup. If this is the case, then please avoid using thetitle: SAP system does not start.

No: -> literally it means that SAP system DID NOT startup. Its startup sequence had some kind of problems.

Lets have a look this startup problem further.

This wiki aims to offer troubleshooting guides for startup problems. With this guide, we hope you can analyze and solve the problem by yourself. In some cases, you may still need to raise an incident ticket(aka message). Then, we hope you can narrow down the problem and provide us detailed information so that we can instantly assist you.

Startup sequence

897933 - Start and stop sequence for SAP systems

In short, SAP system instances can be started up by the following sequence:

Database instance1.Central instance or (A)SCS2.Application servers (Dialog Instances)3.

*Database startup problem is not discussed in this wiki.

The following link explains standard methods to start SAP systems:

http://help.sap.com/saphelp_nw70ehp1/helpdata/en/47/fd7230eca159e8e10000000a421937/frameset.htm

SAP MC, SAPMMC and startsap script needs SAP start service:

http://help.sap.com/saphelp_nw70ehp2/helpdata/en/b3/903925c34a45e28a2861b59c3c5623/frameset.htm

As you can see from above link, many processes are involved in the startup. Among those processes, the sapstartsrv process plays an important role of controlling a SAP instance.

The types of the processes which need to be executed in the system are declared in the instance profile (or start profile). Then, the sapstartsrv process starts up the specified processes.

During this startup sequence, each process generates its trace file on DIR_HOME directory.

Common problems and solutions

Quite common problems are related to the sapstartsrv process, especially following two symptoms: The process doesnt exist and The process exists but does not respond.

The process doesnt exist

On UNIX/LINUX, use an OS command ps to check the process:

ps ef | grep sapstartsrvWhen the process exists, the above command shows like this:

abcadm 7012358 1 0 17:39:33 -

0:05 /usr/sap/ABC/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/ABC/SYS/profile/START_DVEBMGS00_walldorf -D -u abcadm

On WINDOS, use WINDOWS task manager tool to see if the service is running or not.

If the sapstartsrv process doesnt exist, please consult following SAP notes and start up.

1762827 - Startup of Instance Service failed in UNIX or Linux environment

The sapstartsrv process exists but does not respond

SAP does not start - Application Server Infrastructure - SCN Wiki http://wiki.scn.sap.com/wiki/display/SI/SAP+does+not+start

1 of 6 05/12/15 6:01 PM

The OS command PS or the taskmanager shows the process. But the sapstartsrv doesnt respond to any actions. Then, most likey, the UNIX domain socket (/tmp/.sapstream) or Windowsnamed pipe (\\\pipe\sapcontro_) may be broken. If it still exists, please manually delete. And start the sapstartsrv service again.

Other cases

Case 1:

Found the following error information in the startsap.log.

exec(): 0509-036 Cannot load program disp+work because of the following

errors:

rtld: 0712-001 Symbol ixml_iXMLParser_IsNormalizingAtt was referenced

from module disp+work(), but a runtime definition

of the symbol was not found.

rtld: 0712-001 Symbol ixml_iXMLParser_SetNormalizingAtt was referenced

from module disp+work(), but a runtime definition

Root cause:

The library path was not configured correctly after upgrade.

Solution:

Configure the library path.

SAP Note 1265456 SAP system does not start after kernel upgrade

SAP Note 1104735 Upgrade to the new instance-specific directory on UNIX

Case 2:

The sapstartsrv service is up but the dispatcher doesnt startup.

sapcontrol -nr 00 -function GetProcessList

30.11.2013 18:46:21

GetProcessList

OK

name, description, dispstatus, textstatus, starttime, elapsedtime, pid

disp+work, Dispatcher, GRAY, Stopped, , , 26680

igswd_mt, IGS Watchdog, GREEN, Running, 2013 11 30 12:57:32, 5:48:49,

26681

Root cause:

The zombie process of the dispatcher existed.

Solution:

Kill the zombie process.

Case 3:

On NT, frequently failover in few minutes.

Root cause:

The version of the sapstartsrv.exe was different between DIR_CT_RUN and DIR_EXECUTABLE.

Solution:

Copy the sapstartsrv.exe from DIR_CT_RUN to DIR_EXECUTABLE.

SAP Note 1043592 MSCS: Cluster Resource Monitor Crashes on W2K3 SP2

SAP Note 1345206 Handling and preventing sapstartsrv.exe corruptions

SAP Note 1375494 SAP system does not start after applying SAP kernel patch

Case 4:

Startsap.log says:

(1159276) Starting: local /usr/sap/ABC/SYS/exe/run/igswd_mt -

mode=profile pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf

(442404) Waiting for Child Processes to terminate.

(442404) **** 2010/04/10 08:07:11 Child 1155152 terminated with Status

255 . ****

(1155152) **** 2010/04/10 08:07:11 No RestartProgram command for

program 2 ****

The information above is very generic. stderr5 file shows more detailed information.

Command: /usr/sap/ABC/SYS/exe/run/igswd_mt

-mode=profile

pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS00_walldorf


2 of 6 05/12/15 6:01 PM

(1159276) **** Failed to Start Local Command. No such file or directory

Root cause:

The necessary files weren't copied correctly.

Solution:

Copy the igswd_mt to /usr/sap/ABC/SYS/exe/run

Case 5:

After kernel patching, the system doesn't come up but the startsap commands runs successful. The dispatcher was immediately down and no

work process is active.

Stderr2 says:

(1114238) New Child Process created.

(1114238) Starting local Command:

Command: dw.sapABC_DVEBMGS01

pf=/usr/sap/ABC/SYS/profile/ABC_DVEBMGS01_walldorf

Fri Mar 5 08:59:56 2010

Profile configuration error detected, use temporary corrected setup

Shared Pool 40: ipc/shm_psize_40 = 2180000000 (too small)

Shared Pool 40: (smaller than min requirement 2186246040)

Shared Pool 40: (estimated size assumed 2190000000)

*** ERROR => Illegal ShmAdm Slot. Key=2104967296.

*** ERROR => ShmGet: Inv PoolKey=2104967296, Key=40, Flag=1, Ptr=1152921504606843304.

***

Root cause:

Shared memory area may need to be adjusted for new environment (i.e, after patching).

Solution:

Run sappfpar check pf= and adjust ipc/shm_psize_xx to suggested values.

Case 6:

The dispatcher went down right after startup. The dev_disp shows:

rdisp/http_min_wait_dia_wp : 1 -> 1

***LOG CPS=> DpLoopInit, ICU ( 3.4 3.4 4.1) [dpxxdisp.c 1706]

***LOG Q0K=> DpMsAttach, mscon ( CLMAGMACI) [dpxxdisp.c 12674]

*** DP_FATAL_ERROR => DpMsAttach: local hostname 'SRMAGMA98' is resolved

to loopback address (cf. SAP note 1054467 for details)

*** DISPATCHER EMERGENCY SHUTDOWN ***

Root Cause

Incorrect setup of the loopback address.

Solution:

#1054467 Local host name refers to loopback address

Case 7:


*** ERROR => gateway (pid 29622344) died [dpxxdisp.c 16763]

*** DP_FATAL_ERROR => Gateway died with status 2 - I better exit now


The gateway trace dev_rd shows:

***LOG Q0I=> NiIBindSocket: bind (67: Address already in use)

[nixxi.cpp 3740]

*** ERROR => NiIBindSocket: SiBind failed for hdl 9/sock 12

(SI_EPORT_INUSE/67; I4; ST; 0.0.0.0:3300) [nixxi.cpp 3740]


3 of 6 05/12/15 6:01 PM

***LOG S0V=> GwStopGateway, gateway stopped () [gwxxrd.c 13096]

Root Cause:

The service port 3300 was declared twice in the /etc/services file.

sapgw00 3300/tcp

sapgw00s 3300/tcp

Solutoin:

Make sure only one service is available.

sapgw00 3300/tcp

#sapgw00s 3300/tcp

Case 8:


sapstart.log

(11925) Starting local Command:

Command: dw.sapPIP_DVEBMGS00

pf=/usr/sap/PIP/SYS/profile/PIP_DVEBMGS00_unjspfsappip

(11856) Waiting for Child Processes to terminate.

(11856) **** 2013/09/19 10:31:13 Child 11925 terminated with Status 0 . ****

dev_disp

*** ERROR => NiDgHdlBindName: invalid hostname 'localhost' (rc=-2;hdl 1) [nixx.c 4341]

*** ERROR => DpCommInit: NiDgHdlBindName failed: -8 [dpxxdisp.c 10339]

*** DP_FATAL_ERROR => DpSapEnvInit: DpCommInit


Root Cause:

hosts file was modified and localhost was misconfigured.

Solution:

Configure localhost in the hosts file correctly.

Case 9:

The sapstartsrv is up.

When you execute sapcontrol without host opetion, you get: NIECON_REFUSED (WSAECONNREFUSED) error. However, it success with -host opetion.

For example:

sapcontrol -nr 00 -function GetVersionInfo

FAIL: NIECONN_REFUSED (WSAECONNREFUSED: Connection refused),NiRawConnect failed in plugin_fopen()

but

sapcontrol -nr 00 -host saphost.sap.co:50013 -function GetVersionInfo -> works!

Root Cause:

Please refer SAP Note 2062508 sapcontrol commands fail with error "NIECONN_REFUSED (WSAECONNREFUSED: Connection refused)" during upgrade

Solution:

Comment out parameter service/hostname for the SAP start service and restart sapstartsrv.

The sapstartsrv exisits. -function GetProcessList doesn't show programs like dispatcher, gateway, etc.

In normal case, you will see something like this:

----



4 of 6 05/12/15 6:01 PM

GetProcessList

OK


msg_server, MessageServer, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7326

disp+work, Dispatcher, GREEN, Running, Message Server connection ok, Dialog Queue time: 0.00 sec, 2014 02 25 05:45:24, 0:00:53, 7327

igswd_mt, IGS Watchdog, GREEN, Running, 2014 02 25 05:45:24, 0:00:53, 7328

---

-->here, you can see programs (message server, disp+work and IGS) are up and running.

But in this problematic situation, something like....

----


GetProcessList

OK


----

--> No programs are listed .

This is because 'Start_Program_n' command with the program is missing in the profile file which sapstartsrv starts with.

For this 'Start_Programm_n", please refer the following link:

http://help.sap.com/saphelp_nw74/helpdata/en/9c/ac0d50a40af26be10000000a423f68/frameset.htm

(NOTE!!! since kernel 74x release, 'Start Profile' is obsolated. You cannot use 'Start Profile' like ealier release. All the information in Start Profile can be marged to Instance Profile)

After upgrading to 74x system, SAP doesn't startup.

In most cases, the reason is the sapstartsrv is pointing to Start Profile. Start Profile is obsolated since 74x kernel. You need to recofigure your profile.

Delete Start Profile physically from 'profile' dir.In UNIX, adjust /usr/sap/sapservices file.In WINDOWS, adjust the SAP service.Delete UNIX domain socket or Window's named pipe, and restart sapstartsrv.

UNIX Domain Socket .sapstream5xx13 was not enabled...

Case 1:

sapstartsrv process exists but the corresponding UNIX Domain Socket doesn't exist.

Checki sapstartsrv.log. It ends at CCMS agent initialization phase:

cat sapstartsrv.old

---------------------------------------------------

trc file: "sapstartsrv.log", trc level: 0, release: "741"---------------------------------------------------

pid 2134

[Thr 01] Mon Sep 15 10:09:13 2014HistoryLog_Init: logfile (/usr/sap/ABC/SCS00/work/history.glf) already exists, check parameterNo halib defined => HA support disabledCCMS agent initialization for instance type SCS: return code 0.

-->that's all.... sapstartsrv should initialize sapcontrol webservice....

Normally, you should see something like:

CCMS agent initialization for instance type SCS: return code 0.CCMS agent disabled by profile parameter ccms/enable_agent = -1.Initializing SAPControl WebserviceAuto PSE update thread startedStarting AutoRestart threadStarting WebService SSL threadAutoRestart thread startedStarting WebService threadWebservice thread started, listening on port 55013Webservice SSL thread started, listening on port 55014Webservice SSL thread using default SAP SSL credentialTrusted https connect via Unix domain socket '/tmp/.sapstream55014' enabled.Trusted http connect via Unix domain socket '/tmp/.sapstream55013' enabled.

Root Cause:

CCMS agent has been running already by saphostcontrol.

Solution:

set ccms/enable_agent = 0 in the instance profile.

Case 2:


5 of 6 05/12/15 6:01 PM

A trace file called sapstartsrv_ccms.log was generated in work dir. The content of i ends with the information:

INFO: Checking lock file.....

Root Cause:

Please refer SAP note 1916333 sapcontrol error "FAIL: NIECONN_REFUSED" due to agent.log file

Solution:

set ccms/enable_agent = 0 in the instance profile.

No labels

Follow SCNContact Us SAP Help PortalPrivacy Terms of Use Legal Disclosure Copyright


6 of 6 05/12/15 6:01 PM

Documents

SAP Start - Application Server Infrastructure