Learning Life CycleManagement (LCM): CommandLine Security SynchronizationThis purpose of this article is to introduce the command lineLife Cycle Management(LCM) utility in Oracle EPM. The LCM toolcan be used to export and import objects that can be foundwithin the Oracle EPM Environment. This includes Security,Essbase, Hyperion Planning, Financial Management … etc. Asonce gets more familiar with LCM, one comes to realize howpowerful the tool is and how empty life without LCM was.Without LCM some of the more detailed artifacts within anapplication were difficult to move between environments. LCMprovides a centralized mechanism for exporting and importingnearly all of the objects within an Oracle EPM application ormodule. The table below is listed to get an idea of all thefacets of LCM.

Application Artifacts by ModuleModule Artifacts

Shared ServicesUser and Group Provisioning

Projects/ApplicationMetadata

Essbase

Files (.csc, .rpt, .otl,.rul)Data

FiltersPartitions

Index and Page files (driveletters)

Application and DatabasepropertiesSecurity

EAS/Business Rules

RulesLocationsSequencesProjectsSecurity

Hyperion Planning

FormsDimensions

Application PropertiesSecurity

Hyperion FinancialManagement

MetadataData

JournalsForms/Grids

RulesLists

Security

Financial DataQuality Management

MapsSecurityData

MetadataScriptsSecurity

Reporting andAnalysis (Workspace)

ReportsFiles

Database ConnectionsSecurity

The LCM tool is integrated into the Shared Services WebInterface. If can be found under the Application Groups tab.Within the application groups there are three main areas ofinterest:

Foundation – includes Shared Services security such as1.Users/Groups and Provisioning.File System – This is where the exported files will go2.by default. The default location is to be stored serverside, on the Shared Services server in the location:E:\Hyperion\common\import_exportUnder this main folder, the contents are broken out bythe user account that performed the export. Within theexport folder, there is an “info” folder and a“resource” folder. The info folder provides an xmllisting of the artifacts contained within the export.The resource folder contains the actual objects thatwere exported.

The LCM Command line tool provides more flexibilitybecause it can be installed on any machine and theresults can be directed to output to any local folder.Sometimes this is very useful if the Shared Servicesnode is a Unix machine, and the LCM users are unfamiliarwith Unix. Simply install the LCM Command Line Utilityon the Windows machine and redirect its output to alocal Windows folder using the –local command lineoption.

Products and Applications – Each registered product will3.be listed and provide a mechanism to export and import

the respective objects for the associated applications,Essbase, Planning…etc.

Going Command LineThe Shared Services LCM GUI is a great way to become familiarwith the LCM tool. However, when it is time to startautomating LCM tasks and debugging issues, the Command LineLCM utility is very helpful. To get started, the LCM CommandLine tool requires a single command line argument, an xml filethat contains the migration definition. The quickest way toobtain the xml file is to use the Shared Services LCM Webinterface to select the objects you wish, select DefineMigration to pull up the LCM Migration Wizard, and follow theprompts until the last step. Two options are presented,“Execute Migration” or “Save Migration Definition”. Choose“Save Migration Definition” to save the migration definitionto a local file.

That is pretty much all there is to it… move the xml migrationdefinition file to the location you have installed LCM. Forinstance, \Hyperion\common\utilities\LCM\9.5.0.0\bin, open acommand line and run Utility.bat as indicated:[crayon-62493024f4079365107294/]

LCM Example: Synchronizing SharedServices Security between EnvironmentsLCM often requires moving objects and security betweenenvironments, such as from a development environment to aproduction environment. While LCM makes it easy, it is not asstraightforward as simply running an export from oneenvironment and importing into another environment. The reasonis that LCM imports work in a “create/update” mode. In otherwords, the operations performed in LCM are typically additivein nature. While the typical LCM method would capture newusers and new application provisioning, it will not handleremoving user provisioning, removing or changing groups, oressentially removing users from the system. This can be aneasy oversight, but it will ensure that the security becomesout of sync over time and can cause issues as well as securityimplications. At a high level, the steps to sync provisioningusing LCM would be:

Export Users/Groups/Provisioning from Source Environment1.Export Users/Groups from Target Environment2.Delete Using Step 2 Results the Users/Groups in Target3.EnvironmentImport Users/Groups/Provisioning into Target Environment4.

Essentially, Step 1 and 4 are the typical import/exportoperations – where security is exported from one environmentand imported into another environment. However, two additionalsteps are necessary. In Step 3, the users and groups in thetarget environment are deleted, removing provisioning too.

This leaves an empty, clean environment to then importsecurity, ensuring no residual artifacts remain in theenvironment. To use the LCM delete operation, a list of itemsto be deleted must be supplied. This is where Step 2 comes in,a simple export of the Users and Groups in the Targetenvironment will provide the necessary information to provideto Step 3 – deleting the respective users and groups.

Below are some sample XML migration definitions for each step:

Step 1 – Export Users/Groups/Provisioning fromSource EnvironmentNote: By default the results will be sent to the source SharedServices server in the “import_export” directory. You can useLCM to redirect the output to keep the results all in the sameenvironment (the target system) by using the command lineoption [-local/-l] (run utility.bat without any command lineoptions to see help for your version of LCM). Simply redirectthe results into the local folder,\Hyperion\common\import_export, in the Target system.[crayon-62493024f4088395191919/]

Step 2 – Export Users / Groups from TargetEnvironment[crayon-62493024f408b730090475/]

Step 3 – Delete Users/Groups in Target Environment[crayon-62493024f408e894417872/]

Step 4 – Import Users and Groups into Clean TargetEnvironmentThis step assumes that Step 1 was redirected onto the targetenvironment within the import_export directory. The respectivefolder, Step1UsersGroupsSource, can also be manually copied

from the source to the target environment without using theredirection to a local folder technique.[crayon-62493024f4090691344931/]

Troubleshooting with Command Line LCMLCM can be a great tool when it works flawlessly. However, itcan quickly become part of mission critical activities likepromoting artifacts from development to production.Consequently, it is necessary to learn some troubleshootingskills to maintain business continuity using LCM.

Review the output of the LCM operation. Usually it will1.provide some detail about the error that was received.Review the server side Shared_services_LCM.log in2.ORACLE_HOME\logs\SharedServices\SharedServices_LCM.logTurn on debugging for the command line LCM tool. Change3.the line “info” to “debug” in the filesE:\Hyperion\common\utilities\LCM\9.5.0.0\conf in log.xmland hss-log.xml<param name=”Threshold” value=”info” />Use Google, the Oracle Knowledgebase to search for more4.information.Try only a subset of the initial objects. For instance,5.Essbase can export a number of objects, Outline, CalcScripts, Rule Files, Report Scripts, SubstationVariables, Location Aliases, and Security. Try one at atime to determine which part of the whole is failing.Restart the environment. LCM is an emerging technology6.and can sometimes just be in a bad state. I’ve seencountless LCM issues where bouncing the environmentclears the issue up.Look for special characters that might be present in7.your data. LCM is a java tool and uses xml and textfiles to transmit data. There are instances wherespecial characters can mess up the parsing.Look for patches – as mentioned previously, LCM is an8.

emerging technology and is still somewhat buggy(especially older versions). Check release notes inpatches for enhancements/bug fixes in LCM.

Hyperion Troubleshooting andDebugging Guide Part 2 of 2This section will talk about how to dive into debuggingcritical issues with Oracle EPM.

Start a Problem Log

The most useful habit to develop during issue resolution is tostart a detailed log about the issue. Some problems can takedays or weeks to resolve and require trying hundreds ofdifferent prospective resolution attempts. It is easy for a“small” problem to become a long winded issue. Consequently,it is hard to foresee when the issue will resemble theanalogous onion: keep peeling off layers and finding more andmore to fix. If the problem log is created initially, all theimportant details can be captured. Additionally, it is mucheasier to bring others up to speed (management) and createsupport tickets when all of the information is documented.This log should include the error as the end user sees it, theerror from any logs you are able to capture, screenshots,timestamps, and things that you have tried along with theresults.

Reproduce the Issue

The first thing to find out is whether the issue isreproducible. It is very difficult to solve an issue that isnot reproducible. Many errors are simply ‘glitches’ and may

have been caused by a very improbable event, such as adatabase hiccup. For instance, a database problem propagatesinto the Oracle EPM system, forcing it into a bad state. Sucha problem may never produce itself again. Consequently, aninitial step toward resolution is to restart the Oracle EPMservices to bring them back into a ‘known state’. If theproblem is not immediately reproducible after the restart, goback to the problem log and record everything you can. Thistype of issue will need to be profiled over a period of timeto try and discover patterns if it occurs again.

The Numerous Logs

Once it is discovered that the issue is not a simple glitch,it is time to start digging. As mentioned previously, thefirst place to track down the cause of an issue is in thelogs. The logs come in various forms. Here is a generalbreakdown of the log types:

General

Log Type DescriptionWindows Event

ViewerThis is helpful for general systemrelated messages. Also some modulesbuilt on Windows Technology (DCOM)will log messages here. For example,

Financial Management (HFM) andFinancial Data Quality Management

(FDQM).Application Logs The application logs are actually

generated by the Hyperion code itself.These often contain the most useful

information.

ApplicationServer Logs

This type of log pertains to a Javabased Web Application. Most of theHyperion modules with a web basedfront end have Application Server

Logs. The Application Server Logs runwithin the WebLogic, Tomcat, or

WebSphere container.Web Server logs The web server controls the handoff of

web requests between the HyperionModules. The best way to use this logis to look for error codes (404, 401…etc) in the web log and review thecorresponding URL that was used toensure it is correct. Sometimes itmight be obvious that the URL in theweb log has the wrong domain, pointsto the wrong server, or cannot resolve

the context.

Start by reviewing the log for the product where the error isoccurring. The Application Logs and Application Server Logswill be most useful at first. The goal is to find a usefulerror message that can be used in the next process to find aresolution to the problem.

Common Log Locations:

Unfortunately, the actual log locations change drasticallybetween recent versions of Oracle/Hyperion products. As statedbefore, searching for *.log might be useful.

Example Application Server Logs:

EssbaseAdmin

Services

Svr2 /Oracle/Middleware/user_projects/domains/EPMSystem/servers/EssbaseAdminServices0/logs

Workspace Svr1 /Oracle/Middleware/user_projects/domains/EPMSystem/servers/FoundationServices0/logsFinancialReporting

Svr1 /Oracle/Middleware/user_projects/domains/EPMSystem/servers/FinancialReporting0/logs

AnalyticProviderServices

Svr2 /Oracle/Middleware/user_projects/domains/EPMSystem/servers/AnalyticProviderServices0/logs

Web Analysis Svr1 /Oracle/Middleware/user_projects/domains/EPMSystem/servers/WebAnalysis0/logs

Example Application Logs

Reportingand Analysis

Core

Svr3 /Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/

/Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/ReportingAnalysis/stdout_console_default.log

Essbase Svr4 /Oracle/Middleware/user_projects/epmsystem1/diagnostics/logs/essbase/

Sifting Through the Logs:

It helps to know which modules depend on each other in orderquickly pick out the respective log files to analyze. Thebasic idea is to determine which products are interacting andto review each log in detail for messages. It is important toreview the logs of the product not only during runtime (as itis happening), but also during startup. Sometimes the fastestway to cut out the fluff is to stop the services, move ordelete all the existing logs and start the environment backup. This ensures any log messages are relevant to the issue.Alternatively, one has to sift through potentially large logslooking for timestamps to ensure relevance, which can bedaunting.

Product Depends OnSharedServices

Relational Database, MSAD/LDAP

LifecycleManagement

(LCM)

Shared Services, LCM Source/Targetapplications

Essbase Shared ServicesHyperionPlanning

Shared Services, Essbase, Business Rules,Relational Database per App

BusinessRules

Shared Services, Hyperion Planning, Essbase,Relational Database (single database)

HyperionFinancialManagement

Shared Services, Relational Database (singledatabase), DCOM (Event Viewer)

FinancialData QualityManagement

(FDM)

Shared Services, Relational Database perApp, Adapters for Essbase, Planning,

HFM…etc, DCOM (Event Viewer)

StrategicFinance

Shared Services, Relational Database(optional)

DataRelationshipManagement

Shared Services, Database Client, Adapters,DCOM (Event Viewer)

Found an Error Message!

After discovering the error message, the first thing to ask isdoes this message make any sense? Try to use it within thecontext of your problem to solve the issue. Often, it isnecessary to use external resources to resolve the issue. Useresources like Google, the Oracle Support Knowledgebase, andthe Oracle Forums to further research the issue. Most oftenthere will be information regarding your issue.

Note: If possible do not searching using end user messages,i.e. what the user sees when encountering the error. Rather,find a detailed message in the logs. The end user messages areusually very generalized and can provide misleadinginformation because of the vast number of issues which mightmatch the general error message.

If there is still a struggle to discover a useful errormessage, most of the Hyperion modules use a logging mechanismthat can be changed into debug mode. The actual method willdiffer based on product, for instance, most modules use log4jand there is often a .properties file you can change the

logging level from “ERROR” or “WARN” to “DEBUG”. For instance,to enable debugging in Hyperion Planning: log into a Planningapplication, go to Administration -> Manage Properties, Selectthe System tab, Add the property DEBUG_ENABLED with a value ofTrue. After changing the logging level, the service will needto be restarted to reflect the changes. Turning on applicationdebugging should provide more context clues around what theproduct is doing at the time and help pinpoint the error.

Nothing Found…

If these resources do not help, an Oracle Support Ticket maybe required. Additionally, the Oracle Forums can be a goodplace to post a question. When creating a support ticket andposting to a forum, please include as much information aspossible. This is where the Problem Log will come in handy.

This is a good time to look for updates and patches to theproduct. Check for patches and updates onhttp://support.oracle.com. Read the release notes for anythingmatching your problem. Even if there is nothing coming up,some obscure errors can be solved by simply applying a patch.Not all bugs will be in the release notes for the patch.Oracle’s hpatch process is pretty straight forward, but olderenvironments might take some time to apply the patch. Alwaysread the entire release notes and installation instructionsbefore applying a patch. Also, sometimes patches are not asproven as the initial installers. This is because some patchesmay have just been release and only tested with a handful ofclients. So ensure there is a good backup process in case thepatch causes unintended problems. The Oracle hpatch processhas a back out feature, but it is not always useful if thepatch is half way installed and failed.

Finally, the last part of troubleshooting is intuition. Asmore problems are encountered and resolved, one can become

more confident in resolving upcoming issues. There is no wayto have encountered every issue and know the resolution, sothe best that you can do is arm yourself with a good knowledgeof the architecture, have a set of best practices, and lots ofpatience for problem solving.

Oracle EPM Troubleshootingand Debugging Guide (Part 1of 2)This article will discuss some best practices aroundtroubleshooting and debugging your Oracle EPM environment.

Oracle EPM (Hyperion) is a complex system that is composedmany modules that work together to perform differentfunctions. Typically, in design, creating a modular approachis a best practice. However, the Oracle EPM modules werecreated with disparate architectures because they acquiredover time through purchasing technology from many smallercompanies. As a result, the modules interact only at a highlevel, through network APIs. The result is a complex systemwith many breaking points, and error messages that can be lessthan informative. Consequently, troubleshooting and debuggingHyperion is an art form.

The fastest way to debug a critical problem is to have a goodunderstanding of what components represent a known, “good”state. Only by identifying the abnormal element in the systemcan one start to resolve a critical issue.

Below Are Best Practices to Follow to Document a Known, GoodState

What Should Be Running?The first question you should ask is, “is everything up?”Typically you would start by checking the URLs you use foreach product – ensuring you can login. These URLs wouldtypically include the load balancing name.

Example of End User/Administrative URLsSharedServices

http://hyperion.svr.com/interop

HyperionPlanning

http://hyperion.svr.com/HyperionPlanning

EssbaseAdmin

Services http://hyperion.svr.com/easconsole/console.html

Workspace http://hyperion.svr.com/workspace

WebAnalysis

http://hyperion.svr.com/WebAnalysis

FDM http://hyperion.svr.com/HyperionFDM

Set Up Port Monitoring SoftwareYou can also get a ton of info in a quick glance by setting upport monitoring software. Remember to include the relationaldatabase in your monitoring. Sometimes the DBA will take thedatabase down unexpectedly, or the database user IDs willexpire. Additionally, it is easy to have 20-30 Hyperionrelated services or processes running per environmentdistributed among multiple hosts. It’s too time consuming anderror prone to check manually.

An Example of a Custom Java Based Port Monitor

To gather a list of ports within the environment there are afew aids to use for reference. Oracle EPM Version 11 includesa diagnostics web form that will show the status of some ofthe products. This can be found under the Windows Start Menu,under Programs -> Foundation. This might be a good place toget started.

Also, refer to the Oracle EPM Documentation, Install StartHere Document for typical port reference.

http://download.oracle.com/docs/cd/E17236_01/epm.1112/epm_install_start_here.pdf

An Example of Some Common JVM Ports

ApplicationServer

Cluster/Node(s)Port

Workspace svr01 45000

Web Analysis svr01 16000

Financial Reports svr01 8200

Shared Services svr01 58080

Analytic ProviderServices

svr04 13080

Hyperion Planning svr02 8300

Strategic Finance svr04 7750

Essbase AdminServices

svr04 10080

EPMA svr02 19091

Stopping and Starting the EnvironmentOperational procedures are important too. This means having areliable start and stop procedure for the environment. Giventhe complexity of the Hyperion environment, there is no excusefor problems occurring while bringing an environment back upafter routine maintenance. One common issue that comes up is aservice did not start completely, which can be found by usingthe port monitoring method above. Also, sometimes the servicesdo not fully stop – causing residual processes to mess up therestart procedure. This might include extra ESSVR processesindicating the Essbase application did not stop properly, or aJVM process which is hung. As a precaution, check the portmonitoring software after stopping the environment to ensureall the components are indeed down. It is time consuming whenthis type of issue is encountered. In a Windows environmentyou might take the environment down through your normalprocess, then kill any remaining “Java” and “Hyperion” relatedprocesses using task manager. In a Unix environment, afterstopping the environment, “kill -9 -1” can be used as a lastresort. A reboot will always solve this type of issue, thoughnot often actually needed; it might be faster as a last resortfor the unskilled admin needing to fix unexplainable issues.

Functional Validation ScriptFunctional validation is a set of very simple actions that canbe performed within a few minutes to validate thefunctionality of the environment. This is absolutely necessaryto perform before handing the system back over to end usersafter a routine maintenance. This task can be eitherperformed by the Hyperion Admin or handed over to the HelpDesk. It should, at a minimum, include running a few typesreports (HFM, FR, WA), log into Essbase, Planning, HFM, etc…As you discover other reoccurring issues you may want toinclude more specific checks to ensure common “gotchas” areironed out before handing over to the end users.

Knowledge of the ArchitectureThe Hyperion administrator has to be technologically savvy.However, many times Hyperion will tell you exactly what theproblem is. The first place to look when encountering an issueis the logs! Take time to find and document the log files forevery product in your environment. Most administrators are abit intimidated by the number of logs generated by the system,but this is the first place to look. Familiarize yourself withthe various logs in the environment. This includes WebLogiclogs, JVM logs, Hyperion logs, and operating system logs. Ifyou are lost, a good place to start is search for *.log, sortby date, and look at the directory for the product name. Youmight even keep a record of each log after starting theservice successfully so that you can compare to a good stateif there is a problem.Additionally, it is necessary to have context around how theenvironment works. This can be achieved by reading the productmanuals, looking at architecture diagrams, and attendingtraining sessions. Get involved in the environment. Try tounderstand both the technical details (ports, logs, differentcomponents…etc) and the functional basics (create a planningform, edit an outline, load data into Essbase, run a

consolidation) of each Oracle Module.

A Good State: Create a Knowledge BaseHyperion issues commonly come up more than once. Create ashared, searchable, knowledgebase to track issues and theirresolutions. This will help you train your team as it expandsand changes over time.

Final CommentsThe focus of this article was to provide the Oracle EPMAdministrator with best practices on how to document thecurrent state of the Oracle EPM System. This is the mostimportant part of being able to understand complex issues asthey come up. More technical detail will be provided in theOracle EPM Troubleshooting and Debugging Guide (Part 2 of 2).

Navigating Misaligned Menusin Workspace with IE8

If you have recently upgraded your web browser to IE8 and attempted to use

Hyperion Workspace, you’ve likely encountered difficulty in navigating the

interface. When a user clicks on the wheel in the upper left hand corner,

selects “Applications,” they can’t see the menu as it apearrs to be

condensed.

This issue can be quickly resolved by making a browser modification to the IE

security. It is best to run this settings change past your IT department to

ensure it will not open holes in your company’s security profile.

To correct this issue,

1. Open IE8

2. select “Tools”->”Internet Options”

3. Select the “Security” tab and “Custom Level”.

4. You will have 4 zones where security changes can be performed

(Internet, Local Internet, Trusted sites, Restricted sites). The change

can be made to each of these 4 zones if necessary, but it’s possible

only one zone needs modified. Test the combination that works for your

environment and fits your companies IT/security profile.

5. Select the “Internet” zone and scroll down the menu of options to

find “Allow script-initiated windows without size or position

constraints”. This will likely be set to “Disable”… select “Enable” and

click “OK”. (If prompted, accept the change just made to the zone

security).

6. Select “OK” from the security tab to finalize the modification.

7. Test the modification to verify the change worked as intended. As

noted above, you might need to enable this setting on multiple zones in

order for the setting to take effect.