Pacific Gas and Electric Company

Ishaq Mohammed / Anil Saka

Online and Middleware Application Operations

2

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

3

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

4

Company Profile

PG&E SERVICE AREA

IN CALIFORNIA

• Headquarters in San Francisco,

California

• Provides energy to approximately

15 million people

• 70,000 square mile service territory

• Total Assets: $50 billion

• 2012 Net Income: $816 million

• 2012 Capital Expenditures: ~$4.5

billion

5

IT Overview - Infrastructure

• 280+ WebLogic Domains (versions 9.2 through 12.1.1)

• SOA Suite & Oracle Service Bus (OSB)

• WebLogic Portal & Integration

• 970+ WebLogic Servers

• 205 Databases (versions 10g through 11.2.0.3)

• 500+ host servers running Linux and AIX

• Microsoft SQL Server, Apache HTTP Server 2.2 and Sun One.

• 2,000 IT employees – Team of 8 in middleware management

6

IT Overview – MyEnergy Business Application

• MyEnergy application most critical from business perspective

• Account management (e.g. customers can monitor /manage energy use, pay bills online)

• Promotion of energy-savings programs

• Scheduled outages of PG&E services

• 2.2 million active users

• 6,000 users logged in hourly during peak times

7

IT Overview – MyEnergy Architecture Diagram

8

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

9

Challenges

• High cost in manually building 30+ middleware environments each year (~$2.5 million)

• 20% increase in WebLogic landscape past 2 years

• What worked in pre-production sometimes failed to work in production

10

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

11

Solution: Oracle Enterprise Manager to

Provision Middleware Environments

• Modernizes the build process

• Guarantees speed to market delivery

• Eliminates errors

• Reduces labor cost and time spent

• Provides out-of-box monitoring

12

Oracle Enterprise Manager Cloud Control 12c

Deployment

• Four environments for Cloud Control 12c – dev, test, stage and production

• Production environment for Cloud Control 12c:

Management Service 12.1.0.3 Release 3 on Redhat Linux with Active-Standby high availability configured

Management Repository in two node RAC DB 11.2.0.3 on Redhat Linux

Management Agents 12.1.0.3 Release 3 on Redhat Linux and AIX

Fusion Middleware Plug-in 12.1.0.5

13

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

14

Middleware Use Cases

WebLogic Server 12c (WLS) version 12.1.1 Oracle Service Bus 11g (OSB) version 11.1.1.6 WebLogic Portal 10g (WLP) version 10.3.4 Small, Medium Or Large Deployment

15

Middleware Provisioning

• Discover reference Oracle Home/domain

• Create provisioning profile in software library

• Create directives in software library

• Edit predefined deployment procedure

• Test deployment procedure from console to capture input parameters

• Create wrapper script which calls emcli “submit_procedure”

• Enable end users to make requests using PG&E’s Provisioning Portal

16

Prerequisites

• Configure Software Library on OEM Server

• Enable passwordless (ssh keys) connection between Oracle Management Service and new VM

• Update the /etc/sudoers with: weblogic

ALL=(root)/u01/app/Oracle/product/agent12c/sbin/nmosudo • Setup named credentials and privilege delegation.

• Install required packages on the OS, create OS user and

group(webLogic)

• Install and start Oracle Management Agent on new server

17

Sample: Provisioning Profiles

18

Directives

• Provisioning Profiles Capture “Standard” Files and Artifacts of WebLogic Domain[Gold Image].

• Additional Files Specific to WL Domain must be moved to Target Host(s) with File Transfer Step or Directive.

• With Directives you upload required Files to Software Library along with script to move and/or Manipulate Files, called Main file

• Be sure to check “Use Most Recent Version” check box, else your procedure keeps calling older version

19

Sample Directive

20

Customized Deployment Procedure

Oracle-provided deployment procedure named "Provision Middleware" customized to accommodate PG&E requirements Custom Directives (Described Earlier), or Shell Script that Execute Shell Commands and/or call WLST We Generally Added One Directive and 3 to 4 Shell Scripts to Customized Procedures Shell Script Cannot Exceed 4096 Characters. Can Use Runtime Variables in Scripts, e.g. ${target.DOMAIN_HOME_DEST_ADMIN_HOST}

21

Sample Deployment Procedure

22

Options to Provision

Provision using console: • Several clicks and repetitive

process

Provision using EMCLI: • Capture once and execute many

23

EMCLI Wrapper Script

• Generated Properties Files Made Generic by Substituting Actual Values with Specific Keywords:

<Hostname> -> MyHostName <Domain Name> -> MyDomainName <DBSid> -> MyDBSid • Wrapper Scripts Creates Copy of Properties Files and Substitutes New Values

Passed on Command Line • Removes Copy When Completed • Provision New WebLogic Portal Domain Example: sh ./emcliWrapperDB.sh "DB_DATA" "S" “database-host“ "Linux" "WLPDBS" sh ./emcliWrapperWL.sh WLP S EIWLP wlp-mw-host database-host WLPDBS

24

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

25

PG&E Provisioning Portal

• Operating Systems

• SAP Environments

• Oracle Databases

• Oracle Fusion Middleware Environments

ArtifactProvisioner

System Provisioning

Provision Requested

Operating System Artifact

ProvisionRequestedSAP Artifact

Provision RequestedDatabase

Artifact

Provision Requested

Fusion Middleware

Artifact

Capable of Provisioning:

26

PG&E Provisioning Portal Processes

• PG&E has built a portal for request intake and complete orchestration of the provisioning process

• Provides central interface to end users

• Makes REST calls to Pureflex System for OS provisioning

• Installs Management Agent

• Invokes EMCLI scripts to provision middleware environments

27

PG&E Provisioning Portal Features

• Authentication against PG&E Active Directory / SSO

• Ability to notify Requestor/Approver via email

• System provisioning clients pay for the resources

• System provisioning clients are not responsible for the maintenance of any part of the shared infrastructure

• Ability to monitor performance of their workloads to ensure SLAs are met

• Provides at least three deployment options for each artifact in terms of

sizing: small, medium, large

28

One Click!

• Provision Operating System • Install Middleware • Configure Domain • Discover Domain to obtain

out-of-box monitoring

29

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

30

Achieved Results and Improved Business

• Standardized OS, middleware and database images

• Reduced overall provisioning time by 75%

• Saved ~$1 million since rollout

31

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

32

Future Plans

Extend provisioning to more use cases: • Oracle SOA Suite • Oracle Business Process Management Suite • Web Servers

33

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

34

Summary

• PG&E uses Oracle Enterprise Manager Cloud Control 12c deployment procedures with custom steps and scripts to provision all Oracle WebLogic Server, Portal and Service Bus environments

• PG&E has built an end-user portal to provision middleware

infrastructure with a single click • Provisioning brings PG&E a 75% reduction in time to market • Many lines of businesses are enjoying the benefits of lower costs,

faster delivery, and standardized deployments

35

Agenda

• Overview

• Challenges

• Solution

• Middleware Provisioning Use Cases

• PG&E’s Provisioning Portal

• Benefits Gained

• Future Plans

• Summary

• Q/A

36

Thank you for your

participation!

Q/A