Upload
emerson-exchange
View
112
Download
4
Tags:
Embed Size (px)
DESCRIPTION
Presented by Amgen's Auguste Dionne, Dan DiMatteo, and New England Controls' David Maglaya at the 2011 Emerson Exchange in Nashville, Tennessee.
Citation preview
DeltaV Version 10.3.1 Upgrade in 24hrs
DeltaV Version 10.3.1 Upgrade in 24hrs
Version 8.4.2 – Version 10.3.1
PresentersPresenters
� Gus Dionne, Automation ARI
� Dan DiMatteo, IS ARI
� Josh Gangl, OIS Platform Lead
� David Maglaya, Emerson LBP
IntroductionIntroduction
DeltaV Version Upgrade
� Planning the Upgrade
– Objectives, Deliverables, Schedule, Resources
� Executing the Plan
– Developing a Team, Direct and Manage the Execution
Monitoring and Controlling� Monitoring and Controlling
– Verify scope, Integrate Changes
� Closing the Project
– Lessons Learned
These are all requirements of a successful upgrade.
ARI Mfg. Site 75 Acre FootprintARI Mfg. Site 75 Acre Footprint
At a site of this size, planning is key to success
Planning and Defining Our ScopePlanning and Defining Our Scope
� Our goal was to upgrade two (2) DeltaV Systems with
minimum impact to users
� Our constraints were Time (24hrs) , Resources, and
Space
� We needed to upgrade not only the Software Application We needed to upgrade not only the Software Application
but we needed to upgrade most of the hardware that it
would run on
� And we also needed to successfully complete these goals
in the middle of a major project, which had extensive
software impact
Additional ScopeAdditional Scope
� 8 of our 18 Servers would need to be updated from Dell
2850s to R710 (18 servers in the environment– 16 DV
servers plus 2 dedicated domain controllers which were
updated to PE2950 hardware)
� 70 out of 92 controllers would need to be upgraded from
MD’s to MD Plus and MX modelsMD’s to MD Plus and MX models
� All of our 65 OITs would need to be upgraded from 5 and
6 year old models to Optiplex 780s
� Add an additional 19th server to manage Critical Data
Backup Application (CDBA) as part of the upgrade
Tools and ResourcesTools and Resources
� We would need high performance tools to complete the
upgrade in a 24hr window. The tool bag would have the
following +
– Imaging tool for OITs
– A Broadcast Server for deploying images
– Racks for Server buildup and testing– Racks for Server buildup and testing
– Spare OIT hardware for testing
– SI Dongles to turn all OITs to Development stations
– A team of Emerson Support Engineers
– Hard copy checklist for sequential steps
Tools and Resources (additional)Tools and Resources (additional)
– A (Time smart) detailed DeltaV Excel Add-in spreadsheet for
saving and restoring critical controller online parameters
– A controller map depicting model, new Mac address of hardware
and installation location
– A team of hands on Tech’s and Engineers was created– A team of hands on Tech’s and Engineers was created
– A timed schedule was created with each task for each person on
the team
The DeltaV Excel Add In Tool exampleThe DeltaV Excel Add In Tool example
Having a spreadsheet with macros simplifies work and
reduces risk.
The Controller Tracking ID ToolThe Controller Tracking ID Tool
Controller Tracker
The Simple Checklist goes a long wayThe Simple Checklist goes a long way
Organized tasks with priorities are a must for capturing
essentials and ensuring nothing is missed.
The Early ExecutionThe Early Execution
� Offline Migration Testing from 8.4.2 to 10.3.1 at LBP office
� 60+ new OITs would be deployed with the 8.4.2 image
prior to the upgrade day
� 10.3.1 Images would be created and tested in plans for
broadcasting to the new OITs on the upgrade day
� Each new controller to be deployed was labeled and Mac � Each new controller to be deployed was labeled and Mac
address recorded for its intended new location. This saved
the light flashing process at each location for the new
controllers during commissioning.
The Early Execution cont. The Early Execution cont.
� A disk was pulled from the Domain Controller and used to
copy the domain users onto the new DC box, (this was
done for offline code development in the Dev
environment, we used the live DC to migrate the Active
Directory during the outage)
� Critical parameters that needed to be restored would be � Critical parameters that needed to be restored would be
listed in Time Smart XL add in tool
� Server class boxes would be pre-built, racked, stacked
and configured as far as possible prior to execution and
left powered on until upgrade
� The Pro Plus Servers had Database imported at Version
8.4.2 and then the 10.3.1 upgrade wizard run
The Execution prep(the night before) The Execution prep(the night before)
� Critical parameters that needed to be restored would be
uploaded to the XL add in tool with an individual time
record for each time-sensitive value
� Controllers with outputs that needed to be maintained
would have the manual field overrides enabled
� 70 of the 92 controllers would be decommissioned and � 70 of the 92 controllers would be decommissioned and
replaced with the new pre-labeled controllers
The ExecutionThe Execution
� The execution started at 7:00am with the team
concurrently working the plan from a central automation
lab room, server room, and plant floor
� Servers previously racked and stacked are placed online
starting with the Pro Plus boxes
– Legacy DeltaV 8.4.2 PPS servers were “failed”
• Metadata cleanup, FSMO roles seized by dedicated DC• Metadata cleanup, FSMO roles seized by dedicated DC
– New PPS R710 hardware racked & stacked
• DCPROMO / DNS config, FSMO roles transferred to new PPS
The ExecutionThe Execution
� 10.3.1 Images are concurrently broadcasted to multiple
OIT stations seven (7) at a time
� A team of IS engineers are running OIT configuration and
Sys Prep at pre-scheduled OIT locations
The Execution continuedThe Execution continued
� As OITs became available controllers were commissioned
with no field assistance due to Mac address reservation,
then flashed, then downloaded
� Each OIT coming on line was used to RDP to field OITs to
multiply the Flashing capacity of the automation lab (SI)
� Critical online controller parameters were restored using
the DeltaV Time Smart Add-in tool. Online values that were
time sensitive were updated from within Excel, based on
the time the value was stored and the current time of
restoration.
The ClosingThe Closing
� As Systems came back online they were turned over to
Validation then manufacturing
� A Lessons Learned session was held shortly after
Lessons learnedLessons learned
Some Good Points
� Have a well defined plan and schedule
� Trained resources
� Pre building and Server Prep is key to saving time
� RAID 1 (mirroring) with hot spare disk setup to support DC
transfer (easily export Active Directory from dedicated transfer (easily export Active Directory from dedicated
domain controller – Pro Plus and other DeltaV servers use
standard RAID10 config)
� DeltaV Excel Add-in was key in restoring online values
quickly without manual calculations
� Broadcasting of OIT image saved time over upgrade
wizard
Lessons learned cont.Lessons learned cont.
Some Bad Points
� We were overstaffed the morning of outage. Staffing
schedule could be improved, (Pro Plus delay due to Active
Directory DCPROMO issue added to this)
� VX dongles were needed and not known right awayVX dongles were needed and not known right away
� Some user accounts got corrupted due to Schema mis-
match
� Servers pulled from service should have been left in racks
for a week or two since some data configuration recovery
was required
Lessons learned cont.Lessons learned cont.
Some more Bad Points
� Implementing an upgrade with concurrent complex
software changes is extremely difficult
� (Requirements were changed by people outside of core
project team – caused problems with keeping plan locked project team – caused problems with keeping plan locked
down) – point here is to get everyone to commit to
providing requirements up-front and early in the planning
Business Results AchievedBusiness Results Achieved
1. There were no Safety related incidents due to any
upgrade associated actions
2. The DeltaV System went from running version 8.4.2 one
evening to running version 10.3.1 the following evening
3. 18 Servers, 65 OITs, and 70 Controllers were upgraded
(8 R710 replacements and 8 PE2950s were upgraded in
place using Upgrade Wizard, 2 DCs were upgraded to place using Upgrade Wizard, 2 DCs were upgraded to
PE2950 hardware)
4. There was no loss of production outside the planned
outage due to the upgrade
5. There was no loss of utility access due to the upgrade
SummarySummary
� We have done our best to share how we planned,
managed, and executed a successful DeltaV software and
hardware upgrade within a large scale Bio-Tech
manufacturing Facility
� Our benefits are faster, better performing, and more
capable software and hardware to the end user as well as capable software and hardware to the end user as well as
a more robust system for the administrators
QuestionsQuestions
I’d like to take this time to ask if anyone has any questions
that pertain to the project upgrade?
FeedbackFeedback
If any of you would not mind sharing any feedback you might
have we would all like to know if you enjoyed the
presentation or not?
Thank you all so much for attending.
Where To Get More InformationWhere To Get More Information
� Other training sessions
� List books, articles, electronic sources
� Consulting services, other sources