Af11psug Cs05 Bama-companies

8/13/2019 Af11psug Cs05 Bama-companies

1/22

Copyright 2011 Rockwell Automation, Inc. All rights reserved.

Insert Photo Here

Rockwell AutomationProcess Solutions User Group (PSUG)

November 14-15, 2011

Chicago, IL McCormick Place West

A PlantPAx Virtual

Batch Upgrade

Sean Nichols

Controls Engineering Lead

The Bama Companies


2/22

The Bama Companies

History: Founded in the early 20th century

Privately held

All US production based in Tulsa, Oklahoma

Inclusive environment (all stakeholders at every level

involved)

Products made include:

Biscuits

Pizza dough

Hand held pastries

Pie crusts

Breadsticks



3/22

Sean Nichols Bio

Educational

Nuclear power (Navy)

Electrical Engineering from OSU

Extensive Rockwell training

Occupational

5 years with the Naval NPTU

7 years with Bama Companies

5 years maintenance

2 years as Corporate Controls Engineer

Current responsibilities include plant floor controls

and visualization for all Tulsa based plant

locations Personal

Married, 3 kids, one on the way; live in Tulsa,

Oklahoma

Interest include: Family activities, computers


4/22

Process Description

Freezer

Packaging

Ingredients

Mixing

Sheeting Line


5/22

Goal: Address Challenges with Frozen Dough

Pre Improvement State: System could get out of sequence

Had difficulty recovering from power outages

Legacy technology (PLC-5 and RSView32 based platform) Maintenance and troubleshooting had to occur at a Ladder Logic level

Changed programing on average once a day (Bama-tized)

Constantly tweaking to try and make process run better or make recipe changes

Desired Improved State: Environmental and UPS considerations to make system more reliable Improvements for recipe security and operator tracking

S88 based programming (modular equipment/phase coding and flexible recipes)

Complete documentation sets around functional operations

VMware and thin client architecture ( Jointly driven by IT and Engineering ) Electronic hand adds and other SOPs (goal to eliminate batch sheets)

Electronic lot tracking to meet regulatory compliance

Whatever chosen would be planned at other facilities


6/22

Solutions Evaluated

All technology was to remain Allen-Bradley based

Installed base and familiarity with products

Three independent implementations reviewed:1) PC based batch manager integrated with ControlLogix; delivered by a System

Integrator

- Recipe configuration, Campaign management, Batch reporting

2) ConrolLogix and RSView32 system custom created for Bama; delivered by SystemIntegrator

- Lowest HW/SW investment

- Already familiar with the process

- Concerns around bugs; (Bama only knowledge)

3) PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell Automation

- Recognized Industry leader (Price Premium)

- Dedicated Food and Beverage group

- Off the shelf application



7/22

Vendor Selected


PlantPAx (FTBatch, FTView, ControlLogix) system delivered by Rockwell

Automation

Only one to formalize/organize a User Requirements Spec

(Was a billable activity but of value)

System interruption was designed to go into a Hold state

Power outages and brownouts are common at our facilities

Other systems would end up scrapping batches

Flexible recipe management

True S88 compliant offering


8/22

Delivery Team

Bama Foods

Sean Nichols:

Project Manager & EngineeringLead

Nick Huddleston

Process Improvement Mngr.

Operations Lead

John Blaylock

Networking and SW Installs

Additional Bama Staff SQL integration team

4-5 Automation Techs

4-5 Operator Blackhats

Executive Sponsorship (Budget rework)

Rockwell Automation Solutions

Lynn Hecathorn:

Project Manager Dwayne McAlexander:

Lead Engineer

David Keen:

HMI Implementation

Jed Hepfner:

Field Support, Training, Testing,

Startup support


Industrial Controls of Ok.

Troy Bridenstine:

Electrical Foreman


9/22

Project Execution: Foundation Work

User Requirements Specification and Proposal

Interview and captured voice/concerns from IT, Operations (Plant and Production),

Maintenance, and Quality

Allowed for project to be on fast track Knowledge of third party expert (and implementer) was trusted in design

Justified a network audit prior to startup

Contract Negotiations (Good Faith Start)

Mutual investment in creating a sustainable relationship

Required the same engineering lead to design and implement (to the best extent

possible)

Functional Specification Mapped out entire process (Blueprint)

Completed and approved before design/coding started

9 revisions by Bama lead operators, engineering, maintenance (inclusivity)

Contractually binds the deliverables and acceptance criteriaCopyright 2011 Rockwell Automation, Inc. All rights reserved.


10/22

Project Execution: System Topology


19 Logical Racks 500+ Control Modules

9 Weigh Scales

22 Batch Units 105 Phases

25 Active Recipes

1 ESX Server FTView SE VM

FTBatch VM

SQL VM

Citrix Published Apps

10 Thin Clients

Redundant HardDrives

Failover Scheme


11/22

Project Execution: Hardware Design

ControlLogix L74 Controller Supports existing Control Modules plus room

for spare w/ redundancy

Primarily Ladder Logic (standardprogramming language for Bama)

Limited Function Block used for certaincapabilities

Reuse as much RIO as possible

(2)1756 Remote I/O Scanners

1771 Racks

Weigh scales

Local PLC (rack 0) replaced with ASB


12/22

Project Execution: Hardware Design

Added new FLEX I/O to Ethernet

Multiverter (hardware that was not previously

automated) Added Operator Interfaces for Hand Adds

Designed to be PanelView;

Switched to thin client to becompatible with Windows

Domain Servers (required

for tracking scheme)


13/22

Project Execution: Software Design


FTView SE 250 Screen License

RA standard GEMS Library version C2 (~6 module classes)

Event logs from each client and alarm logs are dumped to a SQL database through anODBC connection

FTBatch 22 Units

Incorporated all Hand Adds, Equipment Checks, and SOPs through operator prompting Batch journal is dumped to a SQL database through an ODBC connection

All recipes created by Bama

Found to be much more flexible then previous system

Plans to talk with Product group about ideas for future enhancements

Ultimate recipe and procedural edits will be owned by Quality and Process Improvement

SQL Database Existing System prior to upgrade

Uses Business objects (our ERP)

Data table work required but ultimately working with our existing reports


14/22

Project Execution: Virtual Configuration &

Deployment

Hardware:

Dual ESX servers; Hard drive SAN controllers connected by dual fibers (14 HDspartitioned as 2 arrays)

One HD in each array is a hot spare (Standard RAID technology) Each ESX server has a 1GB switch with dual connections

Redundant fiber to core switch

Off site backup process for all images

Software

System originally built for existing IT/Business infrastructure

Process system added with minimal changes

New VLAN and a few switches standard growth

Not completely new concept (running RSView32 Virtually for 3 years)

Ten (10) Thin clients being managed by single terminal server image Install once, use multiple, replace easily

Only ini file needs to be updated with MAC ID of new hardware

Three (3) virtual servers dedicated to the PlantPAx system

Fourth virtual server (SQL) offsite Three (3) Citrix Applications


15/22

Project Execution: Control Code

Implementation

RA standard GEMS Control Modules version C2

Phase Manager using a sequencer that can be controlled internally or

externally (operators can walk through an entire batch manually if need be)

Supports strutt (scheduled checks/test of equipment)

Needed additional units to insure appropriate arbitration

Comments can be added at any point to the batch record

Broke recipes into a wet mix and mixer class (3 mixers)

Batch ID transfers automatically

Does a material check to make sure product compatibility

Can run multiple products in unison



16/22

Control Module Checkout


Purpose to verify that all GEMS control modules used were compatible with

the 1771 I/O and that the ControlLogix controller can communicate with all

the racks

This process requires downtime as it temporarily puts the new hardware

topography in place and test the system to manage the risk of a swap over

During our specific checkout:

4 racks visible to the PLC5 but invisible to the ControlLogix until we fixed one dipswitch setting and repaired a damaged wire

Found the weigh scales were wired for DH+ instead of RIO

This was a lifesaver for startup because it eliminated hurdles that would

have existed otherwise


17/22


18/22

Startup and Commissioning


Application Training: Valuable but could have been better

No one was previously at formal batch training (informal application training)

Plan was to train first and then do FAT

Was a lecture/watch format with electronic content provided by Rockwell instead of

do format (pros and cons)

7 day shutdown Standard Bama SOPs included Rockwell (and all

other contractors, over 40 total) in planningmeetings to create master schedule and resolveconflicts up front

Used a field resource (Jed) to accompany Dwayneof the 24/7 coverage. Jed had worked on theproject and come onsite with Dwayne for a fewdays beforehand to insure a smooth supportstructure.

Monday startup gave plenty of run time todebug/familiarize and allowed for 24/7 coverage togo to remote support by the weekend


19/22

Lessons Learned


Not all Units are Created Equal: Typical Units = mixers, hoppers, slurry tanks, etc We found others such as a Salt Chute.

Remote ESX Images: Installations and operations from VPNs and firewalls can be a challenge. Using vSphere as a Citrix App

seems to remedy the situation.

Local RA Field Support: Do your best to have Rockwell Automation use a local resource with the appropriate skill set so that when

the delivery team is complete you have expertise around the corner. (We are also attempting to do thisthrough ICO)

Dont use Microsoft auto updates: IT and Eng have an agreement to work on required patches together and determine how to deploy

FAT: Was short as we were trying to train at the same time we

were checking out; should not have tried to address issues

all at once instead of working on a fix at the time.

Virtual Servers: Use Snapshots to keep fresh rollback points.


20/22

Benefits Seen

Still in the evaluation processes but

Looking to see about improvements around power outages

Flexible recipes trials (no salt) completed without interrupting production

Found a lot of hardware and material deficiencies as part of the process

i.e. RPM sensor preventing packing flour sifter at least once since install

Double signoff now needed which improves/enforces QA involvement on signoff

Eliminated paperwork for hand add tracking

Improved manual equipment control capabilities that did not exist before

Improved alarming and troubleshooting capabilities from the plant floor

Inclusivity of system selection and deployment improved plant adoption


21/22

Plans Moving Forward


Offsite data replication to further improve disaster recovery

More in-depth training on batch and phase manager

Pursuing continuing the rollout of the new standards across theorganization

Review new data being collected to see how they may be incorporated into

reporting/process improvements


22/22


Insert Photo Here

Rockwell AutomationProcess Solutions User Group (PSUG)

November 14-15, 2011

Chicago, IL McCormick Place West

Documents

Af11psug Cs05 Bama-companies