Introduction to Perforce for Administrators...• Truncate journal on live server p4d –r c:\p4root –jj • Restore jjpournal to warm spare p4d –r c:\spareroot –jr journal.100

$Page 1: Introduction to Perforce for Administrators...• Truncate journal on live server p4d –r c:\p4root –jj • Restore jjpournal to warm spare p4d –r c:\spareroot –jr journal.100$
Perforce Training Course, 2010.1

1

Introduction to Perforce for Administrators

Introduction

• IntroductionsIntroductions

• Class Schedule

• GUI vs. CLI

• P4Admin Demonstrations

• About the Exercises


2

Course Contents

• Installation and Environment

• Protections

• Backup and Recovery

• Database Maintenance

• Daemons and Triggers

• Multiple and Distributed Depots

• Perforce Jobs


Installation and Environment


3

Server metadata requirements

• RAM ~ 1.5 KB/file

• Disk Space ~ 0.5 KB/file * # users

• CPU ~ Most processors fine

Installing the server

• P4D

• Server environment variables

P4ROOT P4JOURNALP4PORT P4LOG


4

Starting the server program on Unix

Server runs continuously in background...

• ...using environment variables:• Set environment variables• Use the command: p4d &

• using command line arguments:• ...using command-line arguments:

p4d -r /usr/perforce -p 1861 &

Starting the server program on Windows

• Server is installed as service p4s exe• Server is installed as service p4s.exe

• Service starts automatically

• Can run p4d in command prompt window

p4d r C:\P4Server p 1888p4d -r C:\P4Server -p 1888


5

User environment

• Client program environment variables:Client program environment variables:

P4PORT P4USER P4CLIENT

• Server traffic & client workspace views• Server traffic & client workspace views

Mapping files to Perforce file types

• Perforce detects file type text or binary

• Override with default file type mappings:

Typemap:

+m //depot/Jam/MAIN/src/

p4 typemap

+m //depot/Jam/MAIN/src/...binary+l //....pdftext+k //depot/Jam/MAIN/RCSimport/....txtbinary+x //depot/Jam/DEV/proj/executables/...binary+Fl //depot/....gztext+C //depot/....genfile


6

Stopping the server program

p4 admin stopp4 admin stop

Perforce counters

• Used to store server settingsUsed to store server settings

• Perforce counter attributes:• name• numeric value


7

Setting server security level

• Display security counter valuep4 counter security1

• Set security counterp4 counter -f security 2

• Stop and restart server

Enabling process monitoring

• Set monitor counterp4 counter -f monitor 1

• Stop and restart server


8

Monitoring Perforce activity• Show current processesp4 monitor show34 R rlo 00:06:31 sync34 R rlo 00:06:31 sync452 R clarks 00:00:00 monitor

• Show processes and command environmentp4 monitor show -a -e34 p4/2009.2 10.0.0.18 R rl rlws 00:07:37 sync //depot/...

• Mark processes for terminationp4 monitor terminate 34** process ‘34' marked for termination **

p4admin – monitoring servers *live*

• Select multiple server instancesSelect multiple server instances

• Overview of server data

• Display System Info


9

Diagnosing slow server response time

• Network issuesP4PORT=host.domain:1666 vs.

P4PORT=10.0.0.124:1666

• Finding p4 on a network driveg p

p4 -V

Diagnosing server swamp

• Turn on server loggingp4d -vserver=1

• Other processes running on server?

• Virus checkers

• Backup utilities...etc.


10

P4P

• Improves performance at low-bandwidth sitesImproves performance at low bandwidth sites

• Caches frequently-requested file revisions

• Runs on command-line or as Windows service

• No backups required• No backups required

Starting the proxy program

• On Windows install as service p4ps exe• On Windows install as service p4ps.exe

• On Unix:

• Set environment variables

• Or Use command line arguments• Or, Use command line argumentsp4p -d -r /usr/p4cache -p 1668 -t p4home:1666


11

Sample Proxy diagram

P4D

P4 Client

P4P

SlowWAN

FastLAN

P4 Client

P4 Client

New Commands in this Chapter

• p4d

• p4 typemap

• p4 admin stop

• p4 counter -f

• 4 it• p4 monitor

• p4admin

• p4p


12


Protections

User Protections

• Access levelsAccess levels

• The p4 protect command

• The protections table

• Using groups

• Server traffic & protections


13

Access permissions

Level: Allows:• list access to metadata but not file contents• list access to metadata but not file contents

• read viewing file contents but not opening files

• open opening files but not submitting

• write submitting file modifications

• review p4 review command (external application use)

• admin overriding changes to metadata

• super commands that affect server operation

Creating the protections table

p4 protectProtections:write user * * //...super user lisa * //...

access level

user or group

name host files


14

Protections:

Using the protections table

p4 protect

read

write

write

write

read

group

group

user

user

user

p4users

devgroup

mike

mike

emily

95.3.0.0/16

*

*

*

*

//...

//...

-//depot/...

//depot/Jam/MAIN/doc/...

//depot/Jam/MAIN/svr/...

access level

user or group name host files

read

super

user

user

remote

lisa

*

*

-//...

//...

User groups

Si l t ti li ff t f• Single protection lines affect groups of users

• Set maximum data size for commands

• Set login timeout


15

Group: devgroup

Creating and editing groups

p4 group devgroup

p g pMaxResults: UnlimitedMaxScanRows: UnlimitedMaxLockTime: UnlimitedTimeout: 43200Subgroups:

devcontractdevcontractOwners:

rajUsers:

joesharonmike

Group: managers

Setting data access limits

p4 group managers

p gMaxResults: 50000MaxScanRows: 250000MaxLockTime: 30000Timeout: 32400Subgroups:Owners:

jamesUsers:

daveanitakurt


16

Managing groups

• Edit a groupp4 group -a devcontractors

• Delete a groupp4 group -a -d devcontractors

Group devcontractors deleted.

Listing groups

• Show all groupsp4 groups

• Show group membership

p4 groups brunod iadmins

perfuser


17

Listing protections

• Current userp4 protectsp4 protectswrite group p4users * //depot/...super user bruno * //...

• Any user or groupy g pp4 protects –u p4readersread group p4readers * //depot/...

Recommended protections strategy

• Never use * to refer to all userse e use to e e to a use s

• Instead, list all valid Perforce users in groups

• Use the group names instead of *

• This will prevent unauthorized access• This will prevent unauthorized access


18

p4admin - protections and groups *live*

• Create a group for your usersCreate a group for your users

• Assign permissions

• Preview changes to permissions

• Graphical display of permissions


• p4 protectp4 protect

• p4 group

• p4 groups

• p4 protects


19


Backup and Recovery

P4ROOT directory layout

• Database tables:db change db domain db integdb.change db.domain db.integdb.counters db.fix db.jobdb.depot db.fixrev db.jobdescdb.desc db.have db.label...etc.

• Depot directories:Depot directories:depot nu...etc.


20

A few definitions

• JournalLog of updates to metadata

• CheckpointArchive of database

• Server archive verification• Server archive verificationCheck integrity of archived files

Order of backup steps

• Verify database archive integrity

• Checkpoint the database

• Backup using routine procedures...checkpoint

...journal archive

...versioned files


21

Verify server archives are intact

p4 verify Build.com//depot/Jam/MAIN/src/Build.com#7 - edit change 4668

( ) 86638 6 2 86 8 81 2(text) 86638AC6C2BCA86BBEFED8581F424FDC//depot/Jam/MAIN/src/Build.com#6 - edit change 2475

(text) 6D2C49C8DEBC5C3CF41BD87A8355D354//depot/Jam/MAIN/src/Build.com#5 - edit default change

(text) CAA2DC4BFC25A836C5D37E46FB92B016//depot/Jam/MAIN/src/Build.com#4 - edit default change

(text) 9A632B22DED4AB9E1227389E368A725E

...etc.

• Use quiet mode to report only errorsp4 verify -q //...

Checkpointing the database

• Checkpointing...

hi db * fil...archives db.* files...archives the journal...starts a new journal

• Making a checkpoint:p4 admin checkpointorp4d -r /usr/perforce -jc


22

Checkpoint file naming

• First Checkpoint:checkpoint 1checkpoint.1

journal.0

journal

• Second Checkpoint:checkpoint 2checkpoint.2

journal.1

journal

...

• Using a prefixp4 admin checkpoint f:\perforce\apr30

Optional checkpointing features

p4 admin checkpoint f:\perforce\apr30

f:\perforce\apr30.ckp.3f:\perforce\apr30.jnl.2

• Zipping checkpoint filesp4 admin checkpoint zp4 admin checkpoint -zp4d -r C:\perforce -jc -z

C:\perforce\checkpoint.3.gzC:\perforce\journal.2.gz


23

journal.430

p4 admin checkpointBackup Utility

checkpoint.4312

checkpoint.431checkpoint.431

Database DepotDatabase Depot

journal.430

Depot

1

2

Perforceserver

pp

Backup example

• Monday starting at 2:00 AM• p4 verify -q //...• p4 admin checkpoint e:\safe\ch• p4 admin checkpoint e:\safe\ch

ch.ckp.426, ch.jnl.425 created

• Backup Depot files, ch.ckp.426, ch.jnl.425

T d t ti t 2 00 AM

journal

• Tuesday starting at 2:00 AM• p4 verify -q //...• p4 admin checkpoint e:\safe\ch

ch.ckp.427, ch.jnl.426 created• Backup Depot files, ch.ckp.427, ch.jnl.426


24

Perforce database recovery repairs...

• ...inconsistent databases...inconsistent databases

• ...bloated databases

• ...data lost through disk crash

• data lost through accidental user activity• ...data lost through accidental user activity

Recover database from checkpoint & journal

• Stop p4d process, then:cd $P4ROOTcd $P4ROOTcopy db.* <\some\other\dir>\db.*del db.*

• Normal recoveryp4d -r . -jr checkpoint.3 journal

• If last checkpoint and current journal unavailablep4d –r . -jr checkpoint.2 journal.2


25

Offline checkpoint

• Restore checkpoint from live system to backup server.

Ni htl• Nightly:• Truncate journal on live server

p4d –r c:\p4root -jj

• Restore journal to backup serverp4d –r c:\bckroot –jr journal 100p4d –r c:\bckroot –jr journal.100

• Take checkpoint on backup serverp4d –r c:\bckroot –jd chkpt-20090421

Warm spare server • Restore checkpoint from live server to warm spare • Copy over depot files from live server to warm spare

• At regular intervals (nightly for offline checkpoint):• Truncate journal on live server

p4d –r c:\p4root –jj

• Restore journal to warm spare j pp4d –r c:\spareroot –jr journal.100

• Copy depot file changes from live server to warm spare


26

Volume layout for Perforce Volume Sample

LocationContents Backup

NotesPerformance Considerations

Metadata /metadata P4ROOT is here. Backup, but Optimize I/O for Keep disk space utilization at or below 65%

exclude db.* files.

random read/write. Expect 10x to 100x I/O demands compared to the depot volume.

Depot Data /depotdata Versioned files, checkpoints and inactive journals.

Backup entire contents. Consider

Typically more sequentialread/write.

Also admin utilities and scripts.

replicating to a spare machine and / or DR site.

Logs /logs Server logs and active journal.

Backup all. Keep logs as reference.

High performance demands.

Volume Mapping

p4 depot depot

Depot: depotDepot: depotOwner: bobDate: 2009/04/20 17:18:22Description: Production depotType: localAddress: localMap: e:/p4/1/depotdata/depot/...

p4 set –S Perforce P4JOURNAL=f:\p4\1\logs\journal

p4 set –S Perforce P4LOG=f:\p4\1\logs\log


27

Replicating metadata

Master Replica

P4D P4D

• Transfer and replay journal records from the Master to the Replica using p4 replicate

• See Chapter 10, Perforce Replication, in the Perforce System Administrator’s Guide and Perforce Knowledge Base article Perforce Metadata Replication


• p4 admin checkpoint

• p4d -jc• p4d -jc

• p4 verify

• p4d –jr

• p4d –jj

• p4d –jdp4d jd

• p4 depot

• p4 replicate


28


Database Maintenance

Perforce Superuser and Admin Commands

• Resetting users’ passwordsResetting users passwords

• Perforce super and admin flags

• Obliterating files


29

Resetting users’ passwords

p4 passwd username

Enter new password:Re-enter new password:Password updated.

Database cleanup

• Remove old specspp4 user -d usernamep4 client -d client_workspace_namep4 label -d labelname

• Delete empty changelistsp4 change -d changelist#


30

Finding and removing unused specs

• List specs owned by a userp4 clients -u username

• Delete client workspace specp4 client -f -d client_workspace_name

• Delete user specp4 user -f -d username

p4admin – manage users *live*

• Change a user’s passwordChange a user s password

• Delete a user specification


31

Changelist maintenance

U d i d i i• Updating a descriptionp4 change -f changelist#

• Unlocking files p4 unlock -f -c changelist#orp4 –c earl_ws unlock –f //depot/Jam/...

Obliterating files vs. Deleting files

• Delete• Delete...

• ...marks head revision as deleted

• ...keeps revision history

Oblit t ll t f th fil• Obliterate removes all traces of the file


32

Obliterating files

// / / / /p4 obliterate //depot/Jam/MAIN/src/...

p4 obliterate -y //depot/Jam/MAIN/src/jam.ps

p4 obliterate -y //depot/Jam/MAIN/src/rules.c#3


• p4 passwd• p4 passwd

• p4 <command> -f and -d

• p4 obliterate


33


Daemons and Triggers

Daemons and Triggers

• Scripting with Perforce

• Review Daemons

• Triggers


34

Create scripts locally

• User-created scriptsp

• Useful Perforce flags

p4 -G <command>

p4 -s <command>p

p4 <spectype> -o and -i

#!/usr/local/bin/python#Example script named readmarshal.py

import marshal, sys

Example Python script and p4 -G output

true = 1

try:while true:

vars = marshal.load(sys.stdin)print vars

except EOFError:‘’ #note that this is two single-quotes

p4 -G user -o mls | readmarshal.py{'Email': '[email protected]', 'Update': '1997/12/06 11:16:30', 'Reviews1': '//depot/Jam/MAIN/...', 'Reviews0': '//depot/Jam/REL2.1/...', 'FullName': 'Millie Smith', 'User': 'mls', 'code': 'stat', 'Access': ‘2006/08/06 15:16:30'}


35

Write efficient scripts

Mi i i b f 4 d• Minimize number of p4 commands

• Use care when referencing labels

• Narrow scope of commandsp

• Use file to input list

Iterate through results, not commands

Use file to input listp4 -x filelist.txt edit

• Use branch viewp4 diff2 -b pathA-to-pathB


36

Limit references to labels

• Static Labels:Static Labels:• Get file names from label

p4 files //depot/Jam/...@mybuild

• Process results in script

• Use Automatic Labels

Create temporary client workspace spec

• Map only files of interest:Map only files of interest:

View://depot/pathA/src/... //bob-devA/pathA/src/...//depot/pathA/tests/... //bob-devA/pathA/tests/...//depot/pathA/doc/... //bob-devA/pathA/doc/...

• Use in scripts' commandsp4 -c bob-devA sync @pathApackage


37

Metadata tricks

• Merging servers with perfmerge++ g g p g

• Journal tailing for auditing

• Journal tailing for hot standby

• Using vi/sed/awkUsing vi/sed/awk

Review daemons

• E-mail notification of:E mail notification of:• changes submitted

• jobs logged and modified

• Set "Reviews:" field in user profile

• Run the review daemon (in background)


38

Changing your user profile

p4 user

User: bob

Email: [email protected]: 2008/08/09 13:45:59Access: 2008/11/07 16:45:05FullName: Bob EverlyReviews:

//depot/Jam/MAIN/technotes/...//depot/Jam/DEV/projects/proj1/...

Add “Reviews:” field ifrunning review daemon

Review counters

• Store external application sync pointsStore external application sync points

• Counter attributes:

• name

numeric value or string• numeric value or string


39

Displaying counters and values

• Show a counter’s valuep4 counter notifyd678

• List all countersp4 countersnotifyd = 678change = 680job = 12journal = 3monitor = 1security = 2...etc.

Managing counters

• Resetting a counter• Resetting a counterp4 counter bugtrack 792

• Deleting a counterp4 counter -d bugtrackp4 counter d bugtrack


40

What is to be reviewed by whom?

• Find changelists beyond counterp4 review -t notifydChange 679 sam <[email protected]> (Sam Best)

Change 680 eno <[email protected]> (Brian Eno)

• List reviewers for a changelistp4 reviews -c 679sam <[email protected]> (Sam Best)rna <[email protected]> (Ron Ando)

Supported trigger types

• Password authenticationPassword authentication

• Changelist processing

• Form manipulation or validation

• Job fix checking


41

Use triggers to enforce policy

• Authenticate passwords against LDAP server

RELNOTES fil l b i d i h * fil• RELNOTES file always submitted with *.c files

• Submit to “rel1” codeline fixes at least one job

• Add “Reviewed by:” when changelist created

• All *.h files include copyright date

• Begin build after submit to “dev” codeline

• Notify code reviewers when files are shelved

• Prevent users from deleting jobs

Password authentication triggers

• Auth check• Auth-checkVerify against external password manager

• Auth-set

Send new password to external manager


42

Submit triggers

• Change-submitRun after change is created and files lockedRun after change is created and files locked

• Change-contentRun after file transfer

• Change-commitRun after changelist commit

• Submit fails if submit or content trigger returns non-zero value

Shelving triggers

• Shelve-submitR ft h t d• Run after change created

• Shelve-commit

• Run after files are shelved

• Shelve-delete • Run before discarding shelved files

• Shelving fails if a shelve-submit trigger returns a non-zero value.


43

Form triggers

• Form-outRun on generation of form

• Form-inRun on input of form

• In and out triggers may modify forms

Validated form triggers

• Form-saveRun after form parsed, before form saved

• Form-delete

Run after form parsed, before form deleted

• Form-commit• Form-commitRun after form saved


44

Job fix triggers

• Fix-add

Run prior to adding fix

• Fix-delete

Run prior to deleting fixp g

Triggers:d th h k th “ th ld h k % %”

Creating triggers

p4 triggers

passwd auth-check auth “python ldap_check.py %user%”notes change-content //depot/bld/... “relcheck.pl %user% %changelist%”review change-submit //depot/src/*.c “reviewer.pl %changelist%”review change-submit //depot/src/*.h “reviewer.pl %changelist%”new form-out change “addinfo.pl %formfile%”jobs change-submit //depot/rel1/... “/usr/bin/p4/jobs.sh %changelist%”keep form-delete branch “python /usr/bin/spec.py %formfile%”fixone fix-add fix “/usr/bin/p4/checkfix.sh %jobs%build change-commit //depot/src/... “/usr/bin/p4/start.sh %changelist%”build change commit //depot/src/... /usr/bin/p4/start.sh %changelist%jobrep form-commit job “sendtomysql.pl %formfile%”

name path scripttype


45

#!/usr/bin/perl## Example trigger, run using "perl check4reviewer.pl <changelistnumber>".## This runs "p4 describe -s <changelistnumber>" and looks for the string

Example trigger script

# This runs p4 describe -s <changelistnumber> and looks for the string# reviewed by: XXXXX# in the description. If it's not there, the change shouldn't have# been entered because no reviewer was specified in the changelist# description.## Exit status:# 0 - everything passed the check# non-zero - failure#

$ $ 0$changeno = $ARGV[0];$output = `/usr/local/bin/p4 describe -s $changeno`;

if ($output =~ /reviewed by:\s+(\d+)/i) {exit(0); # passed the test!

}print "**** No reviewer id specified in changelist! **** \n";exit(1);


• p4 counterp

• p4 counters

• p4 review

• p4 reviews

• p4 triggers


46


Multiple & Distributed Depots

Creating new depots

p4 depot nu

Depot: nuOwner: bobDate: 2006/06/12 10:15:22Description: Nu project filesType: localAddress: localAddress: localMap: nu/...


47

Working with other depots

• Listing depotsp4 depotsDepot depot 1999/05/26 local depot/... 'Created by eno.'Depot spec 2005/02/24 spec spec/... 'Created by eno.‘Depot nu 2006/06/12 local nu/... ‘Created by eno.’Depot usr 2002/01/10 remote perforce:1888 //usr/...

'Created by eno.'

• Getting files from other depotsp4 sync //nu/Jamgraph/DEV/src/...

Distributed depots

D t i

p4 depot spice

Depot: spiceOwner: bobDate: 2005/05/23 9:01:12Description: Arrakis server’s //depotType: remoteAddress: arrakis:1666Map: //depot/oak/MAIN/

• Integrating from a remote depotp4 integ //spice/src/... //depot/IMPORT/src/...

Map: //depot/oak/MAIN/...


48

Store edited Perforce formsp4 depot allspecs

Depot: allspecsOwner: bobOwner: bobDate: 2008/07/18 10:03:37Description: Specification Form depotType: specAddress: localSuffix: .p4sMap: allspecs/...p p /

• Pre-populate the spec depot:p4 admin updatespecdepot -a

Recovering a stored spec revision

• List revisionsp4 filelog //allspecs/client/bruno_ws.p4s

#4 default change edit on 2008/11/01... #4 default change edit on 2008/11/01... #3 default change edit on 2008/11/01... #2 default change edit on 2008/11/01... #1 default change add on 2008/11/01

• Display content of revisionsp4 print -a //allspecs/client/bruno_ws.p4s

• Replace spec with earlier versionp4 print -q //allspecs/client/bruno_ws.p4s#3

| p4 client -i


49

p4admin – manage depots *live*

• Create depot specificationsCreate depot specifications• Data mining available


• p4 depotp p

• p4 depots


50


Perforce Jobs

Fields:101 Job word 32 required102 l 10 i d

Customizing the job specificationp4 jobspec

102 Status select 10 required103 User word 32 required104 Date date 20 once105 Description text 72 required140 Release select 10 optional

Comments:# Include plentiful comments!# Job: The job name. 'new' generates a sequenced job number.# Your users need to know what the fields mean, for example:# Release: One of ‘4 0’ ‘4 1’ ‘4 2’ or ‘5 0’# Release: One of 4.0 , 4.1 , 4.2 , or 5.0

Values:Status open/cannot_dup/duplicate/in-progress/closedRelease 4.0/4.1/4.2/5.0

Presets:Status open,fix/in-progressDate $nowDescription $blank


51

Jobspec form fields

• Datatypeypword date select line text bulk

• Persistenceoptional default required once always

• Built-in variables$user $now $blank

Editing a Perforce job (with customized jobspec)

p4 job job000053# Include plentiful comments!# Name: The job name.‘new' generates a sequenced job number.# Your users need to know what the fields mean, for example:# Release: One of ‘4.0’, ‘4.1’, ‘4.2’, or ‘5.0’Name: job000053Status: openUser: karenDate: 2009/01/23 10:29:46Date: 2009/01/23 10:29:46Description:

Fix file locking bug.Release: 4.2


52

Defect tracking gateway

• Customize jobspec with defect tracker fields

• Info in both systems kept in sync


• p4 jobspec


53


What’s new in 2010.2

Server to server replication

Database Database

D t

main Perforceserver

replicaPerforceserverp4 pull

Depot Depot


54

Perforce Broker

P4 Cli

PrimaryP4D

P4 Client

BrokerP4 Client

Proxy P4P

ReplicaP4D

P4 Client

Dynamic configuration

• Run commandsp4 configure

replicatepullsecuritymonitor

• Set variables• Set variables• Restrict auto creation of users• Minimum client program version

... etc.


55

Offline storage

• Create a depot of type archiveCreate a depot of type archivep4 depotsDepot Archive 2011/01/04 archive /remote/archive/... Depot depot 2000/01/09 local depot/...

• Archive filesp4 archive –D Archive //depot/Jam/REL2.5/img/jam.pngp p g j p g

• Restore archived filesp4 restore –D Archive //depot/Jam/REL2.5/img/jam.png

Authorization and Changelist server

bruno authorizedh 5399

bruno authorizedh 5399

1

ProjectAP4D

Authorization & Changelist

P4D

p4 submitp4 submit change 5399change 5399

P4 Client

1

ProjectBP4D

p4 changep4 changebruno authorizedchange 5400

bruno authorizedchange 5400

2

2


56

Perforce Highlights

FAST

Easy to learn

Simple to administer

Superior technical support

Cross-platform compatibility

The End

Please fill out class evaluations.

All Perforce manuals and technical notes are available at www.perforce.com.

Report problems and get technical help from [email protected].

Share tips and ideas with other users on [email protected]

Documents

Introduction to Perforce for Administrators...• Truncate journal on live server p4d –r c:\p4root –jj • Restore jjpournal to warm spare p4d –r c:\spareroot –jr journal.100