Engage 2016: Back From the Dead: How Bad Code Kills a Good Server

#engageug

BackfromtheDead:WhenBadCode

KillsaGoodServerEngageUserGroupConference,Eindhoven

March2016

SerdarBasegmez-Developi-@serdar_basegmezWilliamMalchiskyJr.-ESS-@BillMalchisky

#engageug ©2016SerdarBasegmezandWilliamMalchiskyJr.LicensedunderCreativeCommonsBY-NC-SA4.0 2

• Preface

• ChapterI-TheBeginning

• Chapter2-SearchingforClues

• Chapter3-CreaSngaSolidPlaTorm

• Chapter4-TheSoUsideofPerformanceGains

• TheFinalChapter-Results

OurStoryinFortyMinutes


"Ladies and Gentlemen. The story you are about to see is true; the names have been changed to

protect the innocent." --Dragnet

For example... Acme Corporation is now referred to as Acme, Inc.

Disclaimer


• Whatwewillcover

• Problemanalysis

• TroubleshooSngskills

• BestpracSces

• TheperformanceimpactofsubopSmalapplicaSons

• Whatweomi[ed

• Boring,rambling,dry,lectures

• Uselessdrivel

Se^ngExpectaSons


• Preface








• "We'rehavingaproblem.Canyouhelp?"

• "Absolutely.What'shappening?"

• "OurmissioncriScalDBisreally$%&@#$^&ourusers.It'swaytooslow.IttakeslessSmetoreboot[Windows3.1onani386with32MBRAM]thantoopenadocument."

• "Anyideawhatchanged?"

• "Wedon'tknow.Wehavenottouchedthebox."

CustomerCalls


• LackofexperSseand/orknowledge

• Unplannedand/orunexpectedexpansion

• NodedicatedAdministrator

• Nochangemanagement

• Nomonitoring

• Workaroundoverloading

WhyDominoServersFail?


• Preface








• WhilewaiSngforaccess...requestthefollowing

• HelpsestablishthelevelofcriScality

"Round Up the Usual Suspects"

notes.ini log.sfsh tasks top vmstat iosysdf -h User to server ping results

mount swapon -sServer NAB DB copy, sans users


malchw@san-domino:~$ iostat

Linux 3.13.0-83-generic (san-domino) 03/23/2016 _x86_64_ (8 CPU)

avg-cpu: %user %nice %system %iowait %steal %idle

6.21 0.25 3.69 0.51 0.00 89.34

Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn

sda 45.34 2075.44 778.25 6028264 2260469

sdb 0.36 1.52 0.03 4422 80

dm-0 24.51 117.04 186.80 339957 542584

dm-1 16.17 415.61 79.82 1207173 231836

dm-2 17.64 1540.92 511.61 4475713 1485996

malchw@san-domino:~$ vmstat

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

r b swpd free buff cache si so bi bo in cs us sy id wa st

1 0 0 16943764 153144 7941660 0 0 262 98 144 681 6 4 89 1 0

QuickExample-iostat,vmstat


• RunDCT-returnedafewitems,butnothingapplicabletotheperformanceissueexperienced

• CheckDominostats

• Locatedakeyissue-needleinhaystack

• SAIfluctuatedwildly,frequently,plummeSngto18%forminutesonend

• LocateanyrecentNSDfilesforanalysis

Data,DataEverywhere


• Watchtheserverwhennobodyelsedoes

• Lotsofstrangethingshappenonserversovernight

• Observedthesystemprocessingoveronemillionrecordsin:15twiceaweek,atdifferentSmes

• Forexample…nooneatAcme,Inc.knewthisoccurredorwhy

ProTiponDataCollecSon


• Swapspace50%ofinstalledmemory

• Memorywasunder1GBformissioncriScalserver

• SeveralkeyDBscontained100k+docs

• CombinaSoncreatedpagefaulSngplaguefurthererodingperformance

• Systemproperlypatched

• Freespaceadequate

IniSalDataAnalysis-OS


• Obviousbutimportantdatapoints

• Serverlayout

• Whereitemslocated

• Recognizedserver.idfile

• Servertasks

• Contrasttoshtasksrequestedearlier

• Noobviousproblems

IniSalDataAnalysis-Notes.ini


• Agentsrunningallhoursofthenightandday

• AgentsrunningfromDBsacSvelybeingcompacted

• AgentsrunningfromDBswhenupdallandfixuprunning

• Notallscheduledagentsneededtorunallweekend

IniSalDataAnalysis-Amgr


• CompactsSllrunningwhenupdallProgramfires-off

• CompactneverfinishedbeforeexecuSonSmeceilinghit

• LeUlargestDBsinacompletelysubopSmalstate

• Connectedtoserversthatdidnotexist

• ScheduledreplicaSondocuments

• SignificantdelayswithreplicasynchronizaSon

• Ensureddataneverproperlysynchronizedacrossdomain

• CertainconnecSondocumentsonlycoveredtwoDBs

IniSalDataAnalysis-Log.sf


• SeveralbigDBslastfixupcompletedtwoyearsago

• Mostheavilyusedfiles30-75%Used

• Manyviewsmeansclickingoneforcesanewindexbuild

• Nodesign,document,ora[achmentcompression

• DesignservertaskciSngnon-existenttemplates

IniSalDataAnalysis-DBs


• Preface








• Swapspace-Nosetrulethesedays

• 1.5x-2.0xRAMisgoodruleofthumb

• Memory-4GBperprocessoronbusyservers

• VMwarese^ngsifavailable

• AvoidtemptaSonoftoomanyprocessors

• ReviewparSSonsandfreespace

Tier1-OS


• CheckthatpreviousmadesystemchangessSck

• Unfamiliarserverscanexhibitoddbehavior

• CheckTechnotesforanyrecentperformanceissues

• OnceOSisworking,checktoensurethatvirtualizaSonisopSmal

AddiSonalOSConsideraSons


• SpaceproperlyProgramDocuments

• AvoidoverlapwithagentsandotherPrograms

• Pauseagentscheduleduringmaintenance

• Scheduleaweekendtocompletefirstfullmaintenance

• Firstfullcompactwilltakemuchlongerthanyourealize

• Createmaintenancescheduleoftasksagreedtobybusinesslinemanagers

• Ensuresallneededjobsareavailablewhenneeded

Tier2-Domino


• ReviewallenabledDominofeaturestoensurethattheyfuncSonproperly

• SimpleconfiguraSonmiscuescanimpactnegaSvely

• ClusterreplicaSonunabletolocateaclustermember

• DNSerrorscreatelookupdelays

• Removeunneeded,deprecatednetworkports

AddiSonalItemstoFix


• Preface








• DominoAdminhandledthefirstleveltreatment

• Serverperformswell,butnotgoodenough

• Triangulatedtheissuetoamission-criScalapplicaSon

• Nowwhat?

WhereareWe?


• LackofexperSseand/orknowledge

• Developersevolvedfrompowerusers

• Architectureoverloading

• Unplannedand/orunexpectedexpansion

• Undocumentedcodeand/orbusinessprocess

• Nochangemanagement

• Quick&dirtydevelopment

WhyDominoAppsFail?


• Thereisnomagicpillforfindingaperformanceissue

• ManyproblemsarecircumstanSal

• Dependsonwho/when/how…

• RepeaSngtheproblemonacontrolledenvironment

• NeedforProof!

• Themostdifficultpartofthetask

• NeedtobesystemaKcal

DevelopersvsPerformanceIssues


• ResearchandAssessment,

• SpeculaSonforfixes,

• Experiment,

• Prove!

ScienceJustWorks!

http://www.wired.com/2013/04/whats-wrong-with-the-scientific-method/


MethodologyResearch

✤ Symptoms (e.g. logs, performance data, etc.)✤ Story (e.g. user input)✤ Application code

Hypothesis ✤ Speculation on possible reasons✤ Search for ‘Usual Suspects’

Experiment ✤ Testing for possible reasons

Analyze ✤ Check symptoms if fixed

Conclusion ✤ Issue validated and proved to be fixed.


• Whattocollect,basedonthesymptom;

• CPU/memoryload,hangs,spikes,crashes,etc.

• AlltheSme,thesameSmeeverydayorrandom?

• Experiencedbyspecificusers?

• Wearelookingforapa[ernbetweenincidents.

Research&Assessment


Log/NSD/Semaphorefiles

ServerconfiguraSon(inc.notes.ini)

ServermonitoringandstaSsScsdata

Weblogs(forwebapplicaSonissues)

XPagesandOSGilogs(forXPagesspecificissues)

ApplicaSonanddependencies

DataCollecSonChecklist


• SomeSmes,evenopeninginDDEmaycauseissues!

• e.g.XPagescomponentsareautomaScallybuilt

• ApplicaSoncodemighthavesideeffects

• e.g.UpdaSngonanotherdatasource,addingauditlogs,performancedegradaSonontheserver,etc.

• Therewillbedependencies

• Onceisolated,wecanstartinspecSon…

IsolatetheApplicaSon


• DatabasecorrupSons

• @Today/@Nowinviews

• CodesnippetsacSnglikeanadmin

• UpdaSngviews,replicaSngdatabases,runningservercommands,etc.

• CodesnippetsusingtheworstpracSces

• Searchinalargedatabase,wronglooping,etc.

• Anythingthatfitsintothepa[ernifthereisone

• e.g.AnagentmatchingtheincidentSming

UsualSuspects

#engageug

Nothingyet?Diggingdeeper!

33


• DeeperinvesSgaSonneedsateamingeffort

• AdminsandDevelopersshouldcollaborate

• AtestsetuptosimulatetheproducSonenvironment

• Intensive/ControlleddebuggingsessionsinlimitedSmewindows

• SharingexperSse

• ExperimenSngonproducSonshouldbethelastresort

• Oncearepeatableerrorfound,cooperateforasoluSon

TeamUp!


• JVMCrashwiththeHTTPtask

• RandomSmes

• Nopa[erninthelog

• MemorydumpspointaleakintheJVMHeap

• InspectedXPagesapplicaSons,nothingfound

• TriangulatedtheproblemintooneXPagesapp,followingcluesinintensivedebuggingonmemory

• IsolatedtheapplicaSonforaloadtest,nothingfound

• Increasedlogging,tocollectmoredata,nohope!

ExampleCase-Analysis


• CheckedtheserverconfiguraSonandnoSced

• Loggingdataincomplete

• Removedexclusions

• Newlogspointedtheproblem

• SearchingsoUwarecrawlingaspecificpage

• Pagegeneratesstatedata,fillsupthememory

• Simulatedthesamecrashonthetestenvironment

• Onelineofcodefixedtheissue

ExampleCase-ResoluSon


• AmissioncriScalapplicaSonatabank

• WebapplicaSonwith2000+users

• CPUspikesandrandomhangs,mostlyaUernoon

• Logsareclear,nocrashes,noerrormessages

• IsolatedtheapplicaSon,inspectedthe‘usualsuspects’

• FoundawebagentupdaSngaview!

• TriangulatedtheproblemusingweblogsandSEMDEBUG

• But,cannotvalidatetheissueonthetestenvironment…

AnotherCase-Analysis


• CooperatedwiththeDominoAdmin

• DetailedassessmentontheserverconfiguraSon

• Wefoundtheissue!

• “ServerTasksAt14”runninganupdalltask.

• AnotherProgramfilerunningUpdallonaspecificdatabase,every30minutes

• AppliedtothetestplaTorm,validatedbyaloadtest

• Problemsolved!

AnotherCase-ResoluSon


• Preface








• Pagefaultsreducedtozero

• GeneralDBusageandadministraSontasksworkwell

• SAInowover80%

• Weirdovernight(agent)systemoperaSonsresolved

• KeyDBshave93%usedspacethresholdnow

• AllDBscompressed:design,documents,alla[achments

• Programdocuments,agentschedulesalladjusted:finish,nooverlap

QualityAnalysisYieldsQualityResults


NoteonPerformance

Whendoneproperly,fewuserstendtonoScethechange,butifrevertedtheywillallcomplain


Neitheradminnordeveloper

couldsolvealloftheseissuesalone!

Teamworkvs.Performance


• YoucangethelpinspecSngapplicaSonsandservers!

• TheyhavealsohelpedEngage!

BonusSlide

cooperteam MartinScott

teamstudio Ytria


• IBMChampion(2011-2016)

• DevelopiInformaSonSystems,Istanbul

• ContribuSng…

• OpenNTF/LUGTR/LotusNotus.com

• Featuredon…

• EngageUG,IBMConnect,ICONUK,NotesIn9…

• Also…

• BloggerandPodcasteronScienSficSkepScism

SerdarBaşeğmez


• IBMChampion(2011-2016)

• EffecSveSoUwareSoluSons,LLC

• Co-founderofLinuxfestatLotusphere/Connect

• Speakerat20+Lotus/IBMrelatedevents/LUGs

• Co-authoredtwoIBMRedbooks

• Co-wrotetheIBMEducaSonAdministraSoncerSficaSontrackforDomino8.5

WilliamMalchiskyJr.


FollowUp-ContactInformaSon

Serdar Basegmez

[email protected]

@serdar_basegmez Skype: sbasegmez

Bill Malchisky Jr.

[email protected]

@billmalchisky Skype: FairTaxBill

mailto:[email protected]?subject=

mailto:[email protected]

Software

Engage 2016: Back From the Dead: How Bad Code Kills a Good Server