Cyber Grand Challenge Frequently Asked Questions (FAQ) August 3, 2016  

                                                                Defense Advanced Research Projects Agency Information Innovation Office 675 North Randolph Street Arlington, VA  22203‐2114 

               

 

Document Change Summary  

Section  Description  Date 

InitialPublication Nov7,2013Q1 EastCoastCompetitorDaydatesadded Nov14,2013Q1 ModifiedtoincludeWestCoastCompetitorDay

informationandremovedCGCwebsiteURLNov27,2013

Q10 Added‐WhattypeofsecurityvulnerabilitieswillCGCaddress?

Nov27,2013

Q11–Q33 Added Dec17,2013Q34–Q56 Added Dec24,2013Q57–Q59 Added–UpdatetoscoringmethodsandinitialCGC

environmentAPI.Mar10,2014

Q26–Q27 Obsoletedentriesreplacedbyentries58and59. Mar10,2014Q60–Q64 Added Jul24,2014Q65–Q73 Added Aug29,2014Q74 Added Oct21,2014Q75–Q77 Scoredeventupdate1. Nov14,2014Q78–Q79 Tiebreakerpubliccommentupdate1. Feb13,2015Q80–Q92 Scoredeventupdate2. Feb13,2015Q93–Q102 Added Mar18,2015Q103–Q106 Added Apr10,2015Q107–Q111 Added May6,2015Q112 Added May12,2015Q112 Revised May22,2015Q113–Q119 Added May22,2015Q120–Q124 Added May28,2015Q125–Q126 Added May29,2015Q127–Q130 Added Jun2,2015Q131 Added Sep1,2015Q132–Q144 Added Oct20,2015Q145‐Q156 Added Dec10,2015Q157 Added Jan27,2016Q158 Added Mar16,2016Q159–Q161 Added Mar16,2016Q162–Q163 Added Mar25,2016Q164 Added May6,2016Q165–Q170 Added May19,2016Q171 Added May27,2016Q172‐Q174 Added Jun3,2016Q175 Added Jun8,2016

Q176 Added Jun10,2016Q177–Q178 Added Jun23,2016Q179 Added Jun28,2016Q180 Added Jul12,2016Q181 Added Jul15,2016Q182 Added Jul21,2016Q183‐Q184 Added Jul22,2016Q185 PublishCFETeamPhrases Aug3,2016Q186 Added Aug3,2016

Q186:EffectiveAugust3,2016at11:00pmPT,thisFAQisclosed.HowwillnewFAQentriesbepublished?A186:AllnewFAQentrieswillbepublishedintheCGCEventFAQ.[1]https://github.com/CyberGrandChallenge/Event‐FAQ

Q185:WhatwerethecompetitorteamTeamPhrasesusedtocontributetothecalculationofthemasterseed?A185:TheTeamPhrasessolicitedfromfinalistsandusedaccordingtoA176oftheFAQarepublishedinthebelowJSON:[ { "phrase": "defa5b1925203b76ee19bb1102e620754fb655b11b52399da226354630e1f18b61f439b8cb2d520de99589c68fdc5312ab6b229879f7bda06d285cba98a961b7fe63ba3e9b96de11254196e2a73dab6099058af816a747a1182868b868e58eda8206bc33aba51964c4ef77aa4378d5665b66db8b18ae4eb6ed99e560b89ce2467b4bfff16dea49d6dcb88101392f91e0ca4c4ed672e30cc52b3f7c45a0a8d39c7a4b41b83a0f7e00b50c5ce123c38645a7c495b53d32df5b8b57dfb3a0933bce478930cd6b4692e8a57b0c335997c6e86a99114a0ca5c0751118ffe7d989b298d15e5f3df7cd9546290bedb7d79d87f91abafbb4a953078cac4aa53fd10caca1", "name": "CodeJitsu" }, { "phrase": "Remember: Innovation can occur anywhere -- If we knew what we were doing, it would not be called research", "name": "CSDS" }, { "phrase": "Deep Red [team photo: (-(-_(-_-)_-)-)] is excited to participate in the Cyber Grand Challenge. They do not battle with o==]:::::::::::>'s, but with 1's and 0's for treasure and eternal glory. Their specially chosen TeamPhrase shall surely secure their victory.", "name": "DeepRed" }, { "phrase": " ;.\n : . ..\n .NNN. cco XMM k. .\n .MMM. .OK0 kMM l:; l0Wo\n .,;'.,MMM. . . kMW xkl. ,MMo\n '0WMNKNMNMMM: dKK :dMMMMMXO: ,xXMMMWKk, 0MM. ;XKWW :0MMN0x..\n cMMM' lMMMMMX..KMX...lMMMMc.'cWM0 .KMMXl,llWMM: WMM ;0NM0:' ..'MMK'.\n NMMc d0XMMMx. ;MMd .XMMW0, XMMk :MMW0. NMMNMMNx. .,..MMK\n MMMc c:oXM0. .MMl 0:dxMMMWXx:..xOMMMKKXNWMMMWW0'.KMMXMMWO0' WMK\n MMM0c .lXMK .MM; .;dMMMWl.'MMNW0 .KMXl XW0MO. . MM0\n ,NMMd .XMMW. 'MM' oWM; .NMMMx OMNWN. :MMXl. XMMd 0WMMN 'MMX\n kOWMMWXWMXWMM. ;MN cMMXxdNMMMl. xMMMKkd0MMKc XMN' .KMWK. kMWKxo\n : l;Ko'. . .;c .cxxdOO .:odkxol' ,. . . cddl;..\n :kO. . .\n .", "name": "Disekt" }, { "phrase": "\n AAAAA\n AAAAAAAAAAAAAAA\n AAAAAAAAAAAAAAAAAAA\n AAAAAAAAAAAAAAAAAAAAAAA\n AAAAA AAAAAAAAAAA AAAAA\n AAAAAAA AAAAAAAAA AAAAAAA\n AAAAAAAAA AAAAAAA AAAAAAAAA\n AAAAAAAAAA AAAAAAAAAA\n AAAAAAAAAAA AAA AAAAAAAAAAA\n AAAAAAAAAAA A AAAAAAAAAAA\n AAAAAAAAAAA AAAAAAAAAAA\n AAAAAAAAAAA AAAAAAAAAAA\n AAAAAAAAAAAAAAAAAAAAA\n AAAAAAAAAAAAAAA\n AAAAA\n", "name": "ForAllSecure" }, { "phrase": "Shell we play a game?", "name": "Shellphish" }, { "phrase": "GrammaTech and UVA bring you Xandra, Defeneder of Humanity.", "name": "TECHx" } ]

Q184:WenoticedtheTIAPI[1]publishesNNNvalues"between0and100,reflectingpercentage"forpollfeedbackbymultiplyingtheavailabilityscorespecifiedintheCFEscoringdocument[A58]by100.Doesthisconventionholdforscoresaswell(i.e.ascoreof4becomes400intheTIAPI)?A184:Yes[1]https://github.com/CyberGrandChallenge/cgc‐releasedocumentation/blob/master/ti‐api‐spec.txtQ183:IntheFinalistEventInformationdocument,youstate"thepossibilityexiststhatcompetitorHPCaccesswillnotberestoredpriortoCFEcomputation".CanwespecifyapowerupscheduleforourbladesintheeventthatourCRSmustbepoweredupcoldforCFE?A183:No.Q182:CanthesameCSIDappearintwodifferentcompetitions?A182:Noprotectionmechanismexiststhatwouldpreventthis.Q181:We'dliketooptimizeforidenticalbinaries;howmanyofourcompetitorswillfieldChallengeSetscontainingidenticalChallengeBinariesduringCFE?A181:Unknown.Q180:WhatisthetiebreakeralgorithmforCFE?A180:InthehighlyunlikelyeventofatiedscoreinCFE,DARPA/CGCwillusethefollowingtiebreakingalgorithmonthescoresofteamscompliantwiththeCGCRulesForCFE,aTiedBlockisanygroupofscoresinadjacentplacesforwhich1>thescoresareequaland2>theblockoccludesaprizeplace(1st,2nd,or3rd).Forinstance,threeteamstiedfor2nd,3rd,and4thplacewouldconstituteasingleTiedBlock,occluding2ndand3rdprizeplaces.ForeveryTiedBlockattheconclusionofCFEcomputation,thetiebreakingprocessisasfollows:a)Discardonescoredround'sworthofpointsfromeveryCRSinaTiedBlock

fromtheendoftheCFEcomputationrecordb)RemoveanyCRSfromtheTiedBlockwhosescoreisnolongertiedwiththe

Block

c)IfanyTiedBlocksremain,repeatthetiebreakerata)Forexample,attheconclusionofCFEcomputation,1stplaceand2ndplacesystemshavethesamescore,constitutingaTiedBlock.3rdand4thplacesystemsarealsotied,constitutingasecondTiedBlock.Thetiebreakerisrunandthescoresfromthefinalroundareremoved.1stplaceand2ndplacearenolongertied,however3rdplaceand4thplaceremaintied.Forthepurposesofseparating3rdand4thonly,asecondroundisremoved.Thisprocessrepeatsuntil3rdand4thplaceareseparated.IntheeventthattheabovealgorithmcannotseparateaTiedBlockoverthefullCFEcomputationduration,DARPAwillissuearankingbyexpertjudgment.Q179:WilltheCFEEventPlanincludethenumberofroundseachCSwillbefielded?A179:Yes;DARPA/CGCwillneverPlantofieldaCSforlessthan10rounds.Q178:WewerereadingthroughtheCGCRulesandnotedthatAreaofExcellence#5says“mitigatesecurityflaws[using]anetworksecuritydevice”.Dowereallyhavetodealwithalltheproblemsofanetworksecuritydevice?WeuploadedsomeIDSsignaturesthatDoStheIDSdeviceandnoticedthatmax’ingouttheloadonthedevicecausedperformanceimpacttoeveryChallengeSetonourdefendedhost.CouldyouperhapsgiveusavirtualizednetworksecuritydeviceforeveryChallengeSet?A178:No.TheCGCRulesspecifythat“theCRSconfrontstheCFEnetworkfromthevantagepointofarealworldnetworkdefender”usinga“networksecuritydevice”anda“defendedhost”.Thesechallenges:load,resourcecontention,andnetworkconcurrencyarerealworldconstraintsthathavealwaysbeenpartoftheCapturetheFlagtraditionandwereguaranteedbytheCGCRulessincetheinceptionoftheChallenge.Competitorsareassuredthatnolate‐breakingchangeswillbemadetotheRulesthatwouldcreatealessrealisticChallenge. Q177:WhatistheofficialOSkernelofCFE? A177:Nosuchkernelhasbeenspecified,exists,orcanexist.DECREEhasbeenimplementedasalayeroveranopensourceoperatingsystemthatcontinuestoreceiveupdatesandsecuritypatchesfromaglobalcommunityofresearchersonatimelinethatcannotbepredictedbyDARPA/CGC.DARPA/CGCreservestheabilitytocontinuetosecureallpartsoftheCFEarchitectureincludingthekerneluptoandduringCFEagainstanythreattotheintegrityofCFE,toincludesecurityweaknesses,emergentthreats,executiondivergenceattacks,dataandmemoryattacks,hardware/softwareattacksutilizingRowhammer,memorydeduplication,etc.DARPA/CGChascommittedtotheDECREEABIandtheTIAPI;seealsoA137,A174.Anycompetitorapproachwhosesuccessisdictatedbyhardware,firmware,

orkernelerratamayfailinarealworldexercise,toincludedeploymenttomillionsofcomputersortheCFEinfrastructure.Q176:HowwilltheCFEframeworkgeneraterandomnumbersduringtheevent?A176:AseedvaluewillinitializepseudorandomnumbergeneratorsusedinCFE,toincludethePRNGsusedtocreatetheflagpage,pollingschedule,etc.Thisseedvaluewillbearrivedatthroughacalculationconductedinpublicview,describedinthisentry.Thenameofthisseedvalueismasterseed,hereafterMS.EachteamisaskedtoprovidetoDARPA/CGC,bymidnightEDTonJune17,2016,aphrasethatwillcontributetothecalculationofMS.Thecalculationisexplainedbelow,where:

HisdefinedasthebinarydigestoutputofSHA384 HEXisanASCIItransforminwhicheverybyteofinputisconverted

intoa2‐characterhexadecimalrepresentation ASCIINUMisanASCIItransforminwhichthedecimalformofa

numberisprintedinASCII XORisdefinedasbit‐wiseexclusiveor thecommacharacter(“,”)isdefinedasconcatenation.

Thecalculationisasfollows:MS=H(TeamName1,TeamPhrase1)XORH(TeamName2,TeamPhrase2)XORH(TeamName3,TeamPhrase3)XORH(TeamName4,TeamPhrase4)XORH(TeamName5,TeamPhrase5)XORH(TeamName6,TeamPhrase6)XORH(TeamName7,TeamPhrase7)XORH(TeamNameDARPA,TeamPhraseDARPA);DARPAwillcommittoTeamPhraseDARPAonJune10,2016bychoosingarandomnumberrandpublishingHEX(H(H(“DARPA”,TeamPhraseDARPA),ASCIINUM(r)))inthisentry.DARPAwillpublishallcompetitorteamTeamPhrasesaheadoftheCFEintheCGCFAQ.DARPAwillpublishTeamPhraseDARPAandraftertheCFEintheCGCFAQ.

TeamsmaycommunicatetheirTeamPhrasetoDARPA/CGCvia:Email:cybergrandchallenge@darpa.mil(S/MIMEauthsupportedperCQEinstructions).

Post:DARPA/I2O

Attn:CyberGrandChallenge675NorthRandolphStreetArlington,VA22203‐2114

ATeamPhrasemaybeofanylength.TeamsareencouragedtochooseaTeamPhrasethatcanbeexpressedinASCIIandwillsurvivegovernmentreviewforpublicposting.AnyTeamPhrasenotreceivedbymidnightEDTonJune17,2016willbesettotheNULLstring.HEX(H(H(“DARPA”,TeamPhraseDARPA),ASCIINUM(r)))is:

1773ec260768b2d17bd1cdd6fc54a19e619ee0cc58edcab0f19ef2a3212879670ede7121914ed1a55eac4565a0bf88ac Q175:IsDARPA/CGCacceptingsuggestionsforitscryptographiccommitmentstrategyforCFE? A175:TheCFEcommitmentmechanismshavebeendesignedovertwoyearsofChallengeexecutionandcryptographyreviewiscomplete.Nosuggestionswillbeaccepted.Q174:WillthescoringalgorithmdescribedinA58,accessedontheCFEinfrastructureperA13oftheTrialsFAQ[1],andprovidedasanoracletocompetitorsperA137bemodifiedpriortoCFE?A174:No.[1]https://github.com/CyberGrandChallenge/Event‐FAQ/blob/master/event_faq.mdQ173:WillourCRSknowinadvanceonwhichroundaChallengeSetwillberemoved?A173:No.

Q172:WillDARPA/CGCprovideamappingthatidentifieseachopponentCRSduringCFE?A172:No.Q171:WhatistheautonomypolicyforCFE? 

A171:Followingconsiderationofresponsestotheperiodofpubliccomment,carefulobservationofTrials,andcontinuedobservationofSparringPartner,DARPA/CGChasdeterminedthatthefollowingautonomypolicywillgovernCFE: AllroundsofCFEwillbecomputedautonomously.SeealsoSection4oftheCGCRules. Q170:InCFE,ineveryround,willeveryteambescoredonthesamecorpusofChallengeSets?A170:Yes.Q169:WhatisthedurationofthecomputationofCFE?A169:CFEcomputationwillbeginonthemorningofAugust4th,2016,andoccuroverroughlytenhoursofelapsedtime,subjecttoconstraintsdescribedinA168.CFEcomputationisprojectedtofinishduringtheCFEliveevent[1].SuccessfulcomputationofCFErequiresaminimumof40scoredrounds.[1]DetailsontheCFEliveeventaredistributedonwww.cybergrandchallenge.comQ168:WhatisthefixedroundscheduleforCFE?A168:CFEwillnotbecomputedusingafixedroundschedule.CFEwillbecomputedusinganeventplan.Theeventplanwillbeconsultedifanyrisktoeventexecutionisencountered,suchas:‐Naturaldisaster‐Coolantleak‐Powerfailure[1]‐Majorsoftwarefaultgeneratedbyadversariallyformedmachineinputs

(DECREEkernelpanic,etal).‐Thermaloverload‐Threattostorageoftheeventrecord(infrastructurestoragefailure,primary

orsecondary)‐Otherinfrastructurehardwarefailure(suspectedorconfirmed)‐OtherHPChardwarefailure(suspectedorconfirmed)‐Previouslyundiagnosedinfrastructuresoftwarefault

DARPA/CGCwillpublishacryptographiccommitmenttothiseventplananddistributetheplaintextofthiseventplanafterCFE.[1]https://lasvegassun.com/news/2016/may/11/power‐outage‐stops‐play‐for‐40‐minutes‐at‐vegas‐st/Q167:We’veprototypedupsomecodethatsendsallourcellphonesatextmessagewhenSparringPartnermatchesoccur.DoesDARPA/CGCobjecttosemi‐autonomousparticipationinSparringPartnermatches?A167:No.Q166:We’rerunninga“ChaosMonkey”[1]fulltimeonourCRSthatdoesthingslikerandomlycorruptmemory,shutdownnodes,corruptthedisk,andterminateprocesses‐gettingtheCRSreadyforafullyautonomousevent.Unfortunately,ourChaosMonkeyiscausingustoperformsub‐optimallyinourmatchesagainstSparringPartner.WillimperfectplayversusSparringPartnercountagainstourCFEscoreinanyway?A166:No.[1]http://techblog.netflix.com/2012/07/chaos‐monkey‐released‐into‐wild.htmlQ165:We’reconcernedthatourcompetitorsmayanalyzeourreplacementbinaries(RBs)throughtheconsensusevaluationprocessandco‐optportionsofourRB,makingallofourCBs/RBsaCRS‐formedinputemittedontotheCGCnetworkandpotentiallydestinedforacompetitor’sCB.Couldthishappen?A165:Yes.Q164:Onceremoved,willaChallengeSeteverbere‐introducedinCFE?A164:No.Q163:WhathappenswithanewlyintroducedChallengeSet(CS)?A163:InA157normalplaywasdescribed.ForanygivenChallengeSetM,itsinitialroundsarecoveredinthisentryasfollows:

WhenMisfirstintroducedinroundi:

i:Misfielded.ThefirstroundMispolled.i:ThefirstroundinwhichCRSformedinputsforMmaybesubmitted.i+1:ThefirstroundPoVmodulesthatseektoprovevulnerabilityinMmaybethrown.ReplacementChallengeBinariesandIDSrulesfilesforMreceivedperA161priortoroundi+2:

i+2:availableforconsensusevaluation. i+3:fielded.ThefieldingofReplacementChallengeBinariesandIDSrulesfilesincursservicedowntimeinamannersimilartoA157.Q162:HowmanyChallengeSetswillbeactiveatonceduringCFE?A162:Amaximumof30ChallengeSetswillbeactiveduringanygivenround.Q161:IfweuploadabunchofIDSrulesfilesforthesameCS,whichonedoyouuse?A161:Theinfrastructurewillfieldthelastcompleteduploadperround.ThisappliestoReplacementChallengeBinariesaswell. Q160:WillaCSroundscorebezeroperA157ifallCBsandtheIDSrulesfileforthatCSareidenticalbetweenrounds? A160:No. Q159:IsaCRSrequiredtore‐submitidenticalReplacementChallengeBinaries,IDSrulesfilesorPoVmoduleseveryroundaCSisinplay? A159:No. Q158:WhatisDARPA’sprocesstoanswercompetitorquestions?A158:DARPA/CGCisacompetitionwithsubstantialprizesatstake;thisintroducesuniquerequirementsontraditionalsoftwareQ&Aprocesses:

Neverprovidecompetitiveadvantagetoaparticularcompetitor.Allanswerstoallquestionsmustbemadeavailabletoallcompetitorssimultaneously;thisensuresthatnocompetitorhasaccesstomoreinformationaboutthecompetitionthananyothercompetitor.

Neverprovidecompetitivedisadvantagetoaparticularcompetitor.IfacompetitorasksDARPA/CGC“oursecretsauceisketchup,pleasepredictthe

performanceofketchupinCGC”,DARPA/CGCcannotusethequestionaswrittenasrevealingthetextofthequestionwouldrevealthesecretsauceandcreatecompetitivedisadvantage.

DARPA/CGCmustnotrestatepubliclyavailableinformation.Whenaquestion

hasalreadybeenansweredpublicly,restatingtheanswerprovidesthepossibilityofalternateinterpretationsthatharmtheintegrityofthecontest.Forthisreason,DARPA/CGCmustalwaysrepeatexistingpublicanswerswhentheyexist.

DARPA/CGCmustnotdistributeuncertainty.WhenasoftwarebugoccursDARPA/CGCmustindependentlyroot‐causeandauthoritativelyfixpriortoresponding,ascollaboratingwithanycompetitorduringthedebuggingprocesswouldprovidethatcompetitorwithadditionalinsightand,asaresult,competitiveadvantage.

DARPA/CGCmustworktominimizeFAQrewriting.Competitorsdependon

accurateFAQentries;revokinganentryandrewritingitcancausecompetitorstobackupandchangedevelopmentcourse.Forthisreason,manyFAQentriescannotbereleaseduntiltheyarebackedbyrepeatedtesting.

DARPA/CGCmustobserveanapprovalworkflowconsistentwithpublicrelease

ofinformationbyagovernmentagency.Allpubliclypostedanswersflowthroughanapprovalprocessbeforerelease.

CGChasbeenconstructedasatechnologycompetitionthatisgloballyaccessible,fairtoallcompetitorparties,high‐integrity,andastransparentaspossiblewithintheseconstraints.Q157:Howlongareroundsandhowdotheywork?A157:EachroundofCFEplaywillbefollowedbyashort,unscoredbreak.Thedurationofthesebreakswillbeaminimumof30secondswithnomaximumduration.Eachroundofplaywilllastatleast240seconds.Duringabreak,nogameactionswillbeinitiatedbyCFEinfrastructure.ACRSformedinputwhosesubmissioncompletesduringanunscoredbreakwillbesubmittedtotherounddirectlyfollowingthebreak.

Inresponsetofeedbackreceivedduringtheperiodofpubliccomment,DARPA/CGChasformulatedaroundstructuretoenableconsensusevaluation.ForanyChallengeSet,IDSrulesfilesandReplacementChallengeBinariessubmittedduringaroundnarescheduledasfollowsduringnormalplay*: n:CRSformedinputsubmitted.

n+1:DARPA/CGCinspectsCRS‐formedinputs.n+1:ServiceisdowntopollersandPoVs(roundscoreiszero).n+1:IDSrulefileandReplacementChallengeBinariesavailableto

opponentsviaconsensusevaluation.n+2:IDSrulefileandReplacementChallengeBinariesarefielded.

PoVmodulessubmittedduringaroundmarescheduledasfollows: m:CRSformedinputsubmitted. m+1:PoVisthrownasspecified.*normalplaydoesnotincludetheinitialandfinalroundsthataChallengeSetisfieldedduringCFE;CSintroductionandremovalwillbecoveredseparately.Q156:OurCRShadtroublewithnonces,challenge‐responseandconcurrencyinCQEChallengeSets;weareworriedthatthiswillbeanobstacleforourCRSinCFE.WillCFEChallengeSetsincludethesefeatures?A156:Yes.ChallengeSetauthorshavebeeninstructedtoprovideincreasedchallengesetdifficultyfortheCFEphaseofthecompetition.TheguidelinesprovidedtotheChallengeSetauthorscanbefoundathttps://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/submitting‐a‐cb.mdQ155:WillaccesstotheDARPAcloudnodesbemaintained24x7inthemonthspriortoCFE?A155:No.DARPA/CGCwillschedulemaintenancewindowswheneverpossible.DARPA/CGCobservesnofixedmappingbetweenteamsandhardwareandreservestheabilitytore‐provisionwithintheDARPAcloudatanytime.

Q154:WhatshouldIdoifmyDARPAcloudnodesfailduringdevelopmentpriortoCFE?A154:DARPAcloudissuescanbereportedtocybergrandchallenge@darpa.mil.InthecaseofasoftwarefaultDARPA/CGCmayautomaticallyre‐provisionthenode.InthecaseofahardwarefaultDARPAmaypursueremedy,replacement,and/orre‐provisioning.Q153:AretheDARPAcloudnodedisksbackedup?A153:No.CompetitorsareencouragedtofollowcloudcomputingbestpracticesandremainpreparedtodeploytheirCRSonafreshlyre‐provisionedcloud.AutomatedinstallationanddevopsbestpracticesmayassistintheeventofCFEdisasterrecovery.Q152:WhatguaranteesdoesDARPA/CGCmakeagainstHPCnodefailure?A152:DARPA/CGCcannoteliminatethepossibilityofnodefailure.DuringdevelopmentpriortoCFE,replacementnodeswillbeinstalledpercommercialwarrantyaspossible.DuringCFE,failednodeswillnotbereplaced.CRSdesignshouldtakeplaceinformedbythesefacts.Q151:WhatruntimelimitsareplacedonNetworkAppliancerules?A151:EachNetworkApplianceinstancewillutilizeonerulesfile.Foreachinstance,theNetworkAppliancemaintainstwoanalysisringbuffers,oneforincomingbytesreadfromeachsideoftheconnection.Theseringbuffersarecappedat100x1024bytes.Q150:WhatruntimelimitsareplacedonCFEPoVmodules?A150:ThecompetitionframeworkenforcesahardlimitonPoVmodulesof15secondsofwalltimeandlimitsphysicalmemoryusageto64x10242bytesperPoVinstantiation(e.g.aPoV“throw”).Q149:WhatruntimelimitsareplacedonCFEChallengeBinaries?A149:ThecompetitionframeworkenforcesahardlimitonChallengeBinariesof15secondsofwalltimeand3x10243bytesofvirtualaddressspace.Seealso,A58andA119.Q148:WhataretheCFEuploadlimits?A148:TheCFEcompetitionframeworkTeamInterfaceAPI[1]willacceptinputnolargerthan10x10242bytesforPoVmodules,10x10242bytesforNetworkAppliancerulesfilesand50x10242bytesforReplacementChallengeBinaries.

Seealso:Q58,Q77,Q119,andQ132.[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/ti‐api‐spec.txtQ147:WillthevalueofMAX_THROWSintheTeamInterfaceAPIspecificationbehardcodedforCFE?A147:Yes:10.Seealso[1].[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/ti‐api‐spec.txtQ146:DoestheCFEframeworkfacilitatesavingstatebetweenPoVthrows?A146:No.Q145:HowwilltheNetworkApplianceimpactthePerformancecomponentoftheAvailabilityscore?A145:Networkappliancerulesetexecutiontimewillbemeasuredrelativetotheexecutiontimeofareferenceemptyruleset;thesemeasuredincreasesinexecutiontimewillbecombinedwithexistingCBexecutiontimemeasurementstodeterminetheexecutiontimeportionofthePerformancescore.SeealsoA58.Q144:InA58,RetainedFunctionalityisspecifiedintermsofnetworktestcases;arethesenetworktestcases“servicepolls”?A144:Yes.Q143:Canaservicepollfailbytimingout?A143:Yes.Q142:Whyisafacilityprovidedtoquerythelocationofthetype2PoVmemoryrangeandPoVlengthincfe‐pov‐markup‐spec.txt?A142:TheDECREEAPIallowscompetitionshostedusingDECREEtospecifythestartaddressofthetype2PoVmemorylocationandtherequiredsizeofaPoVthroughrequeststothecompetitionframework.ThisflexibilitywillassistinportingtheDECREEframeworktofuturearchitectures.ThisfacilityallowsresearchersutilizingDECREEflexibilityindesigningcurrentandfuturecompetitionsandresearch.DuringtheCyberGrandChallengeFinalEvent,theType2PoVmemoryrangeandPoVlengthwillbeconsistentwiththe“submitting‐a‐cb.md”walkthrough.SeealsoA29.

Q141:WillteamshaveaccesstoascoringoracleduringCFEthatcanbeutilizedpriortofieldingRBs?A141:PerA4,A17,andA137:No.Q140:Isitpossibleforafieldedreplacementbinarytoreceiveanetworkinputthathasnottransitedthenetworkappliance?A140:NoQ139:Canfieldingapoorlyperformingnetworkappliancerulenegativelyaffectavailabilityscore?A139:Yes.Q138:DuringCQE,asecondaryscoringdocumentwasreleasedwhichprovideddegradationcurvesandformulae.WillasimilardocumentbereleasedforCFE?A138:AdditionalinformationaboutCFEscoringwillbereleasedthroughtheFAQ.Q137:Whatadditionalinformationabouttheimpactofnetworkappliancerules,executiontime,etc.ontheAvailabilityscorewillbereleased?A137:DARPA/CGCintendstoallowcompetitorstoanswerthesequestionsindependentlybyprovidingaccesstoacompetitionframeworkoracleviatheDARPAcloud.Atthetimeofthiswriting,constructionofthecompetitionframeworkcontinues.ThisaccesswillbegrantedpriortoCFETrialsandrevokedpriortoCFE.Q136:WhichChallengeSetswereexcludedfromcomputingCQErankinginA130?A136:Theexclusionproofdecryptsto:

Common-name,CSID YAN01_00007,fd0e1101 YAN01_00009,c9967603 YAN01_00010,31502e01 YAN01_00011,77f39101 YAN01_00012,9d97ef01

Thedecryptioncanbeverifiedbytakingtheencryptedbase64blobfromA130anddecryptingitwiththekeyinthebelowscript:

PASS="The best way to predict the future is to invent it.qR59RBQiIbMnuFwJFJpNUHMusu6RjyrGxvDI17v9egiGRloXME9bGjwBjF32bc2qdNqLVwTz3CNuogh1XcWapd5IGQ1VEE4B9/Mt9TAKN0S1P97XZj4WFCH+KAhMzTR3AkYM/mTbtzXfhDzSOEgl7G+7T1IE0uYaYYYuEwm4uL4=" A130_PROOF="U2FsdGVkX1/21aa4u0ZhcTCAyGr8oX7aCimSvKfGG3HFvjy5sCWi7Dh0dY9mb2nizlldmCCeB7g2RbUv3R0PkvFVp+EeTeWj8MpTiBCP0cVS3/uXy2rWIZtkFpOYBXIqYdybwpawzlRshJ8BeA2/WCWBeupEhCWFjJT7XigmLyEZqIujQ/rsWJTLOTfWFWC0" echo "$A130_PROOF" | \ base64 --decode | \ openssl aes-256-cbc -d -pass "pass:$PASS"

Q135:DuringareviewoftheCQEcorpus,ourteamdiscoveredthatsomereplacementbinariesreleasedbyourcompetitorsdisruptthefunctionofourprogramanalysissoftware.We’reconcernedaboutthiseffectduringamoreautonomouscompetition(CFE).TheCGCEventParticipationAgreementexplicitlyallowsthiseffectfromCRS‐formedinputs;willDARPA/CGCprovideadditionalprotectionsagainstthiseffect?A135:No.Q134:Therequirementforfullautonomybringswithitaslewofengineeringdecisions,includingwatchdogtimers,redundantcomponents,parallelexecution,andhighassuranceprogramming.DoesDARPA/CGChavearecommendedpathtowardsfullautonomy?A134:No.SeealsoA4andA17.Q133:WhichCRS‐formedinputsnamedintheCGCEventParticipationAgreementcanbeidentifiedinthecurrentDECREEarchitecture?A133:Atthepresenttime:

‐ Replacementbinaries‐ Dataproducedbyreplacementbinaries‐ Networkappliancerules‐ Trafficmodifiedbynetworkappliancerules‐ Proofsofvulnerabilities‐ Dataproducedbyproofsofvulnerabilities

Q132:IsaCRSpermittedtosubmitpovmlfilesasaProofofVulnerabilitytothecompetitionframeworkduringCFE?A132:No;seealsosection2.2oftheCRSTeamInterfaceAPIspec[1]thatdescribeshowtosubmitaPoVduringCFE.[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/ti‐api‐spec.txtQ131:WhatisthedurationandautonomypolicyoftheCFE?A131:InA58,DARPA/CGCindicatedthatCFEpointswouldaccrue"perChallengeBinaryperround".DARPA/CGCisopeningaperiodofpubliccommentthatwillcloseonSeptember25th,2015.ThisperiodofpubliccommentencouragesfeedbackontheappropriatetimingoftheCFEconsistentwiththegoalsofCGC:theabilitytodemonstrate,iffeasible,real‐timeproactiveandreactivemitigationofnovelsecurityfaultsonmachine‐scaletimelinesmeasuredinseconds.Feedbacktopicsinclude: 1>DurationofCFERounds,inseconds 2>TotalnumberofRoundsinCFE

Note:Feedbackfor1>and2>shouldbepairedtoindicateatotaleventduration.

3>Releaseand/orexpirationscheduleforChallengeSets(CS)inCFE.

4>PolicyonHumanInterference(handsondebugging).ApartiallistoftheorizedpoliciesonCFEdebuggingfollows:

4a>TeamscouldbeallowedtodebugtheirCRSatanytimeduringCFE.Duringdebugging,aCRSwouldbedisconnectedfromtheframeworkandunabletoaccumulatepointsduringdebugging.Afterdebuggingcompleted,anyChallengeSet(CS)previouslyrevealedtothedebuggingteamwouldbefruitofthepoisonedtreeandunavailableforscoringtothatteamfortheremainingdurationoftheevent.Forexample,ifteamalphachosetodebuginthefirstsecondwhenasingleCSwasfielded,thenallpointsforthatCSandtheabilitytofieldthatCSwouldbeunavailabletoalphaforthedurationoftheevent;ateambetathatchosetodebuginthefinalminutewitheveryCSfieldedwouldbeunabletoscoreanyfurtherpointsafterdebugging.

4b>Anumberofscheduleddebuggingbreakscouldbeannouncedpriortotheevent,dividingCFEintoStages.ForeachStage,allpreviousCS'swouldbewipedandfreshCS'swouldbeprovided.

4c>CFEcouldbetotallyautonomous;catastrophicerrorscouldnotbecorrected.

4d>CompetitorswouldbepermittedtofullyinteractwithaCRSforaninitialperiodoftimeattheonsetofCFEtoensurethecorrectoperationoftheCRS.AfterthistimetheCRSwouldcontinueautonomouslyasin4c.

Competitorsareencouragedtoprovidecommentstocybergrandchallenge@darpa.milbySeptember25th,2015.

Q130:WhichChallengeSetswillbeexcludedfromcomputingCQEranking?

A130:SeeFAQA117.TheproofofexcludedCSselectionis:

U2FsdGVkX1/21aa4u0ZhcTCAyGr8oX7aCimSvKfGG3HFvjy5sCWi7Dh0dY9mb2nizlldmCCeB7g2RbUv3R0PkvFVp+EeTeWj8MpTiBCP0cVS3/uXy2rWIZtkFpOYBXIqYdybwpawzlRshJ8BeA2/WCWBeupEhCWFjJT7XigmLyEZqIujQ/rsWJTLOTfWFWC0Q129:AnemptywriteinaPoVXMLpassespoll‐validatebutcausesanexceptionincb‐replay;willthisbefixed?A129:AfixwillbereleasedinafutureversionofDECREE.DuringCQE,competitorsareencouragedtorefertoFAQentryA111withrespecttodebuggingtheirCRSduringtheevent.PriortoCQE,competitorsareencouragedtoensuretheirCRSdoesnotgenerateaPoVwithanemptywritestatementsimilartothebelow: <write><data></data></write>Q128:WhatistheofficialTwitterfeedfortheCGCQualificationEvent?A128:https://twitter.com/DARPA_CGC_CQEQ127:HowdoIconfirmthatIamrunningthelatestversionofDECREE?A127:InthelastDECREEreleaseemail,atest[1]wasissuedtocheckthevagrantversion.Ifyourcurrentsystemrespondswiththeoutputcitedbelow[1][2],thereisnoneedtoupdate.Ifyoursystemrespondswithdifferentoutput,DARPA/CGCadvisesthatyouupdate.[1]TheexpectedoutputofvagrantsshofanupgradedVMwillbe:Linuxcgc‐linux‐packer3.13.2‐cgc#1SMPMonApr1318:33:57UTC2015i686[2]vagrant@cgc‐linux‐packer$cat/etc/decree_versioncqe_development‐vm‐61

Q126:WhereistheCQEchallengebundle?A126:TheCyberGrandChallengeCQEbundleisavailable!PleasedownloadandverifyatyourearliestpossibleconveniencepriortoCQE.Downloadinstructionsareavailable[1].PleasecontinuetomonitortheEventFAQ[2]andtheCQETwitterFeed[3]throughoutCQE.PackageName:cgc_qualifier_event.ar.gz.encPackageMD5:0ec6a5708aea0b0d7c5a9f1c37924423PackageSHA256:8274d4086ee1039e7901b3cc7221fa89e2f923dc5ad39924965f2ff5b110449d[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/cgc‐qualifier‐event‐api.md[2]https://github.com/CyberGrandChallenge/Event‐FAQ/blob/master/event_faq.md[3]https://twitter.com/darpa_cgc_cqeQ125:DoallhardwarecomponentsoftheCRSneedtobephysicallypresentataCQEsitevisit?A125:No.Q124:WereallsubmissionstoScoredEvent2scored?A124:No;somesubmissionstoScoredEvent2werenotscored.Allfinalsubmissionswerescoredandranked.PerA67abesteffortwasmadetocollectandscoremultiplesubmissions.DuringSE‐2,newsubmissionswerecollectedevery10‐20seconds;inthecaseofmultiplesubmissionsfromthesameteamtothesamechallengesetwithinthattimewindow,thelatestsubmissionwasscoredandtherestwerenotscored.Fiveteamswereaffected.Q123:Ismem_use(CB)themaximumRSSsizefortheCBprocess?A123:No;mem_use(CB)iscalculatedcongruentwiththeformulapresentedintheCBscoringwalkthrough[1].[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/scoring‐cbs.mdQ122:Isexec_time(CB)theuserCPUtime+systemCPUtimefortheCBprocess?Justusertime?Orsomethingelse,e.g.wallclock?Ifwall‐clock,whatinterval?A122:exec_time(CB)doesnotincludesystemtimeperA113;exec_timeisusertimemeasuredwithasensorasdescribedinA113.

Q121:Aremem_use(CB)andexec_time(CB)computedfromthevaluesweseeinru_maxrss,ru_minfltand(ru_utime+ru_stime),respectively,intherusagestructurereturnedbyawait3/wait4systemcallfromaparent?A121:Therusagedatastructurereturnedbywait3/wait4containsru_maxrssandru_minfltthatareusedtocomputemem_use(CB),asdescribedinCBscoringwalkthrough[1].DARPA/CGCdoesnotderivetimingmeasurementsfromtherusagedatastructureperA113.[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/scoring‐cbs.mdQ120:HowarethemeasurementsforalltheservicepollsagainstaCBaggregatedintoasinglemem_use(CB)orexec_time(CB)component?Ismem_use(CB)thehighestmaximummemoryusageacrossanyservicepoll/input,orisitanaverage?Likewise,istheexec_time(CB)theaverageacrossservicepolls?A120:ThesecalculationsareperformedinamannercongruentwiththescoringCBswalkthrough[1].[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/scoring‐cbs.mdQ119:FilesizeandmemoryusagerestrictionsimposedonReplacementBinariesappearverysevereandpotentiallynotrepresentativeofreal‐worldconstraints;inthecaseofmemoryusagerestrictions,per‐pagegranularityimposesaheavyoverheadburdenduetorelativelysmallsizeofChallengeBinaries.Cantheselimitsbeadjustedandmemorymeasurementgranularityincreased?A119:CyberGrandChallengedoesnotimposeperformancepenaltiesbymeasuringtheincreaseinperformancebetweenacompetitorsubmittedRBandtheoriginalCB.Performancepenaltiesareassessedbasedonthedifferencebetweenasuccessfullydefended(patched)CBcreatedbyahumanexpertandacompetitorsubmittedRB[1].Thusadefenseisproventoexistandcompetitordefensesaremeasuredagainstknownsuccessfuldefenses.Fromitsinception,theCGChassettheambitiousgoalofmeasuringtheskillofautomatedsystemsagainsttheabilitiesofhumanexperts.Whilesomedefensivecompetitionshaveimposedabsoluteperformancelimitationsof5%orless[2],CGCimposedagracefuldegradationcurveandopenedthiscurvetopubliccommentinearly2014.ToprovidecontinuitytoallcompetitorsthismechanismwillnotbemodifiedaftertheperiodofpubliccommentperA100.Onthetopicofgranularityofmemorymeasurement,theCGCadoptedtheuseofpages,aMMUconstruct;traditionallyhumanexpertshaveskillfullyoptimizedtheuseofpages.

[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/CQE%20Scoring.pdf[2]http://www.microsoft.com/security/bluehatprize/rules.aspxQ118:WhenhandlingmultipleCBsinaCS,isacrashinanyoralloftheCBsconsideredproofofavulnerability?A118:Yes,acrashinanyorallCBsinaCSthatmeetsthecriteriadescribedinA28isconsideredtohaveprovenavulnerability.Q117:WilleveryChallengeSetintheCQECorpusbescoredaspartoftheCQEevent?A117:No.Therewillbeover100ChallengeSetsintheCQECorpus;asmallnumberofthesearepurelydiagnosticandwillbeexcludedfromcomputingCQEranking.ForeachoftheseexcludedCSs:‐ PoVsagainsttheseCSswillnotbepartofCQEConsensusEvaluation‐ CSScoreswillnotbeusedforanypartofCQEcompetitorranking‐ CSScoreswillnotbeconsideredforanypartofsection3.1.3oftheCGCRulesToprovethatexcludedCSswerechoseninadvance,DARPA/CGCwillprovideacryptographicallyverifiableproofofthisselectionpriortoCQE.AfterCQE,aselectiondocumentmatchingthisproofwillbereleased.CompetitorsareadvisedtofieldaCRSthatattemptstosolveeveryCSinCQE.Q116:WhatisthemaximumsizeoftheuncompressedCQEChallengeBundle?A116:Sizewillnotexceed5000x10242bytes.Q115:WhyweretherenovulnerabilitiesthatlackedareferencePoVintherankingofScoredEvent2?A115:DARPA/CGChasrefineditsproceduresfollowingScoredEvent1.NotethatduetothepotentialconflictofinterestinvolvedinmakingCBchangeswhilesimultaneouslyhandlingde‐anonymizedscoringmaterialandPoVs,theDARPA/CGCscoringteamcannotdetermineflawuniquenessorformulateCBrepairsperAttack1inA110.SeealsoA9andA97.Q114:Thepublicspecificationforcb‐replay[1]indicatesthatthereadtagacceptsatimeouttag.Thisdoesnotappeartobeimplemented.Willthecb‐replayimplementationbeadjustedtomatchthespecbeforeCQE?A114:No‐‐themismatchwillbemaintainedthroughCQEforcontinuitypurposes.

[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/replay.dtdQ113:FollowingupA99intheFAQ,isadditionalinformationavailableaboutthecorrelationbetweenoperatingsystemtimingestimatesandDARPA/CGCsensortimingfollowingSE2?A113:TheDARPA/CGCteam'stimingsensorwasdesignedtomeasureprocessexecutiontimewithunprecedentedaccuracyandprecision.Thissensorstopsmeasuringtimeimmediatelyduringthesystemcalltransitionfromusertokernel,andresumesmeasuringtimeimmediatelyduringthetransitionfromkerneltouser.Unfortunately,thetaskclockprovidedbyLinuxisnotasrigidlyaccurateduringthistransitionandcanmeasuresomekernelexecutionasuserexecution,causingdivergencebetweentheDARPA/CGCsensorandestimatesbasedonthesystemclock(usingtheimprovedestimationfacilityinDECREEdescribedinA99).Thisdivergenceincreasesbasedonthenumberandtypeofsystemcallsmadeduringexecution.Therandom()call,inparticular,isamajorsourceofdivergence.ThegraphsprovidedshowthecorrelationresultsoftheDARPA/CGCsensorandthesystemclock,withdivergenceincreasingwhenlargenumbersofsystemcallsaremade.DARPA/CGCexpectsthatformostChallengeSetexecution(unburdenedbyapreponderanceofsystemcalls)thecorrelationbetweenthetaskclockestimateandthehighaccuracyDARPA/CGCsensorshouldbehigh.GraphsofthiscorrelationoversamplesfromtheservicepollsforSE‐2ChallengeSetsareprovidedbelow.PleasenotethedivergentgraphforYAN01_00006,auniqueCBwhoseexecutionispredominantlysystemcalls.Thefollowingplotsshowlinearcorrelation;anrvalue[1]of1.00denotesmaximalpositivelinearcorrelation(optimal).Thex‐axisisgeneratedbytheoutputoftheDARPA/CGCsensor;they‐axisisgeneratedbytheOS‐basedestimatesissuedbyDECREEinA99.Eachdotinthescattergramisasingleservicepoll.[1]http://en.wikipedia.org/wiki/Pearson_product‐moment_correlation_coefficient

Q112:WhatisthemethodologytofindmysubmissionsinthereleasedSE‐2scoringdata?A112:RevisedinFAQv16:ReplacementBinaryscoresandPoVscoreshavebeenreleasedasseparate.csvfilesatcgc.darpa.mil;perthebundledinstructionstheSHA256ofeachRBandPoVisusedtocalculatetheRBandPoVidentifiersusedinthese.csvfiles.Pleasenotethattheseidentifiersarenotprovidedbythesubmission

verificationscript[1];theseidentifiersshouldbecalculatedseparatelyusingthesolutionpackageverificationscript[2].[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/scripts/cqe_submission_verification.py[2]https://github.com/CyberGrandChallenge/cgc‐releasedocumentation/blob/master/scripts/cqe_verify_solution_package.pyQ111:DuringtheCQE,whatkindofhumaninvolvementisallowed?AlthoughwehavereadSection4(FullAutomationRequirement)oftheCGCRules,itremainsunclearwhat"humanassistanceincyberreasoningprocesses"means,exactly.Onascaleofwhatweimaginetobemostreasonabletotheleastreasonable,hereareafewexamplesaboutwhichwearecurious(ofcourse,specificexamplessuchassplittinguppcapsarethingswewillhandleaswehavethoughtofthem,thesearejustexampleswehaveencounteredtohelpmotivateourquestions):*powerorhardwarefailureandwemanuallyrebootservers;*kernelpanicorOOMandwemanuallyrebootservers;*launchingVMs/scripts(suchasforfuzzingorsubmittingbinaries)byhandatthestartoftheCQE;*largepcapcausesustomanuallysplititupbeforefeedingittoourCRS;*trivialbug(suchasmisnamedpythonvariable)inourCRS,sowefixthebugandrestarttheCRS;*binaryusesbehaviorwedidn'taccountfor(suchaswritingtoSTDIN)andwepatchoursystemtoaccountforthis;*buginourCRScausesustomissthefaultingaddressforacrash,sowefixthebugandrerunthatportionoftheCRS;*crashisnotreproducible,sowemanuallyfixitup*CRScannotfindcrash,sowemanuallydoitThefirstthreewecurrentlydeemas"inscope".Thelasttwowecurrentlydeem"outofscope".Thefourinthemiddleweareveryunsureof.A111:DARPA/CGCbelievesthattheissueofhumaninterferenceincyberreasoningprocessesisclear‐cutandcaneasilybedifferentiatedbyanyofourexpertcompetitorsinthisfield.TheDARPACompetitionAgreementgovernshumaninterferenceincyberreasoningfunctions:anyfunctionthatinteractswithChallengeSetsandmakesdecisionsabouthowtoformulateReplacementBinariesandPoV’s.Achangetoareasoningfunctionwillchangetheoutcomeofasystem’sperformanceinanyAreaofExcellence[1].WhenahumanperformsacyberreasoningprocessonthecontentsoftheChallengeBundle,theresultingknowledgeisfruitofthepoisonoustree.Aftergainingsuchknowledgefromanysource,ahumanmaynotmodifytheCRS.

Below,thistestisappliedtothelistofexamplesprovidedinthequestion:

1> Action:powerorhardwarefailureandwemanuallyrebootserverso Humandidnotpossesstaintedknowledge.Nointegrityissue.

2> Action:KernelpanicorOOMandwemanuallyrebootservers

o Humandidnotpossesstaintedknowledge.Nointegrityissue.

3> Action:LaunchingVMs/scripts(suchasforfuzzingorsubmittingbinaries)byhandatthestartoftheCQE

o Humandidnotpossesstaintedknowledge.Nointegrityissue.

4> Action:Manuallyun‐packagingChallengeBundleandfeedingcontentsintoCRS.

o Humandidnotpossesstaintedknowledge.Nointegrityissue.

5> Action:ManuallypackagingresultsfromtheCRSforsubmissiontoDARPA.o Humandidnotpossesstaintedknowledge.Nointegrityissue.

6> Action:LargepcapcausesustomanuallysplititupbeforefeedingittoourCRS

o Humandidnotpossesstaintedknowledge.Nointegrityissue.

7> Action:Trivialbug(suchasmisnamedpythonvariable)inourCRS,sowefixthebugandrestarttheCRS

o ThisexampledescribesanunanticipatedsystembugthatoccurredduetoautomatedprocessingoftheChallengeBundle: IftheCRSsimplycrashesoninputandthebugcanbefixed

throughCRScodeinspection,notaintedknowledgewasinvolved;nointegrityviolation.

IfahumanexaminestheChallengeBundleandperformsacyberreasoningprocesstoarriveattheknowledgeusedtopatchthebug,anintegrityviolationhasoccurred.TheknowledgeusedtopatchthebugderivedfromhumanexaminationoftheChallengeBundle,andisthereforefruitofthepoisonoustree.

Forexample,ifaCRScontainsaflawthatisdesignedtocrashonthefirstCBprocessed,itisnotanintegrityviolationtousethecrashdumptoidentifythisflawaslongasinspectionofthecrashdumpdoesnotyieldtaintedcyberreasoningknowledgegainedfromtheChallengeBundle.HoweverwereateamtoexaminetheChallengeBundleinanattemptto“optimallyfix”thisflaw,perhapsbyinsertingnewPoVseedcases,anintegrityviolationwouldhaveoccurred.

8> Action:Binaryusesbehaviorwedidn'taccountfor(suchaswritingtoSTDIN)andwepatchoursystemtoaccountforthis

o ChallengeSetswhosespecificationisunanticipatedbyteamsmaycausefaultsandshouldbehandledpercase7,above.

9> Action:BuginourCRScausesustomissthefaultingaddressforacrash,sowefixthebugandrerunthatportionoftheCRS

o Inthishypotheticalscenario:

a. Abugexistsinthedynamicanalysis/concreteinputportionofaCRS.

b. ThisbugcausesafalsenegativeinwhichtheCRSdoesnotidentifyacrash.

c. Ahumanusesadebuggertoidentifythefalsenegative.d. Humanusesbugreportfrom9.ctofixtheCRSbugandrestarts

CRS.Anintegrityviolationoccurredinstep9.cwhenahumanperformedacyberreasoningprocessontheChallengeBundle,andthenusedfruitofthispoisonoustreetoidentifyaflawandformulateafixintheCRS.

10> Action:Crashisnotreproducible,sowemanuallyfixitupo Inthishypotheticalscenario:

a. AhumanexaminestheoutputofaCRSinadebuggerortest

harnessbytestingtheCRS‐generatedPoVwiththeCBprovidedintheChallengeBundle.

b. Ahumandetermines,throughiterativetestingandanalysisusingthedatacomponentsin10.a,ahigherreliabilityformulationforthePoV.

c. Usingtheknowledgefrom10.b,ahumanreprogramstheCRStoemitanewlyformulatedPoV.

DuetotheuseoftheCBprovidedbytheChallengeBundlein10.a,thechangemadein10.cisfruitofthepoisonoustree;thisisanintegrityviolation.

11> Action:CRScannotfindcrash,sowemanuallydoit

o HumanPoVgenerationrequirescyberreasoning;thecreationofPoVsisanAreaofExcellence[1];integrityviolation.

[1]AreasofExcellencearedescribedintheCGCRules,Section1.3

Q110:SomeCBsinSE1containedvulnerabilitiesthatwerenotprovenbythereferencePoVset.ACRSthatpatchesaflawprovenbyareferencePoVcanconsistentlyincreasetheReferencecomponentofitsSecurityscore.Ontheotherhand,aCRSthatpatchesaflawnottestedbythereferencePoVsetcanonlyimprovetheConsensuscomponentofitsSecurityscore.Thisinconsistencyappearstoleadtoacompetitionthatmaynotconsistentlydistinguisheffectivepatching.Whycan'ttheDARPAscoringteamdirectlyaddCRS‐discoveredPoVstothelistofreferencePoVsforthepurposeofSecurityscoring?

A110:InordertomitigateAttack3,below.

Q109:WhydoesthescoringsystemdescribedinA58onlyallowforonePoVperCB?

A109:InordertomitigateAttack1andAttack2below.

Q108:Whywasn'taPoVweightedbythenumberofRBsitprovesvulnerable?

A108:InordertomitigateAttack2below.

Q107:Whywasn'tanRB'sSecurityscoreweightedbythetotalnumberofPoVsitdefendsagainst?

A107:InordertomitigateAttack1&Attack4below.

CQEThreatModeling

BelowisalistofmitigatedattacksagainsttheintegrityoftheCGCcompetition.DARPAhasmadeeveryefforttoincreasecompetitorconfidenceincompetitionresultsbymitigatingattacksagainstcompetitionintegrity.CQEscoringistheresultofextensivethreatmodeling.Asubsetofthisthreatmodelingexerciseisincludedinthelistofmitigatedattacksbelowthatareprovidedasareferencepointforanswersaboutspecificscoringdesigndecisions:

Attack1:PoVStuffing

ThisattackrequiresascoringsysteminwhichaCRSmaysubmitandreceivecreditformultiplePoVsthatevaluateasingleCB.Tomountthisattack,aCRSusesknowledgeofaflawtogenerateagreatnumberofPoVsthatproveasingleflawinaCB.Forexample,inthecaseofasimplebufferoverflow,aCRScouldsubmitthousandsofPoVsthatoverflowthesamebufferwithdifferentcontents.ThisattackcannotbeeasilydistinguishedfromaCRSthathasaccomplishedthemoredifficultfeatofgeneratingmultiplePoVsthatprovemultipleflaws.AttemptstodeterminewhenaPoVprovesa"uniqueflaw"requiressubjectivejudgmentsonbehalfoftheDARPA/CGCscoringteam;seeA9.

Attack2:SockPuppetCrashDummies

Inthisattack,aroguecompetitor,inviolationoftheDARPAcompetitionagreement,standsupmultiple"sockpuppet"teams.Asetofthesesockpuppetteamsintentionallyintroduce"keyed"flawsintheirreplacementbinaries(RBs)thatrequireuniqueknowledge(a"key")toprove.InviolationoftheDARPAcompetitionagreement,thesockpuppetssharethisknowledgewiththeroguecompetitor.ThisroguecompetitorprovidesconsensusevaluationPoVsforthe"keyed"flaws,therebyimprovingitsEvaluationscorewithoutactuallyfindinglegitimatesoftwareflaws.Manysubtlevariationsofthissimpleattackexist.

Attack3:ReferenceCollusion

Inthisattack,aroguecompetitor,inviolationoftheDARPAcompetitionagreement,standsupmultiple"sockpuppet"teams.InviolationoftheDARPAcompetitionagreement,thesockpuppetscolludewiththeroguecompetitor,onlyintroducingPoVsthatproveflawsknowntobepatchedbytheroguecompetitor.ThisattackallowsSecurityscoringtobedominatedbyflawschosenbytheroguecompetitor.

Attack4:ConsensusCollusion

Inthisattack,aroguecompetitor,inviolationoftheDARPAcompetitionagreement,standsupmultiple"sockpuppet"teams.EachofthesesockpuppetsintroducesavalidsetofPoVscontaininga"key".InviolationoftheCGCRules,thesockpuppetscolludewiththeroguecompetitor,sharingthe"key"value.TheroguecompetitorisabletoartificiallyimproveitsSecurityscorewithoutactuallymitigatingaflawbysubmittingRBsthatblockanyinputcontainingthe"key”.Manysubtlevariationsofthissimpleattackexist.

Q106:WilladditionalChallengeBinaryspecificationsbereleasedthatexceedthespecificityoftheexistingdocumentation[1]?A106:No.CyberReasoningSystemsshouldreasonaboutsoftwarepertheCGCRules[2].[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation[2]https://cgc.darpa.mil/documents.aspxQ105:HowwillDARPA/CGCprovethatallPoVsscoredduringCQEcouldcauseunlimitedexecutionofarbitrarycode?A105:DARPA/CGCdoesnotassertorintendtoprovethis.Softwarevulnerabilitiescanleadtoavarietyofeffectsrangingfromdenialofservicetocodeexecution;DARPA/CGCwillacceptProofofsuchVulnerabilitiesinSE2andCQEperA28.

Q104:We'reunsurethatourapproachwillworkontheDARPA/CGCscoringapparatus.Canyouguaranteethesuccessofourapproach?A104:No.DARPA/CGCrecommendsthatadventurousorriskyuseofhardwareorsoftwarefeaturesbeevaluatedthoroughlyduringScoredEvents.Commercialsecuritysoftwareisoftenrequiredtooperateinuncertaincomputingenvironmentsandreliesonhardwareinterrogation(suchasCPUID)andsoftwareinterrogation(suchasO/Sversionnumbers)toproceed.Q103:DidDARPA/CGCchangethewaymemoryusageisquantifiedormeasuredinDECREE?A103:No.Memoryusagemeasurementhasremainedconsistentwiththeformulareviewedduringpubliccomment,publishedinA58anddetailedintheDECREEdocumentation[1].Inanattempttoassistcompetitorswithestimatingthisquantity,DARPA/CGCbuiltandcontinuestoupdateawalkthroughtoassistcompetitorsinmakingtheirownmeasurements.OneLinuxquirkthatcouldcausemeasurementproblemsforcompetitorsisthatchildprocessesareassignedamaximumRSSvalueinheritedfromtheirparent.Thiscausesestimationproblemswhenalargeparentprocesslaunchesasmallchildprocess.Inordertoallowcompetitorstomakehigher‐fidelityestimates,thisquirkwasdisabledforCGCprocessesinDECREE[2].DARPA/CGCwillcontinuetoworktocreatethemosthighfidelityimplementationofthescoringmethodologypublishedinA58aspossibleandprovideassistancetocompetitorstomakeaccuratemeasurements.[1]https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/CQE%20Scoring.pdf[2]https://github.com/CyberGrandChallenge/linux‐source‐3.13.2‐cgc/blob/master/fs/binfmt_cgc.c#L300Q102:InSE1,thereleasedreferencesecurityscoresarealleither0.0or1.0.Hasthisbeenconfirmed?A102:Yes.Q101:WhiletraditionallyCapturetheFlagcontestsandDARPAChallengeshavenevergivencompetitorsaccesstothescoringsystemandassociatedsensorspriortocompetition,couldCyberGrandChallengebecomethefirsttoallowopensourceaccesstothescoringsystem?A101:DARPA/CGCwillnotallowcompetitoraccesstothescoringmeasurementcodeanditsassociatedsensorsduetothecurrentlackoftechnologycapableofmakingstrongintegritypromisesaboutexecutablecodesubjecttoadversarialintrospection.DARPA/CGCwillcontinuetouseanopen,competitor‐reviewedscoringalgorithmdescribedinthisFAQ(A58/A59),allowperiodiccompetitor

accesstothescoringsystemthroughthescheduledScoredEvents,andprovidemeasurementassistancetocompetitorsthroughDECREEreleases.Q100:CanthescoringmethodologybeperiodicallyupdatedsimilartotheDECREEreleases?A100:No.Allcompetitorshavehadequalaccessandabilitytocontributetothescoringalgorithmduringitsperiodofpubliccomment.Changingthescoringsystemmid‐competitionisdisruptivetotheoverallcompetitorpoolandwilldegradecompetitorconfidence.Competitorsshouldbeassuredthatscoringfundamentalswillnotbealtered.Q99:We'reusingtheoperatingsystemtomeasureperformanceandtheOSseemstomisssignificantincreasesordecreasesinperformancetiminginourreplacementCBs.WhatdoesDARPAsuggest?A99:DARPA/CGCachievedrepeatable,highprecisionperformancemeasurementsusinguniformhardwareandhardware‐supportedmeasurements.ThismeasurementapparatusbuildsontheworkofLevinthal[1‐2],Duetal[3‐4],Weaveretal[5‐6],andZapanuksetal[7].ToassistcompetitorsinachievingtimingmeasurementsthatmorecloselytracktheDARPA/CGCmeasurementapparatus,DECREEhasbeenupdatedtoprovidehigherprecisiontimingapproximationstothecompetitorsthantheprevious10msresolutionclock.Pleaseseehttps://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/scoring‐cbs.mdDARPA/CGCperformed100testingrunsoftheScoredEvent1challengebundleusingboththeDARPAperformancetimingapparatusandthetask.clockbasedtoolreleasedinDECREE(above).Theresultsofthisexperimentshowthatforagivenchallengebinary,ExecTimeOverheadpossessesameandifferencefromDARPA'sscoringcalculationof0.5%witha95%confidenceintervalof(0.441%,0.576%).ThisexperimentwasperformedonaHaswellmicroarchitecturewith25MBofcacheanda2.3Ghzbaseclock.[1]Levinthal,D.PerformanceAnalysisGuideforIntelCorei7ProcessorandIntelXeon5500processors2009[2]Levinthal,D.CycleAccountingAnalysisonIntelCore2Processors2009[3]Du,J.;Sehrawat,N.&Zwaenepoel,W.PerformanceProfilingofVirtualMachines,SIGPLANNot.,ACM,2011,46,3‐14[4]Du,J.;Sehrawat,N.&Zwaenepoel,W.PerformanceProfilinginaVirtualizedEnvironmentProc.HotCloud2010,USENIX,2010[5]Weaver,V.M.;Terpstra,D.&Moore,S.Non‐DeterminismandOvercountonModernHardwarePerformanceCounterImplementationsInternationalSymposiumonPerformanceAnalysisofSystemsandSoftware,2013,215‐224

[6]Weaver,V.M.&McKee,S.A.CanHardwarePerformanceCountersbeTrusted?InternationalSymposiumonWorkloadCharacterization,2008,141‐150[7]Zaparanuks,D.;Jovic,M.&Hauswirth,M.AccuracyofPerformanceCounterMeasurementsIEEESymposiumonPerformanceAnalysisofSystemsandSoftware,2009,23‐32Q98:We'reconcernedthatDARPA'sperformancemeasurementsarenotrepeatable.Aretheyrepeatable?A98:Yes.Inaprocessknownascontinuousintegrationtesting,theSE1/SE2/CQEscoringprocessanditsassociatedunittestsarecontinuallyre‐testedandcomparedagainsthand‐confirmedperformancemeasurements.Thistestingcorpuswillgrowtoincludeallscoredevents.Theseunittestsincludeallcompetitorsubmissionstodate.SeeA99.Q97:PerthelimitedIPrightsdescribedinA81andtheCGCRulesisthereaconflictofinterestissuewiththeCQEtechnicalpaper;specifically,willthesetechnicalpapersbeusedtomanipulatethecompetitiontodefeatmydescribedapproach?A97:No.DARPA/CGCwascreatedwiththeintentionofbringingthegameofCapturetheFlagtoautomatedsystems.CTFcontestsasawholearedesignedtobeobjectivetestsofskill‐alevelplayingfieldinwhichcompetitorsfromaroundtheglobearetestedsolelyontheirabilitytosolvedifficultreverseengineeringproblems.DARPA/CGCisaCTFcompetitioninbothletterandspirit.Specifically:

DARPA/CGCwillnevertakeanyactiontointentionallydemoteorpromoteanycompetitororsetofcompetitors.

Constructionofthechallengehasbeenundertakeninanefforttomirrorrealworldchallenges.

PertheIPrightsspecifiedintheCGCRulesandA81,neithertheCQEnortheCFEtechnicalreportswillbesharedwithChallengeBinaryauthorperformerteams.

Q96:RegardingtheCQEtechnicalpaper,Iassumeyouwantanoverviewofthetools,approaches,gametheoryandprogramanalysis,butnotafulldesigndocumentanddiscussionofeveryalgorithmandheuristicused.Correct?A96:Yes

Q95:WhathappenswhenDARPAconfirmsacceptanceofourCQEtechnicalpaper?A95:Persection3.1.3oftheCGCrules,CQEtechnicalpaperacceptanceisrequiredinordertoadvanceasafinalistandreceiveCQEprizesinaccordancewithsection3.1.4oftheRules.Q94:Inourtesting,someofourPoVsdon'tseemtoworkreliablyagainstsomeCBs.Whatshouldwedo?A94:DARPAhasdevelopedahigh‐reliabilityPoVharnessforscoringSE1/SE2/CQEthatmaymitigatethisissuesomewhatduringeventscoring.InCFE,PoVreliabilitywillbetheresponsibilityofcompetitors.Q93:Section4oftheCGCRulesrequestsnotificationofpotentialthreatstotheintegrityoftheCGCcompetition.WeareawareofamethodologytocauseatotalmappingofLinuxprocessmemorythatdoesnotcauseLinuxtoreportthememoryasallocatedorconsumed.WillthisthreatentheintegrityoftheCGCcompetition?<METHODOLOGYREDACTED>A93:No.Pleasenotethefollowing:1>DECREEisnotLinux2>DECREEisaspecification3>AreferenceimplementationofDECREEatopLinuxhasbeenreleasedongithub4>DECREEanditsassociatedreferenceimplementationsdonotsupportthereportedshenanigans5>DARPAappreciatescompetitorsupportinconductingahighintegritycompetitionQ92:ThereferencesecurityscoresforreplacementbinariesforCBe7cd3901inScoredEvent1appearedtobeincorrect.CanDARPAconfirmthosescores?A92:AnerrorintheSE1scoringsystemincorrectlyscoredthereferencePoVfore7cd3901asfailingtoprovevulnerability.ThisresultedininflatedreferencesecurityscoresforseveralreplacementCBs.Correctedscoreshavenowbeenpostedatcgc.darpa.mil.DARPAwouldliketothanktheFuzzBombTeamforreportingandhelpingdiagnosethisissue.TheSE2scoringsystemwillincorporatealllessonslearnedfromSE1.CompetitorparticipationandfeedbackcontinuestobeinvaluableontheroadtoCQE.Q91:Willthetest_eventbucketsbemaintainedforteamtestingduringtheevents?A91:Yes,thetest_eventbucketswillexistbeforeandduringscoredevents.CompetitorsareadvisedthesebucketsareatemporaryspaceandfilesmaybedeletedbyDARPAforresourceconsumptionreasonsatanytime.

Q90:WilllibcgcimplementallLLVMintrinsics?A90:No.PriortoCQE,DARPAwillreleasetheversionoflibcgctobeusedduringCQE.Q89:Willthesourcecodetothenewchallengesincludedinthescoredeventsbereleased?A89:Yes,sourcecodewillbereleasedpriortothenextevent.Q88:TheSecurity:ReferencescorereleasedintheSE1scoringappearstohaveamaximumvalueof1.Whyisthis?A88:PerA59,theReferencecomponentoftheSecurityscorehasamaximumvalueof1.Q87:WillDARPAprovidethetotalscoresforthetoprankedteamsinScoredEvent1?A87:No.SeeQ/A68.Q86:ScoredEvent1includedpreviouslyreleased/examinedChallengeBinaries.HowwilltheseChallengeBinariesbetreatedinfutureCGCevents?A86:Definitions:‐ Previouslyreleased&examinedChallengeBinarieswillbereferredtoasOld

CBs‐ ChallengeBinariesreleasedforthefirsttimeduringacompetitioneventwillbe

referredtoasNewCBs.

ScoredEvent2:AmixofOldCBsandNewCBs.OnlyNewCBswillbeconsideredforrankingpurposes.

CQE:ScoringusingNewCBsonly. CFETrials:OldCBsandNewCBsmaybeused. CFE:ScoringusingNewCBsonly.

PleasenotethatCQE&CFEaretheonlyCGCeventsthatinvolveprizeauthority.Q85:Shouldteamsexpectpacketlossinthesamplepacketcapturesreleasedaspartoftheeventchallengebundle?A85:Yes.DARPAisevaluatingthepacketlossrateforScoredEvent2.

Q84:WeresomePoVsubmissionsnotscoredinScoredEvent1?A84:No.AllsubmittedPoVswerescored.InafewcasestheCGCteamhadtofixtheXMLDTDpathofsubmittedPoVsoraddmissing<xml>tags.Pleaserunthescriptfoundathttps://github.com/CyberGrandChallenge/Event‐FAQ/blob/master/cqe‐verify‐solution‐package.pytocalculatethehashesusedinthescoringrelease.Infutureevents,theCGCteamwillnotmanuallycorrectmalformedsubmissions.Q83:I’mtryingtoreproduceresultsonDECREEusingmixedcomponentsfromanoldreleaseandanewrelease.HowshouldIproceed?A83:DECREEanditsassociatedtestsuitesdonotmaintainbackwardscompatibility.CompetitorsshouldusethelatestVMandtoolsuitereleaseavailable.Q82:Arechallengebinarycreatorsallowedtouseinlineassembler?A82:TheguidelinesprovidedtoCBauthorscanbefoundathttps://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/submitting‐a‐cb.mdQ81:WhatisthedistributionoftheCQETechnicalPaperrequiredforteamstoadvancetoCFE?A81:ThisquestionisansweredindetailintheCGCRules,Section5,IntellectualProperty;technicalpapersareprivatecommunicationstoDARPA.DARPAwillobservealllimitationsdetailedinSection5.DARPAwillnotallowaccesstotechnicalpapersbyanypartynotlistedinSection5.Competitorswishingtouseemailencryptionwhensendingtechnicalpapersmayrequestasignedemailfromcybergrandchallenge@darpa.mil;theS/MIMEpublickeyassociatedwiththisemailhasthefingerprint1E48B8A5EFC682F92C867BB6D01A3455.Q80:WilltheCBssettleonastandardlibcreplacementthatwecanrelyuponforanalysis?A80:No.SeeinstructionsprovidedtoCBauthors:https://github.com/CyberGrandChallenge/cgc‐release‐documentation/blob/master/walk‐throughs/submitting‐a‐cb.mdScoringTie‐Breakercomments(AnswersRound1,Q78‐79): Q79:CouldDARPAusethetimeoflastsubmissionasasimpletiebreaker? A79:DARPAdoesnotwishtopenalizeinternationalcompetitorswhomay: resideinlocationswithincreasedtransittimetotheCQEservershaveaccess toslow,lowerbandwidthconnectionsresideinaninconvenienttimezone relativetothestartofCQE.

Q78:CouldthetiebreakeralgorithmfirstremovetheProof‐of‐ Vulnerabilityscoretoincentivizepurelydefensivesolutions? A78:TheSE1tiebreakeralgorithmincreasestheimportanceofthe Defensivescoreinthefirstfourofitsfivecomputationrounds.Q77:MayIsubmitaPoVofarbitrarysize?A77:No.PoVsarelimitedtonogreaterthan10x10242bytes.Q76:WhatconstitutesatestcasewhencalculatingtheretainedfunctionalityscoreinCQE?A76:AtestcaseisasingleservicepollwithinasingleTCPconnection.NoTCPconnectionwillcontainmorethanoneservicepoll.Individualservicepollswilleitherpassorfail.ReplacementChallengebinarieswillbetestedbyatleast1000individualservicepolls.TheseresultswillbeusedtocalculateretainedfunctionalityperA59.Q75:HowwillDARPAcommunicatewiththeteamsduringtheScoredEventsandCQE?A75:Competitorscanemailquestionstocybergrandchallenge@darpa.milduringtheevents.AneventFAQwillbeupdatedasneededongithubathttps://github.com/CyberGrandChallenge/Event‐FAQinordertodisseminateinformationtoallteamsperA70.TheeventFAQwillbeincorporatedintothisdocumentafterthecompletionofeachevent.Q74:IamaforeignnationalwhoiseligibletoparticipatepertheCGCRules.IhavecreatedaUS‐basedLLCwithaUS‐basedRegisteredAgenttoserveastheEntrantformyCGCteam;thisLLCisalsoeligibletoparticipatepertheCGCRules.IsthisapproachcompliantwiththeCGCRules?A74:Yes.Q73:WhathappenswhenaconnectionismadetoaDECREEservice?A73:inetd‐style.Anewinstanceiscreatedtohandlethenewconnection.Thisnewinstanceistorndownaftertheconnectionterminates.Q72:WhattypesofconnectionswillbemadeduringCQEscoring?A72:MultipleconnectionswillbemadefromServicePollers.MultipleconnectionswillalsobemadefromProofofVulnerabilitymodules.ServicePollsandPoVmoduleswillnevershareconnections.

Q71:WhattypesofconnectionswillbemadeduringCFEscoring?A71:Multipleconnectionswillbemadefromservicepollers.Multipleconnectionswillalsobemadefromlogicbuiltbycompetitors.Servicepollsandcompetitorlogicwillnevershareconnections.Q70:WhatotheraccesstoCyberGrandChallengeisavailabletocompetitorsoutsideofthecybergrandchallenge@darpa.milemailboxandtheFAQresponses?A70:Intheinterestsofconductingafairandequitableglobalcompetition,accesstochallengeinformationismadeavailableelectronicallytoallcompetitors.Allcompetitorswhethernextdoororacrosstheglobe,maysubmitquestionsthroughthemailbox,andresponseswillbecommunicatedthroughthisFAQ.Q69:AreCFEfinalistsrequiredtobringhardwaretocompeteinCFE?A69:No.Finalistswillhavetheoptionofeither:1. BringingacompetitionsystemtoCFEinaccordancewithA31,or2. CompetinginCFEonaDARPA‐providedcomputecloudinstanceafterhaving

acceptedtheDARPACloudAgreement.EachDARPA‐providedcomputecloudinstancewillbeontheorderofhundredsofx86‐64cores.FurtherdetailsregardingtheCloudAgreementandsystemspecificationswillbereleasedatalaterdate.Q68:WhatinformationwillbereleasedtocompetitorsafterScoredEvent#1?A68:PleasenotethatinformationreleaseafterScoredEventswillbeentirelydifferentfromthepost‐CQEinformationreleaseaddressedinA25.AfterScoredEvent#1,thefollowinginformationwillbereleasedpublicly:‐ Thenamesoftheseventop‐scoringteamsinrankorder.‐ AlistofSHA‐256hashesforsubmittedChallengeBinariesandtheir

associatedscoresandcorrespondingreferenceCBname.‐ AlistofSHA‐256hashesforPoVsandtheirassociatedscoresand

correspondingreferenceCBname.Pleasenote,thesereleasedhashlistswillnotcorrelatescoredsubmissionstoteams.CompetitorswillberequiredtocalculateSHA‐256hashesoftheirsubmittedinputsinordertodeterminetheirscores.

Q67:HowwillrankingoccurinScoredEvent#1?A67:Multiplesubmissionsmaybescored;hashlistinformationonmultiplesubmissionswillbeavailableviathehashlistformat(A68).Rankswillbedeterminedusingthescoreassignedtoeachteam’sfinalsubmission.Q66:WhatwillCQEChallengeBundlecontain?A66:AtthebeginningofCQE,competitorswillgainaccesstoCQEChallengeBundle(bundlewillcontainacollectionofReferenceCBs,aswellassomepcaprecordingsofsomeservicepollinteractionsbetweenServicePollersandtheseReferenceCBs).Theseservicepollinteractionsamples,wherepresent,arenotguaranteedtobecomplete.Q65:WhatwillScoredEventChallengeBundlescontain?A65:ScoredeventsareintendedtoprovidetechnicalpreparationforCQE;thereforetheScoredEventBundleswillmirrortheformatoftheCQEChallengeBundletothegreatestextentpossible.CompetitorsshouldnotethattheCQEBundlewillbemuchlargerthantheScoredEventBundles.TheseScoredEventBundlesmayalsore‐usepreviouslyreleasedCBs.Q64:WhatisDECREE?A64:DECREEisanopen‐sourceextensionbuiltatoptheLinuxoperatingsystem.Constructedfromthegroundupasaplatformforoperatingsmall,isolatedsoftwaretestsamplesthatareincompatiblewithanyothersoftwareintheworld—DECREEaimstoprovideasaferesearchandexperimentationenvironmentfortheCyberGrandChallenge.DECREEbinariesandsourceareavailable:http://repo.cybergrandchallenge.com/http://github.com/cybergrandchallenge/Q63:HowshouldissuesinDECREEbereported?A63:Emailcybergrandchallenge@darpa.milQ62:WillalladvancedapplicationdefensesthatpreventarbitrarycodefromrunningincreasethesecurityscoreinCQE?A62:No.CGCscoringdoesnotrequirearbitrarycodeexecution,thereforemechanismswhichfrustratearbitrarycodeexecutionwillnotnecessarilypreventscoringevents.InCQE,competitorshavetheopportunitytomitigatedenialofserviceflaws.SeealsoQ4.

Q61:WilltheReferencePatchedCBperformdifferentlythantheOriginalCB?A61:AdiversegroupofsoftwareauthorsarebuildingalargecorpusofCBsforCGCincorporatingmanyclassesofvulnerabilities.TheseCBauthorsarerequiredtoprovideasingleReferencePatchedCBthatpassesthesamefunctionalitytestsuiteastheOriginalCBandisnotsusceptibletoanyofthereferencePoVs.Q60:HowdoestheInterProcessCommunication(IPC)workinChallengeBinaries(CBs)?A60:DECREEprecludescommunicationviasharedmemory,network,orpersistentstoragebetweendifferentCBsaswellasdifferentconnectionsservicedbythesameCB.InordertoofferarichCBportfoliowithbroadCWEcoverageincludingconcurrencyissues,DARPAallowsfortheuseofaCGCIPCmechanismwithinasingleCB,whichworksasfollows.EachCBmaybecomposedofmultiplebinariesrunningindistinctprocesses.TheCGCcompetitionframeworkwilllaunchallofthebinariesassociatedwiththechallenge.Eachoftheseprocesseswillbepre‐connectedwithfiledescriptorstocommunicatewiththeothersviareceive()andtransmit()systemcalls(seeFigure1)inamannerdeterminedbytheCBauthor(seeFigure2).ExampleIPCCBswillbeprovidedpriortothefirstCQEScoredEvent.

Figure1:Filedescriptorconnections

Figure2:ExampleuseofIPCinter‐connections

Q59:WhatisthescoringmethodforCQE?A59:DARPAheldaperiodofpubliccommentfortheCFEandCQEscoringmethods,andfeedbackfromtheCGCcommunitywasreceivedandreviewedbyDARPA.UpdatedscoringmethodsforCQEandCFEwerereleasedonMonday,March10,2014.ThesescoringmethodsareconsistentwiththeCyberGrandChallengeRules.SeealsoQ/A26,27,and30.

CQEscoringistheproductofthreeassessedquantities:Availabilityscore,Securityscore,andEvaluationscore.ThesescoresmaptotheAreasofExcellence(AoE)locatedintheCyberGrandChallengeRulesasfollows:Availability:AreaofExcellence4Security:AreaofExcellence2Evaluation:AreaofExcellence3DuringCQE,theCBsdistributedbyDARPAshallbereferredtoas“referenceCBs”,nottobeconfusedwithreplacementCBssubmittedbycompetitors.PoVsusedbyDARPAforscoringwillbereferredtoas“referencePoVs”.PoVssubmittedbycompetitorsystemstoDARPAduringCQEwillbereferredtoas“submittedPoVs”.DARPAwillscoreonereplacementCBandonesubmittedPoVperreferenceChallengeBinary.CQEScoreswillbeassessedperreferenceChallengeBinary(“CBscore”).EachCBscorewillbecalculatedasfollows:

Availability*Security*Evaluation

Availability:Thisquantityshallvaryasamulti‐stepfunctionbetween0and1,with1beingaperfectscore.PerformanceandRetainedFunctionalitywillbemeasured,withAvailabilitybeingsettotheminimumofthesequantities.‐ PerformanceofasubmittedCBwilldecrementbasedonthegreatestmeasured

increaseinsystemresourceutilization.Thesemeasuredincreasesincludefilesize,executiontime,andmemoryusage.Foreachofthesemeasuredquantities,amaximumacceptableincreaseisindicatedbelow,afterwhichPerformancebeginstodecrement: Filesize:+40% Executiontime:+10% Memoryusage:+10%

‐ RetainedFunctionalitywillbethepercentageoftestcasesthereplacementCB

passes.Competitorsareadvisedthatuseofamulti‐stepfunctionimposesafaster‐than‐linearAvailabilitydropoffbasedondamagetoPerformanceorRetainedFunctionality.Security:Thisquantityshallbedeterminedusingtwoquantities,ReferenceandConsensus:‐ Reference:ThenumberofreferencePoVswhichdonotprovevulnerabilityin

thereplacementCB,dividedbythenumberofreferencePoVs

‐ Consensus:Thisquantitywillbesetto0or1:• 0:AnysubmittedPoVprovedvulnerabilityinthereplacementCB• 1:NosubmittedPoVprovedvulnerabilityinthereplacementCB

IfReferenceiszero,Securitywillbesettozero.IfReferenceisnonzero,Securitywillbecalculatedasfollows: 1+(Reference+Consensus)/2Evaluation:Thisquantitywillbesetto1or2:‐ 1:ThePoVemittedbythisCRSdidnotprovevulnerabilityinthereferenceCB‐ 2:ThePoVemittedbythisCRSdidprovevulnerabilityinthereferenceCBACRS’stotalscoreattheendofCQEshallbethesumofthatCRS’sCBscores.Q58:WhatisthescoringmethodforCFE?A58:DARPAheldaperiodofpubliccommentfortheCFEandCQEscoringmethods,andfeedbackfromtheCGCcommunitywasreceivedandreviewedbyDARPA.UpdatedscoringmethodsforCQEandCFEwerereleasedonMonday,March10,2014.ThesescoringmethodsareconsistentwiththeCyberGrandChallengeRules.SeealsoQ/A26,27,and30.CFEscoringistheproductofthreeassessedquantities:Availabilityscore,Securityscore,andEvaluationscore.ThesescoresmaptotheAreasofExcellence(AoE)locatedintheCyberGrandChallengeRulesasfollows:Availability:AreaofExcellence4Security:AoE2and5Evaluation:AreaofExcellence3CFEscoringwilltakeplaceovermanyrounds.ScoreswillbeassessedperChallengeBinaryperround(“CBroundscore”);itfollowsthatthesumoftheCBroundscoresforasingleCRSwillbethetotalscoreforthatsystem.EachCBroundscorewillbecalculatedasfollows:

Availability*Security*EvaluationAvailability:Thisquantityshallvaryasamulti‐stepfunctionbetween0and1,with1beingaperfectscore.PerformanceandRetainedFunctionalitywillbemeasured,withAvailabilitybeingsettotheminimumofthesequantities.‐ PerformanceofasubmittedCBwilldecrementbasedonthegreatestmeasured

increaseinsystemresourceutilization.Thesemeasuredincreasesincludefilesize,executiontime,andmemoryusage.Foreachofthesemeasuredquantities,amaximumacceptableincreaseisindicatedbelow,afterwhichPerformancebeginstodecrement:

Filesize:+20% Executiontime:+5% Memoryusage:+5%

‐ RetainedFunctionalitywillbethepercentageofnetworktestcasesthe

replacementCBpasses. Competitorsareadvisedthatuseofamulti‐stepfunctionimposesafaster‐than‐

linearAvailabilitydropoffbasedondamagetoPerformanceorRetainedFunctionality.

Security:Thisquantitywillbesetto1or2:‐ 1:AtleastonePoVfromacompetitorprovedvulnerabilityinthisCBduringthis

round‐ 2:NoPoVsfromcompetitorsprovedvulnerabilityinthisCBduringthisroundEvaluation:Thisquantitywillvarybetween1and2:‐ 1:NoPoVsemittedbythisCRSprovedvulnerabilitiesinthisCBserviceon

competitorsystems‐ 1+(x/(N‐1))PoVsemittedbythisCRSforthisCBserviceprovedvulnerabilityin

xcompetitorsystems,whereNisthenumberofCRSsparticipatinginCFE.ACRS’stotalscoreattheendofCFEshallbethesumofthatCRS’sCBroundscores.Q57:WhatistheAPItotheCGCenvironment?A57:ThefollowingClanguagefunctionprototypesareprovided:

void _terminate(unsigned int status); int allocate(size_t length, int prot, void **addr); int deallocate(void *addr, size_t length); int fdwait(int nfds, fd_set *readfds, fd_set *writefds,

struct timeval *timeout, int *readyfds); int random(void *buf, size_t count, size_t *rnd_bytes); int receive(int fd, void *buf, size_t count, size_t

*rx_bytes); int transmit(int fd, const void *buf, size_t count, size_t

*tx_bytes);ThesefunctionprototypesarenotionalandmaybeimprovedduetofeedbackpriortoCGCkickoff.Q56:Canforeignnationalsparticipateinthischallenge?

A56:ThisquestionisaddressedintheCGCRulesSection2andSection6.ForeignnationalsmayparticipateinCyberGrandChallengewithinateamwhichconformstotheCGCRules.Q55:DARPA‐BAA‐14‐05mentionsDARPA‐BAA‐14‐03,whichdescribesthearchitectureframework.WhereisDARPA‐BAA‐14‐03?A55:DARPAanticipatesDARPA‐BAA‐14‐03tobepublishedinthenearfuture.Q54:DoesDARPAhaveacompletegovernmentteamorarethereopportunitiesforCGCsupportindevelopment,judging,operating,etc.?A54:DARPAanticipatesasecondBAAwithotheropportunitieswithinthischallenge.Q53:Canforeignteamsapplyforthefundingalsoorcanteamshaveforeignmembers?A53:ReviewtheeligibilitysectionofDARPA‐BAA‐14‐05(3.1.4)andtheRules(2.1).Q52:Isthis6.1or6.2money?A52:DARPAanticipates6.2.fundsforawardsunderDARPA‐BAA‐14‐05andDARPA‐BAA‐14‐03.Q51:Doesfundamentalversusnon‐fundamentalaffectdesirability?A51:SeeDARPA‐BAA‐14‐05section2.2.Q50:Arethereanyrestrictionsonforeignsubcontractors?Ifso,whataretherestrictions?A50:Seesection3.1.3ofDARPA‐BAA‐14‐05.Q49:Willtheproposalevaluationsfavorsmallbusiness,orisitalevelplayingfieldbasedonmerit?A49:Seesection5ofDARPA‐BAA‐14‐05.Allproposalsareevaluatedonthesamecriteria.Q48:ArethedeliverablesandpaymentpercentagesinDARPA‐BAA‐14‐05fixed,orcanweproposealternatives?A48:Theyarenotional,notfixed.Youcanproposealternatives.

Q47:Canyouclarifythelengthoftheperiodsofperformanceforthebaseandoptionperiods?A47:UnderDARPA‐BAA‐14‐05,eachperiodofperformanceis12months.ThescheduleinDARPA‐BAA‐14‐05isnotional.Planforallactivitiestotakeplacewithintwo12monthphases.Q46:IsitpossibletocombinewithanothergroupaftertheCQE?A46:Yes.Q45:Cananorganizationhavetwoteams,oneforOpentrackandoneforProposaltrack?A45:ThisisexcludedintheRules.Teamsareintendedtobewhollyseparate.

Q44:IfIsubmitaproposaltotheCompetitionBAA(DARPA‐BAA‐14‐05)anddonotgetselected,canIsubmittotheArchitectureBAA(DARPA‐BAA‐14‐03)?A44:There’snothingtopreventyoufromsubmittingtoboth,butyoucannotbeselectedforawardunderboth.IntheeventthataproposersubmitsanotherwiseselectableproposaltobothDARPA‐BAA‐14‐05andDARPA‐BAA‐14‐03,thedecisionastowhichproposaltoconsiderforawardisatthediscretionoftheGovernment.Q43:MustwedeliveraworkingspreadsheetaspartoftheproposalforDARPA‐BAA‐14‐05oristhatjustDARPA’spreference?Yousaiditwouldbe“helpful”versus“required?”A43:Persection4.2.1.2ofDARPA‐BAA‐14‐05,thecostproposalshouldincludeaspreadsheetfile(.xlsorequivalentformat)thatprovidesformulatraceabilityamongallcomponentsofthecostproposal.Thespreadsheetfilemustbeincludedasaseparatecomponentofthefullproposalpackage.Q42:CanwetalktotheContractingOfficerbeforeaproposalissubmitted?A42:ReferenceSection7ofDARPA‐BAA‐14‐05,questionsshouldbesubmittedtoCGC‐CompetitorBAA@darpa.mil.Q41:AretheretwoBAA’santicipatedforthisprogram,theArchitectureBAA(DARPA‐BAA‐14‐03)andtheCompetitionBAA(DARPA‐BAA‐14‐05)?A41:Yes.

Q40:WhatistheeligibilityforusinganOTforprototypes(845)?A40:SeeDARPA’scontractmanagementwebsite(http://www.darpa.mil/Opportunities/Contract_Management/Other_Transactions_and_Technology_Investment_Agreements.aspx)forinformationregardingOTforPrototypeawards.Q39:IstheelectronicsubmittalsystemsimilartoT‐FIMS?A39:Yes.Q38:Couldtheamountsoftheprojectbelargerifanentitysuppliedacostsharebeyondthe$750k?A38:Yes.Q37:WithregardtoSection4.2.1.2.3ofDARPA‐BAA‐14‐05,wherearegovernmentratesandDefenseContractAuditAgency(DCAA)ratesdefined?A37:FARPart42discussesproceduresforestablishingforwardpricingrates.InformationisalsoavailableontheDefenseContractManagementAgency’s(DCMA)Websitehttp://guidebook.dcma.mil/41/.YoudonothavetohaveDCMAapprovedratestoproposeandreceiveanawardunderDARPA‐BAA‐14‐05.Section4.2.1.2.3requiresaproposertojustifyitsproposeddirectlaborratesandprovidesseveralexamplesofhowthatcanbeaccomplished.Q36:WithregardtoSection4.2.1.1.1ofDARPA‐BAA‐14‐05,wherearethetypesofbusinessesdescribed?A36:BusinesssizesaredefinedbytheSmallBusinessAdministration(http://www.sba.gov/content/table‐small‐business‐size‐standards).AdefinitionofHBCUandMinorityInstitutionscanbefoundinDFARS252.226‐7000(http://www.acq.osd.mil/dpap/dars/dfars/html/current/252226.htm#252.226‐7000).Q35:Istherealimittothenumberofteamsawardedortotalamountofgrants?A35:NograntswillbeawardedunderDARPA‐BAA‐14‐05,onlyFirm‐Fixed‐PriceProcurementContractsandOtherTransactions.UnderDARPA‐BAA‐14‐05,DARPAanticipatesmultipleawardsof$750,000perphaseofatwo‐phaseeffort;however,pertheBAA,thenumber/amountofawardswilldependonthequalityoftheproposalsreceivedandtheavailabilityoffunds.

Q34:Willacceptedproposalsbecomepublic?A34.DARPAwillnotpublishawardedproposalsunderDARPA‐BAA‐14‐05.Persection4.2.2oftheBAA,DARPAtreatsproposalsassourceselectioninformation(seeFAR2.101and3.104)andprotectsthemassuch,usingsecurehandlinganddestructionprocedures.Q33:DuringCFE,howwillaCRSmonitorandmodifytraffictoanetworkedhost?A33:Monitor:DuringCFE,eachcompetitorCRSwillreceivearead‐onlystreamofallCompetitorCRSnetworktrafficdirectedtowarditsnetworkhostovertheCFEnetwork.Modify:CompetitorsystemswillbeprovidedwithaccesstoaDARPA‐managednetworkappliancewithinthecompetitionframeworkwhichwillallowfortrafficmodificationbetweentheCFEnetworkandthenetworkhostdefendedbytheCRS.ThemanagedappliancewillusefiltersprovidedbytheCRSthroughtheCompetitionFrameworkAPI.Filtersneednotmodifytraffic;dependingontheirformulationtheymaymodifytraffic,alert,ortakenoaction.LikeChallengeBinaries,filtersprovidedbyaCRSwillbedistributedtoallcompetitorsystemsforpurposesofconsensusevaluation(Shannon'sMaxim).AlertsgeneratedonthemanagedappliancebyfieldedfilterswillbecommunicatedbacktotheCRSthroughtheCompetitionFrameworkAPI.TheseCRS‐providedfilterswillconformtoasubsetoftheopensourceSnortrulesyntax;thissubsetwillbepublishedbyDARPApriortokickoffofCyberGrandChallenge.Q32:HowwillCRShardwareinterfacewiththeCQEevent?A32:CQEisanInternet‐accessibleeventthatwillnotrequireaCRStobephysicallypresentataneventlocation.Section3.1oftheCGCRulesprovidesinformationontheCQECorpusandworkflow.Competitorsshouldnotetherequirementsofthepost‐CQESiteVisitfromtheCGCRules,Section3.1.3.2.Q31:HowwillCRShardwareinterfacewiththecompetitionframeworkduringCFE?A31:ACRSusedinCFEmustfitentirelyinasingle,standard19"42Urack.CFEwilltakeplaceataphysicallocationtobespecifiedbyDARPA.CompetitorsystemsmustbephysicallypresentattheCFEComputeLocationinordertointerfacedirectlywiththecompetitionframework.Furtherdetailsaboutavailablepower,cooling,andinfrastructurewillbepublishedasCyberGrandChallengeprogresses.

Q30:WhenwillCQEandCFEscoringbefinalized?A30:DARPAiscurrentlyholdingaperiodofpubliccommentfortheCFEandCQEscoringmethods.Feedbackshouldbeaddressedtocybergrandchallenge@darpa.mil.ScoringwillbeopenforaperiodofpubliccommentconcludingonFriday,February14,2014.AfinalscoringalgorithmforCQEandCFEwillbereleasedonMonday,March10,2014.Likethecurrentproposedscoringalgorithms,thefinalscoringmethodswillbeconsistentwiththeCyberGrandChallengeRules.SeealsoQ/A26and27.Q29:WhatconstitutesaProofofVulnerabilityinCFE?A29:ProofofVulnerability(PoV)maybedemonstratedduringCFEbyinteractingwiththeCompetitionFrameworkviatwodifferentmethods.Method1:RegistersToprovethataremoteCBisvulnerable,aMethod1PoVshallredirectexecutiontoaninstructionpointervaluenegotiatedwiththeCompetitionFrameworkAPI.Executionmustfaultatthisvalue.Attimeoffault,onegeneralpurposeregistermustalsobesettoasecondvaluenegotiatedwiththeCompetitionFrameworkAPI.Thisredirectionwillcreateaneffectdetectablebyinstrumentationoperatedonthecompetitionframework.Method2:MemoryToprovethataremoteCBisvulnerable,aMethod2PoVwillretrieveavaluepresentinthememoryofacompetitorCB.ThisvaluemustberetrievedfromamemoryrangespecifiedbytheCompetitionFrameworkAPI,mustbeofasizespecifiedbytheCompetitionFrameworkAPI,andmustbesubmittedthroughtheCompetitionFrameworkAPI.Thissubmittalwillcreateaneffectdetectablebyinstrumentationoperatedonthecompetitionframework.Q28:WhatconstitutesaProofofVulnerabilityinCQE?A28:ACQEProofofVulnerabilitywillcauseaCBtocrashinamannerconsistentwithmemorycorruptionflawswithsecurityimpact,e.g.,asegmentationfaultorillegalinstructionfault.

Q27:WhatisthescoringmethodforCQE?A27:ThebelowscoringmethodcandidatewillbeopenforpubliccommentuntilFriday,February14,2014.CQEscoringistheproductofthreeassessedquantities:Availabilityscore,SecurityScore,andEvaluationscore.ThesescoresmaptotheAreasofExcellence(AoE)locatedintheCyberGrandChallengeRulesasfollows:Availability:AreaofExcellence4Security:AreaofExcellence2Evaluation:AreaofExcellence3DuringCQE,theCBsdistributedbyDARPAshallbereferredtoas“referenceCBs”,nottobeconfusedwithreplacementCBssubmittedbycompetitors.PoVsusedbyDARPAforscoringwillbereferredtoas“referencePoVs”.PoVssubmittedbycompetitorsystemstoDARPAduringCQEwillbereferredtoas“submittedPoVs”.CQEScoreswillbeassessedperChallengeBinary(“CBscore”).EachCBscorewillbecalculatedasfollows:

Availability*Security*EvaluationAvailability:Thisquantityshallvaryasamulti‐stepfunctionbetween0and1,with1beingaperfectscore.Performanceandretainedfunctionalitywillbemeasured,withAvailabilitybeingsettotheminimumofthesequantities.CompetitorsareadvisedthatslowingdownthefunctionofareplacementCBwillresultinafaster‐than‐linearAvailabilityscoredropoff.Security:Thisquantitywillbecalculatedasfollows:1+(Reference+Consensus)/2‐ Reference:ThenumberofreferencePoVswhichdonotprovevulnerabilityin

thereplacementCB,dividedbythenumberofreferencePoVs‐ Consensus:

Thisquantitywillbesetto0or1:o 0:AnysubmittedPoVprovedvulnerabilityinthereplacementCBo 1:NosubmittedPoVprovedvulnerabilityinthereplacementCB

Evaluation:Thisquantitywillbesetto1or2:‐ 1:APoVemittedbythisCRSdidnotprovevulnerabilityinthereferenceCB‐ 2:APoVemittedbythisCRSdidprovevulnerabilityinthereferenceCBACRS’stotalscoreattheendofCQEshallbethesumofthatCRS’sCBscores.

Q26:WhatisthescoringmethodforCFE?ThisscoringmethodcandidatewillbeopenforpubliccommentuntilFriday,February14,2014.A26:CFEscoringistheproductofthreeassessedquantities:Availabilityscore,SecurityScore,andEvaluationscore.ThesescoresmaptotheAreasofExcellence(AoE)locatedintheCyberGrandChallengeRulesasfollows:Availability:AreaofExcellence4Security:AoE2,5Evaluation:AreaofExcellence3CFEscoringwilltakeplaceovermanyrounds.ScoreswillbeassessedperChallengeBinaryperround(“CBroundscore”);itfollowsthatthesumoftheCBroundscoresforasingleCRSwillbethetotalscoreforthatsystem.EachCBroundscorewillbecalculatedasfollows:

Availability*Security*EvaluationAvailability:Thisquantityshallvaryasamulti‐stepfunctionbetween0and1,with1beingaperfectscore.Performanceandretainedfunctionalitywillbemeasured,withAvailabilitybeingsettotheminimumofthesequantities.CompetitorsareadvisedthatslowingdownthefunctionofareplacementCBwillresultinafaster‐than‐linearAvailabilityscoredropoff.Security:Thisquantitywillbesetto1or2:‐ 1:AtleastonePoVfromacompetitorprovedvulnerabilityinthisCBduringthis

round‐ 2:NoPoVsfromcompetitorsprovedvulnerabilityinthisCBduringthisroundEvaluation:Thisquantitywillvarybetween1and2:‐ 1:NoPoVsemittedbythisCRSprovedvulnerabilitiesinthisCBserviceon

competitorsystems‐ 1+(x/(N‐1))PoVsemittedbythisCRSforthisCBserviceprovedvulnerabilityin

xcompetitorsystems,whereNisthenumberofCRSsparticipatinginCFE.ACRS’stotalscoreattheendofCFEshallbethesumofthatCRS’sCBroundscores.Q25:WhatwillbepubliclyreleasedPost‐CQE?A25:DARPAintendstoreleasethefollowingitemspost‐CQE:

‐ ReferenceCBs(initialCorpusdistributedforCQE)‐ PoVs,includingbothreferencePoVsandPoVsgatheredduringtheCQE

‐ ReplacementCBsfromtheCQE,includingreferencepatchedCBs‐ PCAPoftrafficusedduringCQEevaluation‐ ReferenceservicepollersforeachCB‐ ReferenceCBsourcecode‐ AdetailedlistofscoresforeachCBforeachfinalist‐ Teamrankings(includingOpenTrackandProposalTrack)

DARPAmaymodifythislistofintendeddeliverablesatitssolediscretion.Q24:Whatinformationaboutchallengebinarieswillbeprovidedaheadoftime(e.g.,sampleinputandresponse;interactionprotocol,APIforservice,etc.)?A24:DARPAwillprovideaninterfacedocumentdetailingthemethodsCBswillusetointerfacewiththeirexecutionenvironment.Q23:Whatwillweknowaboutchallengenetworkconfiguration(e.g.,addressranges)beforethefinalevent?A23:TheCFEnetworktopologywillbeknownpriortoCFE.Inaddition,competitorswillhavetheopportunitytotesttechnologyinteroperabilityduringCFETrials.Q22:Willtheexecutionenvironmentbeprovidedtotheteams?A22:Asampleenvironmentwillbeprovidedpriortotheprogramcommencing(proposaltrackawardshavebeenfinalizedandopentrackteamshavebeenregistered/accepted)intheformofavirtualmachine.Q21:WillsampleinputsbeprovidedwithsomeofthechallengebinariesintheCQEcorpus?A21:Yes.Q20:CansecurereplacementCBsbesubmittedbyaCRSthroughoutCFE?A20:Yes.Q19:WhatistheimpactofsubmittingareplacementCB?A19:ThesubmissionofsecurereplacementsmayberatelimitedbytheCompetitionFrameworkAPI,andfieldingareplacementCBmayimpactserviceavailability.

Q18:Aretherenetworkingconstraintsonpatching?Reachingouttoremoteservers?MayCBscommunicatewiththeCRSwhileexecutingonthenetworkhost?A18:DuringCFE,ChallengeBinarieswillnothavetheabilitytoinitiateconnections.Q17:DuringCFE,fornetworkdefense,willexistingtoolsforscanninganddefending(TCP/UDP/NMAP,wireshark,snort,etc.)work,ormustwedevelopnewtools?Doyouexpecttheteamstodevelopprogramanalysistoolsthemselvesoruseoff‐the‐shelfones?A17:DARPAwillnotdictatewhatautomatedapproachesareacceptablewithinaCRS.Q16:DuringCFE,whatinformation(datasources)willourCRShaveaccessto?SpecificallywillourCRShaveaccesstocrashlogs,coredumps,andfullnetworktrafficfeed?A16:DuringCFE,aCRSwillhaveaccesstoareadonlynetworktap.DuringCFE,aCRSwillhavetheabilitytorequestsomeCBstatusinformationthroughtheCompetitionFrameworkAPI.DatasourcesautomaticallygeneratedbyaCRSinternallywillnotbedictatedbyDARPA.Q15:DuringCFE,howmanynetworkedhostswillcompetitorsberesponsibleformonitoring/protecting?A15:One.Q14:DuringCFE,willcompetitorshaveaccesstothenetworkhost?A14:ACRSwillhavetheabilitytoquerytheCompetitionFrameworkAPIforsomeCBstatusinformation.ACRSwillhavetheabilitytofieldreplacementCBsthroughtheCompetitionFrameworkAPI.Q13:DuringCFE,willyoubeissuingnewbinariestoteamsaftercompetitionstart,orwillyougiveallbinariestoteamsbeforestart?A13:DuringCFE,aCRSwillbenotifiedthataCBisavailablethroughtheCompetitionFrameworkAPI.Q12:WhatprogramminglanguageswillCBsbewrittenin?A12:TheCfamilyoflanguages.

Q11:DoestheU.S.GovernmentassertanyintellectualpropertyrightstoCRSsourcecodedevelopedbyopentrackcompetitors?A11:No.Q10:WhattypeofsecurityvulnerabilitieswillCGCaddress?A10:CGCChallengeBinariesshallcontaintraditionalmemorycorruptionflaws.AsubsetofrelevantflawtypesdrawnfromtheMITRECommonWeaknessEnumerationentriesasfoundonhttp://cwe.mitre.org/follows;teamsareencouragedtomakeuseofthislistasastartingpoint,notareference.CWE‐120:BufferCopywithoutCheckingSizeofInput('ClassicBufferOverflow')CWE‐121:Stack‐basedBufferOverflowCWE‐122:Heap‐basedBufferOverflowCWE‐123:Write‐what‐whereConditionCWE‐124:BufferUnderwrite('BufferUnderflow')CWE‐128:Wrap‐aroundErrorCWE‐129:ImproperValidationofArrayIndexCWE‐130:ImproperHandlingofLengthParameterInconsistencyCWE‐131:IncorrectCalculationofBufferSizeCWE‐134:UncontrolledFormatStringCWE‐135:IncorrectCalculationofMulti‐ByteStringLengthCWE‐147:ImproperNeutralizationofInputTerminatorsCWE‐158:ImproperNeutralizationofNullByteorNULCharacterCWE‐170:ImproperNullTerminationCWE‐190:IntegerOverfloworWraparoundCWE‐191:IntegerUnderflow(WraporWraparound)CWE‐193:Off‐by‐oneErrorCWE‐194:UnexpectedSignExtensionCWE‐195:SignedtoUnsignedConversionErrorCWE‐196:UnsignedtoSignedConversionErrorCWE‐401:ImproperReleaseofMemoryBeforeRemovingLastReferenceCWE‐409:ImproperHandlingofHighlyCompressedData(DataAmplification)CWE‐415:DoubleFreeCWE‐416:UseAfterFreeCWE‐457:UseofUninitializedVariableCWE‐466:ReturnofpointervalueoutsideofexpectedrangeCWE‐467:Useofsizeof()onaPointerTypeCWE‐468:IncorrectPointerScalingCWE‐469:UseofPointerSubtractiontoDetermineSizeCWE‐763:ReleaseofInvalidPointerorReferenceCWE‐786:AccessofMemoryLocationBeforeStartofBufferCWE‐787:Out‐of‐boundsWriteCWE‐788:AccessofMemoryLocationAfterEndofBufferCWE‐805:BufferAccesswithIncorrectLengthValue

CWE‐806:BufferAccessUsingSizeofSourceBufferCWE‐822:UntrustedPointerDereferenceCWE‐823:UseofOut‐of‐rangePointerOffsetCWE‐824:AccessofUninitializedPointerCWE‐825:ExpiredPointerDereferenceQ9:WhatconstitutesasoftwareflawinCyberGrandChallenge?A9:DARPACGCwillnotprovideaformaldefinitionofasoftwareflaw;thisquestionliesoutsidethescopeofthechallenge.TheCGCwilloperateinthetraditionofexistingcybercompetitions:aflawisprovenwhenaninputdeliveredfromthenetworktoaflawedsoftwareprogram(CB)createsaneffectdetectablebyinstrumentationoperatedbythecompetitionframework.CGCChallengeBinarieswillcontainmemorycorruptionflawsrepresentativeofflawscategorizedbytheMITRECWE1,however,CompetitorSystemsmayproveanysoftwareflawtheydiscoverthroughautomatedreasoning.AlistofrepresentativeCWEcategorieswillbereleasedpriortothekickoffofCyberGrandChallenge.Q8:WhatplatformwillCGCrunon?A8:CGCChallengeBinaries(CBs)willbeincompatiblewithanyknownOSarchitecture.CBswillruninanenvironmentcustombuiltforthecompetition.Knowledgeoftheoperatingsystemwillnotbeinscopeforthecompetition;rather,CGCrequiresacompetitionsystemtoreasonaboutthefunctionofcompiledbinariesreceivinginputsfromthenetwork.CBswillnotconformtoanycurrentlyknownapplicationlayerprotocols.CBprotocolknowledgemustbegeneratedautomaticallybycompetitionsystemsduringCGCeventsthroughaprocessofautomatedreasoningaboutsoftware.TheseconstraintswillensurethatallknowledgeinusebycompetitionsystemsduringCGCeventsisgeneratedviaautomaticprocesses.Q7:WhatCPUarchitecturewillCGCrunon?A7:Forthepurposeofmaximizingaccessibilityandparticipation:Intelx86,32‐bit.Q6:Whatcompilerwillbeusedtobuildthebinaries?A6:CGCwilldistributeareferencecompilertoolchainpriortochallengekickoff.However,challengebinariesmaybeproducedbyanycompilerincludingthereferencecompiler.

1http://cwe.mitre.org/

Q5:Duringthefinalevent,whathappenswhenmyCompetitionSystemfieldsanewChallengeBinary?A5:DuringCFE,inordertoenactdefenses,aCRSmaychoosetoreplaceaCBwithanewlysecuredversion.TofieldareplacementCB,aCRSmustsubmitthereplacementthroughanautomatedAPIoperatedbythecompetitionframework.ThecompetitionframeworkwilldeploythereplacementbinaryonbehalfoftheCRStoitsnetworkedhost.Additionally,thecompetitionframeworkwillmakeacopyofthereplacementCBavailabletoallcompetitorsystemsforthepurposesofconsensusevaluation(Shannon’sMaxim).Oncedeployed,replacementCBswillberequiredtofunctionasself‐containedreplacementswithoutcustomdependencies,libraries,etc.Q4:I'minterestedinadvancedapplicationdefenses.WillthesebepartofCGC?A4:DuringCFE,systemswillhavetheabilitytodeploynetworkdefensesaswellasapplicationdefenses.Todeployapplicationdefenses,competitionsystemsmayanalyzeCBsandfieldsecurereplacements.DuetothecompetitivenatureofCGC,DARPAexpectsthatcompetitorswillfieldmanyapproachesofvaryingtype,advancement,andefficacy.Q3:WhatlimitationsareimposedonreplacementCBsduringCFE?A3:DuringCFE,thecompetitionframeworkwillmonitortheavailabilityandcorrectfunctionofeachCB.IfaCRSdeploysreplacementCBsthatdegradeCBfunctionbyimpactingperformance,correctnessofCBresponses,ortheabilitytoservicenetworkrequests,anegativeimpactonscoringisexpected.SimilarconstraintswillbeimposedonreplacementCBsduringCQEscoring.Q2:IntheCGCRules,AreaofExcellence2specifiesAutonomousPatching.DoesthismeanaCyberReasoningSystem(CRS)isrequiredtoisolateandremoveflaws,ormayaCRSfieldanysecurereplacementChallengeBinary(CB)?A2:DuringtheCGCQualificationEvent(CQE)andFinalEvent(CFE),CBswillbeevaluatedbasedonavailability,correctfunction,andthemitigationofflaws,asdescribedintheCGCRulesandthisFAQ.NospecificrequirementsareimposedontheformulationmethodforsecurereplacementCBs.Q1:AreyouplanninganIndustryDayforcompetitors?A1:TwoCompetitorDaysessionsareplanned,oneontheEastCoast,andoneontheWestCoast.

‐TheEastCoastCompetitorDaysarecurrentlyscheduledforDecember3and4,2013attheDARPAConferenceCenter,675NorthRandolphStreet,

Arlington,VA22203.Note:theseconddaywillbearepeatofthefirstdaytoaccommodateregisteredattendees.Availabilityisonafirst‐come‐first‐servedbasis.AllregistrationswillbefortheDecember3sessionuntilcapacityisreached;atthatpoint,registrationswillbefortheDecember4session.Pleasevisithttp://www.sa‐meetings.com/darpacgccompetitordayformoreinformationandtoregister.‐TheWestCoastCompetitorDayiscurrentlyscheduledforDecember9,2013attheWestinSt.Francis,335PowellSt,SanFrancisco,CA.Availabilityisonafirst‐come‐first‐servedbasis.Pleasevisithttp://www.sa‐meetings.com/darpacgccompetitordaywestformoreinformationandtoregister.