Dominion Assure 1.3 EAC Modification Certification Test Plan · Page 1 of 28 Template Rev 05-04 Dominion Assure 1.3 EAC Modification Certification Test Plan DVS-CPVS11DOMINI003-CTP-01

ECO Certification Test Plan Report No. DVS-CPVS11DOMINI003-CTP-03

Page 1 of 28 Template Rev 05-04

Dominion Assure 1.3 EAC Modification Certification Test Plan

DVS-CPVS11DOMINI003-CTP-01

Prepared for:

Vendor Name Dominion Voting Systems

Vendor System Assure 1.3

EAC Application No. DVS1101

Vendor Address 1201 18th Street, Suite 210 Denver, CO 80202

Prepared by:

216 16th St. Suite 700

Denver, CO 80202 303-575-6881

www.SLIGlobalsolutions.com

Accredited by the Election Assistance Commission (EAC)

for Selected Voting System Test Methods or Services

Dominion Voting Assure 1.3 Modification

Certification Test Plan

Assure 1.3 Modification Certification Test Plan v1.2 Document No. DVS-CPST11DOMINI005-CTP-01


Man

agin

g te

chno

logy

ris

k

Copyright 2011 by SLI Global Solutions, Inc.

All products and company names are used for identification purposes only and may be trademarks of their respective owners.

Revision History Date Release Author Revision Summary Dec. 12th, 2011 1.0 M. Santos Initial Draft

Dec. 22nd 2011 1.1 M. Santos Incorporated agreed to edits from Dominion

Jan 10th, 2012 1.2 M. Santos Incorporated comments from EAC

Jan 20th, 2012 1.3 M. Santos Incoporporated additional comments from EAC

Jan 24th, 2012 1.4 M. Santos Update to attachment A proprietary level

Disclaimer The information reported herein must not be used by the client to claim product certification, approval, or endorsement by NVLAP, NIST, or any agency of the Federal Government.

Trademarks SLI is a registered trademark of SLI Global Solutions, Inc. All products and company names are used for identification purposes only and may

be trademarks of their respective owners.





Man

agin

g te

chno

logy

ris

k

TABLE OF CONTENTS 1 INTRODUCTION .................................................................................................................................... 5

1.1 CERTIFICATION TEST PLAN ATTACHMENTS .......................................................................................... 5 1.2 REFERENCES ..................................................................................................................................... 5 1.3 TERMS AND ABBREVIATIONS ................................................................................................................ 6 1.4 PROJECT OVERVIEW ........................................................................................................................... 7 1.5 PURPOSE ........................................................................................................................................... 7 1.6 SCOPE OF TESTING ............................................................................................................................ 7 1.7 TESTING RESPONSIBILITIES ................................................................................................................. 8

1.7.1 Project Schedule ...................................................................................................................... 8 1.7.2 EAC & Manufacturer Dependencies ........................................................................................ 9

1.8 TARGET OF EVALUATION DESCRIPTION .............................................................................................. 10 1.8.1 GEMS .................................................................................................................................... 10 1.8.2 AV-OSX ................................................................................................................................. 10 1.8.3 AV-TSX BallotStation ............................................................................................................. 10 1.8.4 AV-OS PC .............................................................................................................................. 11 1.8.5 ABasic Script for Vermont...................................................................................................... 11 1.8.6 Documentation Updates ........................................................................................................ 11

1.9 CHANGE CONTROL AND CONFIGURATION MANAGEMENT ..................................................................... 12

2 PHYSICAL CONFIGURATION AUDIT ............................................................................................... 13

2.1 TDP DOCUMENTATION ANALYSIS & REVIEW ...................................................................................... 13 2.2 SOURCE CODE REVIEW & ANALYSIS .................................................................................................. 14 2.3 TRUSTED BUILD ................................................................................................................................ 15

3 MATERIALS REQUIRED FOR TESTING ........................................................................................... 16

3.1 SOFTWARE/FIRMWARE ...................................................................................................................... 16 3.1.1 Manufacturer Software/Firmware .......................................................................................... 16 3.1.2 Additional Supporting Test Software ..................................................................................... 16

3.2 EQUIPMENT ...................................................................................................................................... 16 3.2.1 Manufacturer Equipment ....................................................................................................... 16 3.2.2 COTS Equipment ................................................................................................................... 17 3.2.3 Additional Supporting Test Equipment .................................................................................. 17

3.3 TEST MATERIALS .............................................................................................................................. 17

4 TEST SPECIFICATIONS ..................................................................................................................... 18

4.1 REQUIREMENTS ................................................................................................................................ 18 4.2 HARDWARE CONFIGURATION AND DESIGN ......................................................................................... 18

4.2.1 Description of Configuration .................................................................................................. 18 4.2.2 System Review ...................................................................................................................... 18

4.3 ANALYSIS OF THE MODIFICATIONS FOR TESTING ................................................................................ 19 4.3.1 GEMS .................................................................................................................................... 19 4.3.2 AV-OSX ................................................................................................................................. 20 4.3.3 AV-TSX BallotStation ............................................................................................................. 20 4.3.4 AV-OS PC .............................................................................................................................. 20 4.3.5 ABasic Script for Vermont...................................................................................................... 20 4.3.6 Documentation ....................................................................................................................... 21

4.4 TESTING STRATEGIES ....................................................................................................................... 21 4.4.1 Modification Validation Testing .............................................................................................. 21 4.4.2 Integration Testing ................................................................................................................. 23

5 TEST DATA ......................................................................................................................................... 24





Man

agin

g te

chno

logy

ris

k

5.1 DATA RECORDING ............................................................................................................................ 24 5.2 TEST DATA REDUCTION .................................................................................................................... 24 5.3 TEST VALIDATION ............................................................................................................................. 24

6 TEST PROCEDURE AND CONDITIONS ........................................................................................... 25

6.1 FACILITY REQUIREMENTS .................................................................................................................. 25 6.2 TEST SETUP ..................................................................................................................................... 25 6.3 TEST SEQUENCE .............................................................................................................................. 26 6.4 ERROR RECOVERY ........................................................................................................................... 26

7 TEST OPERATIONS PROCEDURES ................................................................................................. 27

8 APPROVAL SIGNATURES ................................................................................................................. 28





Man

agin

g te

chno

logy

ris

k

1 INTRODUCTION

This Certification Test Plan outlines the test approach SLI Global Solutions will follow when performing Certification Testing on the Dominion Assure 1.3 EAC Modification project against the 2005 Voluntary Voting System Guidelines (VVSG). Per section 4.4.2.3 of the EAC Testing and Certification Program Manual, all testing on the modifications to the system will be tested to the 2005 VVSG; however, pending successful completion of this test campaign, the system will only be granted a 2002 VSS certification since the system, as a whole, will not be tested to the 2005 VVSG. The purpose of this document is to provide a clear understanding of the work SLI will conduct and a detailed plan outlining the test effort.

This campaign will test an upgrade by Dominion Voting Systems to their Assure 1.2 voting system, including:

GEMS software application

OSX firmware

Ballot Station firmware

AV-OS PC firmware

ABasic script for the State of Vermont

Accompanying documentation updates

When the testing is complete, SLI will submit a Modification Certification Test Report that details all test results and findings from the test effort, as well as a recommendation to the EAC.

1.1 Certification Test Plan Attachments The following attachments apply to this Certification Test Plan:

1. Attachment A – Documentation and Functional Discrepancy Report – May contain proprietary information (future submission)

1.2 References The following is a listing of all documents that contain material used in preparation of this test plan. 1. Election Assistance Commission VVSG, 2005 Version 1.0. Volumes I and II. 2. NIST NVLAP Handbook 150: 2006. 3. NIST NVLAP Handbook and 150-22: 2008.





Man

agin

g te

chno

logy

ris

k

4. EAC Testing and Certification Program Manual, United States Election Assistance Commission, 2007

5. SLI Quality System Manual, Revision v1.14, prepared by SLI, dated November 28th, 2011

6. EAC Notice of Clarification 09-001 Clarification of the Requirements for Voting System Test Laboratories (VSTLs) Development and Submission of Test Plans

1.3 Terms and Abbreviations The following terms and abbreviations will be used throughout this document:

Table 1 – Terms and Abbreviations

Term Abbreviation Description AccuVote®-OS AV-OS Optical scanner utilized by the Assure 1.3 voting

system AccuVote®-OSX AV-OSX Optical scanner utilized by the Assure 1.3 voting

system AccuVote®-TS R6 AV-TS R6 DRE device utilized by the Assure 1.3 voting systemAccuVote®-TSX AV-TSX DRE device utilized by the Assure 1.3 voting systemDirect Recording Electronic

DRE Touch screen device used by a voting system that allows voters to cast a ballot electronically

Equipment Change Order

ECO This signifies a minor update to the voting systems hardware inventory.

Precinct Count Scanner

PCS A precinct-count optical scanner is a mark sense-based ballot and vote counting device located at a precinct and is typically operated by scanning one ballot at a time.

Protective System Counter

PSC Counts the total number of ballots ever cast on the machine, as opposed to the ballots cast for a specific election

Standard Lab Procedure

SLP A SLI formally documented methodology for performing a given procedure in a test campaign

Visually Impaired BallotStation

VIBS A system of headphones and ergonomically designed keypad that allows those with limited or no vision to listen as a recorded ballot is narrated, then make choice selections on the keypad rather than the touch screen itself.

Voting System Test Lab

VSTL Test laboratory accredited by the National Voluntary Laboratory Accreditation Program (NVLAP) and the US Election Assistance Commission to test voting systems.





Man

agin

g te

chno

logy

ris

k

Term Abbreviation Description Voting Test Specialist VTS An SLI employee within the Compliance division

that has been qualified to perform EAC voting system certification testing.

1.4 Project Overview This test plan outlines the approach SLI will implement in certification testing of the Dominion Assure 1.3 EAC Modification voting system project against the pertinent requirements of the federal 2005 VVSG, Volumes 1 and 2. This Test Plan is for a Modification certification effort that Dominion has initiated in order to apply updates to the GEMS software application, OSX firmware, Ballot Station firmware, AV-OS PC firmware, and an ABasic report script for the State of Vermont, as well as accompanying documentation updates. Functional testing will be implemented in order to determine the modified system’s fitness.

1.5 Purpose The purpose of this EAC Test Plan is to create clear and precise documentation of the processes that SLI, as an EAC accredited VSTL, will use throughout the course of modification conformance testing.

This Test Plan:

Summarizes the modifications and scope of the changes

Analyzes the modifications’ impact

Defines the overall test approach

Identifies required voting system hardware and software to be tested, and in what configurations

Defines the types of tests to be performed

Defines the process for recording and reporting test results

1.6 Scope of Testing SLI will provide testing on the Dominion Assure 1.3 EAC Modification project based on the guidelines established for Modification verification testing as defined by the EAC’s Voting System Testing and Certification Program Manual v 1.0. This effort will include all required levels of software, firmware, and system testing required to demonstrate that the Dominion Assure 1.3 modifications are adequately validated against the requirements of the 2005 VVSG and Voting System Testing and Certification Program Manual v 1.0.





Man

agin

g te

chno

logy

ris

k

SLI’s major task categories for Modification verification testing, as defined by SLI’s National Voluntary Lab Accreditation Program (NVLAP) audited and approved Quality System Manual, the Voting System Testing and Certification Program Manual v 1.0, and the 2005 VVSG, include:

Physical Configuration Audit (PCA)

o Documentation review and assessment. o Software and Hardware Configuration Audit. Verification of hardware

physical and functional configurations.

Functional Configuration Audit (FCA)

o Functional testing of the associated voting system components will be performed to validate functionality and system level integration. This testing includes formal test execution, utilizing each component associated to the modifications, as outlined in this test plan.

Management of Dominion Assure 1.3 supplied deliverables, SLI’s test artifacts, and software, firmware, hardware and system test configurations.

Generation of detailed and repeatable test modules based on SLI’s formal Test Methods, to ensure the voting system is tested against all applicable requirements of the 2005 VVSG and associated Dominion Assure 1.3 modification specific requirements in a manner consistent with SLI’s Test Methods.

Reporting of all test results.

SLI will develop and submit to the EAC a final test report for the Dominion Assure 1.3 modifications.

1.7 Testing Responsibilities The following schedule describes the high level tasks and assigned personnel titles that will be involved in the Certification Test effort of the Dominion Assure 1.3 EAC Modification project.

1.7.1 Project Schedule

The project schedule consists of the components listed in the following subsections.

1.7.1.1 Owner Assignments

Test Module Development and Validation will be conducted by Voting Test Specialists (VTS), with oversight provided by the Test Manager.





Man

agin

g te

chno

logy

ris

k

Test Suite Development and Validation will be conducted by Voting Test Specialists, with oversight provided by the Test Manager.

Formal Test Execution will be conducted by Voting Test Specialists, with oversight provided by the Test Manager.

1.7.1.2 Test Module Development and Validation

Test Modules will be developed to provide repeatable detailed test steps. The modules are defined at a basic level in SLI’s formal Test Methods and each module is designed to be usable in any suite that employs its functionality. This re-usability reduces the development time associated with creating test suites. The modules will be validated prior to Formal Test Execution to ensure accurate testing of the modifications. Under the VVSG, this portion of the certification is considered to be within the Functional Configuration Audit (FCA).

1.7.1.3 Test Suite Development

A test suite will be developed to test the key areas of the voting system that are impacted by the included modifications. The test suite will contain multiple test modules providing clear and traceable information. Under the VVSG, this portion of the certification is considered to be within the FCA.

1.7.1.4 Formal Test Execution

Official Test Execution of the validated test suites and modules will be conducted against the modified voting system components to verify the updated component’s compliance with the VVSG requirements. Under the VVSG, this portion of the certification is considered to be within the FCA.

1.7.2 EAC & Manufacturer Dependencies

The Test Plan will require EAC approval prior to finalization.

Dominion will be required to provide all documentation, equipment and supporting materials identified as part of the modification.

In addition, Dominion is required to provide training on the voting system and support throughout the life of the project.

Dominion will be required to provide the test election definition for use in the modification test. The election definition requirements only pertain to the need for any type of election that supports use of the AV-OS, AV-OSX, AV-TSX and AV-TS R6. The election definition will consist of any size election that has at least one split precinct, as well as a precinct with more than one AV-OS device assigned to it.





Man

agin

g te

chno

logy

ris

k

1.8 Target of Evaluation Description This Modification Certification pertains to updates made by Dominion Voting Systems to their Assure 1.3 voting system, specifically updates to the GEMS software application, OSX firmware, AV-TS (X and R6) Ballot Station firmware, AV-OS PC firmware, and an ABasic script for the State of Vermont.

1.8.1 GEMS

GEMS was updated to account for the following:

The Cards Cast report was updated to resolve an issue where the total number of registered voters was not accurately reported for split precincts.

An occasional poster sharing violation was leaving the application in a state that did not accurately reflect the status of the memory card information being uploaded. The system accumulates the data in a two step process where the data is uploaded from the memory card and then posted to the database. If the upload was successful then the green arrow was displayed, even if the posting failed.

An installation program was updated with a new splash screen and agreements page to reflect Dominion Voting Systems instead of Premier Election Solutions.

A runtime error is addressed by a documentation update implemented to prevent the runtime error from occurring. The error was caused by permissions for a folder not having inheritable right propagated to its sub-folders.

1.8.2 AV-OSX

AV-OSX was updated to account for the following:

The Protective System Counter (PSC), which counts the total number of ballots ever cast on the machine, as opposed to the ballots cast for a specific election, was archiving the PSC only during graceful shutdowns. A hardware reset was reverting the PSC to its previously saved count without the current count being archived.

Date presented on results tape can be incorrect, as it was using the system date (UTC) instead of the local date

1.8.3 AV-TSX BallotStation

AV-TSX BallotStation was updated to account for the following:





Man

agin

g te

chno

logy

ris

k

A voter, or poll worker, is unable to cancel an in process ballot when in VIBS mode, where the device is set up for a visually impaired voter and the screen is hidden. (This issue does not occur when not in VIBS mode)

1.8.4 AV-OS PC

AV-OS PC was updated to account for the following:

Incorrect “machine id” and “copy number” are printed on reports for multiple tabulators in a single precinct

1.8.5 ABasic Script for Vermont

An ABasic reporting script customized for the state of Vermont was added to the GEMS install

1.8.6 Documentation Updates

The following documents were modified for Assure 1.3:

ASSURE 1.3 Product Overview Guide To include the new software and documentation version numbers in the system

ASSURE 1.3 Matrix To include the new software and documentation version numbers in the system

GEMS Reference Guide 1.21.6 or later To include the description of the new operation of the AVServer Console

GEMS System Administrator’s Guide 1.21.3 or later To include the description of the of the permissions settings to prevent the GEMS Runtime Error

ABasic 2.2.5 Reports Guide To include the Vermont ABasic report script

DRS PhotoScribe PS900 iM2/PS960 Hardware Guide To include the new PS960 Model B information – this configuration has already received a de minimis approval by the EAC through Dominion’s ECO PS960-001

The following documentation is also modified: ASSURE 1.3 System ID Guide

o Once the hash values are known for the trusted build files, they will be entered into this document and the software version numbers will be updated





Man

agin

g te

chno

logy

ris

k

o Also to be included in this document is the new PS960 Model B unit identification information – this configuration has already received a de minimis approval by the EAC through Dominion’s ECO PS960-001

1.9 Change Control and Configuration Management The SLI project team follows a standard change control and Configuration Management (CM) process. This specifies the methods used by SLI to ensure changes are managed and controlled effectively and efficiently and defines the process for receipt, check-in, storage and disposition of hardware, TDP and non-TDP Documents.

The CM process specifies the procedures used by SLI to identify, monitor, and control versions of all of the stated EAC project deliverables, ensuring that the items are constantly and reliably managed throughout the engagement to assure changes are managed and will not result in invalidated or wasted testing efforts.

A summary of these tasks includes:

Receive and review TDP documents: o Delivery department will perform a check-in of the documents

received.

o Documents are placed on the controlled server in the specific project’s TDP folder identified by the delivery date.

o Delivery department notifies Test Manager (TM) of delivery via email.

o TM assigns VTS personnel to perform the documentation review.

o VTS personnel perform a Physical Configuration Audit (PCA) review of any pertinent documentation.

Receive Hardware o The HW Specialist or designee checks in equipment that is delivered

by Dominion.

o Operational Status Checks are performed on all delivered hardware.

o The equipment is stored in a secure room with controlled access.





Man

agin

g te

chno

logy

ris

k

2 Physical Configuration Audit

2.1 TDP Documentation Analysis & Review SLI will complete an assessment of the deliveries of the Technical Data Package of the Dominion Assure 1.3 voting system, as detailed in section 1.8.

Each document included in the delivery of the voting system TDP will be reviewed for compliance to the 2005 VVSG, Volume 2, Sections 2.2 through 2.13 and Volume 2, Section 6.6, relative to any newly implemented updates. The documents and the intended review criteria are listed below:

ASSURE 1.3 Product Overview Guide

o Volume 2, requirement 2.1.1.2

ASSURE 1.3 Matrix o Volume 2, requirement 2.1.1.2

GEMS Reference Guide 1.21.6 or later o Volume 2, requirement 2.8.5.b&d and 2.5.6.2.a,b,c&f

GEMS System Administrator’s Guide 1.21.3 or later o Volume 2, requirement 2.8.5.a

ABasic 2.2.5 Reports Guide o Volume 2, requirement 2.1.1.a&b

DRS PhotoScribe PS900 iM2/PS960 Hardware Guide o Volume 2, requirement 2.1.1.2

ASSURE 1.3 System ID Guide o Volume 2, requirement 2.4.2

System Change Notes o Volume 2, requirement 2.1.1.2 and 2.13





Man

agin

g te

chno

logy

ris

k

2.2 Source Code Review & Analysis SLI conducted a source code review of all modified source code submitted in the delivery of the voting system TDP for compliance to the VVSG, version 2005, v1 sections 5.0-5.27 and v2, Section 6.6. The files listed were compared to the original files in the 1.2 code base, to see what had been changed. those changes were reviewed. The modules listed below were reviewed with no discrepancies noted. Source code for GEMS 1.21.6

Files modified: • PosterDlg.h • PosterDlg.cpp • GEMSDoc.cpp • DownloadTCPConn.cpp • DownloadDlg.cpp • DownloadAVSPort.cpp • CBDetail.h • CardCastReport.cpp • CardCastReport.h

Installation resources for GEMS

Files modified: • Splash Screen (GEMS).bmp • premier_sla.rtf • premier_sla_2.rtf

Source code for ABasic

List of the files associated • 195usvt.abs • vt_prnraces.abs

Source code for AV- OSX 1.2.7

Files modified: • settings.cpp • abasicreport.cpp

Source code for AV-OS PC 1.96.14

Files modified: • abinterp.c • ac2000.h

Source code for BallotStation 4.7.10





Man

agin

g te

chno

logy

ris

k

Files modified: • BallotDlg.cpp • VoteController.cpp

2.3 Trusted Build As a Modification version of the Assure 1.2 only the applications listed in section 2.2 – Source code Review and Analysis were included in the Trusted Build procedure. Within that selection, GEMS, the ABasic script for Vermont and AV-OSX were included in the SLI Trusted Build, which leveraged an iBeta Trusted Build image to create the environment. BallotStation and AV-OS PC were built previously by iBeta, and are included in the Assure 1.3 voting system implementation. For the Trusted Build, the modified 1.3 files were moved into a 1.2 code base, replacing the original files. This way it is verified that all files are from 1.2, except those that are explicitly replaced.





Man

agin

g te

chno

logy

ris

k

3 MATERIALS REQUIRED FOR TESTING

This section outlines the materials required in the testing of the Assure 1.3 voting system, as outlined in the Test Plan.

3.1 Software/Firmware The following software/firmware is required for the execution Dominion Assure 1.3 EAC Modification project tests. This includes all supporting software such as operating systems, compilers, assemblers, application software, firmware, any applications used for burning of media, transmission of data or creation/management of databases.

3.1.1 Manufacturer Software/Firmware

The following table details the portions of the Assure 1.3 system that will be exercised in the testing of the modifications.

Table 1 – Manufacturer Software/Firmware

Application Version

GEMS software version 1.21.6

AV-OS PC firmware version 1.96.14

AV-OSX firmware version 1.2.7 AV-TSX DRE firmware version 4.7.10 AV-TS R6 DRE firmware version 4.7.10 ABasic script for state of Vermont in GEMS 1.21.6

3.1.2 Additional Supporting Test Software

No additional supporting test software will be utilized in this certification test campaign.

3.2 Equipment The following equipment is required for the execution of testing. This includes system hardware, general purpose data processing and communications equipment.

3.2.1 Manufacturer Equipment

The following manufacturer equipment will be used in testing:

AV-OS optical scanner, any certified model





Man

agin

g te

chno

logy

ris

k

AV-OSX optical scanner, any certified model AV-TSX DRE, any certified model AV-TS R6 DRE, any certified model

3.2.2 COTS Equipment

The following Commercial Off-the-Shelf equipment will be used in testing:

One COTS server (Windows Server 2003 SP2), used for implementing VCProgrammer, Key Card Tool, GEMS applications

Media reader/writers for writing/reading from and to GEMS Printer for printing reports

3.2.3 Additional Supporting Test Equipment

No additional support equipment will be used in testing.

3.3 Test Materials The following test materials are required for the execution of testing including, as applicable, test ballot layout and generation materials, test ballot sheets, test ballot cards and control cards, standard and optional output data report formats, and any other materials used in testing.

Election definition

Ballots

Memory cards

Printer paper rolls

Poll worker access smart card

Administrator smart card

Polling place voter smart card





Man

agin

g te

chno

logy

ris

k

4 TEST SPECIFICATIONS

The following are the specifications for testing to be conducted on the Dominion Assure 1.3 voting system. The specifications contain details on the focus of testing, configuration(s), and the functions to be tested.

4.1 Requirements The Dominion Assure 1.3 voting system modifications will be tested to 2005 VVSG requirements, as indicated in section 4.4 below.

4.2 Hardware Configuration and Design The Dominion Assure 1.3 voting system will be configured to implement each relevant modified software/firmware item and any associated hardware.

Each piece of hardware will be validated to have only the specified firmware and data files for execution of duties.

All PCs used in testing will be returned to a pristine condition including reformatting of the hard disk, re-installation of applicable operating systems as well as re-installation of all manufacturer identified supporting software and updates.

4.2.1 Description of Configuration

The system will be set up for Modification and Integration Testing utilizing the components listed below:

An EMS workstation with the GEMS software installed & configured

One AV-OS optical scanner

One AV-OSX optical scanner

One AV-TSX DRE

One AV-TS R6 DRE

4.2.2 System Review

This section describes the workings of the Assure 1.3 voting system modifications and will assist the reader with understanding the flow of data and system processes.

The Assure voting system is a full featured voting system capable of handling a variety of election types. The election management system software GEMS is used for election definition and database and ballot definition creation for General, Open Primary and Closed Primary elections. Once election data is created through the





Man

agin

g te

chno

logy

ris

k

GEMS software, it is installed on the AV-TSX and AV-TS R6 DREs, as well as the AV-OS and AV-OSX precinct count voting devices. Ballots created from the GEMS election definition are scanned on the supported devices, accumulating vote data. The DRE devices can be used to tabulate and transmit vote data to the GEMS consolidation, tabulation and reporting software installed and configured on a workstation.

4.3 Analysis of the Modifications for Testing An analysis was performed on the modifications incorporated into the Assure 1.3 voting system, which include updates to the GEMS software application, OSX firmware, Ballot Station firmware, AV-OS PC firmware, as well as an ABasic script for the State of Vermont.

4.3.1 GEMS


The Cards Cast report was updated to resolve an issue where the total number of registered voters was not accurately reported for split precincts. The system was updated to recognize when a split precinct is being reported and to accumulate the total number of registered voters from each split. This is an isolated fix that is adequately tested by reporting on both split precincts and non-split precincts.

Interruptions in memory card uploads were leaving the application in a state that did not accurately reflect the status of the memory card information being uploaded. The system, though recognizing that the upload/post process had been interrupted, had only one visible status once the process was begun: a green up arrow. The system has been modified to now have an intermediate status of a white up arrow once the upload of data is completed, and then the green up arrow once the posting of the data is completed.

The Install program was updated with a new splash screen and agreements page to reflect Dominion Voting Systems instead of Premier Election Solutions. The updates here are solely cosmetic changes that should have no impact on the operation of the installation.

A runtime error during application startup is addressed by a documentation update. The documentation in “DVS M.51 GEMS System Admin Guide” has been updated with a modified configuration of the permissions for an application folder that prevents the runtime error during system startup. This fix has been applied in the field and is now being formally introduced into system documentation.





Man

agin

g te

chno

logy

ris

k

4.3.2 AV-OSX


Issue where system counter will reset to zero: A hardware reset was affecting the Protective System Counter (PSC), which was archiving the counter only during graceful shutdowns. The hardware reset was zeroing the counter without the current count being archived. The fix for this issue is to now archive the counter after each ballot cast. The modifications made to the PSC affect what the system looks for to trigger an archive of the count. Now, instead of waiting for a signal that the system is gracefully shutting down, the system looks for each event of a ballot being cast.

The date presented on results tape could be incorrect, as it was using the system date (UTC) instead of the local date. If generated in a time zone that was still in the current day, while the UTC time zone had transitioned into the next day, the results tape would have an incorrect date stamp. The fix for this issue was to reference the local date instead of the UTC date, so that when run in the same scenario that illustrates the issue, the results tape now reflects the correct date.

4.3.3 AV-TSX BallotStation


Unable to cancel ballot when in VIBS (ballot hidden) mode: When a ballot is in a hidden–audio only–mode, the ballot was not able to be cancelled. The “Cancel Ballot” button is designed to enable a poll worker to cancel the ballot; however, in the hidden-audio only- mode the button was not functioning such that the cancel process was not occurring, and the system would seem to freeze. The modification for this issue enables the Cancel Ballot button to effectively close an in-process ballot.

4.3.4 AV-OS PC


When multiple devices were assigned to a single precinct, the label printed out for each device showed the precinct ID, but incorrect values for the machine specific identifying codes. The modifications incorporated for this issue provide a unique identifying code for each device assigned to a precinct.

4.3.5 ABasic Script for Vermont

An ABasic reporting script customized for the state of Vermont was added to Assure 1.3 to provide functionality to a current customer. An ABasic report script,





Man

agin

g te

chno

logy

ris

k

compatible with the Assure ABasic format, was added for the State of Vermont and that state’s reporting requirements on undervote/overvotes /blank-votes. Vermont requires that, for each race, the script reports overvotes separately and reports only blank votes/undervotes as undervotes,

4.3.6 Documentation

The documents listed in section 1.8.6 will be reviewed to verify that all changes made correspond to the specified changes, also listed in section 1.8.6. All changes will also be validated to conform to applicable VVSG requirements.

4.4 Testing Strategies Testing will implement two suites: one validating the modifications implemented within Assure 1.3, and the other executing an integration test to verify that the overall system integrity remains intact.

4.4.1 Modification Validation Testing

4.4.1.1 GEMS


For the Cards Cast report updated to resolve an issue where the total number of registered voters was not accurately reported for split precincts: This testing will validate that the Cards Cast report now accurately reports the total number of registered voters in split precincts, as well as verifying that the totals for a non-split precinct continue to be reported correctly. VVSG 2005 requirement 2.2.4.b is the referenced requirement for verification of this modification.

For the issue of interruptions in memory card uploads leaving the application in a state that did not accurately reflect the status of the memory card information being uploaded: This testing will validate that the status for each phase is correctly reflected to the user such that when a data upload is successfully completed, the white up arrow is displayed, and when, and only when, the posting of the data is successfully completed, the green up arrow is displayed. VVSG 2005 requirement 2.2.4.a is the referenced requirement for verification of this modification.

For the install program being updated with a new splash screen and agreements page to reflect Dominion Voting Systems instead of Premier Election Solutions: This testing will validate that the install runs correctly, with the new splash screen displaying and the new software licensing agreement presented without impact on the installation itself. VVSG 2005





Man

agin

g te

chno

logy

ris

k

requirement 2.2.3.a is the referenced requirement for verification of this modification.

For the runtime error during application startup addressed by a documentation update: The configuration of the application will be implemented as per the updated documentation. VVSG 2005 requirement 2.2.4.a is the referenced requirement for verification of this modification.

4.4.1.2 AV-OSX


For the issue where system counter will reset to zero: The testing for this modification will verify that the archive occurs after each ballot cast, and that an unexpected power down does not cause the PSC to reset to zero. VVSG 2005 requirement 2.1.8.b is the referenced requirement for verification of this modification.

For the date presented on the results tape potentially being incorrect due to the system date (UTC) being used instead of the local date: This testing will verify that the results tape utilizes the local time/date information instead of the UTC time/date. VVSG 2005 requirements 2.2.4.b and 2.2.5.g are the referenced requirements for verification of this modification.

4.4.1.3 AV-TSX BallotStation


For the issue of being unable to cancel ballot when in VIBS (ballot hidden) mode: The testing for this fix will verify that the Cancel option works at all key points in the ballot casting process, both in VIBS mode and in non-VIBS mode. VVSG 2005 requirements 2.3.3.3.j and 4.1.4.3.c.i are the referenced requirements for verification of this modification.

4.4.1.4 AV-OS PC


For the issue of when multiple devices are assigned to a single precinct, the label printed out for each device showed the precinct ID, but incorrect values for the machine specific identifying codes: This testing will verify that when multiple devices are assigned to a single precinct that they are each provided with a unique identifier. VVSG 2005 requirement 2.2.5.b is the referenced requirement for verification of this modification.

4.4.1.5 ABasic Script for Vermont

For the ABasic reporting script customized for the state of Vermont added to Assure 1.3: This testing will include downloading the report script to a





Man

agin

g te

chno

logy

ris

k

precinct count voting device and printing test totals reports on that voting device.. VVSG 2005 requirement 2.4.3.a is the referenced requirement for verification of this modification.

4.4.2 Integration Testing

An Integration test will be implemented to validate that the system’s basic integrity remains intact for Assure 1.3. VVSG 2005 requirement Vol. 2, 6.3 is the referenced requirement for verification of this activity. An election definition will be imported, with election information disseminated to AV-OS, AV-OSX, AV-TSX and AV-TSR6 precinct devices. Ballots will be cast on each device, then vote data will be transmitted back to GEMS, where it will be accumulated and tallied, with a final results report generated. Areas not exercised as part of the integration test include the Assure Security Manager, PCS, ExpressPoll, AV-OS central count, KeyCardTool, VCProgrammer, Voter Card Encoder, OSAA, AIMS, Voter Assist Terminal BMD, and AccuView Printer Module, as these areas are considered to be removed enough from the implemented modifications that they are not impacted at any level.





Man

agin

g te

chno

logy

ris

k

5 TEST DATA

5.1 Data Recording SLI will cast votes in a pre-selected manner and verify that the expected data is accumulated, tallied and reported accurately.

Testing information is recorded in the test suites, as well as in test notebooks, which are utilized according to SLI’s standard lab procedure SLP-VC-30 - Test Notebooks.

5.2 Test Data Reduction SLI processes the test data by manually recording input data into each pertinent module within the Test Suites as well as the exact output that is generated, e.g., the vote counts when the data is consolidated.

5.3 Test Validation Upon completion of the test suite development, SLI will conduct validation by walking through the test steps using the equipment to ensure the steps are complete and accurate. The validation run will help eliminate any human error or potential issues due to inconsistency between documentation and actual application. Any discrepancies found during the validation process will be recorded following the SLI approved discrepancy process.





Man

agin

g te

chno

logy

ris

k

6 TEST PROCEDURE AND CONDITIONS

This section describes the test conditions and procedures for execution of test suites. If a particular sequence is mandatory for the execution of suites, a rationale will be given. Additionally, this section is used to describe procedures for setting up equipment that will be utilized in the execution of the test suites.

6.1 Facility Requirements Testing will be performed on site at SLI in Denver, Colorado.

All TDP and test documentation is stored on site at SLI’s facility in a secure project directory on SLI’s secure Voting server.

6.2 Test Setup The following process will be utilized in the setup of the Assure 1.3 system in preparation for the testing of the implemented modifications, as well as performance of the integration test.

One GEMS environment will be set up and launched to validate that the application comes up. A zero report will also be generated to validate that the database does not contain any data that would potentially compromise the test.

One AV-OS will be set up and activated. A zero report will be generated to validate that no votes are contained within the memory of the device, which would potentially compromise the test.

One AV-OSX will be set up and activated. A zero report will be generated to validate that no votes are contained within the memory of the device, which would potentially compromise the test.

One TSX DRE will be set up and activated. A zero report will be generated to validate that no votes are contained within the memory of the device, which would potentially compromise the test.

One TS R6 DRE will be set up and activated. A zero report will be generated to validate that no votes are contained within the memory of the device, which would potentially compromise the test.

Successful completion of operational status checks on each device will indicate whether the system is ready for test execution.





Man

agin

g te

chno

logy

ris

k

6.3 Test Sequence First, one test suite will be formally executed to validate the modifications implemented within the Assure 1.3 voting system. Second, one test suite will be formally executed to verify the integrity of the Assure 1.3 voting system as an integrated system.

6.4 Error Recovery In the event that an error is encountered during testing, SLI will follow the SLP for Anomalies. An error will be treated as an anomaly until properly researched in an effort to determine the cause. The Test Specialist will record all information gathered during analysis of an anomaly in the Test Notebook per the SLP for Test Notebooks.

Once researched, if an anomaly is determined to be an issue of non-conformance with the voting system under test to the applicable VVSG requirements, a discrepancy will be opened per SLI’s discrepancy process. If the cause of the anomaly cannot be determined, it will be categorized as a discrepancy with the applicable information being properly documented per the Discrepancy SLP. Discrepancies will be uploaded to the EAC’s VRT.

If an anomaly occurs during testing and is not reproducible or found to be caused by human error (i.e., incorrect test steps or sequence), the system or components under test will be analyzed by the Test Specialist with support from the Test Manager and manufacturer if required. (Note: all testing is halted during analysis.) The Test Specialist will restore the environment to operating condition and determine what steps must be taken to validate the environment is in the same state as when testing was halted. This may require an additional Operational Status Check or retesting if the integrity of the test results is found to be impacted.





Man

agin

g te

chno

logy

ris

k

7 TEST OPERATIONS PROCEDURES

An inventory will be performed to verify the voting equipment received contains hardware and software elements as defined in the TDP prior to commencement of testing.

Throughout the testing effort, test suites and modules will be marked as follows:

Accept – Test is accepted as successful. Reject – Test is rejected as unsuccessful. NT – Not Testable is used for test modules that cannot be executed. For

example, if failure of one test module precludes attempting subsequent test modules, the latter will be marked as NT.

Test results with either Reject or NT will include comments by the VTS explaining the reason for the result.

Issues encountered during review and testing will be documented on the Discrepancy Report. Issues that do not conform to the requirements of the VVSG, version 2005 will be marked as Documentation Discrepancies or Functional Discrepancies. (A discrepancy occurs when any application or component does not meet defined requirements or specifications.)

Issues that are encountered during testing or documentation review, but are not addressed by an applicable standard, will be added to the Discrepancy report and noted as Informational. The vendor has the option whether to address Informational issues. All responses provided by the vendor are noted in the Discrepancy Report attachment to the Certification Test Report.





Man

agin

g te

chno

logy

ris

k

8 Approval Signatures

SLI:

Client:

Name: Ian S. Piper

Title: Director, Certification

Date: January 24, 2012

End of Certification Test Plan

VSTL Director

Date

lhoppert

Traci_Sig

lhoppert

January 24, 2012