Abstract - NetApp Community · SEC 17a-4(f) Compliance Assessment Abstract ... 12 2.4 Serialize the ... Securities & Exchange Commission for 17a-4 Records

SEC 17a-4(f) Compliance Assessment

AbstractThis Technical Assessment is a compliance assessment of whether the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities meet the requirements and conditions of SEC Rule 17a-4(f).

It is the conclusion of Cohasset Associates that the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities meet all of the requirements for which they are directly responsible as they relate to the recording, identification, retention protection and availability of electronic records with a time-based retention period as specified in SEC Rule 17a-4(f).

Cohasset Associates

3806 Lake Point Tower505 N. Lake Shore DriveChicago, IL 60611 USA

www.cohasset.com

312-527-1550

2.0

Guiding the Way to Successful Records & Information Management

Technical Report

December 2011

Prepared by Cohasset Associates, Inc.

Technical ReportSEC 17a-4(F) Compliance Assessment: NetApp SnapLock Compliance Storage Solution and NetApp Deduplication Storage Capabilities

Table of Contents • 2

Table of Contents

1. Introduction .......................................................................................................................................3 1.1 The Electronic Storage Requirements of the Securities & Exchange Commission

for 17a-4 Records ............................................................................................................................................3

1.2 NetApp SnapLock Compliance Storage Solution and

NetApp Deduplication Capabilities ............................................................................................................5

1.3 Assessment and Technical Assessment .....................................................................................................6

2. Compliance Assessment with SEC Rule 17a-4(f) ...........................................................................7 2.1 Structure and Organization of Cohasset’s Assessment ........................................................................7

2.2 Non-rewriteable, Non-erasable Format ...................................................................................................7

2.3 Verify Automatically the Quality and Accuracy of the Recording Process ............................... 12

2.4 Serialize the Original and Duplicate Units of Storage Media ........................................................ 18

2.5 Store Separately a Duplicate Copy .......................................................................................................... 19

3. Conclusions ......................................................................................................................................22

Appendix A. Deduplication Capabilities – An Overview ..................................................................23

End Notes ..............................................................................................................................................25

About Cohasset Associates, Inc. .........................................................................................................26


Introduction • 3

1. Introduction

This section sets the context for this technical assessment. It identifies a) the SEC’s regulatory foundation for allowing e-records to be retained on a variety of electronic storage media, and b) the storage system that is the subject of Cohasset’s assessment against these SEC electronic storage media regulations.

1.1 The Electronic Storage Requirements of the Securities & Exchange Commission for 17a-4 Records

Records retention requirements for the U.S. securities broker-dealer industry are stipulated by the Securities & Exchange Commission (“SEC”) Regulations 17 CFR 240.17a-3 and 17 CFR 240.17a-4 (the “Rule” or “Regulation”), adopted on February 12, 1997. Within this regulation, Rule 17a-4(f)1 expressly allows records to be retained on electronic storage media, subject to meeting certain conditions.

Three foundational documents collectively define and interpret the specific regulatory requirements that electronic storage systems must meet in order to be SEC compliant under Rule 17a-4(f). They are:

• The Rule itself,

• The associated SEC Interpretive Release No. 34-44238, Commission Guidance to Broker-Dealers on the Use of Electronic Storage Media under the Electronic Signatures in Global and National Commerce Act of 2000 with Respect to Rule 17a-4, dated May 1, 2001 (the “2001 Release”), and

• Release No. 34-47806, Electronic Storage of Broker-Dealer Records, dated May 7, 2003 (the “2003 Release”).


Introduction • 4

In the Rule and the two subsequent interpretative releases, the SEC clearly states that the use of electronic storage media and devices, to the extent that they can deliver the prescribed functionality, satisfy the stipulations of Rule 17a-4.

Rule 240.17a-4(f) specifically states:

The records required to be maintained and preserved pursuant to § 240.17a-3 and § 240.17a-4 may be immediately produced or reproduced on “micrographic media” (as defined in this section) or by means of “electronic storage media” (as defined in this section) that meet the conditions set forth in this paragraph and be maintained and preserved for the required time in that form [emphasis added].

(1) For purposes of this section:

* * * * * (ii) The term electronic storage media means any digital storage medium or system and, in the case of both paragraphs (f)(1)(i) and f(1)(ii) of this section, which meets the applicable conditions set forth in this paragraph (f).

The 2003 Release further clarifies that implementation of rewriteable and erasable media, such as magnetic tape or magnetic disk, may meet the requirements of a non-erasable, non-rewriteable recording environment – to the extent that they deliver the prescribed functionality and so long as appropriate integrated control codes are in place. The 2003 Release states:

A broker-dealer would not violate the requirement in paragraph (f)(2)(ii)(A) of the rule if it used an electronic storage system that prevents the overwriting, erasing or otherwise altering of a record during its required retention period through the use of integrated hardware and software control codes.

The key words within this statement are “integrated” and “control codes.” The term “integrated” means that the method used to achieve a non-rewriteable, non-erasable recording environment must be an integral part of the recording hardware and software. The term “control codes” indicates the acceptability of using attribute codes (metadata) that are integral to the hardware and software of the recording process in order to protect against overwriting or erasure of any records.


Introduction • 5

Examples of integrated control codes that could be applied towards providing a non-rewriteable, non-erasable recording process are:

• A retention period during which records cannot be erased,

• A unique record identifier that differentiates it from all other records, and

• The date/time of recording (the data/time of recording and the unique identifier serve in combination to “serialize” a record).

The 2003 Release specifically notes that recording processes or applications which merely mitigate the risk of overwrite or erasure (rather than prevent them), such as relying on access control security, will not satisfy the requirements of Rule 17a-4(f).

An important associated requirement of Rule 17a-4(f)(2)(i) is that a member, broker or dealer wanting to store their 17a-3 and 17a-4 records electronically must notify its “examining authority” ninety (90) days prior to employing any technology other than WORM optical media. Examining authorities are self-regulatory organizations (SROs) under the jurisdiction of the SEC such as the New York Stock Exchange (NYSE) and Financial Industry Regulatory Authority (FINRA).

1.2 NetApp SnapLock Compliance Storage Solution and NetApp Deduplication Capabilities

NetApp SnapLock Compliance Storage Solution

The NetApp SnapLock Compliance Storage Solution was designed to meet the requirements of the SEC Rule 17a-4(f) in that it provides for a specific area of the storage system (a NetApp Compliance Storage Volume) to be designated for recording and protecting electronic record files in a non-rewriteable, non-erasable (or write-once, read-many [WORM]) format. Within a SnapLock Compliance Storage Solution, integrated control codes are utilized to ensure that the integrity of stored records is protected for a specified retention period.

NetApp Deduplication Capabilities

The NetApp Deduplication capabilities, which can be employed in conjunction with a SnapLock Compliance Storage Solution, are designed to ensure that the overall integrity of the recorded files and their associated metadata is preserved. The deduplication approach employed by NetApp uses hash digests to determine if recorded blocks of data are potential duplicates. When potential duplicates are identified, a byte-for-byte comparison of the complete record block is


Introduction • 6

performed prior to releasing storage space for one of the duplicate blocks. The full metadata, which contains the integrated protection control codes, for all record blocks is retained and protected as non-erasable and non-rewriteable.

1.3 Assessment and Technical Assessment

To obtain an independent and objective assessment of the NetApp SnapLock Compliance Storage Solution and Deduplication capabilities relative to meeting the requirements set forth in SEC Rule 17a-4(f), NetApp Corporation (“NetApp”) engaged Cohasset Associates, Inc. (“Cohasset”), a highly respected consulting firm with specific knowledge, recognized expertise and more than 30 years of experience regarding the legal technical and operational issues associated with the records management practices of companies regulated by the SEC and SROs.

Cohasset’s assignment was to:

• Assess the ability of the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities to meet the requirements of all the relevant conditions of Rule 17a-4(f), and

• Prepare this technical assessment regarding that assessment.

This assessment represents the professional opinion of Cohasset Associates and should not be construed as an endorsement or rejection by Cohasset of the NetApp SnapLock Compliance Storage Solution product and Deduplication capabilities or any other NetApp products. The information utilized by Cohasset to conduct this assessment consisted of a) oral discussions, b) product requirements c) functional description documents and d) other directly related materials provided by NetApp. It also included information acquired by Cohasset from publicly available sources.

Additional information about Cohasset is provided in Section 3 of this assessment.

The content and conclusions of this assessment are not intended and should not be construed as legal advice. Relevant laws and regulations are constantly evolving and legal advice must be tailored to the specific circumstances of the laws and regulations for each organization. Therefore, nothing stated herein should be substituted for the advice of competent legal counsel.


Compliance Assessment • 7

2. Compliance Assessment with SEC Rule 17a-4(f)

This section presents Cohasset’s assessment of the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities relative to meeting the electronic records storage requirements of SEC Rule 17a-4(f) that are considered to be the responsibility of the recording and storage solution.

2.1 Structure and Organization of Cohasset’s Assessment

The assessment of each relevant requirement in Rule 17a-4(f) is structured into four parts:

Compliance Requirement – Definition of the specific SEC regulatory requirements that must be met in order to utilize electronic records storage media in the retention of 17a-3 and 17a-4 records;

Compliance Assessment – Cohasset’s assessment of the degree to which NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities comply with the Rule;

NetApp Compliance Storage Solution and Deduplication Capabilities – Description of the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities that enable them to meet the specific 17a-4(f) requirements; and

Other Considerations – Identification of actions (if any exist) that may need to be performed in order to meet the requirements of the Rule.

2.2 Non-rewriteable, Non-erasable Format

Compliance Requirement 17a4(f)(2)(ii)(A)

Preserve the records exclusively in a non-rewriteable, non-erasable format.

As set forth in Section III(B) of the 2001 Release, this requirement “is designed to ensure that electronic records are capable of being accurately reproduced for later reference by maintaining the records in unalterable form.”



The following statement in the 2003 Release further clarifies that certain implementations of rewriteable and erasable media, such as magnetic disk or magnetic tape, would meet the requirements of a non-erasable, non-rewriteable recording environment provided a) they deliver the prescribed functionality, and b) that functionality is delivered for as long as appropriate integrated control codes are in place:

A broker-dealer would not violate the requirement in paragraph (f)(2)(ii)(A) of the rule if it used an electronic storage system that prevents the overwriting, erasing or otherwise altering of a record during its required retention period through the use of integrated hardware and software control codes.

Compliance Assessment

It is Cohasset Associates’ opinion that the NetApp SnapLock Compliance Storage Solution capabilities meet this requirement of the SEC Rule to preserve each record file stored in a non-erasable, non-rewriteable environment for the designated retention period.

Where NetApp Deduplication capabilities are employed in conjunction with a SnapLock Compliance Storage Solution, it is Cohasset’s opinion that the deduplication methodology, particularly the byte-for-byte comparisons performed on each potentially duplicate record block to determine deduplication, does not compromise the requirement of this Rule for maintaining the non-erasable, non-rewriteable recording status of each record file for the designated retention period.

NetApp SnapLock Compliance Storage Solution Capabilities

The NetApp SnapLock Compliance Storage Solution capabilities provide for a specific area of the storage system (a volume) to be designated for recording electronic record files in a non-rewriteable, non-erasable (or write-once, read-many [WORM]) format. Within a SnapLock Compliance Volume, integrated control codes are utilized to ensure that the integrity of stored records is protected for a specified retention period. The NetApp SnapLock Compliance Storage Solution (vs. the NetApp SnapLock Enterprise version) incorporates controls that do not allow the NetApp Compliance Storage Solution system administrator to reduce the retention period or to erase protected records prior to the expiration of the retention period.



The following points need to be taken into account to fully understand the non-rewriteable, non-erasable capabilities of NetApp SnapLock Compliance Storage Solution:

• During administrative configuration and setup a specific area of a NetApp storage system is defined as a Compliance Volume. Designation as a SnapLock Compliance Volume provides the environment for non-erasable, non-rewriteable control codes to be recorded and applied.

• In order to protect each record file written to a NetApp SnapLock Compliance Storage Solution against erasure or overwrite during the required retention period, the content, records management or file management application must execute the following two commands during the recording process:

■ A time-based1 retention period, in the form of a retention expiration date,2 must be sent to the NetApp SnapLock Compliance Storage Solution for each record file, and

■ The write permission must be removed, thereby disallowing the overwriting of the record file.

These commands should be executed in back-to-back sequence in order to assure immediate protection of each record file according to the conditions of the Rule.

• A SnapLock Compliance Volume also can be set up to “Auto Commit” records that have been recorded to a SnapLock Compliance Volume but have not been committed to a non-erasable, non-rewriteable (WORM) state. In this mode, based on an administratively established “time delay”, each record file is automatically committed as a WORM file by applying the Default retention period and removing the write permission for that file.

• A SnapLock Compliance Storage Solution provides for retention controls to be established at the time of administrative setup for default, minimum and maximum retention periods.

■ The Default retention period delineates a retention period that will be assigned to a record file if the records management or file management application does not send a command that sets the retention period. The default retention period ensures that all protected records will have a retention period associated with them. If a default retention period is not defined at setup, then the Maximum retention period is assumed as the default.

■ The Maximum retention period (default value is 30 years) is designed to protect against retention periods being set that are well beyond the required maximum retention period. The highest possible value for the maximum retention period is 70 years.



■ The Minimum retention period assures that a retention period is not assigned that is lower than the minimum required retention period for the records that are being recorded.

• The Default, Minimum and Maximum retention period can be extended. They also can be lowered by the member, broker or dealer SnapLock Compliance Storage Solution administrator to accommodate potential changes in regulatory retention periods that are lower than the values originally set or to allow for additional record file types with a lower retention period to be written to a SnapLock Compliance Storage Solution.

■ Changes to the administratively defined Default, Maximum and Minimum retention periods do not affect the retention periods assigned to existing record files. Thus, changes apply only to new, “this day forward” records being committed to the system.

• The retention period of recorded non-erasable, non-rewriteable record files may never be shortened by the records management or file management application or by the system administrator.

• The retention period can be extended for a specific record file by sending a revised retention period instruction to the NetApp SnapLock Compliance Storage Solution that is longer than the current retention period.

• Once the control codes for each record file are in place, the NetApp SnapLock Compliance Storage Solution protects the records against overwrite, modification or erasure by rejecting any request it receives from the records management application or file management application to overwrite or delete records prior to the expiration of the designated retention period. When deduplication has been employed, no data blocks are allowed to be deleted until the retention periods in all associated block metadata “nodes” have expired.

• Records are protected against deletion by a NetApp SnapLock Compliance Storage Solution system administrator prior to the expiration of the retention period.

• To ensure that the retention period is not compromised, a software-based retention clock is initialized at the time of administrative setup of the NetApp SnapLock Compliance Storage Solution. Once set, the internal retention clock cannot be modified. This internal SnapLock Compliance Storage Solution-specific retention clock prevents the premature deletion of records that may arise from certain actions such as a system administrator inadvertently or intentionally setting the hardware storage system clock forward which may cause retention periods to be shortened.



NetApp Deduplication Capabilities

NetApp Deduplication capabilities can be used in conjunction with a NetApp SnapLock Compliance Storage Solution to optimize the available storage by consolidating duplicate data blocks of stored record files to a single instance of that data block. To fully understand the non-rewriteable, non-erasable aspects related to NetApp Deduplication (see Appendix A for an overview of deduplication), the following points need to be taken into account:

• The NetApp Deduplication process occurs locally “within” the hardware and software of the NetApp storage management system. Its processes operate on the SnapLock Compliance Storage Solution data blocks based on predefined steps and predetermined algorithms. Therefore, the process of deduplication is not influenced by external applications or human intervention (other than administrative setup and scheduling or initiating the start of the process).

• The NetApp Deduplication process works at the level of each “data block”3 that represents a portion of a record file, not on the complete record file. Each data block is referenced by the “node” of metadata that, among other information, contains the integrated control codes used to protect each record block against erasure or overwrite for the assigned retention period.

• During the deduplication process, the metadata containing the integrated protection control codes and the reference or pointer to each physical data block of a record file are not deduplicated or consolidated. Therefore, all of the integrated control codes (retention period, read-only attribute, file identification) for every data block of a record file written to storage are preserved for the assigned retention period.

• During the deduplication process, when two blocks of data are suspected of being duplicates due to the result of a comparison of calculated hash digests or fingerprints stored in a catalogue of block hash digests, a byte-for-byte comparison is undertaken to determine if they are exact duplicates.

■ If the suspected duplicate data blocks are found to be exact duplicates then the reference pointer of the second data block is redirected to point to the first data block. The reference metadata, including protection control codes, for both data blocks are retained and protected. The storage space for the second data block is released for use by the storage system – in essence, the duplicate second data block is deleted.



■ If the suspected duplicate blocks are found to not be exact duplicates, then both data blocks and their reference metadata are preserved.

• Should there be a need to “reduplicate” the data blocks to their original state, such as for purposes of migrating the record file to a system that does not support deduplication, NetApp provides a capability to reduplicate blocks of data that contain more than one reference pointer and restore the reference pointers and associated protection and identification metadata to each block, thereby reestablishing the original recorded state of the data blocks and metadata reference pointers prior to deduplication.

Other Considerations

The system administrator for a member, broker or dealer is procedurally responsible for appropriately configuring and setting up the NetApp SnapLock Compliance Storage Solution to ensure that electronic records are protected in accordance with the appropriate retention periods as defined in SEC Rules 17as-3 and 17a-4.

2.3 Verify Automatically the Quality and Accuracy of the Recording Process

2.3.1 Compliance Requirement 17a-4(f)(2)(ii)(B)

Verify automatically the quality and accuracy of the storage media recording process.

The intent of SEC Rule 17a-4(f)(2)(ii)(B) is to ensure that the media recording process is accurate to a very high degree and, therefore, the recorded information is of the highest accuracy and quality. The objective of this subsection of the SEC Rule is to provide the utmost confidence that all records read from the storage media are precisely the same as those recorded. There are two points in time where it is important to verify the quality and accuracy of record file data:

• During the recording process – when the record files are written to the magnetic storage media, and

• Post recording – when the record files are retrieved or periodically verified throughout the required retention period.

2.3.2 Compliance Assessment

Cohasset believes that the NetApp SnapLock Compliance Storage Solution, in conjunction with state-of-the-art magnetic disk technology, meets the SEC requirement to verify the accuracy and completeness of the recording process.



2.3.3 NetApp SnapLock Compliance Storage Solution Capabilities

The NetApp Data ONTAP Operating System underlying the SnapLock Compliance Storage Solution is comprised of three layers as it relates to verifying the quality and accuracy of the recording process:

• The File System Layer – processes requests for record file writes and reads from the client application and maintains the metadata associated with each record file;

• The RAID4 Layer – receives the record file from the File System Layer where it transforms the file into logical data blocks, adds RAID DP (dual parity) error correction information to each block, issues the write or read request to the Storage Layer, receives uncorrectable read errors from the storage layer and performs rebuilding, as required, of the data blocks using the RAID error correction information; and

• The Storage Layer – adds a checksum5 for each block of data to be recorded, validates the checksum upon retrieval and communicates write and read commands to the physical magnetic disk drive controller.

The SnapLock Compliance Storage Solution uses magnetic disk drives that employ state-of-the-art monitoring and control methods and error detection and correction or reporting technology to ensure that data is accurately and completely recorded.

The methods and technology utilized by SnapLock Compliance Storage Solution to verify the accuracy and quality of the recorded data are described below.

2.3.3.1 Verification During the Recording Process

The goal of a recording process is: write the record files (received from the host/client application) accurately and completely on the storage media. The SnapLock Compliance Storage Solution uses state-of-the-art magnetic disk drives that employ well established error detection and correction (ECC6) technology and monitoring and control methods to ensure that data is accurately and completely recorded. The ability to automatically detect recording and retrieval errors of various types – seek errors, write errors, read errors, etc. – and, through retries and error correction methods, attempt to successfully record the data, is an integral part of magnetic disk drive technology. When the magnetic disk drive is unable to correct an error, it notifies the NetApp DataONTAP Storage Layer which then undertakes steps to rewrite the data or to rebuild the data using RAID error correction information (discussed in more detail later).



During the recording process the magnetic disk drive continuously validates that the electronics and mechanics of the disk drive are properly working. Recording errors are typically caused by a malfunction of the electronics or mechanics of the magnetic disk drive, media defects or environmental conditions, e.g., temperature or humidity, which affect the recording process. When a magnetic disk drive encounters a recording error in a sector on the media, it retries the action multiple times to successfully complete the process. If unsuccessful, the magnetic disk drive either attempts to rewrite the data in a different sector of the magnetic disk media or reports the error to the Data ONTAP Storage Layer which may redirect the data to be recorded to different sector on the media or, if unable to successfully record the data, notify the system administrator of the problem. This process assures that all record file data written by the SnapLock Compliance Storage Solution to a magnetic disk drive are accurate.

The SnapLock Compliance Storage Solution uses non-volatile random access memory (NVRAM), operating on a battery backup, to store pending write and read operations. Should a major failure occur during the recording process (such as a power outage or other major system interruption), all pending write or read operations are protected from incomplete execution or loss. When the SnapLock Compliance Storage Solution is restarted (following the major system interruption), it then properly executes the pending write or read operations stored and protected in NVRAM.

Also, during the recording process, checksums are used to ensure that the integrity of the data is preserved when passed from one layer of the SnapLock Compliance Storage Solution to another. The checksums allow for the detection and correction of any inter-component or inter-step errors. Should a checksum error be encountered, the error would be reported to the system administrator for appropriate action, such as restarting the storage system. Because these internal checksums are only used to validate the movement of data between or among storage layers, they are not retained as part of the recorded data.

The following additional data is recorded to the magnetic disk media by the SnapLock Compliance Storage Solution that support the error detection and correction of record files: a) a checksum is generated and stored with every block of data that is recorded. This checksum is verified by the Storage Layer upon retrieval of every block of data, and b) RAID DP dual parity error correction information is generated and stored for each block of data written to two separate (dual) magnetic disk drives. Should uncorrectable errors be reported by the magnetic disk drives or should a checksum verification identify an uncorrectable read error, the RAID DP information facilitates the rebuilding of in-error blocks of data. More detail regarding the recording and use of this information is provided in the next section.



2.3.3.2 Verification Over Time

The capabilities of the magnetic disk drive technology and the SnapLock Compliance Storage Solution collectively address both the verification of record file integrity and the correction of detected errors during the period established for retaining each record file. The integrity verification technologies are applied at two points in time: a) each time a record file is retrieved, and b) periodically over time, independent of record file retrieval.

Periodically, the SnapLock Compliance Storage Solution reads back (retrieves) and verifies every block of recorded data to assure that the integrity of record files has been preserved which is particularly important for record files that are infrequently retrieved (this process also is known as a scan or scrub of the data). The default frequency of these periodic scrubs is once a week. An alternative frequency can be defined by the system administrator of the SnapLock Compliance Storage Solution.

2.3.3.2.1 Verifying Record File Integrity

The verification of recording accuracy during retrieval (whether as a request from a host/user application or during a scan process) is initially performed by the magnetic disk drives using the ECC information written with each data sector at the time of recording. If a read error is detected, the read request is retried a predetermined number of times. If the read-error persists, then the ECC is utilized to correct the error. Should an uncorrectable read error occur (an error that cannot be corrected after multiple retries and applying ECCs), then the error is reported to the SnapLock Compliance Storage Solution where RAID DP information is utilized to rebuild and rewrite the in-error blocks to new magnetic disk locations..

Storage industry laboratory testing information and empirical statistical calculations indicate that the level of uncorrectable read errors in state-of-the-art magnetic disk drives is extremely low – in the range of 1 bit in 100 trillion (100,000,000,000,000) bits (equivalent to one byte in 12.5 trillion bytes or 12.5 terabytes).

While the predicted uncorrectable read error level is extremely low, there are other electronic and mechanical errors that may occur with magnetic disk drives that can be monitored and managed to enhance a storage system’s ability to proactively undertake the replacement of a drive, rather than have it fail unexpectedly. (see section 2.3.3.3 Error Monitoring and Reporting below).



2.3.3.2.2 Recovering In-Error Record File Data

When an uncorrectable read error is reported by a magnetic disk drive to the SnapLock Compliance Storage Solution, NetApp RAID DP information is utilized to rebuild the block(s) of data containing the errors. RAID DP is NetApp’s unique implementation of RAID 6 – which uses two parity drives to correct errors that require rebuilding of data. RAID DP error correction information provides for the rebuilding of in-error data ranging from single blocks of data up to the failure of two full magnetic disk drives. This is particularly important when the storage system monitoring and analysis determines that a magnetic disk drive should be proactively replaced or when the failure of one or even two magnetic disk drives occurs unexpectedly.

If the reported errors cannot be corrected using RAID DP, the options for recovery from the duplicate copy are described in Section 2.5 – Store Separately a Duplicate Copy.

2.3.3.3 Error Monitoring and Reporting

When uncorrectable write or read errors occur with magnetic disk drives, one of the problems may be a complete failure of the drive or media due to a number of potential causes, such as magnetic head failure, media defects, gouging of the media by the recording head, failure of a spindle or actuator motor, etc.

In the past, the approach for dealing with major disk drive failures has been reactive: a) wait until the unexpected failure occurs, then b) replace the magnetic disk drive and c) rebuild or recover the stored data.

The SnapLock Compliance Storage Solution uses a proprietary process for collecting information regarding recoverable magnetic disk errors (those errors that were automatically corrected by the magnetic disk drive or NetApp DataONTAP storage layer) and uses this information to predict the potential failure of individual magnetic disk drives. This process regularly collects information regarding each magnetic disk drive’s recoverable error conditions, such as write errors, read errors, seek errors, not-ready errors, etc. Then a threshold-based heuristic analysis system is employed that takes into account the time between errors as well as the total number of errors in an attempt to project whether a specific disk drive is likely to fail and should be proactively replaced. When the magnetic disk drive is replaced, the data on the replacement drive is either rebuilt using RAID DP or recovered from the duplicate copy.

NetApp’s error monitoring and heuristic analysis process provides for proactive management of potential magnetic disk drive failures which can substantially decrease the risk of potential data loss or complex data rebuild or recovery of record files.



2.3.4 Other Considerations

Cohasset Associates’ conclusion is that given the collective capabilities of a) today’s state-of-the-art magnetic disk recording technology (including error detection and correction capabilities) and b) the advanced error correction capabilities of SnapLock Compliance Storage Solution, the NetApp Compliance Storage Solution meets the requirements of the Rule.

Cohasset recommends that the member, broker or dealer utilize the latest in accurate and reliable magnetic disk technology, as described in this paper, and employ the most advanced capabilities of the SnapLock Compliance Storage Solution functionality, particularly RAID DP, to assure the highest level of compliance with the Rule.

This assessment is based on the following trends and observations regarding the overall trustworthiness of the magnetic disk recording process:

• The magnetic disk drive technology utilized in the SnapLock Compliance Storage Solution has undergone significant advances over time (i.e. electronic controls, error detection and correction, and monitoring capabilities). Collectively, these advances provide an extraordinarily high degree of accuracy and completeness during the recording and read back process.

• In recent years, many technological advances (e.g., RAID 6 or NetApp’s RAID DP) have been made that enhance the correction of in-error blocks of data – ranging from single data blocks or multiple blocks of data up to the failure of two full magnetic disk drives – when reading record files from or conducting a scan or scrub of magnetic disk media.

• The electronic information storage industry’s reputation and livelihood requires the ongoing achievement of two key performance standards. The first: all record files must be accurately and completely recorded to magnetic storage in a high quality manner. The second: any errors found subsequent to the recording process must be either fully corrected or recovered from a duplicate copy.

• For many decades now, the largest corporate and public entities globally have relied on the quality and accuracy of magnetic disk storage – appropriately backed up for recovery – and advanced storage and file systems to store, protect and retrieve their mission-critical and vital business and regulatory data and document-based records.

During this time, the technologies utilized to achieve a very high degree of accuracy in the magnetic disk recording process have rapidly advanced – with the most dramatic improvements occurring in the past 5-10 years. Today’s state-of-the-art magnetic storage



recording processes are considered very trustworthy by the commercial and government organizations that universally rely on them for storing their business and mission critical information.

As magnetic disk technology and storage and retention management technology and best practices continue to evolve, the member, broker or dealer should, when reasonable, consider adopting them to assure that their compliance with the Rule is supported by the strongest technology and practices available.

• Lastly, SEC Requirement 17a-4(f)(3)(iii) “Store separately from the original a duplicate copy of the record stored on any medium acceptable under 240.17a-4 for the time required” ensures that recorded record files can be recovered in the event of a major or catastrophic failure.

Twenty years ago, when optical disk drives and media were in the early stage of technology evolution, an automatic read-after-write capability significantly enhanced their ability to verify the quality and accuracy of the optical recording process – and, therefore, it was specifically included in the SEC’s no-action letter of 1993 as well as its 1997 rule change regarding the safeguards necessary for the storage of electronic records created pursuant to Requirements 17a-3 and 17a-4.

While an automatic and immediate verification of the recorded data may enhance the overall accuracy and quality of the magnetic disk media recording process, it is Cohasset Associates opinion that the error handling and error correction associated with state-of-the-art magnetic disk technologies coupled with the SnapLock Compliance Storage Solution error detection and correction capabilities (particularly RAID DP), meet or exceed levels acceptable to be compliant with the Rule.

Further, in accordance with the SEC’s ongoing commitment to ensure that the requirements of Rule 17a-4(f)(3)(iii) “not be technology restrictive”, Cohasset believes that state-of-the-art magnetic storage technologies and, specifically, the SnapLock Compliance Storage Solution, provide a technology solution that satisfies the original intent and spirit of the Rule.

2.4 Serialize the Original and Duplicate Units of Storage Media

Compliance Requirement 17a-4(f)(2)(ii)(C)

Serialize the original, and, if applicable, duplicate units of storage media, and time-date for the required period of retention the information placed on such electronic storage media.



This requirement, according to Section III(B) of the 2001 Release, “is intended to ensure both the accuracy and accessibility of the records by indicating the order in which records are stored, thereby making specific records easier to locate and authenticating the storage process.”

While this requirement is thought to specifically address tracking the individual units of media related to micrographic storage, the Rule may be satisfied for electronic records by capturing index or metadata associated with each electronic record that both “uniquely” identifies the record, and also associates a “date of recording” with each record.


Cohasset believes that the NetApp SnapLock Compliance Storage Solution capabilities meet the SEC requirement to serialize both the original record file and each duplicate copy of the record file stored.


The NetApp SnapLock Compliance Storage Solution capabilities serialize and time-date each record file by:

• Storing and maintaining in its file system directory and in each block metadata node the unique file name identifier as transmitted by the records management or file management application along with the date/time of recording.

• Storing and maintaining the unique record file name and date/time of recording when a duplicate copy of a record file is made.


There are no other considerations related to this requirement.

2.5 Store Separately a Duplicate Copy

Compliance Requirement 17a-4(f)(3)(iii)

Store separately from the original a duplicate copy of the record stored on any medium acceptable under 240.17a-4 for the time required.

The intent of this requirement is to provide an alternate storage source for accessing the record should the primary source be compromised.



Note: A “duplicate copy” is different from a backup copy in the sense that the duplicate copy is the recording of a real-time copy of the record or recording a one-for-one “journal” copy of the record that is never overwritten or erased. Backup copies, on the other hand, may be overwritten as they are “rotated” on a periodic basis.


Cohasset believes that the NetApp SnapLock Compliance Storage Solution capabilities comply with this SEC requirement.


The following are the NetApp SnapLock Compliance Storage Solution capabilities that support meeting this requirement of the Rule:

• The NetApp SnapLock Compliance Storage Solution software can be configured to automatically replicate the entire record file content and associated metadata nodes or the incremental portion of the record file content and associated metadata nodes of a NetApp SnapLock Compliance Storage Solution to another NetApp SnapLock Compliance Storage Solution, either locally or remotely configured.

• NetApp provides four replication and backup products or methods for performing the replication and backup function for a SnapLock Compliance Storage Solution: SnapMirror, SnapVault, FlexClone and the “vol copy” command.

• When a SnapLock Compliance Volume is replicated or backed up with these NetApp replication and backup products, the replicated copy, whether deduplicated or not, contains all of the metadata nodes that contain the integrity and retention protection codes and the content for each record file stored plus the SnapLock Compliance Clock software and setting. Replicating the SnapLock Compliance Clock allows the retention periods to be actively managed on the replicated copy.

• If an unrecoverable read error occurs on the primary NetApp SnapLock Compliance Storage Solution, the following options are available:

■ The system administrator can identify and restore the in-error records to a new storage location on a SnapLock Compliance Volume,

■ The entire duplicate copy of record files can be restored to a new SnapLock Compliance Volume,



■ the in-error records on the duplicate NetApp SnapLock Compliance Volume and restore them to the primary NetApp SnapLock Compliance Volume, or

■ The duplicate copy of the records stored on a second NetApp SnapLock Compliance Volume could be reconfigured as the primary file storage system. The reconfigured primary file storage system then would be replicated to another NetApp SnapLock Compliance Volume as the duplicate copy.


A member, broker or dealer also has the option to create an archival duplicate copy of the records by copying the record file content stored in a NetApp SnapLock Compliance Storage Solution to a storage media compliant with the SEC Rule.


Conclusions • 22

3. Conclusions

It is Cohasset Associates’ conclusion that the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities meet all of the requirements that are their direct responsibility as it relates to the recording, identification, retention protection and availability of electronic records as specified in SEC Rule 17a-4(f).

This Technical Assessment has addressed whether the NetApp SnapLock Compliance Storage Solution and NetApp Deduplication capabilities meet the requirements and conditions of SEC Rule 17a-4 (f).

Cohasset found the NetApp SnapLock Compliance Storage Solution:

• Provides the required non-rewriteable, non-erasable retention protection for the complete spectrum of broker-dealer records with a time-based retention – pursuant to Rules 17a-3 and 17a-4,

• Provides for verification of the accuracy and quality of the recording process, in conjunction with the capabilities of the latest magnetic disk technology and NetApp’s RAID DP.

• Preserves the unique file identifier transferred from the network file management application during the recording process and appends a date-of-recording thereby giving each record file a unique identification, and

• Incorporates additional protection mechanisms, such as an independent Compliance Clock, to ensure that the records are also protected from inadvertent or intentional administrative actions – actions that could lead to deletion or modification of any records in the file storage system during the specified retention period.

Where NetApp Deduplication capabilities are employed in conjunction with a SnapLock Compliance Storage Solution, it is Cohasset’s opinion that the deduplication methodology, particularly the byte-for-byte comparisons performed on each pair of potentially duplicate record blocks and the retention of all metadata reference pointers for all record blocks, does not compromise the requirement of this Rule for maintaining the non-erasable, non-rewriteable recording status of each record file.


Appendix A • 23

Appendix A - Deduplication Capabilities - An Overview

In the context of digital storage, data deduplication is a process for finding duplicate digital objects and storing only a single physical copy of those objects. The primary purpose for deduplication is to reduce the amount of storage space required to store digital data. Reducing objects that have been or may be stored multiple times to a single object reduces the storage space required and, thereby, increases the amount of space available to store more digital objects for longer periods of time. This can be a critical requirement for digital objects that either have a high potential for duplication (e.g. e-mail and attachments) or have a longer retention period.

The process of identifying duplicate objects typically involves calculating a hash digest or fingerprint (using a standard or proprietary algorithm or formula) of an object, then using that hash digest or fingerprint to compare to the hash digest or fingerprint of other objects. When the comparison results in the determination that duplicates exist, then a process is employed to reassign the metadata pointer of a second duplicate to the first duplicate and release the storage occupied by the second. Comparing hash digests or fingerprints to determine duplicates is deemed much faster than comparing the full object, particularly where the deduplication is done in-line or in real-time with the writing of the digital object to storage.

There are two notable data integrity aspects related to most deduplication processes, including the NetApp Deduplication capabilities:

• Deduplication processes, methodologies and algorithms are generally intrinsic to the storage system and, therefore, the ability of a human to influence the determination of which objects are duplicates and how to reassign the metadata pointers is virtually impossible – particularly with the very high volume of record files that are regularly being recorded.

• The metadata (object identifier or reference pointer to the physical data block, date recorded, retention period, etc.) associated with every object (prior to deduplication) is retained in order to be able to “piece together” all of the single-store physical objects required to reproduce a complete object or record file based on an external request for an object or record file.


Appendix A • 24

While there are a number of methodologies for performing deduplication, two are particularly notable:

• Catalog-based indexing that typically is combined with a post-storage deduplication process. This is the methodology that is employed in the NetApp Deduplication capabilities. Using this methodology, a hash digest or fingerprint is calculated during the recording process for every block (see footnote 2) of a digital object or record file stored and added to an aggregated catalog of block-level hash digests. Then a post-storage process is initiated that sorts the catalogue of hash digests and compares them to determine potential duplicate blocks. When potential duplicate blocks are found, a byte-for-byte comparison is performed to determine if indeed they are duplicates. Where a duplicate is determined, then the metadata reference pointer for the second block is redirected to the first block and the storage area for the second block is released back to the storage system. Because a byte-for-byte comparison is ultimately used to determine a duplicate, and in order to facilitate a higher object throughput at the time of recording, a smaller hash digest is generally employed (e.g. a 32 bit hash digest).

• Another method is Content-addressable storage (CAS). In CAS, the hash digest of a digital ob-ject generally is calculated in-line during the time of recording and immediately compared to other hash digests of digital objects stored. If a duplicate is found, then the metadata for the new digital object is stored and the pointer in the metadata is directed to the existing digital object. Because the initial hash digest calculation is the ultimate determinant regarding the po-tential existence of a duplicate, a much longer hash digest is utilized (e.g. 128 bit hash digest).

One of the risks of deduplication is that the hash digest process may produce the exact same result for objects that actually are not duplicates, i.e., while the resulting hash digest is the same, there are elements of the content that are different. It is generally considered that the stronger the hash digest algorithm employed (typically exemplified by a longer bit-count result, e.g., 128 bit result vs. a 32 bit), the lower the probability that a false duplicate will occur. One method to eliminate the possibility of a false duplicate occurring is the NetApp Deduplication approach whereby a byte-for-byte comparison is made of the data in each pair of blocks that are suspected of being duplicates.

While there are a number of other criteria that can be examined when determining the effectiveness and efficiency of deduplication processes, the primary criteria from a regulatory and records management perspective is to prevent an object that is not the duplicate of another object from being released to the storage system - for all intents and purposes, deleted. The NetApp Deduplication process addresses this criteria by requiring that a byte-for-byte comparison be performed for all digital object data blocks that are suspected of being duplicates (by virtue of having identical hash digests or fingerprints stored in the catalogue).

The major points of the NetApp Deduplication capabilities and the process that address the requirements of the SEC Rule are covered in Section 2.2


End Notes • 25

End Notes 1. The NetApp SnapLock Compliance Storage Solution currently supports only time-based retention (i.e., for a specified

period from the time after the file is recorded). Event-based retention (i.e., indefinite retention until a specified event occurs, followed by a specified retention period) is not currently supported.

2. The retention expiration period is contained in the “file last accessed time” or “atime” file protocol attribute that is sent with the record to the NetApp SnapLock Compliance Storage Solution by the records management or file management application.

3. A “block” is the unit of information that is used for the physical recording of data on most magnetic disk storage systems. Therefore, a record file typically consists of multiple blocks of information, e.g., a record that was 100,000 bytes in length would be recorded as 25 physical blocks of data assuming each block was 4,000 bytes in length. Associated with each block is a “node” of metadata that, among other attributes, contains a “pointer” to the associated physical block of data.

4. Redundant Array of Independent Disks (RAID): a method for recording data to magnetic disk devices that provides for various levels of error correction and read or write performance improvements. Data blocks are recorded (striped) across multiple physical magnetic disk units so that the failure of any stripe on a physical magnetic disk drive does not cause failure of a complete record file. RAID error correction parity information recorded in conjunction with each block of data allows for rebuilding of data within one or more blocks of data. More advanced RAID technologies such as RAID 4, RAID 6 and NetApp’s RAID DP (dual parity) utilize one or more parity disks which allow the rebuilding of in-error data for up to one or two physical disk drive devices.

5. Checksum (may also be referred to as a cyclic redundancy checksum or CRC): the sum of a group of data items, which sum is used for verification purposes. Checksums are used in detecting errors so that correcting mechanisms can be initiated.

6. ECC or error correcting code: codes generated during the recording process by the magnetic disk drive which are used to check and possibly correct any potential errors that may have been introduced into the data over time. ECC codes are calculated using an algorithm based on the content of the data being recorded to a sector of the magnetic disk. ECC codes differ from checksums in that the ECC is integral to the magnetic disk drive whereas a checksum is calculated based on the content of a block of data by the storage layer and sent to the magnetic disk with the block of data.


About Cohasset • 26

About Cohasset Associates, Inc.

Cohasset Associates, Inc. (www.cohasset.com), is one of the nation’s foremost consulting firms specializing in records and information management. Now in its fourth decade of serving clients throughout the United States, Cohasset Associates provides award-winning professional services in three areas: management consulting, education and legal research.

Management Consulting: The focus of Cohasset Associates’ consulting practice is improving the programs, processes and systems that manage document-based information. Cohasset works to provide its clients with cost-effective solutions that will both achieve their business objectives and meet their legal/regulatory responsibilities. This ranges from establishing effective corporate records management programs to planning state-of-the-art electronic records systems.

Education: Cohasset Associates is renowned for its longstanding leadership in records manage-ment education. Today, Cohasset’s educational work is centered on its annual National Conference for Managing Electronic Records (MER), which addresses the operational, technical and legal issues associated with managing the complete life cycle of electronic records (www.merconference.com). The MER sessions also are available at RIM on Demand. (www.rimeducation.com/videos/rimondemand.php)

Legal Research: Cohasset Associates is nationally respected for its leadership on records management legal issues – from retention schedules to the use of alternative media to paper for storing document-based information.

For more than twenty years, Cohasset Associates has been a “thought leader” in records and information management. Cohasset has been described as the only management consulting firm in its field with its feet in the trenches and its eye on the horizon. It is this blend of practical experience and a clear vision of the future that, combined with Cohasset Associates’ commitment to excellence, has resulted in Cohasset Associates’ extraordinary record of accomplishments and innovation.

This technical assessment and the information contained in it are copyrighted and are the sole property of Cohasset Associates, Inc. Selective references to the information and text of this technical assessment are welcome, provided such references have appropriate attributions and citations. Permission is granted for in-office reproduction so long as the contents are not edited and the “look and feel” of the reproduction is retained.

Documents

Abstract - NetApp Community · SEC 17a-4(f) Compliance Assessment Abstract ... 12 2.4 Serialize the ... Securities & Exchange Commission for 17a-4 Records