GRAD 521, Research Data Management Winter 2014 – Lecture 4

Amanda L. Whitmire, Asst. Professor

Organizing Your Data

Lesson 4 OutlineFile-naming conventionsData organizationStandard operating procedures: keeping a [lab] notebook

File naming conventions

File-naming strategies (1)

1. BE CONSISTENT• Have conventions for naming:

(1) Directory structure(2) Folder names(3) File names

• Always include the same information (e.g. date and time)

• Retain the order of information(e.g. YYYYMMDD, not MMDDYYY )

File-naming strategies (2)2. BE DESCRIPTIVE• Try to keep file and folder names under 32 characters• Within reason, Include relevant information such as:

Unique identifier (ie. Project Name or Grant # in folder name) Project or research data name Conditions (Lab instrument, Solvent, Temperature, etc.) Run of experiment (sequential) Date (in file properties too) When using sequential numbering, make sure to use leading zeros to allow

for multi-digit versions. For example, a sequence of 1-10 should be numbered 01-10; a sequence of 1-100 should be numbered 001-010-100.

No special characters: & , * % # ; * ( ) ! @$ ^ ~ ' { } [ ] ? < > - + / Use only one period and before the file extension

(e.g. name_paper.doc NOT name.paper.doc OR name_paper..doc)

File naming conventions

Project_instrument_location_YYYYMMDDhhmmss_extra.ext

Index/grant conditions Leading zero!

s/n, variable Retain order

7

File naming strategies

Order by date:19550412_notes_MassObs.docx

19550412_questionnaire_MassObs.pdf

19631215_notes_Gorer.docx

19631215_questionnaire_Gorer.pdf

Order by subject:Gorer_notes_19631215.docx

Gorer_questionnaire_19631215.pdf

MassObs_notes_19550412.docx

MassObs_questionnaire_19550412.pdf

Order by type:Notes_Gorer_19631215.docx

Notes_MassObs_19550412.docx

Questionnaire_Gorer_19631215.pdf

Questionnaire_MassObs_19550412.pdf

Forced order with numbering:01_MassObs_questionnaire_19550412.pdf

02_MassObs_notes_19550412.docx

03_Gorer_questionnaire_19631215.pdf

04_Gorer_notes_19631215.docx

US standard MM/DD/YY DD/MM/YY YYYY-MM-DDJanuary 12, 2011 01/12/11 12/01/11 2011-01-12

February 15, 2011 02/15/11 15/02/11 2011-02-15March 27, 2011 03/27/11 27/03/11 2011-03-27April 11, 2011 04/11/11 11/04/11 2011-04-11May 20, 2011 05/20/11 20/05/11 2011-05-20June 6, 2011 06/06/11 06/06/11 2011-06-06July 18, 2011 07/18/11 18/07/11 2011-07-18

August 7, 2011 08/07/11 07/08/11 2011-08-07September 9, 2011 09/09/11 09/09/11 2011-09-09October 14, 2011 10/14/11 14/10/11 2011-10-14

November 24, 2011 11/24/11 24/11/11 2011-11-24December 18, 2011 12/18/11 18/12/11 2011-12-18

January 19, 2012 01/19/12 19/01/12 2012-01-19February 27, 2012 02/27/12 27/02/12 2012-02-27

March 15, 2012 03/15/12 15/03/12 2012-03-15April 27, 2012 04/27/12 27/04/12 2012-04-27May 11, 2012 05/11/12 11/05/12 2012-05-11June 17, 2012 06/17/12 17/06/12 2012-06-17July 16, 2012 07/16/12 16/07/12 2012-07-16

August 4, 2012 08/04/12 04/08/12 2012-08-04September 14, 2012 09/14/12 14/09/12 2012-09-14

October 26, 2012 10/26/12 26/10/12 2012-10-26November 26, 2012 11/26/12 26/11/12 2012-11-26December 8, 2012 12/08/12 08/12/12 2012-12-08January 22, 2013 01/22/13 22/01/13 2013-01-22

February 20, 2013 02/20/13 20/02/13 2013-02-20March 26, 2013 03/26/13 26/03/13 2013-03-26April 18, 2013 04/18/13 18/04/13 2013-04-18May 23, 2013 05/23/13 23/05/13 2013-05-23

Dates listed in order of collection

On using number orderin file names…

US standard MM-DD-YYYY DD-MM-YYYY YYYY-MM-DDJanuary 12, 2011 01-12-2011 12-01-2011 2011-01-12January 19, 2012 01-19-2012 19-01-2012 2012-01-19January 22, 2013 01-22-2013 22-01-2013 2013-01-22

February 15, 2011 02-15-2011 15-02-2011 2011-02-15February 20, 2013 02-20-2013 20-02-2013 2013-02-20February 27, 2012 02-27-2012 27-02-2012 2012-02-27

March 15, 2012 03-15-2012 15-03-2012 2012-03-15March 26, 2013 03-26-2013 26-03-2013 2013-03-26March 27, 2011 03-27-2011 27-03-2011 2011-03-27April 11, 2011 04-11-2011 11-04-2011 2011-04-11April 18, 2013 04-18-2013 18-04-2013 2013-04-18April 27, 2012 04-27-2012 27-04-2012 2012-04-27May 11, 2012 05-11-2012 11-05-2012 2012-05-11May 20, 2011 05-20-2011 20-05-2011 2011-05-20May 23, 2013 05-23-2013 23-05-2013 2013-05-23June 6, 2011 06-06-2011 06-06-2011 2011-06-06

June 17, 2012 06-17-2012 17-06-2012 2012-06-17June 26, 2013 06-26-2013 26-06-2013 2013-06-26July 16, 2012 07-16-2012 16-07-2012 2012-07-16July 18, 2011 07-18-2011 18-07-2011 2011-07-18July 22, 2013 07-22-2013 22-07-2013 2013-07-22

August 4, 2012 08-04-2012 04-08-2012 2012-08-04August 7, 2011 08-07-2011 07-08-2011 2011-08-07August 8, 2013 08-08-2013 08-08-2013 2013-08-08

September 9, 2011 09-09-2011 09-09-2011 2011-09-09September 14, 2012 09-14-2012 14-09-2012 2012-09-14September 19, 2013 09-19-2013 19-09-2013 2013-09-19

October 14, 2011 10-14-2011 14-10-2011 2011-10-14October 26, 2012 10-26-2012 26-10-2012 2012-10-26

On using number orderin file names…

If we sort by MM/DD/YY,dates are out of order.

US standard MM-DD-YYYY DD-MM-YYYY YYYY-MM-DDAugust 4, 2012 08-04-2012 04-08-2012 2012-08-04

June 6, 2011 06-06-2011 06-06-2011 2011-06-06August 7, 2011 08-07-2011 07-08-2011 2011-08-07August 8, 2013 08-08-2013 08-08-2013 2013-08-08

December 8, 2012 12-08-2012 08-12-2012 2012-12-08September 9, 2011 09-09-2011 09-09-2011 2011-09-09December 10, 2013 12-10-2013 10-12-2013 2013-12-10

April 11, 2011 04-11-2011 11-04-2011 2011-04-11May 11, 2012 05-11-2012 11-05-2012 2012-05-11

January 12, 2011 01-12-2011 12-01-2011 2011-01-12September 14, 2012 09-14-2012 14-09-2012 2012-09-14

October 14, 2011 10-14-2011 14-10-2011 2011-10-14February 15, 2011 02-15-2011 15-02-2011 2011-02-15

March 15, 2012 03-15-2012 15-03-2012 2012-03-15July 16, 2012 07-16-2012 16-07-2012 2012-07-16June 17, 2012 06-17-2012 17-06-2012 2012-06-17April 18, 2013 04-18-2013 18-04-2013 2013-04-18July 18, 2011 07-18-2011 18-07-2011 2011-07-18

December 18, 2011 12-18-2011 18-12-2011 2011-12-18January 19, 2012 01-19-2012 19-01-2012 2012-01-19

September 19, 2013 09-19-2013 19-09-2013 2013-09-19February 20, 2013 02-20-2013 20-02-2013 2013-02-20

May 20, 2011 05-20-2011 20-05-2011 2011-05-20January 22, 2013 01-22-2013 22-01-2013 2013-01-22

July 22, 2013 07-22-2013 22-07-2013 2013-07-22May 23, 2013 05-23-2013 23-05-2013 2013-05-23

November 24, 2011 11-24-2011 24-11-2011 2011-11-24March 26, 2013 03-26-2013 26-03-2013 2013-03-26June 26, 2013 06-26-2013 26-06-2013 2013-06-26

On using number orderin file names…

If we sort by DD/MM/YY,dates are out of order.

US standard MM-DD-YYYY DD-MM-YYYY YYYY-MM-DDJanuary 12, 2011 01-12-2011 12-01-2011 2011-01-12

February 15, 2011 02-15-2011 15-02-2011 2011-02-15March 27, 2011 03-27-2011 27-03-2011 2011-03-27April 11, 2011 04-11-2011 11-04-2011 2011-04-11May 20, 2011 05-20-2011 20-05-2011 2011-05-20June 6, 2011 06-06-2011 06-06-2011 2011-06-06July 18, 2011 07-18-2011 18-07-2011 2011-07-18

August 7, 2011 08-07-2011 07-08-2011 2011-08-07September 9, 2011 09-09-2011 09-09-2011 2011-09-09October 14, 2011 10-14-2011 14-10-2011 2011-10-14

November 24, 2011 11-24-2011 24-11-2011 2011-11-24December 18, 2011 12-18-2011 18-12-2011 2011-12-18

January 19, 2012 01-19-2012 19-01-2012 2012-01-19February 27, 2012 02-27-2012 27-02-2012 2012-02-27

March 15, 2012 03-15-2012 15-03-2012 2012-03-15April 27, 2012 04-27-2012 27-04-2012 2012-04-27May 11, 2012 05-11-2012 11-05-2012 2012-05-11June 17, 2012 06-17-2012 17-06-2012 2012-06-17July 16, 2012 07-16-2012 16-07-2012 2012-07-16

August 4, 2012 08-04-2012 04-08-2012 2012-08-04September 14, 2012 09-14-2012 14-09-2012 2012-09-14

October 26, 2012 10-26-2012 26-10-2012 2012-10-26November 26, 2012 11-26-2012 26-11-2012 2012-11-26December 8, 2012 12-08-2012 08-12-2012 2012-12-08January 22, 2013 01-22-2013 22-01-2013 2013-01-22

February 20, 2013 02-20-2013 20-02-2013 2013-02-20March 26, 2013 03-26-2013 26-03-2013 2013-03-26April 18, 2013 04-18-2013 18-04-2013 2013-04-18May 23, 2013 05-23-2013 23-05-2013 2013-05-23

On using number orderin file names…

If we sort by YY/MM/DD,dates are in order.

On usingleading zeroes in file names…

Pair & share – 10 minutes

5 minutes: generate potential file names for your data & document your strategy

3 minutes: share your strategy with a partner & hear what they came up with

2 minutes: volunteer to share your idea with the class

Project_instrument_location_YYYYMMDDhhmmss_extra.ext

Index/grant conditions Leading zero!

s/n, variable Retain order

FileOrganization

Data organization

AGU presentation

Class presentation

OS presentation

Presentations

Ocean Sciences Class AGU

be thoughtful

be consistent

document your approach

When naming & organizing your

files and folders…

Documenting your process

Why keep a notebook?

1. To provide yourself with a record of how your data were collected, organized and processed. Yes, you will forget if you don’t write it down.

2. To provide yourself with a central & physical location for protocols, results, and general happenings.

3. To encourage a thoughtful process. The act of writing things out forces you to explore and clarify ideas.

4. To enable continuity in the face of unexpected events.

5. To establish a legal and scientific provenance (historical record) of your work

What to use as a notebook?

1. A notebook with a stitched binding.

2. Don’t be temped by the “carbon copy” types.

3. A larger size is generally better, but preferences vary.

4. Better quality paper is more expensive, but is worth it.

5. A notebook with pre-numbered pages is also worth it.

6. The kind that lays flat when open is nice to have.

7. Electronic notebooks (ELNs) are hugely variable and often discipline-specific. Do your homework if you want to go that route.

What to write with?

PEN

What should go into it?

Everything that is directly

relevant to your work.

You tell me.

Suggested notebook content

• Notes on all discussions and thoughts on the experimental/research goals

• Protocols for experiments or for acquiring data (if using 2ndary data). Tape in printed versions if you want to.

• Provide justifications for method, source of data (how did they generate the data?).

• Annotate all calculations so that anyone could understand what you did. Remember units!

• Provide full details on experimental/research organisms/subjects.• If performing lab experiments w/reagents, list details like manufacturer,

batch, etc.• Record equipment details. • Capture field, laboratory, or interview conditions.

Suggested notebook content

• Your personal description may be relevant. Maybe. If it could be, write it down.

• Write the names of people who have assisted you, for future acknowledgement.

• Make quick drawings set-ups, locations, etc.• Take photos and paste them in.• Record file names of data, where they are stored and backed up. • Get into the habit of making daily entries – ALWAYS date your entries,

including YEAR.• If you record log data into a digital spreadsheet, record the event in your

notebook.• Detail your mistakes.• Use military time, or at least AM/PM

Suggested notebook content

• All entries should be un-obscured by attached graphs, data sheets, photographs, etc.

• When you include photographs, graphs, product labels, etc., use glue to cleanly and permanently attach all edges. Do not use tape.

• Avoid making entries that are unrelated to your project.• Include the full names and contact information for all collaborators• At the conclusion of your experiment, write or print out a full directory

of all electronic files that relate to your experiment.• Describe the locations of all computer media, data binders, seeds,

samples, etc.. • Typically, the laboratory notebook should stay in the laboratory where

the experiments were conducted (i.e., with your mentor or employer).• Digital cameras are really cheap, so if you do expensive, important

science I recommend that you have 1 camera per project. At the end of each work day the researcher should take a photograph of each page.

In summary:

Write downAll The Things

Homework will be posted on Blackboard after class

Due before class Tuesday

Write a summary (1-2 paragraphs) describing your current or expected

approach for documenting and organizing your research activities.

Here’s our brainstorm photo