Data goodness

Mostly in black and white

By Dom

You must love your data!

• Lost data : • Current imaging data in BRIC cost ~£5.1M, just for scanning costs! (2011)

no research no publications

no jobs no PhDs!Sad Dom

• Look after your data! – It looks after you

• Happy Dom

Data Storage

• Home directories:– ISIS home, U Home

» Not for large amounts of imaging data

• Projects directory– ISIS, V: Big stuff goes here

• If you require large amounts of space – E.g. > 50 GB

– LET ME KNOW IN ADVANCE!

Server goodness

• Why is the server a good place to store data?

• Mirror and parity - some errors - data can be easily recovered

– BACKUPS:• Tape backups, daily - 1 month retention• if you have funding, processed data can be mirrored off site• raw data is always mirrored offsite (ECDF) by default

– Desktop PC's• not reliable - no mirroring, no parity - some errors - data is lost

(Often all of it)• Network backups often fail

– Machines turned off, Network busy– moving to a new system when I get time!

Data love• Curation: Do this as you work!

• Plan your data use

– Use meaningful folder names– Make 'README.txt' files with dates, names of students/employees

involved, references to software, scripts and versions, purpose of experiment/processing.

– Be tidy with your data - tidy up occasionally – Friday afternoon - quick tidy up– Big tidy up at end of experiment/ project/ phase/ year

• BE CAREFUL, don’t rush

• Data, spreadsheets, databases– Anonymisation– *** Repatriation keys***

Code and Scripts

• Coding:• Testing

– Make sure that the software you are using does exactly what you think it does!

» Check every step for every image!

– Do not use hard coded paths• Use versioning software (ECDF)

Safe data is Happy data!