AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego

Projectcode: dibbs.seedme.org/downloads

or

drupal.org/projects/foldershare

drupal.org/projects/smalldata

Trialwebsite: sandbox.seedme.org

Projectwebsite:dibbs.seedme.org

FolderShare:BuildingadatasharingcloudonDrupal8forresearchers

AboutmeAmitChourasiaSanDiegoSupercomputerCenter@UCSanDiego•  VisualizaJonscholar/evangelist•  Lecturer/Instructor•  Interests

–  HighperformancecompuJng–  Datastewardship–  Computergraphics

Drupalusersince~2006|version4.7.4•  Personalwebsite(Drupal4-8:150pages)•  Projectwebsite(Drupal5:50,000+pages)•  SeedMe2cloudservice(Drupal7:150,000+pages)•  SeedMe2plaCorm(Drupal8:AnGcipate1M+items)

Seeking:PHPprogrammers

Desired:DeepknowledgeofDrupal8core

PresentaJonOverview1.  Background&moJvaJon2.  Architecture3.  SeedMe2pla\orm

FolderShare:Virtualfilesystem»  EnJtydatamodel&accesscontrol»  Filemanagement&security»  ViewsintegraJon»  UI&Commandplugins»  Fileforma_ers»  Webservices

SmalldatamoduleandAPI–  VisualizaJon

4.  Targetusers/Usecases5.  Screenshots6.   Demo

SeedMe1a.k.a.StreamencodeexploreanddisseminateMyexperiments•  BasedonDrupal7•  InproducJonasPla\ormasaService(PaaS)•  Videoencodingwasthemainfocus

SeedMe2:DatasharingbuildingblocksEvoluJonoftheoriginalSeedMeproject(Completerewrite)•  BasedonDrupal8•  IncorporatesuserfeedbackfromoriginalSeedMeproject•  BuiltfordistribuJonandextension•  Datasharinganddatamanagementisthemainfocus

SeedMeProject

SeedMe2’sfocus

EnablerapidaccesstodataConsumabledataCanbehandledbystockwebbrowser(Upload/Download,<2GBperfile)Displayableonmanydevices(PhonetoPC)

Datamanagementstumblingblocks

Transfer Storage CollaboraJon AutomaJonAccesscontrol

ButwhataboutPresentaJonandDiscovery?

Datamanagementstumblingblocks

Transfer Storage CollaboraJon AutomaJonAccesscontrol

Issuesduetocontentdispersion

DescripJoninsomeone’smindDataintheCloud Discussiononemails

ThreeD’s:Data,DescripGon,Discussion

FilesystembasedsoluGons

RelatedsoluJons

ContentmanagementsystembasedsoluGonsHubZeroFigShare

MiddlewareGlobusIRodsNEWT

SoZwarerepositoriesGitHubSVNCVS

FilehosGngClouddrives(Dropbox,etc)WebDAV

ToolsSCPFTP

LimitaGonsofexisGngsoluGons Lackextensibility Lacksupportforrichcontent(descripJon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracJonviawebbrowser&commandline&API Resourcerestricted

Workflow

Update as desired

Description

CreateProject

Sharing

Add foldersView

SearchDownload

Web browser, Command line, REST or App

Sign In

1 2

3

4

Upload files

Users

Drupal8

Architecture

Modules•  Virtualfilesystem

•  Accesscontrol•  Hierarchicalstorage•  Commandplugins•  UIanddisplay•  Search/index•  Webservices

•  Fileforma_ers•  QuickVisualizaJon

Webserver(Apache+PHP)

Database(MySQL)

Webbrowser

RESTclients

Commandline

Smalldata(PHPlibrary)

Drupal8ContributedModules

e.g.Federated

AuthenGcaGonviaOIDCmodule

ProjectcontribuJons

SeedMePla\ormEcosystemDrupal(ContentManagementSystem)•  Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonwebakerWordpress&Joomla)•  Modulararchitecturewithlargeecosystem(over1,000contributedmodules)•  Largedeveloper&supportcommunity(4,000contributorstocore+thousandsmore)•  Securityadvisoryandupdatesforcoreandstablecontributed

moduleseverymonth•  VersaJledeploymentopJons(personalhosJng,insJtuJonalhosJng,cloudhosJng)

FolderSharemodule•  Requireddependencies(11-AllinDrupalcore)

–  DateJme–  Field–  File–  Image–  Link–  Media–  OpJons–  Text

•  OpGonaldependencies(3-AllinDrupalcore)–  Comment–  Help–  RESTfulwebservices

•  HTTPbasicauthenJcaJon•  SerializaJon

–  Search

–  Filter–  System–  User–  Views

•  Coremodulesrecommended–  Texteditor–  FieldUI(maybe)–  ViewsUI(maybe)

•  Contributedmodulesrecommended–  Realname–  RESTUI(maybe)–  SmallData(forquickvisualizaGonofCSV,JSONfiles)

FolderSharemodule

•  Virtualfilesystem(fieldable):–  EnJtytype&API– Accesscontrols– Usagetracking–  Views,displays,breadcrumbs,forms–  Pluginsforfieldforma_ers,search,views,acJons,andqueueworkers

•  Configurablebysites–  e.g.Keywords,comments,flags,DOIs

•  Extensiblebydevelopers

Files&folders

•  Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder

– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs

parenJdpointstoimmediateparent

rooJdpointstotopfolder

Abstractedfilestorage

FoldersexistinthedatabaseEveryuploadedfilehasaFileID(sequenGal)InternalfileorganizaJonandstoragebasedon16bitpa_ernBitpa_ern0000000000000000Physicalhierarchyas0000/0000/0000/0000/file_idEachunderlyingfolderstores9,999filesTotalfilehandling:65,535*9,999=655,284,465(~655million)

Accesscontrols

•  Drupalaccount-based•  Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor

•  TopfoldercontrolsenJrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess

Filestorage

•  Foldersonlyexistindatabase•  Filesdescribedindatabase&storedondisk

•  Diskdirectory!==folderhierarchy– Be_erforsecurityandloadbalancing– Fileshavegeneratednames

•  Avoidscharactersetandnamelengthlimits

– Fileshavenoextensions•  AvoidsaccidentalserverexecuJonof“.php”,etc.

Views

•  Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages

•  Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand

•  ThreeUIvariants:– NoscripJng– ScripJngbutnoAJAX– ScripJngwithAJAX

Plugins

•  Fieldforma_ers– Foldernames,enJtyreferences,MIME-typeicons

•  Search–  Indexandpresentresults

•  Queueworker– Updatefolderhierarchysizesinbackground

•  AcJons&customcommands– MenuUIitemstoadd,delete,etc

TotallinesofphpcodeNode:25,639(Drupalcore8.5.0)Foldershare:50,156(Alpha1version)

0

20,000

40,000

60,000

80,000

100,000

120,000

140,000

v1 v2 v3 v4 v5 alpha

Line

sofcod

e

Release

Foldersharecodemain

PHP

Docs

0

2000

4000

6000

8000

10000

12000

v1 v2 v3 v4 v5 alpha

Line

sofcod

e

Release

Foldersharecodemisc

JS

YML

CSS

TWIG

TXT

Codetrivia

FoldershareAPIDocumentaJon

SmallDataAPI&Module

•  Structureddataparsers&writers– Tables,trees,andgraphs–  JSON,CSV,TSV,TXT,etc.

•  Fieldforma_ers– Light-weightvisualizaJon– Lineplots,barcharts,piecharts,etc.

SmallData

FORADMINISTRATORS

FolderShareconfiguraGonAdminmenuStructure>FolderShare

FolderShareconfiguraGonFieldsManagefields,forms&display

ConfiguraJonlocatedinadminmenuStructure>FolderShare

FolderShareconfiguraGonFilesStoragelocaJon&uploadrestricJons

FolderShareconfiguraGonInterfaceCommandplugins

FolderShareconfiguraGonListsManagelisJngoffileandfolders

FolderShareconfiguraGonSearch(opGonal)

FolderShareconfiguraGonSecurityManagesharingcapabiliJes

FolderShareconfiguraGonWebservices(opGonal)ManageRESTcapabiliJes

FolderShareRESTseengsRequiresRESTUIcontributedmoduleManageRESToperaJons

FolderShareUsageAdminmenuReports>FolderShareusage

FolderSharepermissions

FORUSERS

Topfoldersownedbyyou

Topfolderssharedwithyou

Publictopfolders

Menu

Theselistsdisplaytoplevelfolders

FoldersmayhaveadescripJon

MenuopJons–withnoselecJon

MenuopJons–withselecJon

Sharingformtorestrictaccess

SortablelisJngoffilesandfolders.

Differentusers

Breadcrumbsshowspath

EveryfolderandfilemayaddadescripJon.(Forma_edtextfieldakaBodyfieldinDrupal’sNode)

AddcustomsfieldssuchasCommentstoFolderShare.

TheFolderShareenGtyisfieldable.

MenuopJonschangeonselecJon

ViewsubfolderBreadcrumbsshowspath

VisualizaJonscanbeswitchedinteracJvelytodifferentcharttypes

QuickvisualizaJonofCSV&JSONfiles

SamplecommandlineinteracJonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

SamplecommandlineinteracJonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'

help

ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"

mkdir --help mkdir "/test" mkdir "/test/data"

upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"

download --help download "/test" "/Users/amit/downloads"

rm --help rm -rf "/test”

SeedMe2–Targetusers/Usecases•  Researchers

–  CollaboraJonhuborpersonaldashboard•  Projectrepositories

–  IncludeprojectspecificcustomizaJon(e.g.taxonomy,keywords)•  Developers

–  IntegrateyourscienJficapplicaJon•  Sciencegateways

–  Datasharing–  Datapublishing–  Dataescrowservice

•  CyberInfrastructureproviders–  OfferSeedMepla\ormtoyouruserbase/communiJes

Realusagebyphysicists

Deployment/serviceopJonsDIY-Runowninstance(Yourbranding+yourdomain)•  Onyourownhardware•  CondohardwareProvider/vendorrunsaninstance•  YourinsJtuJon•  CIcenters•  Commercialvendors

Asacloudservice(Ourbranding&ourdomain)•  dibbs.seedme.org

Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org

Tryityourself!

DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads

Team

AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego

SeedMe

AcknowledgementsAllusersandapplicaJonintegratorsfortheirvaluablefeedback

NaGonalScienceFoundaGon

ThismaterialisbaseduponworksupportedbytheNaJonalScienceFoundaJonunderGrantNo.ACI-1235505andACI-1443083

"Anyopinions,findings,andconclusionsorrecommendaJonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaJonalScienceFoundaJon."

DEMO

Talk to usamit sdsc.edu

•  Keen to learn about potential FolderShare uses cases in your work•  Seeking: PHP programmers Desired: Deep knowledge of Drupal 8 core •  Single sign-on: Checkout http://www.cilogon.org/drupal

Projectcode: dibbs.seedme.org/downloads

or

drupal.org/projects/foldershare

drupal.org/projects/smalldata

Trialwebsite: sandbox.seedme.org

Projectwebsite:dibbs.seedme.org