View
3
Download
0
Category
Preview:
Citation preview
AmitChourasia,DavidNadeau&MichaelNormanSanDiegoSupercomputerCenter,UCSanDiego
Projectcode: dibbs.seedme.org/downloads
or
drupal.org/projects/foldershare
drupal.org/projects/smalldata
Trialwebsite: sandbox.seedme.org
Projectwebsite:dibbs.seedme.org
FolderShare:BuildingadatasharingcloudonDrupal8forresearchers
AboutmeAmitChourasiaSanDiegoSupercomputerCenter@UCSanDiego• VisualizaJonscholar/evangelist• Lecturer/Instructor• Interests
– HighperformancecompuJng– Datastewardship– Computergraphics
Drupalusersince~2006|version4.7.4• Personalwebsite(Drupal4-8:150pages)• Projectwebsite(Drupal5:50,000+pages)• SeedMe2cloudservice(Drupal7:150,000+pages)• SeedMe2plaCorm(Drupal8:AnGcipate1M+items)
Seeking:PHPprogrammers
Desired:DeepknowledgeofDrupal8core
PresentaJonOverview1. Background&moJvaJon2. Architecture3. SeedMe2pla\orm
FolderShare:Virtualfilesystem» EnJtydatamodel&accesscontrol» Filemanagement&security» ViewsintegraJon» UI&Commandplugins» Fileforma_ers» Webservices
SmalldatamoduleandAPI– VisualizaJon
4. Targetusers/Usecases5. Screenshots6. Demo
SeedMe1a.k.a.StreamencodeexploreanddisseminateMyexperiments• BasedonDrupal7• InproducJonasPla\ormasaService(PaaS)• Videoencodingwasthemainfocus
SeedMe2:DatasharingbuildingblocksEvoluJonoftheoriginalSeedMeproject(Completerewrite)• BasedonDrupal8• IncorporatesuserfeedbackfromoriginalSeedMeproject• BuiltfordistribuJonandextension• Datasharinganddatamanagementisthemainfocus
SeedMeProject
SeedMe2’sfocus
EnablerapidaccesstodataConsumabledataCanbehandledbystockwebbrowser(Upload/Download,<2GBperfile)Displayableonmanydevices(PhonetoPC)
Datamanagementstumblingblocks
Transfer Storage CollaboraJon AutomaJonAccesscontrol
ButwhataboutPresentaJonandDiscovery?
Datamanagementstumblingblocks
Transfer Storage CollaboraJon AutomaJonAccesscontrol
Issuesduetocontentdispersion
DescripJoninsomeone’smindDataintheCloud Discussiononemails
ThreeD’s:Data,DescripGon,Discussion
FilesystembasedsoluGons
RelatedsoluJons
ContentmanagementsystembasedsoluGonsHubZeroFigShare
MiddlewareGlobusIRodsNEWT
SoZwarerepositoriesGitHubSVNCVS
FilehosGngClouddrives(Dropbox,etc)WebDAV
ToolsSCPFTP
LimitaGonsofexisGngsoluGons Lackextensibility Lacksupportforrichcontent(descripJon,discussion,etc…) Lackindependentdevelopersupport Lack3wayinteracJonviawebbrowser&commandline&API Resourcerestricted
Workflow
Update as desired
Description
CreateProject
Sharing
Add foldersView
SearchDownload
Web browser, Command line, REST or App
Sign In
1 2
3
4
Upload files
Users
Drupal8
Architecture
Modules• Virtualfilesystem
• Accesscontrol• Hierarchicalstorage• Commandplugins• UIanddisplay• Search/index• Webservices
• Fileforma_ers• QuickVisualizaJon
Webserver(Apache+PHP)
Database(MySQL)
Webbrowser
RESTclients
Commandline
Smalldata(PHPlibrary)
Drupal8ContributedModules
e.g.Federated
AuthenGcaGonviaOIDCmodule
ProjectcontribuJons
SeedMePla\ormEcosystemDrupal(ContentManagementSystem)• Widelyusedinindustry,academiaandgovernment(thirdmostpopularCMSonwebakerWordpress&Joomla)• Modulararchitecturewithlargeecosystem(over1,000contributedmodules)• Largedeveloper&supportcommunity(4,000contributorstocore+thousandsmore)• Securityadvisoryandupdatesforcoreandstablecontributed
moduleseverymonth• VersaJledeploymentopJons(personalhosJng,insJtuJonalhosJng,cloudhosJng)
FolderSharemodule• Requireddependencies(11-AllinDrupalcore)
– DateJme– Field– File– Image– Link– Media– OpJons– Text
• OpGonaldependencies(3-AllinDrupalcore)– Comment– Help– RESTfulwebservices
• HTTPbasicauthenJcaJon• SerializaJon
– Search
– Filter– System– User– Views
• Coremodulesrecommended– Texteditor– FieldUI(maybe)– ViewsUI(maybe)
• Contributedmodulesrecommended– Realname– RESTUI(maybe)– SmallData(forquickvisualizaGonofCSV,JSONfiles)
FolderSharemodule
• Virtualfilesystem(fieldable):– EnJtytype&API– Accesscontrols– Usagetracking– Views,displays,breadcrumbs,forms– Pluginsforfieldforma_ers,search,views,acJons,andqueueworkers
• Configurablebysites– e.g.Keywords,comments,flags,DOIs
• Extensiblebydevelopers
Files&folders
• Childrenpointtoparents– ParentIDsenablefastqueriesforallchildrenofafolder
– RootIDsenablefastqueriesforaccesscontrolsandbreadcrumbs
parenJdpointstoimmediateparent
rooJdpointstotopfolder
Abstractedfilestorage
FoldersexistinthedatabaseEveryuploadedfilehasaFileID(sequenGal)InternalfileorganizaJonandstoragebasedon16bitpa_ernBitpa_ern0000000000000000Physicalhierarchyas0000/0000/0000/0000/file_idEachunderlyingfolderstores9,999filesTotalfilehandling:65,535*9,999=655,284,465(~655million)
Accesscontrols
• Drupalaccount-based• Permissions+accesscontrollistontopfolders– Listofusersthatcanviewandauthor
• TopfoldercontrolsenJrehierarchy– SimplerthandesktopOSes– Similartofilesharingservices– Fasttocheckaccess
Filestorage
• Foldersonlyexistindatabase• Filesdescribedindatabase&storedondisk
• Diskdirectory!==folderhierarchy– Be_erforsecurityandloadbalancing– Fileshavegeneratednames
• Avoidscharactersetandnamelengthlimits
– Fileshavenoextensions• AvoidsaccidentalserverexecuJonof“.php”,etc.
Views
• Listpersonal,public,andsharedfiles&folders– Pages&embeddedviewsinfolderpages
• Integrateddesktop-likeUI– Selectfilesandfolders– Thenchoosemenucommand
• ThreeUIvariants:– NoscripJng– ScripJngbutnoAJAX– ScripJngwithAJAX
Plugins
• Fieldforma_ers– Foldernames,enJtyreferences,MIME-typeicons
• Search– Indexandpresentresults
• Queueworker– Updatefolderhierarchysizesinbackground
• AcJons&customcommands– MenuUIitemstoadd,delete,etc
TotallinesofphpcodeNode:25,639(Drupalcore8.5.0)Foldershare:50,156(Alpha1version)
0
20,000
40,000
60,000
80,000
100,000
120,000
140,000
v1 v2 v3 v4 v5 alpha
Line
sofcod
e
Release
Foldersharecodemain
PHP
Docs
0
2000
4000
6000
8000
10000
12000
v1 v2 v3 v4 v5 alpha
Line
sofcod
e
Release
Foldersharecodemisc
JS
YML
CSS
TWIG
TXT
Codetrivia
FoldershareAPIDocumentaJon
SmallDataAPI&Module
• Structureddataparsers&writers– Tables,trees,andgraphs– JSON,CSV,TSV,TXT,etc.
• Fieldforma_ers– Light-weightvisualizaJon– Lineplots,barcharts,piecharts,etc.
SmallData
FORADMINISTRATORS
FolderShareconfiguraGonAdminmenuStructure>FolderShare
FolderShareconfiguraGonFieldsManagefields,forms&display
ConfiguraJonlocatedinadminmenuStructure>FolderShare
FolderShareconfiguraGonFilesStoragelocaJon&uploadrestricJons
FolderShareconfiguraGonInterfaceCommandplugins
FolderShareconfiguraGonListsManagelisJngoffileandfolders
FolderShareconfiguraGonSearch(opGonal)
FolderShareconfiguraGonSecurityManagesharingcapabiliJes
FolderShareconfiguraGonWebservices(opGonal)ManageRESTcapabiliJes
FolderShareRESTseengsRequiresRESTUIcontributedmoduleManageRESToperaJons
FolderShareUsageAdminmenuReports>FolderShareusage
FolderSharepermissions
FORUSERS
Topfoldersownedbyyou
Topfolderssharedwithyou
Publictopfolders
Menu
Theselistsdisplaytoplevelfolders
FoldersmayhaveadescripJon
MenuopJons–withnoselecJon
MenuopJons–withselecJon
Sharingformtorestrictaccess
SortablelisJngoffilesandfolders.
Differentusers
Breadcrumbsshowspath
EveryfolderandfilemayaddadescripJon.(Forma_edtextfieldakaBodyfieldinDrupal’sNode)
AddcustomsfieldssuchasCommentstoFolderShare.
TheFolderShareenGtyisfieldable.
MenuopJonschangeonselecJon
ViewsubfolderBreadcrumbsshowspath
VisualizaJonscanbeswitchedinteracJvelytodifferentcharttypes
QuickvisualizaJonofCSV&JSONfiles
SamplecommandlineinteracJonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
SamplecommandlineinteracJonfoldershare --helpfoldershare --host http://demo.seedme.org --user dave --password ’cliRocks!'
help
ls --help ls / ls -l "/Classification Collection" ls -l "/Classification Collection/Preliminary Results"
mkdir --help mkdir "/test" mkdir "/test/data"
upload --help upload "plots/villi.png" "/test" upload "plots/composite.png" "/test” Upload "sample-small-data/OpenGL mesh memory use.csv" "/test/data" Upload "sample-small-data/Image classification breakdown schedule.json" "/test/data"
download --help download "/test" "/Users/amit/downloads"
rm --help rm -rf "/test”
SeedMe2–Targetusers/Usecases• Researchers
– CollaboraJonhuborpersonaldashboard• Projectrepositories
– IncludeprojectspecificcustomizaJon(e.g.taxonomy,keywords)• Developers
– IntegrateyourscienJficapplicaJon• Sciencegateways
– Datasharing– Datapublishing– Dataescrowservice
• CyberInfrastructureproviders– OfferSeedMepla\ormtoyouruserbase/communiJes
Realusagebyphysicists
Deployment/serviceopJonsDIY-Runowninstance(Yourbranding+yourdomain)• Onyourownhardware• CondohardwareProvider/vendorrunsaninstance• YourinsJtuJon• CIcenters• Commercialvendors
Asacloudservice(Ourbranding&ourdomain)• dibbs.seedme.org
Explorerswelcome(webbrowserneeded)Trialwebsite:sandbox.seedme.org
Tryityourself!
DevelopersinvitedDownloadcode:h_ps://dibbs.seedme.org/downloads
Team
AmitChourasia,DavidNadeau,DmitryMishin&MichaelNormanSanDiegoSupercomputerCenter|UniversityofCaliforniaSanDiego
SeedMe
AcknowledgementsAllusersandapplicaJonintegratorsfortheirvaluablefeedback
NaGonalScienceFoundaGon
ThismaterialisbaseduponworksupportedbytheNaJonalScienceFoundaJonunderGrantNo.ACI-1235505andACI-1443083
"Anyopinions,findings,andconclusionsorrecommendaJonsexpressedinthismaterialarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNaJonalScienceFoundaJon."
DEMO
Talk to usamit sdsc.edu
• Keen to learn about potential FolderShare uses cases in your work• Seeking: PHP programmers Desired: Deep knowledge of Drupal 8 core • Single sign-on: Checkout http://www.cilogon.org/drupal
Projectcode: dibbs.seedme.org/downloads
or
drupal.org/projects/foldershare
drupal.org/projects/smalldata
Trialwebsite: sandbox.seedme.org
Projectwebsite:dibbs.seedme.org
Recommended