Upload
others
View
3
Download
0
Embed Size (px)
Citation preview
5 years of DaiquiriLessons learned and further developments
Jochen Klar
Leibniz-Institute for Astrophysics Potsdam (AIP)
[email protected] / @jochenklar
1 / 15
DaiquiriA framework for the publication of scientific databases
Allows for highly customizable web applications
Based on a common easily maintainable code base
Separated into an app and the daiquiri library
Features:
SQL web interface to relational databases
User management and user registration work flow
Metadata management and access control
First impression in-browser plotting
Table download and file service
Employs VO protocols and standards
2 / 15
Public databases in astronomySloan Digital Sky Survey (SDSS) and Millennium simulation
Image credit: David Kirkby
3 / 15
Databases hosted @ AIPRAVE database query interface for DR3 (Siebert et al., 2011)
MultiDark Database (Riebe et al., 2011)
RAVE (2013), CosmoSim (2014), APPLAUSE (2015), Gaia@AIP (2016)
4 / 15
4 / 15
4 / 15
4 / 15
4 / 15
Daiquiri v1aka legacy-daiquiri (2012 - 2016)
Written in PHP using the Zend framework
Front-end first written using jQuery, later AngularJS 1 and Bootstrap 2
Using:
MySQL 5.5 as database server
PHP-SQL-Parser to parse queries
MySQL Query Job Queue for asynchronous jobs
paqu for parallel access to sharded databases (Cosmology, Gaia)
mysqldump-vo to create download files
Gearman to create download files asynchronous
5 / 15
Language and Frameworkthen: PHP and the Zend framework version 1.12
PHP: fast to get into, easy to deploy, very popular, but [insert rant here]!
Zend: insufficient ORM, no templating beyond PHP, security pitfalls, noecosystem
6 / 15
Language and Frameworkthen: PHP and the Zend framework version 1.12
PHP: fast to get into, easy to deploy, very popular, but [insert rant here]!
Zend: insufficient ORM, no templating beyond PHP, security pitfalls, noecosystem
now: Python and Django
Python: todays preferred scripting language, widely used in astronomy
Django: full MVC framework with everything included, huge community
Django REST framework: de-facto standard for REST interfaces in Django
django-allauth: local and social authentication, registration work flows
astropy: community python library for Astronomy
...
6 / 15
Front-endthen: Angular 1 and Bootstrap 2
AngularJS 1: JavaScript front-end framework, templates, 2-way binding
Bootstrap 2: CSS and JS for layout, forms, modals, buttons, navbars, ...
7 / 15
Front-endthen: Angular 1 and Bootstrap 2
AngularJS 1: JavaScript front-end framework, templates, 2-way binding
Bootstrap 2: CSS and JS for layout, forms, modals, buttons, navbars, ...
now: Angular 1 and Bootstrap 3
AngularJS 1: still awesome
Bootstrap 3: responsive layout, mobile friendly
7 / 15
Front-endthen: Angular 1 and Bootstrap 2
AngularJS 1: JavaScript front-end framework, templates, 2-way binding
Bootstrap 2: CSS and JS for layout, forms, modals, buttons, navbars, ...
now: Angular 1 and Bootstrap 3
AngularJS 1: still awesome
Bootstrap 3: responsive layout, mobile friendly
Front-end web development is evolving fast
While the number of ways to organize JavaScript is almost infinite, here are some toolsthat help you build single-page applications.
https://github.com/showcases/front-end-javascript-frameworks
7 / 15
Understanding queriesthen: PHP-SQL-Parser
a lot of custom code needed to get the necessary information from the query
no support for ADQL, therefor no support for TAP
8 / 15
Understanding queriesthen: PHP-SQL-Parser
a lot of custom code needed to get the necessary information from the query
no support for ADQL, therefor no support for TAP
now: queryparser based on Antlr
Antlr: parser generator for structured text or binary files
queryparser: Antlr generated python code to parse query strings
ADQL translator using ADQL grammar to translate to MySQL syntax
MySQL parser using MySQL grammar for parse a MySQL query
open source and available on GitHub and PyPI (python2, python3)
Using mysql_sphere to translate ADQL functions into MySQL
developed and maintained by Gal Matijevic (AIP)
8 / 15
Asynchronous Jobsthen: MySQL Query Job Queue
MySQL plugin implementing a job queue inside the database
using UDF like qqueue_addJob() and tables like mysql.qqueue_jobs
needs to be maintained along MySQL/MariaDB versions
9 / 15
Asynchronous Jobsthen: MySQL Query Job Queue
MySQL plugin implementing a job queue inside the database
using UDF like qqueue_addJob() and tables like mysql.qqueue_jobs
needs to be maintained along MySQL/MariaDB versions
now: Celery and RabbitMQ
Celery: asynchronous task queue queue in Python, widely adopted
RabbitMQ: message broker in Erlang
redis: in-memory data structure store for task results, can also be used forcaching
systemd: new init system for Linux, make it easy to deploy daemons
9 / 15
Downloading tablesthen: Gearman and mysqldump-vo
Gearman: to create download files asynchronous
mysqldump-vo: to create custom download formats:
MySQL dumps
CSV
VOTable, VOTable Binary, VOTable Binary2
10 / 15
Downloading tablesthen: Gearman and mysqldump-vo
Gearman: to create download files asynchronous
mysqldump-vo: to create custom download formats:
MySQL dumps
CSV
VOTable, VOTable Binary, VOTable Binary2
now: Celery and RabbitMQ and old-school unix pipes
mysqldump database_name table_name | some_magic > table_name.csv
10 / 15
Daiquiri v2aka django-daiquiri (since 2016)
Python (>= 2.7 and >= 3.4) using the Django framework (1.11)
Responsive front-end written in AngularJS 1 and Bootstrap 3
Using:
MariaDB 10.1
queryparser and Antlr to parse and translate queries
Celery, RabbitMQ, redis and systemd for asynchronous tasks (not only queries)
old-school unix pipes to create files using mysqldump
open source and available on GitHub and PyPI
11 / 15
Featuresimplemented
SQL query interface (with examples, job list, plotting, ...)
Customizable data query forms
Contact messages to the support staff
DOI integration and landing pages for databases and tables
File service and zip-archive creation
Registration and log-in using Oauth2 (facebook, twitter, GitHub, Google, ORCID)
WordPress as CMS for project presentation and documentation
VO protocols:
Data Access Layer Interface (DALI)
Table Access Protocol (TAP)
Universal Worker Service Pattern (UWS)
12 / 15
Featuresupcoming
File access, filtering and download for observatory archives
Cut-out service for images and datacubes
FITS tables download
Use of sharded databases (paqu v2)
Management of collaboration meetings
VO protocols:
Simple Image Access (SIA)
Simple Spectral Access (SSA)
Provenance Data Model
13 / 15
Try django-daiquiri!as a user
Development version of the Gaia@AIP Services: https://gaia.aip.de/dev
as a provider
README: https://github.com/aipescience/django-daiquiri/blob/master/README.rst
git clone https://github.com/aipescience/django-daiquiri-app appcd app; python3 -m venv env; source env/bin/activatepip install django-daiquiri mysqlclientcp config/settings/sample.local.py config/settings/local.pymkdir log download
./manage.py sqlcreate # shows commands for MariaDB
./manage.py migrate # creates database and tables
./manage.py migrate --database=tap # creates TAP_SCHEMA
./manage.py createsuperuser # creates admin user
./manage.py runserver # runs a development server
14 / 15