Embed Size (px)
Citation preview
The Data Accessing and Processing Stack of J-TEXT Cloud Database
W. Zheng, Q. Liu, M. Zhang, K.X.Yu, K.H.W
State Key Laboratory of Advanced Electromagnetic Engineering and Technology in Huazhong University of Science and Technology, Wuhan 430074, ChinaSchool of Electrical and Electronic Engineering, Huazhong University of Science and Technology, Wuhan 430074, China
Contact us: [email protected]; [email protected]
2. Data accessing and processing flow in JCDB
1. JCDB API
IntroductionPurpose:
Access control
Friendly web scope
Batch processing
High performance
Modular design
Main Content:
1. JCDB API 2. Data flow in JCDB3. JCDB plugins4. JCDB Web API5. Conclusion
High performance
Local data (Raw data, engineering data,
calculation data) -> Access Control (Authorization)
-> Plugins (Processing read and write request) ->
API ( Low-level transactions )-> Storage System (
Database or File)
Extensive functionality
Remote data ( Phone or Web)-> Web API ( Web
request)-> Access Control (Authorization
&Authentication )-> Plugins ->API -> Storage
System
Low-level implementation includes metadata management, storage engine interfaces.CRUD: Creat, Read, Update, Delete and Batch process (Duplicate and Delete recursively )Data model: Experiment( like folders), Signal ( like flies), Payload ( transparent for users)Permission: Assign Permission, Authorization, Authentication.Owner | Group | OthersRead | Write | Modify <Tom, Read | Write>
MongoDB
Storage Engine
J-TEXT Experiment
Shot101
Shot 102
Preset
Diagz
Ip
DY
DX
PayLoad
Samples
PayLoad
Samples
PayLoad
Samples
PayLoad
Samples
PayLoad
Samples
PayLoad
Samples
……
3. JCDB Plugins
JCDB Core Service
JCDB API
IQueryPlugin
Storage Engine
IDataTypePlugin
Query Plugins
Data Type Plugins
Query Plugins
Data Type Plugins
Build Signal/Experiment queries
Build Storage Engine arguments
Entities & Queries
Samples
Storage Engine Interface
Users
Read/WriteRequests
PathQueryPlugin
/entity/path/?name=* #query all root nodes
/entity/path/exp1/IP #query node by path /exp1/IP
/entity/path/exp1?name=*
#query sub children under exp1
/entity/path/exp1?name=*&recursive=true
#query all children and sub-children under exp1
/entity/path/exp1?name=sig1 #query child named sig1
IDQueryPlugin
/entity/id/?name=*
/entity/id/{id}
/entity/id/{id}?name=*
/entity/id/{id}?name=*&r
ecursive=true
/entity/id/{id}?name=sig1
ExpressionPlugin
Signal1*Cos()+Signal2/lg5
FixedWaveDataTypePlugin
Partial Read & Write data
5. Conclusion Provide flexible and extendible metadata
management and search based on MongoDB
Cassandra powered parallel storage engine optimized for long pulse data acquisition.
Good parallel write performance. Customizable plugins and storage engines RESTful API and web GUI
4. JCDB WEB API
• RESTful API
• Responsive web design
• Streaming read and write
• Permission control
Phone↑
WEB →
←Functional area
