Upload
andrey-moskalenko
View
22
Download
2
Embed Size (px)
Citation preview
U n i v e r s i t y L O G O
Testing and Developing Tools to Promote the Reproducibility of Computational
Research Andrey Moskalenko
Center for Theoretical and Computational Materials ScienceDaniel Wheeler | Faical Yannick P. Congo
Reproducible Research• Main Areas:
• Computational• Experimental
•Context of the Project• Simulation Management • Sumatra and CoRR• Benchmark Phase Field Problem• Conclusion
Table of Contents
U n i v e r s i t y L O G O
• Context of the Project
• Simulation Management • Sumatra and CoRR• Benchmark Phase Field Problem• Conclusion
Table of Contents
U n i v e r s i t y L O G O
Simulation Management The GoalComputational Research Now
U n i v e r s i t y L O G O
Current available tools
Workflow Tools
Wrapping Tools
Execution Control
Version Control
RobustCommand lineWeb integrationHighly collaborative
Not suitable for capturing execution context
Suitable for recording stable automated executions
Provides log, search and view of execution history
Capture entire simulation context
Version environmentsCollaborative
Not collaborative with current tools
Not robust or ubiquitous
Not suitable for log, search and view of history
Suitable for building pipelines of distinct tasks
Enables a clear division of tasks for non-experts
Black box design for each section of the pipeline
Monolithic in nature encouraging isolated ecosystem of tools
• Context of the Project• Simulation Management
• Sumatra and CoRR• Benchmark Phase Field Problem• Conclusion
Table of Contents
• Context of the Project• Simulation Management
• Sumatra and CoRR• Benchmark Phase Field Problem• Environment and Examples• Conclusion
Table of Contents• Context of the Project• Simulation Management
• Sumatra and CoRR• Benchmark Phase Field Problem • Conclusion
Table of Contents
U n i v e r s i t y L O G O
Sumatra and CoRR
- What is it good for?1
- What are the limitations?
U n i v e r s i t y L O G O
Ease of access
Record structureUser
interface
Sumatra and CoRR
- What is it good for?1
- What are the limitations?
- Autonomous- Local and cloud storage- Continuously recording- Compatible - click-and-run
2
Sumatra and CoRR
dt = 1Equation = f()while elapsed_time is less than desired_duration:
result1 = equation.solve(dt = dt, solver = LinearPCG)result2 = equation.solve(dt = small_dt, solver = LinearPCG)
if result1 does not meet tolerance * result2:decrease dt and solve againelse:increase dt and solve againExtract data
U n i v e r s i t y L O G O
EnvironmentWorkflow Definition Jupyter Notebook aka iPython Notebook
libraries GitHub Cluster
• Context of the Project• Simulation Management • Sumatra and CoRR
•Benchmark Phase Field Problem• Conclusion
Table of Contents
• Context of the Project• Simulation Management • Sumatra and CoRR
•Benchmark Phase Field Problem• Conclusion
Table of Contents
U n i v e r s i t y L O G O
Analysis – phase-field model
2 Test CoRR and Sumatra functionality
1 Performance evaluation
3 Results
1 Performance evaluation
U n i v e r s i t y L O G O
Analysis – phase-field model
Results
U n i v e r s i t y L O G O
Why is reproducibility a difficult task?
• Versions and updates• Legality• Hardware• Python libraries and dependencies • Time drain
U n i v e r s i t y L O G O
• Context of the Project• Simulation Management • Sumatra and CoRR• Benchmark Phase Field Problem
•Conclusion
Table of Contents
U n i v e r s i t y L O G O
Conclusion
2 Problem: CHiMaD benchmark problem Solution: CoRR
1 Could you reproduce our phase-field results?
3 More work to be done in both areas
U n i v e r s i t y L O G O
Acknowledgements
2 MML Thermodynamics and Kinetics group
1MentorsDaniel Wheeler, Ph.DFaical Yannick P. Congo, Ph.D
3 Anushka Dasgupta
4 All who made NIST SURF possible