Network of Excellence Benchmark

Carlos Oscar SorzanoTechn. Director I2PC

Natl. Center Biotechnology (CSIC)

NoE Benchmark

NoE Benchmark

• There was a defined task.• The community accepted the challenge.• The result of the challenge could have served

as a starting point to improve 3D angular orientation.

NoE Benchmark• The challenge did not arise from the

community: although the community accepted it.

NoE Benchmark• Evaluation was a black box: the evaluation

Figure of Merit was unknown a priori.

NoE Benchmark• There was not a clear interchange standard:

there was no way of testing that the submitted results where actually in the correct format, coordinate system, Euler signs, …

NoE Benchmark• There was a single dataset: there was no

exploration of different packages performing better under different circumstances.

NoE Benchmark• Egos were too high

NoE Benchmark• It was a single shot.

1891: 10.8s1906: 10.6s1911: 10.5s1924: 10.4s1930: 10.3s1936: 10.2s…2009: 9.58s

For future benchmarks• Well defined task.• Well defined interchange format.• Task proposed by the community.• Different datasets.• Transparent evaluation with pre-test

datasets.• Acceptance of competition as a healthy way

to improve.• Possibility to withdraw from competition.• Competitions should be repeated over time.