View
7
Download
0
Category
Preview:
Citation preview
Deliverable 13.12
The SIAM User Forum
Report
D r . L e o n H e m p e l
L a r s O s t e r m e i e r
T o b i a s S c h a a f
T e c h n i c a l U n i v e r s i t y B e r l i n
SIAM Security Impact Assessment
Measures
User forum report
Project number
261826
Call (part) identifier
FP7-Security-2010-1
Funding scheme Collaborative Project
1
TABLE OF CONTENTS
1. Introduction
1. Objectives 2
2. User forum outline 2
3. Summary of results 3
Feedback on: General 3
Feedback on: Functionalities of the AST 4
Feedback on: SMT Typology 5
Feedback on: Definitions of Roles 5
Feedback on: Questions 5
Feedback on: Reporting 6
4. Conclusion 7
5. Appendix 8
2
INTRODUCTION
1. OBJECTIVES
This report summarizes the results of the second user forum, conducted in Berlin on
October 31st 2013. After consulting the European Commission and leading
practitioners in the field of mass transportation, the SIAM consortium decided to
combine the four user forums into one international forum to enable in-depth
discussion within a heterogeneous group of international participants about the
assessment methodology, tools, and content developed by the SIAM consortium.
Presented was an alpha-version of the ‘SIAM Assessment Support Toolkit’ (SIAM
AST) which is based on the ‘SIAM Wireframe’ presented in the first round of user
fora in December 2012 and January 2013. The outcomes of these fora (D13.11)
enabled the adjustment of requirements by the potential users and the creation of
the SIAM AST. The toolkit, although in a stage of development, allowed the users to
interact with and inspect all phases and steps of the assessment support process and
simulate a full assessment. Any feature not fully available yet was indicated
sufficiently such that users could understand what this feature was intended to do.
In contrast to the first wireframe, it was now possible to present to the participants a
comprehensive set of assessment questions which the SIAM partners have been
developing within the course of the year. The objectives of the forum were to elicit
additional feedback on the revised toolkit features and to evaluate the assessment
questions.
2. User forum outline
The user forum was conducted at the “PC College” training facility in Berlin,
Germany. The facility had provided a local area network within which the SIAM AST
infrastructure could be simulated, by means of a local web application server
(provided by KU).
3
After a short introduction by the coordinator, the AST toolkit was presented by
Ronald Grau of Kingston University, covering a brief overview on the general ideas,
system architecture, and some of the important theoretical underpinnings of the
assessment support functionality.
Each participant was provided with a user account for the system and assigned a
specific role within the AST, fitting their professional expertise. It was then presented
a scenario that illustrated the security problem which induced the assessment
process. The user forum case featured an example scenario concerning an increased
need to detect unattended luggage and objects in a mass transportation facility. At
the heart of the proposed solution was a range of advanced computer vision
technology. This could be implemented into the existing CCTV system at the facility -
capable to recognise and track people and objects based on different algorithms
processing the CCTV footage. The technology provider, impersonated by Graeme
Jones of Kingston University, suggested a solution that would invite further
capabilities, with different implications on the technical requirements as well as on a
range of different trust and freedom infringement issues. After creating a new
assessment case with the toolkit and completing the configuration phase together
with the audience, the participants were asked to answer a carefully prepared set of
questions that were presented on their computer.1
3. Summary of results
Feedback on: General
During their trial with the AST, participants were asked to give their opinion on the
overall the structure and presentation of the tool. Because the AST and its
conceptual underpinnings were presented in detail, participants had no problem
understanding the methodology and structure of the tool. Some noted that without
the presentation they may have had more difficulties. This points towards a need to
provide sufficient documentation and help in the toolkit, such as an overview of the
assessment support process implemented in the toolkit2, some of the specific
1 For screenshots please see appendix 2 Configuration phase - assessment phase - reporting phase
4
concepts utilized3, as well as the overall assessment methodology applied. The latter
issue will be addressed in guidelines in the handbook being created in work package
12.
Furthermore experts asked for further definitions of terminology related to scientific
language as well as for the topics and aspects in the respective tasks. In general, the
tool was perceived as a bit unbalanced with regard to the content, as freedom
infringement-related questions were dominant in the set and not all filters were
working yet. As a result of this, it was suggested later on that an alternative way of
navigating through the questions could be provided by means of a graphical
interface which makes transparent the different semantic relations between topics
and tasks.
Feedback on: Functionalities of the AST
The AST shall contain further options and helpful tools to enable users to actually
find answers and use the AST according to its purpose. So far the toolkit which
contains methodologies that have been developed in other work packages of SIAM
(e.g. workshop organization, threat assessments) is located at the top bar of the AST.
The user forum participants saw the need to have the tools and advice present in the
context of each individual question.
One of the key functions of the AST is the possibility of delegating questions to other
AST participants. This function was introduced to make it possible for a user to “work
off” all posed questions, even if the knowledge of the current user is not sufficient
and the methodologies cannot give enough advice to this user to find suitable
answers. The user forum participants very much appreciated this function but
highlighted that it needs more information than just the names of the AST
participants at the delegation tool (currently, only the names of the other actors
were offered for selection as possible recipients of a delegated question). Some
users showed interest in the possibility to implement the internal messaging system
offered by the AST into their own organisation. It was debated whether providing
very detailed information about the other users would encourage AST participants to
3 Tasks – Topic –Aspect – Question
5
delegate questions more frequently if they do not suppose these in their field of
competence - rather than finding useful answers by themselves. This indeed would
reduce the purpose of the AST in increasing reflexivity in the assessment process.
The consortium and the user forum participants discussed that this could be
contained if delegated questions were somehow flagged to the assessment leader
and the reflexivity score reduced.
One more idea put forward was a tool that could arrange a meeting between all AST
participants, which highlighted the necessity of personal contact and coordination in
assessment processes in general.
Feedback on: SMT Typology
The SMT typology was not instantly obvious to some users. Understanding the
typology is critical, especially for the assessment leader, whose task it is to configure
an assessment case. Part of this task is to decompose the technological solution
considered in a case into its technological parts and to assign the appropriate SMT
classes, which then partially determine which assessment questions are provided to
the assessment participants. It was recognized that sufficient documentation and
training needs to be provided to the assessment leaders so that they can perform
this task.
Feedback on: Definitions of Roles
In the configuration phase the assessment leader invites other actors to participate
in the assessment process. To each participant one particular role will be allocated.
This will also determine which questions will be posed to that participant reflecting
his or her professional role and competences. In the current specification, the
consortium decided to allocate only one role to one participant, but practitioners
raised the question that in some companies, especially in SMEs, one person can have
multiple roles in the process. Furthermore, it became evident that a new role is
required to address the problem that some questions are too complex in the judicial
context. Hence, the role of the ‘lawyer’ will be added to target all legal questions
that cannot be answered by participants that have only a general judicial knowledge.
6
Feedback on: Questions
The overall opinion was, that sometimes questions were too complicated in the
formulation, or the level of information required to answer them was perceived too
high. This was often the case when questions about legal compliance were posed.
However, SIAM aims to increase reflexivity such that situations should be avoided
where users can simply ignore questions they do not want to deal with. A solution
for this problem is that the SIAM consortium will establish a set of broader questions
that will be posed to everyone and then subsequent question in respect of deeper
understanding will be assigned to particular roles. In the case of the legal questions
the already mentioned new role ‘Lawyer’ will be introduced. It became also clear
that some questions need further context information to enable users to find the
answers for those questions. However, testing the tool with users exposed that
answering questions with only yes or no will not produce the necessary knowledge.
As a solution, it was considered that available answer options may be enhanced with
an additional text field to justify or explain an answer given. The purpose of this is to
make users shift their answer habits from a normative towards evidence-based
reasoning. The toolkit must provide necessary assistance in the form of
supplementary information such as tools or methodological advices that help users
find answers. A further outcome of the user forum was that the questions must be
very clear in distinguishing between passengers and employees. The frameworks for
these two groups are too different.
Feedback on: Reporting
After the completion of the assessment phase it is the assessment leaders’ task to
summarize all the given answers into a coherent report. The SIAM AST provides a
comprehensive editing tool where the answers given by all participants are
presented, ordered by task, topic, and aspect. The assessment leader can then
formulate summary statements on the various issues which are then printed in the
assessment report. The user forum highlighted the importance that the assessment
leader should be given a guideline on how to summarize all the answers, in order to
make sure all issues are addressed appropriately. Assessment leaders have a
powerful function in the toolkit, because in the end they decide which opinions get
7
emphasized in the report. To balance that role, SIAM introduced an “Observer” AST
function. Users of this role are not directly participating in the assessment, however,
they are able to audit the reporting activities performed by the assessment leader.
There also must be further explanations on the reflexivity scores that indicate how
many divergence roles have come into play in the AST.
4. Conclusion
The second user forum presented a well structured alpha version of the AST. The
suggestions, criticisms and ideas given as feedback at the first user fora were used to
improve the toolkit substantially. Fortunately, the SIAM consortium was able to
acquire some of the participants from the last user fora and supplement them with
renowned experts from leading scientific institutions and other practitioners. This
was the first time where user not related to the project would test the tool and
evaluate its content. This provided valuable insight into their interaction with such a
tool and gave pointers towards issues where SIAM still has to be improved.
All adaptions will take place in the course of the coming months. SIAM will then
present an operational beta version of the AST at the ‘Computers, Privacy & Data
Protection’ (CDPD) international conference in Brussels on January 22nd, 2014.
8
5. Appendix
Screenshots of the Assessment Support Toolkit (AST) presented at the User forum
October 31st, 2013.
Login Screen
Case Dashboard
9
Establishment of a new case
10
11
12
Technology depiction
13
Actor involvement
14
Assessment phase
15
Recommended