Interpretation of complex situations in a semantic-basedsurveillance framework

Carles Fernandez a,!, Pau Baiget a, Xavier Roca a, Jordi Gonzalez b

a Computer Vision Centre, Edifici O. Campus UAB, 08193 Bellaterra, Spainb Institut de Robotica i Informatica Ind. UPC– CSIC, 08028 Barcelona, Spain

a r t i c l e i n f o

Article history:Received 14 April 2008Accepted 29 April 2008

Keywords:Cognitive vision systemSituation analysisApplied ontologies

a b s t r a c t

The integration of cognitive capabilities in computer vision systems requires both toenable high semantic expressiveness and to deal with high computational costs as largeamounts of data are involved in the analysis. This contribution describes a cognitivevision system conceived to automatically provide high-level interpretations of complexreal-time situations in outdoor and indoor scenarios, and to eventually maintaincommunication with casual end users in multiple languages. The main contributionsare: (i) the design of an integrative multilevel architecture for cognitive surveillancepurposes; (ii) the proposal of a coherent taxonomy of knowledge to guide the process ofinterpretation, which leads to the conception of a situation-based ontology; (iii) the use ofsituational analysis for content detection and a progressive interpretation of semanticallyrich scenes, by managing incomplete or uncertain knowledge, and (iv) the use of such anontological background to enable multilingual capabilities and advanced end-userinterfaces. Experimental results are provided to show the feasibility of the proposedapproach.

& 2008 Elsevier B.V. All rights reserved.

1. Introduction

A notable evolution has occurred in the fields ofartificial intelligence and cognitive science during the lastyears [26]. Symbolic and connectionist approaches, tradi-tionally antagonistic views of intelligence, are currentlysharing common ground and seen as complementary forthe development of hybrid intelligent systems. Especiallyin modern cognitive science, interdisciplinarity plays afundamental role in the design of cognitive models,capable of reasoning and adapting under strong con-straints of uncertainty.

Cognitive systems, unlike traditional intelligent ma-chines, do not pursuit reasoning as an end in itself, nor tryto design generalized models or absolute truth. Instead,

they highlight the need to use situated frameworks toenable actions which are desirable in concrete, naturalcontexts and toward specific goals. These systems in-corporate plausible computational mechanisms whichapproximate human-like cognitive operations of percep-tion, reasoning, decision, learning, reaction, or commu-nication, in order to enhance the human capacity torecognize and interpret meaningful content in largecollections of information acquired from diverse sources.

We aim to design the integrative architecture of acognitive vision system (CVS), which extracts descriptionsof interpreted human behaviors and complex situationsfrom recorded video sequences. We focus on controlledscenarios of a restricted discourse domain, such aspedestrians interacting with vehicles and static objectsin inner-city scenarios, see Fig. 1. The CVS is thought tohave multilingual capabilities and an interface withseveral modalities of communication with external users,especially involving natural language (NL) interaction.

Contents lists available at ScienceDirect

journal homepage: www.elsevier.com/locate/image

Signal Processing: Image Communication

ARTICLE IN PRESS

0923-5965/$ - see front matter & 2008 Elsevier B.V. All rights reserved.doi:10.1016/j.image.2008.04.015

! Corresponding author. Tel.: +34 935813072; fax: +34935811670.E-mail address: perno@cvc.uab.es (C. Fernandez).

Such characteristics demand a series of considerationsto fulfill regarding the integration of different types ofknowledge: symbolic and non-symbolic representations;episodic and semantic content; metric-temporal andlinguistic-oriented predicates. This article discusses thedifferent modules implied in the semantic levels of theconsidered system, describing the alternatives which havebeen chosen to clarify the structure of the domainknowledge, and to devise effective cooperation betweenthe representation formalisms appearing.

This contribution enhances current research on se-mantics in multimedia services by proposing a model ofhigh-level architecture, which enables the integrationof semantic information, its interpretation in situateddomains, and the interaction with external end-users.This has been particularly applied to the analysis ofinformation received from multiple cameras (active andpassive) placed in the considered scenarios. Such anarchitecture is open to the addition of new sources, e.g.audio information.

The paper is organized as follows: next section reviewssome work related to the field of this project. Section 3structures the CVS by proposing a comprehensivemultimodal architecture. Following chapters are struc-tured according to the semantic expressivity reachedby the modules described in them: first, Section 4introduces the basic terminology of concepts selected forthe system and compiled into an ontology, which ispresented as a convenient tool for semantic integration.Different representational formalisms are chosen assuitable for the different types of knowledge implied inthe classification. Section 5 discusses the processesimplied at the conceptual level, which carry out tasksfor conceptualization, reasoning, and inference over thequantitative knowledge obtained from the visual sub-system. Section 6 describes how higher interpretationsare obtained, as a result of a situational analysis at thebehavioral level using contextual and intentionalmodels. After that, Section 7 gives a general descriptionof the user interaction level, which is designed to pro-vide a complete interface of communication with a

final user of the system. Section 8 shows experi-mental results for the main stages discussed, andSection 9 concludes the article by highlighting someimportant considerations and pointing out future lines ofwork.

2. Related work

The system conceived in this article especiallybuilds upon the work done by Nagel, who has activelyinvestigated for decades the field of CVS applied tovehicular traffic surveillance [17,2]. He tackles thehigh-level analysis of visual occurrences by means offuzzy logic inference engines, and derives the results tothe generation of NL textual descriptions. An enhance-ment of this original scheme has been developed byGonzalez, which enables to enlarge the domain of aCVS towards the analysis of general human behaviorsin image sequences, in what has been called humansequence evaluation (HSE) [9]. Our system builds upon thisframework.

The evaluation of human behaviors in image sequencesis a commonly required task for applications such assurveillance, content-based retrieval of documents, oradvanced interfaces related to cognitive fields. Never-theless, while low-level visual techniques have beenactively investigated for decades, high-level processinghas acquired significant attention in the field of visionespecially in the last years, as stated in [20]. Theautomatic analysis and description of temporal eventswas tackled many years ago by Marburger et al. [16], whoproposed a NL dialogue in German to retrieve informationabout traffic scenes. More recent methods to identifyhuman activities in video sequences have been reportedby Kojima et al. [13], which are based in concepthierarchies of actions to recognize interesting elementsand developments in a scene, particularly interactionsbetween people and objects. In [6], Buxton reviewsprogress in generative models for advanced CVS to explainactivities in dynamic scenes, observing applications suchas education, smart rooms, and also surveillance systems.The study of interactions among moving people andobjects is undertaken under statistical approaches forhigh-level attention and control. Most approaches do notemphasize the episodic properties of analyzed behaviors;instead, we define an independent stage to analyze theevolution of situations and their contextualization. Theintegration of asserted facts to interpret a certain situationis made using a fuzzy metric-temporal engine, whichtakes both geometric and temporal considerations intoaccount.

There have also been intense discussions about how tointerrelate the semantic information extracted from videosequences. The aceMedia integrated project intends tounify multimedia representations by applying ontology-based discourse structure and analysis to multimediaresources [14]. Crowley proposes some conceptual frame-works towards the understanding of observed humanactivity, including interactions [7]. He suggests a modelconsisting of a set of roles to be accomplished by entities;

ARTICLE IN PRESS

Fig. 1. Snapshot from the recordings on one scenario covered by thesystem. In this scene, a pedestrian (Agent 22) is stealing an object(Object 23) from another pedestrian (Agent 2).

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 555

specific configurations of interrelated entities playingroles conform the so-called situations. The EU ProjectActIPret uses semantic-driven techniques to automaticallydescribe and record activities of people handling tools inNL, by exploiting contextual information towards sym-bolic interpretation of spatiotemporal data [25]. Itsreasoning engine focuses on the coordination of visualprocesses to obtain generic primitives from contextualcontrol. The intelligent multimedia storytelling systemCONFUCIUS interprets NL inputs and automatically gen-erates 3D animation and speech [15]. Several methods forcategorizing eventive verbs are discussed, and the notionof visual valency is introduced as a semantic modelingtool. In [20], Park and Aggarwal discuss a method torepresent two-person interactions at a semantic level, alsoinvolving user-friendly NL descriptions. Human interac-tions are represented in terms of cause–effect (event)semantics between syntactical agent–motion–target tri-plets. The final mapping into verb phrases is based onsimultaneous and sequential recognitions of predefinedinteractions. Concerning the semantic mappings of NLsentences, it is also interesting to mention ProjectFrameNet [5], which has built a lexical resource forseveral specific languages such as English, Spanish, Ger-man, or Korean, aiming to list the acceptable semantic andsyntactic valences of each word in each of its contexts.However, our requirement for multilingual capabilitiesdemands a neutral semantic stage which holds languageindependency; this has been solved by defining a concepthierarchy which is not based on eventive verbs, but on thegenerality of situations, that allows structuring thedomain knowledge from a more comprehensive point ofview.

Current CVSs build on both purposive and reactive dataflows, which incorporate techniques from several visionand reasoning levels. Most authors agree that mechanismsfor the evaluation, gathering, integration and activeselection of these techniques are fundamental to attainrobust interpretation of dynamic information [25,10].These needs for coordination of contextual knowledgesuggests to single out specific stages for semanticmanipulation. Although many advanced surveillancesystems have adopted semantic-based approaches to facehigh-level issues related to abstraction and reasoning, theuse of ontologies at high levels of such systems is onlynow beginning to be adopted. Following these premises,the structure of the proposed system is based on amodular architecture, which allows both top-down andbottom-up flows of information, and has been designed tointegrate ontological resources for cooperation with thereasoning stage.

3. An architecture for CVS

The cognitive system presented in this article performsHSE and is built upon three disciplines, namely computervision, knowledge representation, and computationallinguistics. It follows the multilevel architecture forhuman-oriented evaluation of video sequences proposedin [9], known as HSE. Its modular scheme suggests a

bidirectional flow of communication between consecutivelayers, in order to:

(1) Evaluate the information at a certain cognitive stage,and generate new descriptions oriented to modules athigher levels (bottom-up data flow). The producedresults come after the analysis of knowledge validatedby lower levels, in combination with predefined goal-oriented models and the history of asserted facts up tothe moment.

(2) Send high-level productions and inferences back in areactive manner, to support low-level processing andguide mechanisms for evaluation (top-down dataflow).

This composed configuration results in an active chain ofcooperative interactions among the different modules forvisual, conceptual, and linguistic processing, see Fig. 2.Following the HSE scheme, we define several levels ofsensors and actuators: in the first place, the active sensorlevel (ASL) consists of a distributed layout of static andactive Pan–Tilt–Zoom cameras which constantly providethe vision system with image sequences from distinctviewpoints. At the image signal level (ISL), a segmentationprocess is performed, which allows to detect significantinformation within the image data. Three levels ofgranularity are considered for the detection of agenttrajectories, body postures, and facial expressions. Next,image features resulting from the segmentation processare manipulated by means of tracking procedures at thepicture domain level (PDL), which maintains the identifica-tion of targets. The information about detected features isforwarded at each time-step to the scene domain level(SDL), in which a supervisor process takes control over thetrackers and integrates data into a unique database ofquantitative information. At this point, the geometric datafor each detected agent over time are available within aground-plane representation of the controlled scenario.

The syntactical interoperability among the modules isaccomplished by using XML for data exchange purposes.In addition, XML schemas have been selected to providecontent control over the communication, yet granting thescalability and flexibility of this process. On the otherhand, the capabilities for semantic interoperability havebeen introduced into the system architecture by means ofan ontology, which restricts the validity and relationshipsof the involved semantic terms to the specific workingdomain. Next section describes this idea in more detail.

We proceed on the basis that visual and structuralinformation consisting of geometrical values are availableover time. Details about the extraction of visual informa-tion that we use can be found in [12,22]. Next sections willdeal in more detail with the high-level stages of thesystem, namely conceptual predicates (CPL), behaviorinterpretation (BIL), and user interaction (UIL) levels.

4. Knowledge representation

The main motivation for the use of ontologies is tocapture the knowledge involved in a certain domain of

ARTICLE IN PRESSC. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569556

interest, by specifying some conventions about the contentimplied by this domain. Ontologies are especially used inenvironments requiring to share, reuse, or interchangespecific knowledge among entities involved in differentlevels of manipulation of the information.

There exist many approaches for the ontologicalcategorization of visually perceived events. An extensivereview is done in [15], from which we remark CaseGrammar, Lexical Conceptual Structures, Thematic Proto-Roles, WordNet, Aspectual Classes, and Verb Classes,which focus on the use of eventive verbs as mainrepresentative elements for classifying types of occur-rences. As an extension, our approach relates eachsituation from an ontology with a set of required entities,which are classified depending on the thematic role theydevelop. The main advantage of this approach is anindependency of the particularities of verbs from aconcrete NL, thus facilitating addition of multiple lan-guages.

The design of the ontology for the described CVS hasbeen done putting especial effort on the definition of theknowledge base. Description logic allows us to structure thedomain of interest by means of concepts, designing sets ofobjects, and roles, denoting binary relations betweenconcept instances [3]. Specifically, our domain of interest isrepresented by a knowledge base K ! hT;Ai, whichcontains two different types of knowledge:

" A TBox T storing intensional knowledge, i.e. a set ofconcept definitions which classify the terminologicalinformation of the considered domain. In practice, wesplit the terminology into several TBoxes (i.e. taxo-nomies), according to the semantic nature of theparticipants for each set. Some of the main importantsets are Event/Situation-TBox (see Table 1), Entity-TBox, and Descriptor-TBox (see Table 2).

" An ABox A storing assertional knowledge, i.e. factualinformation concerning the world state and the set of

ARTICLE IN PRESS

Situation Analysis(STP, HLSP)

Basic relationsand inferences (STP)

Dialog Manager(HLSP)

Conceptual Planner (HLSP, STP)

BodyTracker

(XML/XSD)

Behavioral,Situational Models

(HLSP)

Conceptual DB

ASL

PDL

CPL

UIL

SDL

ISL

BIL

View Calibration and Tracker Integration(XML/XSD)

Face Tracker

(XML/XSD)

Agent Tracker

(XML/XSD)

Distributed PTZ-camera system

SpatiotemporalHuman and Scene

Models (STP)

MetricTemporal

DB

NL Generation(HLSP, NL)

Generation ofVirtual Environments

(HLSP, STP)

NL Understanding(NL, HLSP)

Final User

Ontological KB (TBoxes, ABox)

BodyDetection

Face Detection

Agent Detection

CO

NC

EPTU

AL

LEVE

LSU

SER

INTE

RAC

TIO

NLE

VEL

VISI

ON

LEV

ELS

Fig. 2. General architecture for the considered system. The knowledge representation formalism used by each module is enclosed in parenthesis. A deeperanalysis over these formalisms is done throughout the following sections.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 557

individuals which can be found in it. This extensionalknowledge will be first instantiated by reasoning andinference stages dealing with First-Order Logic,and then introduced into the relational databaseby means of concept assertions, e.g. pedestrian

(Agent1) and role assertions, e.g. enter(Agent2, Crosswalk).

The ontology language we use has been restricted to theSHIF family (a.k.a. DL-Lite), which offers conceptsatisfiability and ABox consistency to be log-space

computable, thus allowing the relational database tohandle in practice large amounts of data [1].

Talmy organizes conceptual material in a cognitivemanner by analyzing what he considers most crucialparameters in conception: space and time, motion andlocation, causation and force interaction, and attentionand viewpoint [24]. For him, semantic understandinginvolves the combination of these domains into anintegrated whole. Our classification of situations (i.e. theEvent/Situation-TBox, the central element in our ontology)agrees with these structuring domains: we organizesemantics in a linear fashion, ranging from structural

ARTICLE IN PRESS

Table 1Taxonomy containing some concepts from the Event/Situation-TBox

owl:Thing

Event/Situation

ContextualizedEvent

ObjectInteraction ceLeaveObj cePickUpObj

LocationInteraction ceAppear ceCross ceEnter ceExit ceGo ceOnLocation

GroupInteraction ceGrouped ceGrouping ceSplitting

AgentInteraction ceGoAfter ceFight

BehaviorInterpretation

bAbandonedObjbDangerOfRunoverbTheftbWaitForSomebodybWaitToCrossbYieldbChasebEscapebSearchForbUseMachine

Status

Action

aBendaHeadTurnaHit aKick aPunch aShoveaRunaSitaSquataStandaWalk

Activity

ActivityPedestrian ypAccelerate ypMove ypStop ypTurn ActivityVehicle yvAccelerate yyBrake yvSteer yvStop

Expression

eAngry eCurious eDisgusted eFrightened eHappy eImpatient eNormal eSad eSurprised

Table 2Taxonomies showing highlighted concepts from the Entity-Tbox (left) and the Descriptor-TBox (right)

Agent

Vehicle

NonStandardVehicle AnimalVehicle EmergencyVehicle Ambulance FireEngine PoliceCarStandardVehicle Bicycle Bus Motorbike RegularCar Tramway Truck Van

Pedestrian

CrowdPedestrianGroupSinglePedestrian Face Limbs Torso

Object

owl:Thing

Entity

GenericLocation Source Destination Locus Area ParticularLocation PedestrianCrosswalk Road Sidewalk WaitingArea Table VendingMachine

Location

PickableObjectScenarioObject

MovableObject

SpatialDescriptor QuantityDescriptor TemporalDescriptor

Descriptor

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569558

knowledge in vision processes (quantitative pose vectors)to uncertain, intentional knowledge based on attentionalfactors (high-level interpretations). It is structured asfollows, see Table 3:

" First, Pose Vectors are collections of detected staticconfigurations for the tracked elements, such aspositions or orientations at a given time-step. No classis created for them, since semantics is only present inform of structural information by means of quantita-tive values.

" The Status class contains metric-temporal knowledge,based on the information provided by the consideredtrackers: body, agent, and face. Its elements representdynamic interpretations of the spatial configurationsand trajectories of the agents. Some examples includeto detect that a pedestrian is turning left, or that a caris accelerating.

" The ContextualizedEvent class involves semantics at ahigher level, now considering interactions amongsemantic entities. This knowledge emerges after con-textualizing different sources of information, e.g. ‘sitdown’—‘bus stop’, or ‘wave hand’—‘open mouth’, thatallows for anticipation of events and reasoning ofcausation.

" Finally, the BehaviorInterpretation class specifies eventinterpretations with the greatest level of uncertaintyand the larger number of assumptions. Intentional andattentional factors are considered, here the detection ofremarkable behaviors in urban outdoor scenarios forsurveillance purposes.

This classification of knowledge will guide the process ofinterpretation. It can be seen that this proposal takes into

account all levels of extraction of visual informationwhich have been thought for the CVS—i.e. agent, body,face, and relation with other detected objects, agents, andevents—and also suggests a proper way of managing thedifferent stages of knowledge. This categorization con-siders the relevance of the retrieved information, somehierarchical degrees of perspective, and also the level ofsubjectiveness required for a scene interpretation, as willbe explained in the following sections.

5. Conceptual reasoning

The acquisition of visual information produces anextensive amount of geometric data, considering thatcomputer vision algorithms are applied continuously overthe recordings. Such a large collection of results turns outto be increasingly difficult to handle. Thus, a process ofabstraction is needed in order to extract and manage therelevant knowledge derived from the tracking processes.The question arises how these spatiotemporal develop-ments should be represented in terms of significance,also allowing further semantic interpretations. Severalrequirements have to be accomplished towards thisend [11]:

(1) Generally, the detected scene developments are onlyvalid for a certain time interval: the producedstatements must be updated and time-delimited.

(2) There is an intrinsic uncertainty derived from theestimation of quantities in image sequences (i.e.the sensory gap), due to the stochastic properties ofthe input signal, artifacts during the acquisitionprocesses, undetected events from the scene, or falsedetections.

ARTICLE IN PRESS

Table 3A knowledge-based classification of human motion representations

Dynamic interpretaton ofstatic elements

Disambiguation using information from other levels

Array of StaticConfigurationsover time( Pose Vectors )

Status

Contextualized Event

standing, moving head up and down

sequence of expressions

serious expr.:worry? /

concentration?

trajectory oflocations

stopped:sitting? /

standing?

standing infront of a

scene object

staring atscene object

nod? /search for

something?

sequence ofpostures, gestures

Description Examples for: Agent Body Face

Interpreted BehaviorHypothesis ofinterpretation using completescene knowledge

"A person is searching for information on the timetables of the subway station"

This terminology, which structures the Event/Situation-TBox, will guide the process of interpretation.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 559

(3) An abstraction step is necessary to obtain a formalrepresentation of the visual information retrievedfrom the scene.

(4) This representation has to allow different domains ofhuman knowledge, e.g. analysis of human or vehicularagents, posture recognition, or expression analysis, foran eventual semantic interpretation.

Fuzzy metric temporal horn logic (FMTHL) has beenconceived as a suitable mechanism to solve each of theaforementioned demands [23]. It is a rule-based inferenceengine in which conventional logic formalisms areextended by a temporal and a fuzzy component. This lastone enables to cope with uncertain or partial information,by allowing variables to have degrees of truth orfalsehood. The temporal component permits to representand reason about propositions qualified in terms oftime. These propositions are represented by means ofconceptual predicates, whose validity is evaluated at eachtime-step.

Fig. 4 shows a diagram of the different tasks managedby the inference engine. All sources of knowledge aretranslated into this logic predicate formalism for thesubsequent reasoning and inference stages. One of thesesources is given by the motion trackers in form of agentstatus vectors, which are converted into has_statusconceptual predicates [4]:

t ! has_status (Agent, X, Y, Theta, V) (1)

These predicates hold information for a global identifica-tion (instance id) of the agent #Agent$, his spatial locationin a ground-plane representation of the scenario #X;Y$,and his instantaneous orientation #Theta$ and velocity #V$.A has_status predicate is generated at each time-step foreach detected agent. In addition, certain atomic predicatesare generated for identifying the category of the agent, e.g.pedestrian(Agent) or vehicle(Agent). The resultingcategories are selected from primitives found in theEntity-TBox. Similarly, the segmented regions from the

scenario are also converted into logic descriptors holdingspatial characteristics, and semantic categories from theLocation-TBox are assigned to them:

point (14, 5, p42)

line (p42, p43, l42)

segment (l31, l42, lseg_31)

crosswalk_segment (lseg_31) (2)

As detected entities are automatically classified by themotion trackers, also assigning concepts from the Loca-tion-TBox to regions of the scenario can be well accom-plished in an automatic manner. Each instance holdsseries of semantic properties, being these elements fromthe ABox, which can relate the instance to a particularconcept after a classification process. Therefore, onlymethods for the obtention of semantic features arerequired, which can be based upon the analysis oftrajectories.

The abstraction process is thus applied over theinformation obtained both from the scenario and fromthe agents, i.e. the categorized segments from theconsidered location and the agent status vectors gener-ated. Quantitative values are converted into qualitativedescriptions in form of conceptual predicates, by addingfuzzy semantic parameters from the Descriptor-TBox suchas close, far, high, small, left, or right. The addition of fuzzydegrees allows to deal with the uncertainty associated tovisual acquisition processes, also stating the goodness ofthe conceptualization. Fig. 3 gives an example for theevaluation of a has_speed predicate from an assertedhas_status fact. The conversion from quantitative toqualitative knowledge is accomplished by incorporatingdomain-related models to the reasoning system [18].Hence, new inferences can be performed over an instan-taneous collection of conceptual facts, enabling the deri-vation of logical conclusions from the assumed evidence.Higher-level inferences progressively incorporate morecontextual information, i.e. relations with other detected

ARTICLE IN PRESS

always( has_speed(Agent,Value) :- has_status(Agent,X,Y,Theta,V) , associate_speed(V,Value)).

always( associate_speed(V,Value) :- degreeOfValidity(V,-0.83,-0.27,0.28,0.83), Value = zero ; degreeOfValidity(V,0.28,0.83,2.78,5.56), Value = small ; (...) degreeOfValidity(V,-16.67,-12.5,-5.56,-2.78), Value = normal ; degreeOfValidity(V,12.5,13.89,100.0,100.0), Value = high ; (...)

PREDEFINED LOGIC MODELS

has_speed(Pedestrian1, high).has_speed(Pedestrian3, zero) .

QUALITATIVE RESULT

QUANTITATIVE KNOWLEDGEhas_status(Pedestrian1, 0.320, 5.423, 0.344, 15.432). has_status(Pedestrian3, 6.655, 2.650, 1.971, 0.305) .

V=15.432 100% ’high’ 16% ’normal’

V=0.30585% ’zero’ 15% ’small’

0.280 0.83 12.5 16.713.9

V

Degree of Validity

zero normal highsmall

Fig. 3. Conversion from quantitative to qualitative values. (a) The input status vectors contain information for the agents. As a result, qualitativedescriptions are represented logically. (b) FMTHL includes fuzzy mechanisms accepting more than one single interpretation, since it confers degrees ofvalidity to values on uncertain ranges.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569560

entities in the scenario. This spatiotemporal universe ofbasic conceptual relations supplies the dynamic inter-pretations which are necessary for detecting events withinthe scene, as described in the taxonomy.

We refer to those predicates expressing uniquelyspatiotemporal developments as spatiotemporal predicates(STP). More specifically, STP facilitate a schematic repre-sentation of knowledge which is time-indexed andincorporates uncertainty. Hence, all those concepts inthe Event/Situation-TBox which can be inferred only usingthese constraints are enclosed under this category. STP donot consist only of atomic predicates, but they can beproduced after an interpretation based on temporal-geometric considerations. Next example shows FMTHLinference rules for the STP similar_direction(Agent,Agent2):

always(similar_direction(Agent, Agent2):-

has_status(Agent,_,_,_,Or1,_),has_status(Agent2,_,_,_,Or2,_),Dif1 is Or1 - Or2,

Dif2 is Or2 - Or1,

maximum(Dif1, Dif2, MaxDif),

MaxDif o 30 ).

6. Behavior interpretation

An independent stage is implemented for achievingeffective modeling of behaviors and complex situations.The concurrence of hundreds of conceptual predicatesmakes necessary to think of a separate module to dealwith new semantic properties at a higher level: someguidelines are needed to establish relations of cause,effect, precedence, grouping, interaction, and in generalany reasoning performed with time-constrained informa-tion at multiple levels of analysis, i.e. the contextualiza-tion and interpretation proposed in Section 4.

We introduce the concept of high-level semanticpredicates (HLSP) as those which express semantic rela-tions among entities, at a higher level than metric-temporal relations. They result from applying situationalmodels over STP. These new constraints embed restric-tions based upon contextualization, integration, and inter-pretation tasks. Hence, the set of HLSP reaches the highest

account of semantics, in the cognitive sense that each oneof them implies a perceived situation or behavior which ismeaningful and remarkable by itself in the selecteddomain. HLSP have been chosen as central elements inthe semantic environment of the CVS, for them beinglanguage-independent and suitable for a neutral frame-work between vision and linguistics.

The tool which has been chosen to enable behaviormodeling of the HLSP is the situation graph tree (SGT), see[2,9]. The SGT is a hierarchical classification tool used todescribe behavior of agents in terms of situations they canbe in. These trees contain a priori knowledge about theadmissible sequences of occurrences in a defined domain.Basing on deterministic models built upon elements of theontology, they explicitly represent and combine thespecialization, temporal, and semantic relationships ofthe conceptual facts which have been asserted.

The semantic knowledge related to any agent at a givenpoint of time is contained in a situation scheme, whichconstitutes the basic component of a SGT, see Fig. 5. Asituation scheme can be seen as a semantic function thatevaluates an input consisting of the conjunction of a set ofconditions—the so-called state predicates—and generateslogic outputs at a higher level—the action predicates—once all the conditions are asserted. Here, the actionpredicate is a note method which generates a semanticannotation in a language-oriented form, containing fieldsrelated to thematic roles such as Agent, Object or Location,which refer to participants of the Entities-TBox in theontology.

ARTICLE IN PRESS

Spatiotemporal / Structural analysis Contextual / Behavioral analysis

Temporally-validfacts

Always-valid facts(prior knowledge)

ConceptualInferences Linguistic-oriented

high-level predicates

Action-orientedhigh-level predicates

SceneModel

AgentStatus

Conceptualdatabase(FMTHL)

SituationGraph Tree(From

motiontrackers)

Graph traversal(Conversion to FMTHL)

(NL generation)

(Generation of virtual environments)

Fig. 4. The FMTHL inference engine manages (i) the conceptualization step for the motion data, (ii) the assertion of new facts given a series of atomicpredicates, and (iii) the application of situational models from the BIL.

similar_direction(Agent,Agent2)has_speed(Agent, high)has_speed(Agent2, high)

note (chase(Agent, Agent2))

SITUATION_ID

FMTL predicates

High-Level Semantic Predicate

1

Fig. 5. Situation scheme from a SGT. In this example, a set of conditionsin form of STP enable the generation of a HLSP, encoded in form ofreaction predicates.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 561

On the other hand, the temporal dimension of thesituation analysis problem is also tackled by the SGT. Asseen in Fig. 6, the situation schemes are distributed alongthe tree-like structure by means of three possibledirectional connections, the particularization, prediction,and self-prediction edges. Particularization edges allow toinstantiate more specific situations once the conditions ofa general situation have been accomplished. On the otherhand, prediction edges inform about the followingadmissible states within a situation graph from a givenstate, including the maintenance of the current state bymeans of self-prediction edges. Thus, the conjunction ofthese edges allow defining a map of admissible pathsthrough the set of considered situations. A part of a basic

SGT is shown in Fig. 7, which illustrates a model toidentify situations such as an abandoned object or a theft.

As previously shown in Fig. 4, the behavioral modelencoded into a SGT is traversed and converted into logicalpredicates, for automatic exploitation of its situationschemes. Once the asserted spatiotemporal results arelogically classified by the SGT, the most specializedapplication-oriented predicates are generated as a result.These resulting HLSP predicates are indexed with thetemporal interval in which they have been the persistentoutput of the situational analysis stage. As a result, thewhole sequence is split in time-intervals defined by thesesemantic tags. These intervals are individually cohesiveregarding their content.

ARTICLE IN PRESS

Specialization edge

Prediction edge

Self-prediction edge

Situation graph

Situation scheme

2has_speed (Agent, zero)

note( stopped (Agent))

AgentStopped

1

is_active (Agent)

NO_ACTION_PREDICATES

AgentInScene

1

has_speed (Agent, Vel)

NO_ACTION_PREDICATES

AgentMoving

1

has_speed (Agent, high)

note( moving_fast (Agent))

AgentMovingHighVel

1

is_turning (Agent, Direction)

note( turning (Agent, Direction))

AgentTurning

1

2

2

2

Fig. 6. A naive example of a SGT, depicting its main components. Specialization edges allow particularizing a general situation scheme with one of thesituations within its child situation graph, in the case that more information is available. Prediction edges indicate the situations available from thecurrent state for the following time-step; in particular, self-prediction edges hold a persistent state.

Fig. 7. This situation graph is evaluated when the system detects that an object has been left by the pedestrian who owns it. The set of conditions areFMTHL predicates, the reaction predicate is a note command which generates a high-level semantic tag.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569562

By describing situations as a conjunction of low-levelconditions, and interrelating those situations among themusing prediction and specialization edges, the contextua-lization stage described in the taxonomy of situations isaccomplished. On the other hand, since the HLSP actionpredicates are modeled depending on the application, aparticular attentional factor is established over theuniverse of occurrences, which can be understood as theinterpretation of a line of behaviors, for a concrete domainand towards a specific goal.

The results obtained from the behavioral level, i.e. theannotations generated by the situational analysis of anagent, are actually outputs of a process for contentdetection. From this point of view, a SGT would containthe classified collection of all possible domain-relatedsemantic tags to be assigned to a video sequence. Inaddition, the temporal segmentation of video is alsoachieved: since each HLSP is associated with the temporalinterval during which it has been generated, a videosequence can be split into the time-intervals which hold apermanent semantic tag. Some experimental resultsregarding situational analysis are presented in Section 8.

7. User interaction

A fundamental objective of cognitive systems is toachieve effective human–machine interaction, in order toenhance human capability and productivity across a largecollection of endeavors related to a definite domain.Toward this end, several interfaces of communicationare required, which make knowledge accessible toexternal users and permit them to take control over theimplicit operations at each level. Ontologies play afundamental role, especially regarding NL understandingtasks, since they restrict the domain of validity of thelinguistic queries and organize the knowledge to achievean effective process of information retrieval.

In the case of the CVS discussed here, the multilingualinterface of communication with a final user has beenconceived to support two types of human–machineinteractions:

(1) Generation of textual descriptions in NL, in a multi-lingual environment.

(2) Understanding of user queries written in NL, also formultiple languages.

NL generation has been seen as a process of choice,whereas NL understanding is best qualified as one ofmanagement of hypothesis towards reaching the mostprobable interpretation of a linguistic input [21]. Adetailed description of the multilingual NL generationmodule for the system is presented in [8].

Regarding the NL understanding module, it has beenconsidered convenient for it to enable communicationwith an external user at different levels of interaction:

" Answering questions about previous or current devel-opments detected in a scene, such as ‘‘How manypedestrians are there in the crosswalk at the moment?’’,

or ‘‘Please tell me about any potentially dangerousbehaviors occurred within the last 3 h’’.

" Carrying out specific commands to control and guidethe system. A primary interest for the CVS is to act overits set of active cameras, so that the regions of interestcan be focused and best exploited. Accepting sentenceslike ‘‘Zoom in on the last pedestrian’’ is interesting forthis purpose. However, more general directives shouldbe possible, such as ‘‘Concentrate only on the peopleappearing by the right side’’.

" The user also has the possibility to enhance or modifythe employed knowledge base, e.g. by including newinformation about the scenario or renaming identifiers.

The use of ontological semantics, and in particular theapplication of DL reasoning tasks over the relationaldatabase, has been proven to be an efficient solutionwhen dealing with large amounts of data. Specifically,instance checking capabilities permit to perform tasks suchas checking out whether all defined concepts admitindividuals (consistency), finding the most specific con-cept an individual is instance of (realization), or retrievingthe list of all individuals which instantiate a particularconcept (retrieval), among others [3].

The ontology reduces the domain of validity under-taken by the universe of user queries, and makes itpossible to restrict them to a reduced space of situations.By asserting concepts from the knowledge base, theinstantiated elements from the ontology are related toparticular knowledge sources, such as the factual databaseand the onomasticon [19], see Fig. 8. The factual databasemakes the information asserted at the ABox permanentlyavailable, so that an awareness of the occurred facts overtime can be accessed as needed. The onomasticon consistsof a repository of names, used to relate the instances ofconcepts from the ontology to the most suitable identi-fiers or referring expressions, keeping in mind to maintainthe coherence of the semantic discourse. The onomasticondirectly points to elements of the factual database, forinstance stating that Agent4 was the individual from theAgent class (defined in the ontology) which participatedin a chase occurrence, instantiated in the factualdatabase from frames 1241 to 1250 of an outdoor scenevideo sequence.

Fig. 9 shows a directed scheme of the process designedfor the conversion from NL user queries to specific actionsto carry out. First, the NL understanding module parsesthe NL query basing uniquely on linguistic models. The apriori information from the scene, i.e. elements of thescenario and types of events or situations expected, isparticularly provided by a multilingual lexicon. After that,the Dialog Manager module selects a suitable goal queryby means of a pattern matching operation over thedisambiguated tree structure resulting from parsing. Thelist of currently admissible goal queries for the system ispresented in Table 4. The resulting goal query is forwardedto a Conceptual Reasoner, which performs a search overthe factual database, aligns the information, and decidesthe action to perform according to the plan associated toeach goal. The possible final actions include presenting

ARTICLE IN PRESSC. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 563

ARTICLE IN PRESS

Semanticknowledge

Episodicknowledge

entitiessituations/events

descriptors

ONTOLOGY

FACTUALDATABASE

(...)

tonomasticon

lexicon

Fig. 8. When identifying a situation, the links among the different participants of the ontology, fundamentally entities, occurrences, and descriptors areinspected. Entries from the onomasticon and the lexicons are identified as afore known entities (such as names for particular locations or for agentcategories). Instantiated situations are collected into the factual database for episodic awareness and remembering.

Multilingual Lexicon

A priori scene knowledge:Situations(S), Locations(L)

Onomasticon

Instantiated scene elements:Agents/Objects (A), Time (T)

Ontology

FactualDatabase

NL query

Goal query: GoalType(T,S,A,L)

Disambiguated tree structure

NL Understanding(Parsing)

Dialog Manager(Goal selection)

EXTERNAL USER

Goal-oriented action

Plan Library<Goal, Plan>

Episodic searchConceptual Planner(Action planning

for goals)

Instantiation

System DBsVision SystemNLG Module

Fig. 9. Scheme of the process designed to transform NL user queries in goal-oriented actions.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569564

ARTICLE IN PRESS

Fig. 10. Set of semantic annotations produced for the outdoor scene, which have been automatically generated for the fragment of recording comprisedbetween frames 450 and 1301.

Table 4List of admissible goal queries

QUERIES

COMMANDS

KNOWLEDGE BASEMODIFICATION

add { T, S, A, L }remove { (T), (S), (A), (L) }rename { A }

list { (T), A }list { (T), S, (A), (L) }assert { (T), S, (A), (L) }query { T?, (S), (A), (L) }query { (T), S?, (A), (L) }query { (T), (S), A?, (L) }query { (T), (S), (A), L? }

restrict { A }restrict { L }restrict { (T), S, (A), (L) }show { (T), (S) }

"How many thefts have been seen?"list { S=bTheft }

"Has there been any abandoned object?"assert { S=bAbandonedObject }

"What is the last person doing in the table?"query { S=?, A=Agent8, L=tables }

"When was the machine kicked?"query { T=?, S=aKick, L=VendingMachine }

"Concentrate on pedestrians appearing from the right part"restrict { S=ceAppear, A=SinglePedestrian, L=right }

"Show me persons appearing in the last 500 frames"show { T=(668,1168), S=ceAppear, A=SinglePedestrian }

"An object was abandoned on frame 1300"add { T=(1300,1300), S=bAbandonedObject }

Type of query Admissible syntax

In the current implementation, a goal query reduces the candidates in a semantic frame to a four-tuple consistent of Time (T), Situation (S), Agent/Object(A), and Location (L). Some examples for goal interpretations of NL input queries are shown, too.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 565

information to the user by means of the NL generatormodule, forwarding specific actions to the active visionsystem, or modifying current information in the systemdatabases.

8. Experimental results

Figs. 10 and 11 show current experimental results, inwhich a collection of HLSP have been successfullygenerated for sequences recorded in outdoor and indoorsurveilled scenarios, respectively.1 The collection of HLSPdescribes interactions among the involved entities, viz.agents, objects, and locations, and also interpretations ofbehaviors in the case of complex occurrences. Somecaptures showing the results after tracking processes havebeen provided, too, for illustration purposes. The numberof frame appears in front of each produced annotation,and also in the upper-right corner of each capture.

Detections of new agents within the scene have beenmarked in blue, annotations for activating predefinedalerts have been emphasized in red.

The outdoor scene was recorded with four staticcameras and one active camera. The video sequencecontains 1611 frames (107 s) of 720% 576 pixels, in whichpedestrians, pickable objects, and vehicular traffic areinvolved and interrelated in a pedestrian crossing scenar-io. A total of three persons, two bags, and two cars appearon it. The events detected within the scene range fromsimple agents entering and leaving the scenario tointerpretations of behaviors, such as objects beingabandoned in the scene, a danger of runover between avehicle and two pedestrians, or a chasing scene betweentwo pedestrians.

The indoor scene was also recorded with four staticcameras and one active camera. The scene contains 2005frames (134 s) of 1392% 1040 pixels, in which threepedestrians and two objects are shown interrelatingamong them and with the elements of a cafeteria, e.g. avending machine, chairs, and tables. The events instan-tiated in this case include again agents appearing andleaving, changes of position among the different regions of

ARTICLE IN PRESS

Fig. 11. Set of semantic annotations produced for the indoor scene, which have been automatically generated for the fragment of recording comprisedbetween frames 150 and 1839.

1 The sequences presented are part of the data set recorded for theHERMES Project (IST 027110, http://www.hermes-project.eu), which willbe made available to the scientific community.

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569566

the scenario, sit-down and stand-up actions, and behaviorinterpretations such as abandoned objects (in this casethis is deduced once the owner leaves the surveilled area),the interaction with a vending machine, and violentbehaviors such as kicking or punching elements of thescenario.

The proposed approach for situation analysis is capableof carrying and managing confidence levels, obtained atthe conceptual stage in form of degrees of validity for theFMTHL predicates. Nevertheless, the current implementa-tion relies on the assertion of those predicates associatedwith the highest confidence values, in order to avoid acombinatorial explosion of solutions. As a consequence,only one HLSP is produced by the SGT at each frame,which permits to associate each predicate with an intervalof validity.

Part of the evaluation has been accomplished by meansof NL input queries over the two presented scenes. At thisregard, a list of 75 possibly interesting NL questions orcommands to formulate have been proposed by a group of15 persons. The current capabilities have been restrictedto those user inputs representable by the set of goalqueries described in the previous section. Complexinput queries such as those related to pragmatic content,e.g. ‘‘Why has the second person come back?’’ or ‘‘How isthe last pedestrian crossing the road?’’, cannot be answeredby the system at present and will be tackled in furthersteps.

Other evaluation results for the current implementa-tion have highlighted that an increment of complexityespecially affects two tasks in the high-level architecture:the evaluation of FMTHL predicates by the inferenceengine and the access to the ontology. An increment oflength for the recorded sequences results in an exponen-tial growing of the instantiated elements in the concep-tual database, and as a consequence a higher increment inthe computational time for the SGT traversal. Theseresults encourage the use of heuristic methods to solvethese difficulties.

When an alarm is missed from the vision levels, thehierarchical structure of the SGT simply does not in-stantiate a situation, since one of its required stateconditions is not accomplished. If the rest of informationdoes not allow to reach a certain level of specialization fora situation, then its parent situation will be asserted.Otherwise, a general situation will be asserted due to thelack of information. Thus, the more exhaustively wedefine the hierarchy of a SGT, the more robust will bethe system in front of missing information, but the moreexpensive it will be the cost in terms of computation.

A similar consideration has to be done regarding falsealarms: the SGT will instantiate a wrong situation onlywhen the false information agrees with the sequence ofadmissible states defined in the tree by means of theprediction edges. This way, the robustness of the situa-tional analysis is given by the SGT based on both thetemporal and specialization criteria. The generation ofincorrect information depends of both the sensory gap(bad information provided by the vision acquisitionsystems) and the semantic gap (incorrectness or incom-pleteness of the models at high level).

These experimental results for the situational analysishave been obtained using the F-Limette2 inference enginefor FMTHL and the SGTEditor3 graphical editor for SGTs.On the other hand, the implementation of the ontologyand the query system have been developed using theProtege4 ontology editor and the Jena5 Semantic WebFramework.

9. Conclusions

We have presented the high-level architecture of acognitive vision system (CVS) for surveillance purposes,which intends to extract plausible interpretations ofoccurrences appearing in video sequences from a definitedomain. The CVS presented is designed in structuredlevels of reasoning that allows extracting interpretationsat different semantic levels. In addition, a relationaldatabase management system has been described in afirst step, which has been considered from an ontologicalsemantics perspective, in order to offer broad practicalcoverage for complex situations requiring to process largeamounts of data.

The proposed taxonomical building of the events in adomain implies different levels of interpretation, rangingfrom basic actions and events (e.g. walk, run, turn) tocontextualized events and a more scenario-specific inter-pretation of behaviors (e.g. meeting, giving way, chasing).The basic levels for the status of agents are defined usinghuman motion models, thus being applicable in a generalway. The compilation of new behaviors for a new scene isespecially focused on increasing or modifying the otherlevels, which are more dependant on the particularworking scenario.

Modeling an ontology permits to reduce the complex-ity associated to the multilingual dimension of the system.It also allows to clarify and simplify the design andimplementation of components to bridge the semanticgap. Ontologies are particularly useful in the applicationfield of NL understanding, since they make easier thecategorization of a discourse and play a great role indisambiguation. Another important benefit is to restrictthe domain of acceptance for the different forms ofsemantic representation, since the constraints applied tothe terminology fix the validity of the situations to detect.This way, mechanisms for prediction based on restrainedbehavioral models can be developed.

As stated in [19], changes in the topology of theontological hierarchy and in the distribution of knowledgedo not hold special significance. What has been realized asmuch more crucial has been to focus on coverage, in orderto find the most suitable grain size of the possiblesemantic representations with relation to the needs ofthe concrete application, e.g. how particular have to be theinterpretations in order to detect a given set of occur-rences. The main idea is to model those behaviors

ARTICLE IN PRESS

2 http://cogvisys.iaks.uni-karlsruhe.de/Vid-Text/f_limette/index.html3 http://cogvisys.iaks.uni-karlsruhe.de/Vid-Text/sgt_editor/index.html4 http://protege.stanford.edu/5 http://jena.sourceforge.net/

C. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 567

requiring a more subjective interpretation in a way suchas they are not wrongly extended to general situations,and also making easy deducible situations not requireexcessively detailed information. That is why the de-scribed approach has been designed to work at differentlevels of representation regarding the generality ofsituations, and the reason for the general architecture tohave been conceived in terms of collaborative modules.

By increasing the complexity of the scenes in terms ofnumber of cameras and number of agents, we actuallyincrease the difficulties to the tracking systems. If the visionlevels concerning tracking and calibration tasks are robustenough, the upper levels of the system should be able todeal with this kind of complexity, only by paying a highercomputational time. Nevertheless, the behavior of crowdsor large groups of agents has not been analyzed yet, and ithas to be included as future work: the exponential growingof complexity for certain situations could easily complicatereal-time performance at the semantic levels. In order tosolve such difficulties, new approaches need to be investi-gated, firstly involving the exploitation of confidence levelsresulting from the conceptual reasoning stage.

The addition of alternative domains, such as soccer,will be tackled in the future. In this regard, one of the mosttime consuming tasks will be deciding upon a finaltaxonomy of accepted situations, which is both based onthe available tracking capabilities and the complex designof the situational models. Managing large group interac-tions and prioritizing the situations of interest are issuesthat will probably represent a great cost, too, once thefunctionality of the tracking system is granted. The rest oftasks follow the described approach.

One immediate application is related to the field ofsemantic indexation. The ontology of situations providesthe space and validity of possible annotations for videosequences related to the domain. In the specific imple-mentation, a SGT acts as an actual content classifier, whichcharacterizes the temporal structure of video sequencesfrom a semantic perspective. Thus, the resulting predi-cates can be identified as high-level semantic indexes,which can facilitate further applications such as searchengines and query-based retrieval of content.

The approach suggests enhancements for the automa-tization of many tasks, such as the automatic categoriza-tion of semantic regions in the scenario, or regardingdiscourse goal recognition at a pragmatic level, forexample. Experimental results show that the bidirectionalcollaboration between ontological resources and reasoningstages at different representation levels provides expres-sive results. Nevertheless, one fundamental requirement isto establish a pondered trade-off between the use ofreasoning capabilities, which enable powerful but rigidinterpretations, and the flexibility of the relational data-bases, in which an increase of complexity notablydiminishes the effectiveness of the cognitive system.

Acknowledgments

This work is supported by EC Grants IST-027110 for theHERMES project and IST-045547 for the VIDI-video

project, and by the Spanish MEC under ProjectsTIN2006-14606 and CONSOLIDER-INGENIO 2010(CSD2007-00018). Jordi Gonzalez also acknowledges thesupport of a Juan de la Cierva Postdoctoral fellowship fromthe Spanish MEC.

References

[1] S. Abiteboul, R. Hull, V. Vianu, Foundations of Databases, Addison-Wesley Publishing Co., London, 1995.

[2] M. Arens, H.-H. Nagel, Behavioral knowledge representation for theunderstanding and creation of video sequences, in: Proceedings ofthe KI 2003, vol. 2821(1), Springer, Berlin, Heidelberg, New York,2003, pp. 149–163.

[3] F. Baader, D. Calvanese, D. McGuiness, D. Nardi, P. Patel-Schneider(Eds.), The Description Logic Handbook, Cambridge UniversityPress, Cambridge, 2003.

[4] P. Baiget, C. Fernandez, X. Roca, J. Gonzalez, Automatic learning ofconceptual knowledge for the interpretation of human behavior invideo sequences, in: Proceedings of the 3rd Iberian Conference onPattern Recognition and Image Analysis (Ibpria 2007), Girona,Spain, Lecture Notes in Computer Science, vol. 4477(1), Springer,Berlin, 2007, pp. 507–514.

[5] C.F. Baker, C.J. Fillmore, J.B. Lowe, The Berkeley FrameNet project, in:Proceedings of the COLING-ACL, Montreal, Canada, 1998.

[6] H. Buxton, Learning and understanding dynamic scene activity: areview, Image Vision Comput. 21 (1) (2003) 125–136.

[7] J.L. Crowley, Situated observation of human activity, in: 1stWorkshop on Computer Vision for Interactive and IntelligentEnvironment (CVIIE 2005), IEEE Computer Society, Lexington, KY,USA, 2005, pp. 97–108.

[8] C. Fernandez, P. Baiget, X. Roca, J. Gonzalez, Natural languagedescriptions of human behavior from video sequences, in:30th Annual German Conference on AI (KI 2007), Osnabruck,Germany, Lecture Notes in Computer Science, Springer, Berlin,2007.

[9] J. Gonzalez, Human sequence evaluation: the key-frame approach,Ph.D. Thesis, Universitat Autonoma de Barcelona, Barcelona, Spain,2004, ISBN 84-933652-2-X.

[10] G. Granlund, Cognitive vision systems, in: H.-H. Nagel, H.I.Christensen (Eds.), Organization of Architectures for CognitiveVision Systems, Springer, Heidelberg, 2006, pp. 37–55.

[11] M. Haag, W. Theilmann, K. Schafer, H.-H. Nagel, Integration of imagesequence evaluation and fuzzy metric temporal logic programming,in: Proceedings of the 21st Annual German Conference on AI,Springer, London, UK, 1997, pp. 301–312.

[12] I. Huerta, D. Rowe, M. Mozerov, J. Gonzalez, Improving backgroundsubtraction based on a casuistry of colour-motion segmentationproblems, in: Proceedings of the 3rd Iberian Conference onPattern Recognition and Image Analysis (Ibpria 2007), Girona,Spain, Lecture Notes in Computer Science, Springer, Berlin,2007.

[13] A. Kojima, T. Tamura, K. Fukunaga, Natural language descriptionof human activities from video images based on concepthierarchy of actions, Internat. J. Comput. Vision 50 (2) (2002)171–184.

[14] I. Kompatsiaris, Y. Avrithis, P. Hobson, M.G. Strinzis, Integratingknowledge, semantics and content for user-centred intelligentmedia services: the aceMedia project, in: Proceedings of Workshopon Image Analysis for Multimedia Interactive Services (WIAMIS’04),Lisboa, Portugal, 2004, pp. 21–23.

[15] M. Ma, P. Mc Kevitt, Visual semantics and ontology of eventiveverbs, in: Proceedings of the 1st International Joint Conference onNatural Language Processing, vol. 1(1), 2004, pp. 278–285.

[16] H. Marburger, B. Neumann, H.J. Novak, Natural language dialogueabout moving objects in an automatically analyzed traffic scene, in:Proceedings of the IJCAI-81, Vancouver, 1981.

[17] H.-H. Nagel, Steps toward a cognitive vision system, AI-Magazine 25(2) (2004) 31–50.

[18] H.-H. Nagel, R. Gerber, Representation of occurrences for roadvehicle traffic, AI-Magazine 172 (4–5) (2008) 351–391.

[19] S. Nirenburg, V. Raskin, Ontological Semantics, MIT Press, Cam-bridge, MA, 2004.

[20] S. Park, J.K. Aggarwal, Event semantics in two-person interactions,in: Proceedings of the Pattern Recognition, 17th InternationalConference on (ICPR’04), vol. 4(1), IEEE Computer Society,Washington, DC, USA, 2004, pp. 227–230.

ARTICLE IN PRESSC. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569568

[21] E. Reiter, R. Dale, Building Natural Language Generation Systems,Cambridge University Press, Cambridge, UK, 2000.

[22] D. Rowe, I. Rius, J. Gonzalez, J.J. Villanueva, Improving trac-king by handling occlusions, 3rd ICAPR, UK, LectureNotes in Computer Science, vol. 2(1), Springer, Berlin, 2005,pp. 384–393.

[23] K. Schafer, C. Brzoska, ‘‘F-limette’’ fuzzy logic programmingintegrating metric temporal extensions, J. Symbolic Comput. 22(5–6) (1996) 725–727.

[24] L. Talmy, Toward a Cognitive Semantics, Concept StructuringSystems, vol. 1, Bradford Book, 2000.

[25] M. Vincze, W. Ponweiser, M. Zillich, Contextual coordination in acognitive vision system for symbolic activity interpretation, in:Proceedings of the 4th IEEE International Conference on ComputerVision Systems, vol. 1(1), IEEE Computer Society, Washington, DC,USA, 2006, p. 12.

[26] R.A. Wilson, F.C. Keil (Eds.), The MIT Encyclopedia of the CognitiveSciences, Bradford Book, 2001.

ARTICLE IN PRESSC. Fernandez et al. / Signal Processing: Image Communication 23 (2008) 554–569 569