TRANSFORM DST Deployment Guide

v1.0

Content  

1   Introduction  ................................................................................................................................  4  

1.1   Intended  audience  ..............................................................................................................................  4  

2   Glossary  .......................................................................................................................................  4  

3   Hardware  Requirements  ..............................................................................................................  4  

4   Software  Requirements  ...............................................................................................................  5  

4.1   Specification  ........................................................................................................................................  5  

4.2   Installation  ..........................................................................................................................................  5  

4.2.1   Operating  System  ......................................................................................................................  5  

4.2.2   Java  Version  ...............................................................................................................................  5  

4.2.3   RDBMS  PostgreSQL  ...................................................................................................................  6  

Enabling  PostGIS  ...........................................................................................................................................  8  

Upgrading  PostGIS  ........................................................................................................................................  8  

4.2.4   Application  Server  Tomcat  ........................................................................................................  8  

4.2.5   Webserver  Apache  ....................................................................................................................  9  

4.2.6   Geoserver  ..................................................................................................................................  9  

4.2.7   Liferay  Portal  ...........................................................................................................................  10  

4.2.8   TRANSFORM  specific  software  ................................................................................................  12  

4.3   Data  ...................................................................................................................................................  13  

4.3.1   Base  data  (OpenStreetMap  etc.)  .............................................................................................  13  

4.3.2   Transform  data  sets  (per  city)  .................................................................................................  13  

4.3.3   Harmonizing  of  data  ................................................................................................................  18  

4.3.4   Lessons  learned  .......................................................................................................................  19  

4.3.5   Outlook  ....................................................................................................................................  20  

List  of  Figures: Figure 1: Liferay directory structure ................................................................................................. 11  

Figure 2: Example of worker properties file ..................................................................................... 12  

Figure 3: Example of worker entries for Liferay in the sites-enabled/default file on the server ...... 12  

Figure 4: Excerpt of city staging scheme ......................................................................................... 15  

Figure 5: Consolidated data model .................................................................................................. 16  

Figure 6: Import workflow to consolidated scheme .......................................................................... 17  

Figure 7: Amsterdam: Data on (very detailed) building data ........................................................... 18

Figure 8: Copenhagen: Only data for a selected number of buildings was available for the tool development ............................................................................................................................. 18  

Figure 9: Vienna: Only block based data was available for a small part of the city (Aspern Seestadt) .................................................................................................................................. 18  

Figure 10: Hamburg: Data was available for blocks in one district (Wilhelmsburg) ......................... 18  

Figure 11: Genoa: Data was available as coordinates for public lighting installations for the whole city ............................................................................................................................................ 19  

Figure 12: Data was available on building level for one district (Grand Lyon) ................................. 19  

1 INTRODUCTION  

This document is the technical description (“Deployment Guide”) for the TRANSFORM DST. It describes the hard- and software requirements to get the system up and running.

1.1 Intended  audience  

The Deployment Guide is intended for technical staff responsible for the installation of hard- and software in the cities interested using the tool.

2 GLOSSARY  

Apache     Webserver  Software  CPU   Central  Processing  Unit  (processor)  DST   Decision  Support  Tool  GB   Gigabyte  Liferay   Portal  Software  

Portlet  Portlets  are  pluggable  user  interface  software  components  that  are  managed  and  displayed  in  a  web  portal.    

PostGIS   Spatial  extension  for  PostgreSQL  PostgreSQL   Relational  database  system  RAM   Random  Access  Memory  RDBMS     Relational  database  management  system  Tomcat   Application  Server  

3 HARDWARE  REQUIREMENTS  

The TRANSFORM DST is running as a web application on a web server and should run on decent web server hardware to produce a smooth user experience.

Since the DST has been developed as a Liferay Portlet the system needs to be able to run the application container properly. The following is the minimum requirements as suggested by the TRANSFORM development team (further information can be found e.g. here: http://www.liferay.com/products/liferay-portal/tech-specs). The suggested hardware requirements are also fitting the specifications of the used database (PostgreSQL) as well as the web server (Apache). For each of the applications it is most important to have them configured properly which will be describe below in some detail in the software section. These are the minimum hardware requirements:

CPU Quad core CPU (>2,5 GHz)

RAM >8GB

Disk Space >500GB (depending on the used geographic data in the database, see below)

4 SOFTWARE  REQUIREMENTS  

4.1 Specification  

In order to be able to be running the DST several software components need to be pre-installed on the server. This is a list of (exemplary) components which are mandatory to run the tool.

Operating System (OS) 64Bit Linux (optional: Windows Server)

Java Version JDK >1.6

Database PostgreSQL > 9.1

Webserver Apache >2.2

Application Server Tomcat >7.0

Portal Software Liferay >6.1

Transform Specific

Portlet DST Frontend Portlet

Servlet Transform Connector

4.2 Installation  

4.2.1 Operating  System  

The DST will be most probably be installed on an already existing web server. This server should be running an 64bit operating system (preferably Linux but a Windows server is also suitable). The installation of such an operating system is beyond the scope of this deployment guide.

4.2.2 Java  Version  

In order to be able to run the application server Tomcat (see below) the system needs to be running a recent Java version, preferably Oracle® Java in a version of 1.7 or above. Keep in mind that you might be already running Java. To check this issue the following command on the command line:

java -version

If you see an output like:

java version "1.7.0_25"

Java(TM) SE Runtime Environment (build 1.7.0_25-b15)

Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

you should be able to run Tomcat, otherwise install Java like this:

The installation of Java varies widely on different operating systems. Here are some examples of how to install Java on Debian/Ubuntu Linux and Windows® machines:

Debian/Ubuntu: On the command line run the following command:

sudo apt-get install oracle-java7-installer

On Windows®: For a recent version of Oracle® Java point your browser here:

https://java.com/de/download/index.jsp

and download the Java installer to your hard drive. After the download you should have an executable installer file. To install Java just click this installer twice to execute it and then follow the instructions on the screen. After the installation has finished you should be able to run Java programs on your Windows® computer.

4.2.3 RDBMS  PostgreSQL  

The DST stores its data in a database. There are many different database systems on the market but for the DST the TRANSFORM team has chosen the PostgreSQL database system since it provides also a spatial extension (PostGIS, described below).

This deployment guide will only explain the standard way of installing the database system software on your server which should lead to a system capable of running the DST. Keep in mind, though, that there are numerous ways to fine tuning a database system like PostgreSQL for higher workloads etc. which is beyond the scope of this guide. In order to do so please refer to internet searches on how to fine tune PostgreSQL.

To install PostgreSQL do the following:

Debian/Ubuntu: On the command line run the following command:

sudo apt-get install postgresql

On Windows® browse to the following site:

http://www.postgresql.org/download/windows/

and download the Enterprise installer for your operating system. If you have done so execute the installer and follow the instructions on your screen. The Enterprise installer includes also PostGIS (see next chapter) which you can tick during the installation process to include the installation on your hard drive.

4.2.3.1 Spatial  extension  PostGIS  

PostGIS is a so called extension to your PostgreSQL database to enable the handling of geographic/geometric objects in the database.

To install it do the following:

Debian/Ubuntu: Prerequisite on Linux is that you need to have PostgreSQL installed before you can install PostGIS. So, after you have installed PostgreSQL see preceding chapter, issue the following command:

sudo apt-get install postgis

On Windows®: To install PostGIS on Windows® either use the Enterprise installer (see above) and tick the PostGIS installation during the installation process (this is the preferred way of installing PostGIS)

OR point your browser to the following site:

http://download.osgeo.org/postgis/windows/

and download the PostGIS version apprpopriate to your PostgreSQL version (i.e. if you are running PostgreSQL 9.1 choose the directory pg91 and download e.g. postgis-pg91-binaries-2.0.6w64.zip if you are on a 64bit Windows® Server machine). Unzip the downloaded file and and run the executable.

To spatially enable a database in PostgreSQL with PostGIS functionalities the following procedures need to be applied after installing PostGIS (both on Linux and Windows® machines):

The PostGIS site cites the following1:

These instructions are for PostgreSQL 9.1 and higher, PostGIS 2.0 and higher that is compiled with raster support. Note: if you have postgis, without raster support, you can not use CREATE EXTENSION. Refer to PostGIS install.

1 Source: http://postgis.net/install/, 05.12.2014

Enabling  PostGIS  

PostGIS is an optional extension that must be enabled in each database you want to use it in before you can use it. Installing the software is just the first step. DO NOT INSTALL it in the database called postgres.

Connect to your database with psql or PgAdmin. Run the following SQL: -- Enable PostGIS (includes raster) CREATE EXTENSION postgis; -- Enable Topology CREATE EXTENSION postgis_topology; -- fuzzy matching needed for Tiger CREATE EXTENSION fuzzystrmatch; -- Enable US Tiger Geocoder CREATE EXTENSION postgis_tiger_geocoder; Upgrading  PostGIS  

To upgrade PostGIS, you first have to install the latest binaries and then upgrade each database you have PostGIS installed in

For example connect to database you want to upgrade and if you just installed binaries for 2.1.3 You can upgrade from 2.0 to 2.1, 2.2 et.c using this approach. To go from 1.* to 2.* you need to do a hard upgrade. Refer to PostGIS install for more extensive instructions. Note: that as of PostGIS 2.1.3 and PostGIS 2.0.6, you need to set environment variables to get full features. -- Upgrade PostGIS (includes raster) ALTER EXTENSION postgis UPDATE TO "2.1.4"; -- Upgrade Topology ALTER EXTENSION postgis_topology UPDATE TO "2.1.4"; -- Upgrade US Tiger Geocoder ALTER EXTENSION postgis_tiger_geocoder UPDATE TO "2.1.4";

4.2.4 Application  Server  Tomcat  

The DST is running as a Portlet in the Liferay portal (see below). In order to be able to be running Liferay an application server software (Java container) needs to be installed on the server. There are different application server software on the market but the TRANSFORM team has chosen Apache Tomcat since there is a bundled version available that combines Liferay and Tomcat in one package for easier installation. In order to get Tomcat and Liferay running you need to download this bundle. To be consistent this will be explained in chapter 4.2.6 Liferay Portal.

If you are already running Tomcat on your system you need to deploy Liferay within your installation. This will also be covered in chapter 4.2.7.2.

4.2.5 Webserver  Apache  

Apache is the component responsible for serving web content to the clients (such as HTML pages etc.). For security reasons it is also used as a proxy to forward request to Tomcat. If you are running a web server you most probably will have Apache already running there. If not do the following:

Debian/Ubuntu: Issue the following command on the command line:

sudo apt-get install apache2

in order to be able to use Apache as a proxy for Tomcat you need to install mod-jk like this:

sudo apt-get install libapache2-mod-jk

On Windows® you need to do the following:

The easiest way to get Apache running on Windows® is to download a binary package from these sites:

Webserver Apache2 v2.2:

http://www.apachehaus.com/cgi-bin/download.plx#APACHE22VC09

mod-jk for Windows®

http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/windows/tomcat-connectors-1.2.39-windows-x86_64-httpd-2.4.x.zip

or if you are running on another operating system visit:

http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/windows/#warnings

After having installed the above packages you will need to create a so called worker in mod_jk and Apache2 which will handle the forwarding of resources. To create such a worker in Apache follow these instructions:

http://tomcat.apache.org/connectors-doc/generic_howto/quick.html

4.2.6 Geoserver  

The Transform DST is -to a great extent- based on spatially explicit data which is served via a server application called Geoserver. This application is able to visualise spatial datasets stored in

(but not limited to) PostGIS on the PostgreSQL databases. To be run it needs an application server as Java container. In our case we chose Tomcat since it was already bundled with Liferay (see next chapter).

To install Geoserver you will need to go to its download site (http://geoserver.org/download/) and download the latest stable version of Geoserver (at the time of writing v2.6.1).

Important: In the case of the DST it is recommended that you download the Web Archive (war) version of the Geoserver application to be able to deploy it in the already existing Tomcat installation to be found here: http://sourceforge.net/projects/geoserver/files/GeoServer/2.6.1/geoserver-2.6.1-war.zip/download

If your Tomcat is configured for auto-deployment you will just need to copy this war file into the webapps folder of your Tomcat installation and Geoserver should be deployed. To be able to reach the installation via the internet you will need to create a separate entry in the mod_jk worker file (see previous chapter).

After successful installation you should be able to reach Geoserver via the following URL:

http://localhost:8080/geoserver

or (via mod:jk worker):

http://[yourhost]/geoserver

4.2.7 Liferay  Portal  

The Liferay Portal is the environment the Transform DST is running in. It is a Java application that is run as a web application through an application server like Tomcat. The easiest way to install it is to download a Liferay/Tomcat bundle (see next sub chapter).

4.2.7.1 Standalone  installation  (Tomcat  –  Liferay  bundle)  

If you do not already have an instance of Tomcat running on your system or you would like to run a version dedicated to Liferay alone on your system you can download a bundle from the following location:

http://www.liferay.com/de/downloads/liferay-portal/overview

You can either choose the “Commutiy Edition” (free) or the “Enterprise Subscription” which you will have to pay a fee for. The Community Edition provides everything we need for running the Transfrom DST. To download the bundle click the drop down list box and choose “Bundled with Tomcat” then click “Go”. After the download has finished you will have to unzip the downloaded zip file on the filesystem of your server.

You will get the following directory structure:

Figure 1: Liferay directory structure

To start the Liferay portal you need to change to the following directory:

/var/lib/liferay-portal-6.1.2-ce-ga3/tomcat-7.0.40/bin

And issue the following command:

./startup.sh

This should start Tomcat and deploy the Liferay portal. After successful start the portal should be available at:

http://localhost:8080/web/

After ensuring that Liferay is reachable at this location you will have to add an entry to the mod_jk worker file (see Figs. 2 and 3).

Figure 2: Example of worker properties file

Figure 3: Example of worker entries for Liferay in the sites-enabled/default file on the server

4.2.7.2 Installation  as  Web  Archive  (war)  

If you are already running Tomcat on your system please follow the steps provided by the Liferay team to be found here:

http://www.liferay.com/de/documentation/liferay-portal/6.2/user-guide/-/ai/installation-and-setup-liferay-portal-6-2-user-guide-15-en

4.2.8 TRANSFORM  specific  software    

4.2.8.1 DST  Frontend  Portlet  

The Transform DST Portlet can be deployed with the provided war file (TransformFrontend-portlet-6.1.1.1.war). This file can be deployed as follows:

1. Stop Tomcat.

2. Delete existing deployment. If you have previously deployed "foo.war" in TOMCAT_HOME/webapps, then it has been unpacked into webapps/foo/... You must delete this directory and all its contents. On Unix, this can be done with “rm -r $TOMCAT_HOME/webapps/foo”

3. Copy WAR file to TOMCAT_HOME/webapps/.

4. Start Tomcat.

To change the database settings of the deployed war file (Tomcat unpacks the warfile into a directory), go to ROOT/WEB-INF/classes (ROOT of your unpacked webapp). Create a file called portlet-ext.properties as described in http://www.liferay.com/community/wiki/-/wiki/Main/Database+Configuration

4.2.8.2 Transform  Connector  (for  mapping  functionalities)  

The Transform DST includes mapping functionalities to visualise and work with spatial data stored in the PostGIS database. To let the DST interface interact with the data in the database the DST uses a servlet (the TransformConnector) which is provided as a war file (TransformConnector_v[…].war). This file can be deployed automatically in the Tomcat webapp directory (which is the same procedure as the deployment of the Transform Frontend portlet. See previous chapter).

The Transform Connector is using PHP for the database connections for its graphical user interface (GUI) elements. To be able to run PHP code within Tomcat please refer to the following site:

http://php-java-bridge.sourceforge.net/doc/tomcat6.php

4.3 Data    

4.3.1 Base  data  (OpenStreetMap  etc.)  

The mapping functionality of the DST is on the one hand using so called base maps which serve the purpose to let the user orient his/herself on the map. This purpose is served by OpenStreetMap data being loaded from the Geofabrik site into the GUI.

4.3.2 Transform  data  sets  (per  city)  

Each city provided more or less complex datasets to realize the implementation of certain measures according to their needs. The datasets are stored in a particular staging area in order to do the needed pre-processing e.g. completing and cleansing of the data. Therefore six different city schemes within the PostgreSQL database lportal were created. The database for keeping all

the data is the same as it used for running the Liferay instance. Within this database, the following schemes with the objective to keep the staging data are currently available:

• amsterdam • copenhagen • genua • hamburg • lyon • public • vienna

The data within the public scheme contains all the entities which are created and used by the Liferay instance and a consolidated data model, star schema (Error! Reference source not found.) integrating all the available data from all six cities.

The import of the data to the city staging areas (schemes) is done by the OSS application QGIS if the data sources are comprised of georeferenced datasets e.g. shapefiles or linked datasources e.g. Web Map Service. In case of textual based data e.g. CSV, XLS data files this is done by the commercial database administration and development SW Navicat. Navicat provides also the functionality for importing various different file formats to a DB by a so called Import Wizard.

Within the Figure 4 an excerpt of the current status of available staging data within the DB is shown (Amsterdam and Hamburg):

Figure 4: Excerpt of city staging scheme

Figure 5: Consolidated data model

Based on the data within the staging schemes various SQL scripts are developed to create appropriate temporary result tables providing the necessary data for implementing the on top applied measure definitions.

Afterwards the cleaned and completed data within the six city stages is imported to the consolidated scheme (Fig. 5). This is done by hand crafted DML SQL scripts, because of the very individual structure and content of the data provided. The basic workflow although can be described as follows (Fig. 6Error! Reference source not found.):

1. import the name of the source table and the mapping of source key and surrogate key to the table trnsfrm_keymap table

2. import the geometric objects (features) with additional meta information about level of detail (LOD) to the trnsfrm_feature table

3. import all the LOD related additional information to generic or city specific LOD entities (tables), e.g. district, block, building, network information

4. import all the available given energy related information to the generic or city specific entities, e.g. electricity, gas, warm water, heat consumption

Figure 6: Import workflow to consolidated scheme

For each new city

Pre-processed city data table/DML SQL

Finished

Key Mapping

Features

Generic and specific level of detail (LOD) meta information

Energy consumption related information

These four steps and the step for providing the appropriate staging area have to be done for each new city to be imported to the system. The analysis work to be able to interpret, cleanse and complete the given data has to be done in advance.

Without an pre-defined process starting on the city side taking into account the restrictions and possibilities on the organizational levels point of view and enabling governance processes the technical process enabling a frequent and lower effort data transition workflow stays an high effort and very individual less efficient task. For the sake of the given scientific project with the objective to show the possibilities of such a system it is although shown that the requirements implementing the measures and running the simulation can be reached.

4.3.3 Harmonizing  of  data  

As described in the previous chapter the data sets provided by the Transform partner cities varied not only hugely in their content but also in their spatio-temporal resolution. Often the data sets differed within each city (e.g. data sets on building block, street level, district and city level). Often detailed data had been provided without context information (meta-data) which would have been needed to develop a measure (e.g. gas consumption data were delivered without building information). Figures 7 to 12 display the great differences in resolution within the data sets for each city.

Figure 7: Amsterdam: Data on (very detailed) building data

Figure 8: Vienna: Only block based data was available for a small part of the city

(Aspern Seestadt)

Figure 9: Hamburg: Data was available for blocks in one district (Wilhelmsburg)

Figure 8: Copenhagen: Only data for a selected number of buildings was available

for the tool development

Within WP3 a process has been developed to harmonize those data:

1. Request cities to provide homogenous data sets

2. Request cities to provide context information

3. Do an online research to acquire needed context information

4. Pre-process data to assure spatial and temporal homogeneity

Furthermore due to the constellation of the project, data have been provided on a city level (WP2) and often in-depth information on a neighbourhood-level (WP4 – SULs).

The heterogeneity had several reasons: Besides a lack of European or even national standards, privacy issues lead to huge diversity of data quality and format. In one example a data set had been created by the project partner but due to the national law the data could not be used outside of the municipality and therefore only default values have been used in the Decision Support Tool.

4.3.4 Lessons  learned  

The data collection for the development of the TRANSFORM DST was an iterative and sometimes cumbersome task since TRANSFORM had to deal with 6 different city administrations which meant 6 different cultures and privacy legislations as well as 6 different levels of maturity when it comes to the monitoring and availability of energy related data within the city administrations. In most cases some of the relevant data was not in the hands of the city but was owned and collected by the local energy provider who was reluctant (sometimes not allowed) to deliver such needed datasets.

As stated in the last chapter the delivered datasets were of different quality and spatio-temporal resolution: Some datasets were based on yearly sums of energy consumption numbers, some had a finer resolution. Some data sets where data from the near past (2013), some were quite old ones which showed that there is a huge need for standardised datasets in order to foster an in depth look and a development of CO2 reduction measures and to have them shared between cities.

Figure 11: Data was available on building level for one district (Grand Lyon)

Figure 10: : Genoa: Data was available as coordinates for public lighting installations

for the whole city

4.3.5 Outlook  

As said in the last chapter from the data perspective the TRANSFORM project showed that there is a huge need for standardised data sets when it comes to comparing performances of the energy consumption (and production) within the different European cities. The TRANSFORM project therefore proposes the following:

Since the energy related data in the European cities is inhomogeneous at the moment we will need

• The adoption of already established standards and protocols within the energy data community. Comparable data will help cities analyse their situation and will strengthen results in this area

• open and shared data initiatives to be joined by the cities and data will have to be made public in form of Open Government Data and Linked Open Data

• energy providers should be “forced” to deliver needed datasets if they are necessary for public interests, i.e. the development of measures to reduce CO2 emissions