49
Business Intelligence and Reporting Tools Evaluation: BIRT vs Pentaho by ZHI ZHANG (100985308) WENJING ZHANG (7352330) A Project Report submitted in fulfillment of the requirements for CSI5112 SOFTWARE ENGINEERING Approved by: Professor Daniel Amyot April 17, 2015

Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

  • Upload
    others

  • View
    8

  • Download
    2

Embed Size (px)

Citation preview

Page 1: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

Business Intelligence and Reporting Tools Evaluation: BIRT vs Pentaho

by

ZHI ZHANG (100985308)

WENJING ZHANG (7352330)

A Project Report

submitted in fulfillment of the requirements for CSI5112 SOFTWARE ENGINEERING

Approved by: Professor

Daniel Amyot April 17, 2015

Page 2: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– ii–

Executive Summary

The primary objective of this report is to select a BI reporting tools that satisfy our organization's need

to deliver enterprise portal applications to our customers. More specifically, we performed comparative

analysis on behalf of Ivy Technologies, an independent software vendor that provides IT management

solutions for the retail industry. The company is hoping to find a proper BI reporting tool that enables

their developers to efficiently deliver customized reporting portal for the clients, so as to improve the

performance and functionality of client's current reporting systems.

It is for this reason that we conduct comparative study on two of the most popular and powerful tools

by including a series of testing, analysis and critiques. This report adopted the criteria-based assessment,

which is a quantitative method for software selection and evaluation. We obtained our evaluation

results both from the experimentations conducted using two products, and the reviews from Innovent

Solutions [4]. In addition we made reference to documentations from the open source community,

which covers a great deal of information on specific features.

Our evaluation shows that both products satisfied Ivy Technologies’ basic requirements for a BI

reporting tool, with BIRT scores a bit higher than Pentaho by 6%. As open source tools, however, some

advanced features are not available in these community versions. Based on the evaluation results, we

made the decision to recommend BIRT to Ivy Technologies due to its superior usability and richer

functionality. Since our evaluation does not include scalability testing, further investigation needs to be

conducted if Ivy Technologies adds new requirements in terms of scalability and integration.

damyot
Highlight
damyot
Highlight
damyot
Highlight
damyot
Highlight
damyot
Highlight
Page 3: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– iii–

Revision History

Version Description Summary of changes Date

V 1.0 Initial Draft Created outline 2015-3-10

V 1.1 Revised Draft Experiment and results documented 2015-3-28

V 1.2 Revised Draft Content revised 2015-04-04

V 2.0 Final Document Document ready for final review 2015-04-17

Page 4: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– iv–

Table of Contents Executive Summary ...................................................................................................................................... ii

Revision History ........................................................................................................................................... iii

List of Figures ............................................................................................................................................... vi

List of Tables ............................................................................................................................................... vii

Glossary ..................................................................................................................................................... viii

1. Introduction .......................................................................................................................................... 1

1.1 Business Context ........................................................................................................................... 1

1.2 Evaluation Goals ............................................................................................................................ 1

1.3 Evaluation Scope ........................................................................................................................... 2

2. Evaluation Criteria ................................................................................................................................ 3

2.1. Overview ....................................................................................................................................... 3

2.2. Report Designer Components ....................................................................................................... 4

2.3. Data Source ................................................................................................................................... 5

2.4. Charting Criteria Evaluation .......................................................................................................... 6

2.5. Parameterization and Output Format Criteria Evaluation ............................................................ 7

2.6. Analytical Functionality Criteria Evaluation .................................................................................. 7

2.7. Usability Criteria Evaluation .......................................................................................................... 8

2.8. Installation and Implementation Criteria Evaluation .................................................................... 9

2.9. Miscellaneous Criteria Evaluation ............................................................................................... 10

3. Methodology ...................................................................................................................................... 11

4. Evaluation of Pentaho Report Designer ............................................................................................ 12

4.1. Report Designer Components Criteria Evaluation ...................................................................... 13

4.2. Data Source Criteria Evaluation .................................................................................................. 15

4.3. Charting Criteria Evaluation ........................................................................................................ 17

4.4. Parameterization and Output Format Criteria Evaluation .......................................................... 19

4.5. Analytical Functionality Criteria Evaluation ................................................................................ 20

4.6. Usability Criteria Evaluation ........................................................................................................ 21

4.7. Installation and Implementation Criteria Evaluation .................................................................. 22

4.8. Miscellaneous Criteria Evaluation ............................................................................................... 23

4.9. Evaluation Summary ................................................................................................................... 24

5. Evaluation of BIRT .............................................................................................................................. 25

5.1. Report Designer Components Criteria Evaluation ...................................................................... 27

Page 5: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– v–

5.2. Data Source Criteria Evaluation .................................................................................................. 29

5.3. Charting Criteria Evaluation ........................................................................................................ 29

5.4. Parameterization and Output Format Criteria Evaluation .......................................................... 31

5.5. Analytical Functionality Criteria Evaluation ................................................................................ 33

5.6. Usability Criteria Evaluation ........................................................................................................ 35

5.7. Installation and Implementation Criteria Evaluation .................................................................. 36

5.8. Miscellaneous Criteria Evaluation ............................................................................................... 37

5.9. Evaluation Summary ................................................................................................................... 38

6. Conclusions and Recommendations .................................................................................................. 40

Reference .................................................................................................................................................... 41

Page 6: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– vi–

List of Figures Figure 4-1 PRD UI with Welcome Window ................................................................................................. 12

Figure 4-2 Preview report in html output format ....................................................................................... 13

Figure 4-3 Data source connection in PRD ................................................................................................. 16

Figure 4-4 Chart editor in PRD .................................................................................................................... 17

Figure 4-5 Parameter editor in PRD ............................................................................................................ 19

Figure 5-1 BIRT Report Designer UI ............................................................................................................ 25

Figure 5-2 Create a new Data source .......................................................................................................... 26

Figure 5-3 Define a new Data set ................................................................................................................ 26

Figure 5-4 Preview report in web browser ................................................................................................. 27

Figure 5-5 Architecture Overview of BIRT .................................................................................................. 28

Figure 5-6 Interactive chart by adding script .............................................................................................. 30

Figure 5-7 Cascading parameter group in BIRT .......................................................................................... 32

Figure 5-8 Define Data Cube in BIRT ........................................................................................................... 34

Page 7: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– vii–

List of Tables Table 2-1 Evaluation Criteria Categories ....................................................................................................... 3

Table 2-2 Criteria Impact Levels and Numerical Weights ............................................................................. 4

Table 2-3 Description of the Report Designer Components Criteria ............................................................ 4

Table 2-4 Description of Data Source Criteria .............................................................................................. 5

Table 2-5 Description of Charting Criteria .................................................................................................... 6

Table 2-6 Paremeterization and Output Format Criteria Evaluation ............................................................ 7

Table 2-7 Analytical Functionality Criteria Evaluation .................................................................................. 7

Table 2-8 Usability Criteria Evaluation .......................................................................................................... 8

Table 2-9 Installation and Implementation Criteria Evaluation .................................................................... 9

Table 2-10 Miscellaneous Criteria Evaluation ............................................................................................. 10

Table 3-1 Criteria Evaluation Score ............................................................................................................. 11

Table 4-1 Report Designer Components Criteria Evaluation ...................................................................... 14

Table 4-2 Data Source Criteria Evaluation .................................................................................................. 16

Table 4-3 Charting Criteria Evaluation ........................................................................................................ 18

Table 4-4 Paremeterization and Output Format Criteria Evaluation .......................................................... 20

Table 4-5 Analytical Functionality Criteria Evaluation ................................................................................ 20

Table 4-6 Usability Criteria Evaluation ........................................................................................................ 21

Table 4-7 Installation and Implementation Criteria Evaluation .................................................................. 23

Table 4-8 Miscellaneous Criteria Evaluation ............................................................................................... 24

Table 5-1 Report Designer Components Criteria Evaluation ...................................................................... 28

Table 5-2 Data Source Criteria Evaluation .................................................................................................. 29

Table 5-3 Charting Criteria Evaluation ........................................................................................................ 30

Table 5-4 Parameterization and Output Format Criteria Evaluation .......................................................... 32

Table 5-5 Analytical Functionality Criteria Evaluation ................................................................................ 34

Table 5-6 Usability Criteria Evaluation ........................................................................................................ 35

Table 5-7 Installation and Implementation Criteria Evaluation .................................................................. 36

Table 5-8 Miscellaneous Criteria Evaluation ............................................................................................... 37

Table 6-1 Evaluation Scores Comparison .................................................................................................... 40

Page 8: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

– viii–

Glossary

ad-hoc query The process by which a user can create a new report to answer a specific question

immediately.

BI Short for Business Intelligence. It is a set of business applications and technologies for

gathering, storing, analyzing, and converting data into information to enable future

business strategies and decisions to be made.

Dashboard It is a user interface that organizes and presents information in an easy-to-read format

and help organizations align people’s actions with strategy by tracking and analyzing key

business metrics and goals.

Data cube A set of data that is usually constructed from a subset of a data warehouse and is

organized and summarized into a multidimensional structure defined by a set of

dimensions and measures.

Filter A saved subset of information pulled from your database in a query that is based on

certain criteria.

MDX It is the query language for multidimensional databases. It is equivalent to SQL.

OLAP Short for On-Line Analytical Processing. It is an approach to building databases that are

optimized for analytical purposes.

Portal A web site that typically provides personalized capabilities to its visitors.

PRD Pentaho Report Designer

UI User Interface

XML Short for Extensible Mark-up Language. It offers a widely adopted standard way of

representing text and data in a format that can be processed without much human or

machine intelligence.

Page 9: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

1

1. Introduction

1.1 Business Context This project intends to evaluate business intelligence and reporting tools for Ivy Technologies. Ivy

Technologies is an independent software vendor that provides IT management solutions for the retail

industry. Since last year, Ivy Technologies has received requests from clients asking for services

regarding data analysis and online reporting. Most of these clients complain about poor performance,

lack of visualization and little customization of their current reporting system, for example, end users

were forced to manually modify report output to cope with specific queries. Given the situation, Ivy

Technologies decided to build up a new team to deliver enterprise portal applications that offer robust

reporting framework and analytics capabilities. The management team are considering the possibility of

utilizing a BI application tool to develop customized reporting portal that can access client's existing

databases, view query results from a browser, and be able to design the report framework based on

end-user needs. Currently Ivy Technologies does all its web application developments in Eclipse IDE,

therefore the preference is Eclipse-based integrated open source tools.

1.2 Evaluation Goals The purpose of this evaluation is to perform objective comparison between two open source BI

reporting tools, Pentaho and BIRT, and find the best possible choice that match the client’s requirement

and can be used in production environment.

By considering the problem mentioned in our business context above, we list the following

requirements that must be satisfied for a suitable candidate:

Provide common report designer components;

Support multiple data sources;

Support common chart types;

Provide common analytical functionality;

Support multiple platform; and

Support deployment onto application servers.

The tool should also meet several soft goals include:

Ease of use via simple user interface;

Provide sufficient documentation manuals and online repositories/FAQs/Forums;

Low product cost; and

Easy to maintain and update.

The winner must pass all mandatory requirements and get higher final score.

damyot
Highlight
damyot
Highlight
damyot
Highlight
Page 10: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

2

1.3 Evaluation Scope There are a number of open source business intelligence and reporting tools available in the market,

such as BIRT, Jaspersoft, Pentaho, SpagoBI, just to name a few. After our initial filter, we narrowed down

our choice to two open source BI reporting tools, BIRT and Pentaho. The reason is that Jaspersoft focus

on nicely formatted printed output of report rather than extensive use of charting, while SpagoBI is a

100% open source BI tool with no commercial edition, which limits the choice of our client when they

can afford to upgrade to an enterprise edition with new features and better performance.

Our investigation into these two tools will be limited to a comparative evaluation of the functionalities

of community versions, namely Pentaho Report Designer 5.2 and BIRT Version: 4.4.2. The commercial

versions provide extended features based on these open source platforms, and will be mentioned but

will not get tested in this report. Community versions of two tools will be installed on windows

platforms and evaluated by two separate arbiters.

damyot
Highlight
References?
Page 11: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

3

2. Evaluation Criteria This section describes the set of evaluation criteria used to assess the business intelligence and

reporting tools: Pentaho Report Designer and BIRT. These criteria were selected to reflect the needs of

our client.

2.1. Overview When assessing the quality of software, it is always good to balance between hard objectivity and

subjective individual user experience. This report will adopt the criteria-based assessment which is a

quantitative method for software selection and evaluation. A criteria-based assessment will look at the

quality of software in a number of areas which stated in ISO/IEC 9126 Software engineering – Product

quality[1]. There are 27 criteria in total used for assessment and grouped into the following 8 categories

outlined in table 2-1.

Table 2-1 Evaluation Criteria Categories

Criteria Category Description

Report Designer Components Focuses on evaluating the tool’s user interface components and advanced functional components. Refer to section 2.2 and Table 2-3 for the complete list of criteria.

Data Source Focuses on evaluating the tool’s supported data source and the way used to refine that data source. Refer to section 2.3 and Table 2-4 for the complete list of criteria.

Charting Focuses on evaluating the tool’s charting functionality. Refer to section 2.4 and Table 2-5 for the complete list of criteria.

Parameterization & Output Format Focuses on evaluating the tool’s parameterization and output format which enables report readers to customize the content of the output. Refer to section 2.5 and Table 2-6 for the complete list of criteria.

Analytical Functionality Focuses on evaluating the tool’s ability to analyze multidimensional data. Refer to section 2.6 and Table 2-7 for the complete list of criteria.

Usability Focuses on evaluating the tool’s user experiences and available support offered. Refer to section 2.7 and Table 2-8 for the complete list of criteria.

Installation &Implementation Focuses on evaluating the tool’s ease of installation and implementation. Refer to section 2.8 and Table 2-9 for the complete list of criteria.

Miscellaneous Focuses on evaluating the tool’s costs and stability. Refer to section 2.9 and Table 2-10 for the complete list of criteria.

Page 12: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

4

According to the evaluation methodology, each criterion is assigned a relative weight (1-Nice to Have, 5-

Important, 10-Crucial), based on the relevance and importance of that specific criterion to the product

requirements and desires. The detailed description is listed in Table 2-2.

Table 2-2 Criteria Impact Levels and Numerical Weights

Criteria Impact Level Weight Description

Crucial 10 This is the minimum requirement which must be satisfied by the tool.

Important 5 This is an important or highly desired factor for potential solution.

Nice to Have 1 This is a low impact factor which will have little impact on decision.

2.2. Report Designer Components For our evaluation goals, we want to find a suitable solution which can provide flexible reporting and

printing functionality using data from multiple source and supports output to display devices, printers,

PDF, XHTML, XML, and CVS files. In that case, we want to evaluate what common and advanced

components of report designer contained, in order to fulfil the complex tasks as mentioned above.

Table 2-3 lists two evaluation criteria under report designer components category.

Table 2-3 Description of the Report Designer Components Criteria

Report Designer Components Criteria

Impact Level Description

Common Report Designer Components

10-Crucial This criteria describes the product’s user interface common components it must contained, in order to help our client to design and publish any style of report easily. This is evaluated based on testing whether the product contains the following components in order to fulfil the most common reporting requirements, such as: Report Editor, Palette, Data Explorer, Property Editor, Preview, Expression Builder, Chart Builder and Script Editor. We will create a couple of sample reports which contains different features in order to cover the above components as much as possible to see whether these reports can published without problem.

Advanced Components 5-Important This criteria describes the product’s advanced functional components that desired to have.

Page 13: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

5

Report Designer Components Criteria

Impact Level Description

This is evaluated based on testing whether the product contains the following components, such as: Sub-reports, Side-by-side report, Cross-tabs, Horizontal panning, Multi-column layout, CSS controlled format, etc.

2.3. Data Source For our business context, clients can have a wide variety of databases, therefore we would like the

product to support as many types of data sources as possible. This is also an important category in our

evaluation as many advanced reporting features are based on an appropriate data source.

Table 2-4 lists three evaluation criteria under data source category.

Table 2-4 Description of Data Source Criteria

Data Source Criteria Impact Level Description

Support Multiple Data Sources 10-Crucial This criteria describes whether multiple data sources and queries can be supported in one report. This evaluation is based on assessing how to include multiple data source in one report and how many types of data source can be used in the product. According to our client’s requirement, the suitable tool should cover as much as possible common data source types. In addition, does it have a (graphical) query designer is also an essential aspect to considerate.

Support for Joining Multiple Data Sources in the Designer

5-Important This criteria describes whether joining multiple data sources can achieved in the product. This evaluation is based on assessing how to join multiple data sources in the product.

Support Non-JDBC Data Sources 5-Important This criteria is asking whether the product support to connect non-JDBC data sources or any advanced data source, such as CSV, EJB, Excel, Hibernate, JNDI, JSON, Mondrian, MangoDB. This is evaluated based on how many different types of non-JDBC data source can the product supported and assessing the ease of connection.

damyot
Highlight
Avoid...
damyot
Highlight
List them!
damyot
Highlight
Avoid
Page 14: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

6

2.4. Charting Criteria Evaluation Based on the client’s expectation, charting has been considered to be one of the most important criteria

when it comes to decision.

Table 2-5 lists five evaluation criteria under data source category.

Table 2-5 Description of Charting Criteria

Charting Criteria Evaluation Score Observations and Comments

Support for Common Chart Types

10-Crucial This criteria describes the product’s common chart types it must contained. This is evaluated based on testing whether the product contains the following chart types, such as: 2D, 3D, Pie, Multi-pie, Bar, Stacked Bar, Bar XY Line, Line XY Area, Area XY, Stacked Area, Bar Line, Bubble, Scatter Plot, Multi-Axis. Besides, the ease of creating a chart is also a major factor to be assessed here.

Chart Interactivity 5-Important This criteria describes the product’s ability in term of interactivity function in charting. This is evaluated based on assessing whether the product contains interactivities, such as: mouse-over, tool tips, drill-through, hyperlinks, hide/show, etc.

Advanced Charts 5-Important This criteria describes how many types of advanced charts the product contained, in order to meet the various needs of clients. The advanced chart types which our client is looking for are: Radar, Gantt, Ring, Tube, Cone, Waterfall, Difference chart. This is evaluated based on what type of advanced charts does the product have apart from the common types mentioned above and whether these charts are satisfied the client’s needs.

Variety of Chart Theme 1-Nice to Have This criteria describes the product’s choice of chart theme. This is evaluated based on how many chart theme does it offer.

Support for Scripting 5-Important This criteria describes the product’s feasibility to support scripting language in charting. This is evaluated based on testing whether the

damyot
Highlight
damyot
Highlight
Much better.
Page 15: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

7

Charting Criteria Evaluation Score Observations and Comments

product contains the function that enable the user to directly type in code from a supported scripting language and how many scripting languages are supported, which JavaScript is a mandatory requirement here.

2.5. Parameterization and Output Format Criteria Evaluation Instead of generating multiple reports to share a same layout and probably the same data,

parameterization is used to maintain the output of report that enables the report reader be able to

change the structure or value of some of the data.

Table 2-6 lists three evaluation criteria under Parameterization and Output Format category.

Table 2-6 Parameterization and Output Format Criteria Evaluation

Parameterization & Output Format Criteria

Evaluation Score Observations and Comments

Static Parameters Selection 10-Crucial This criteria describes the product’s ability to query certain parts of data by setting parameters. This is evaluated based on testing how many value types of parameter offered by the product, whether the default value can be changed based on the formula we specified.

Dynamic Parameters Selection 5-Important This criteria describes the product’s ability to add dynamic interactivity to a published report such that when a user executes or views it, he can specify how to constrain certain parts of the query data. This is evaluated based on testing the above features in our sample reports.

Multiple Output Formats 5-Important This criteria describes which output formats are available. This is evaluated based on how many types of output formats offered in the product. The minimum requirement on output formats included: HTML, PDF, Excel and XML.

2.6. Analytical Functionality Criteria Evaluation Table 2-7 lists five evaluation criteria under analytical functionality category.

Table 2-7 Analytical Functionality Criteria Evaluation

Page 16: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

8

Analytical Functionality Criteria Evaluation Score Observations and Comments

Common Aggregations 10-Crucial This criteria describes the common aggregation type each tool contains. The minimum requirement includes: sum, average count, min, max, and variance. This is evaluated based on testing whether the tool satisfies the minimum requirements and how many aggregations the tool can offer.

User-defined Aggregations 5-Important This criteria describes the product’s ability on user-defined aggregations. This is evaluated based on whether the user can define aggregations by using script language.

Slice & Dice 5-Important These analytical functional criteria is used to test whether any data mining/analysis function is built-in the produce. These criteria can be seen as an advanced functionality which our client considered to be desired to have within the product and can help our client to view the report through a dynamic approach. This is evaluated based on testing or assessing whether the followed functionalities are included in the produce: slice & dice, drill-down & roll-up, sorting and ranking.

Drill-down & Roll-up 5-Important

Sorting & Ranking 5-Important

2.7. Usability Criteria Evaluation Table 2-8 contains the list of usability criteria for this evaluation. User efficiency is always a crucial factor

to include for all evaluation. We have also include the documentation and support criteria, in order to

discuss how well or bad an open source tool can offer in term of available support.

Table 2-8 Usability Criteria Evaluation

Usability Criteria Evaluation Score Observations and Comments

User Efficiency 5-Important This criteria describes the product’s most functions and available actions are designed to be easy accessible and easy to understand. This is evaluated based on:

How long it will take to set up the environment?

Does the graphical user interface considered user-friendly?

Whether sufficient template and examples are available?

Whether prior technical knowledge is

damyot
Highlight
Expectations?
Page 17: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

9

Usability Criteria Evaluation Score Observations and Comments

required to create reports?

Does provide any preview regarding to query on data source, charting and report?

Product Documentation 5-Important This criteria describes the product’s availability of documentations as an open source tool. This is evaluated based on how many documentations offered on web, does it free and does it updated frequently, etc.

Support 1-Nice to Have This criteria describes the product’s availability of support as an open source tool. This is evaluated based on:

How well supported in terms of forum activity?

How many new posts per day in forum?

Does user need to pay for the support?

2.8. Installation and Implementation Criteria Evaluation Table 2-9 lists four evaluation criteria under installation and implementation category.

Table 2-9 Installation and Implementation Criteria Evaluation

Installation and Implementation Criteria

Evaluation Score Observations and Comments

Supports Multiple System Platforms

10-Crucial This criteria describes the product’s feasibility to run on which platform. This is evaluated based on:

Can the product be used on all three mainstream platforms: Windows, Linux and iOS?

What is the minimum system requirements to run this tool?

What is the minimum memory needed to run this tool?

Deploying Report Onto an Application Server

10-Crucial This criteria describes the product’s ability to deploy report on an application server. This is evaluated based on assessing:

Whether the product can deploy a report through an application server?

Does the application server is free of charge

Page 18: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

10

Installation and Implementation Criteria

Evaluation Score Observations and Comments

Support IDE Plug-in 1-Nice to Have This criteria describes the product’s feasibility of running as IDE plug-in. This is evaluated based on whether the product support IDE plug-in.

No Compilation Before Running Report

1-Nice to Have This is evaluated based on running sample reports, in order to assess whether compilation required before running it.

2.9. Miscellaneous Criteria Evaluation Table 2-10 lists two evaluation criteria under miscellaneous category.

Table 2-10 Miscellaneous Criteria Evaluation

Miscellaneous Criteria Evaluation Score Observations and Comments

Product Cost 1-Nice to Have This is evaluated based on how much does it cost to acquire the product and other associated products.

Product Stability 5-Important This criteria describes the stability of the product. This is evaluated based on observations of the product's performance in our experimental test environment. We will test the product by playing with it in a normal way or in an unusual way to test whether it will crash or not. And we will also judge it by checking how often the product gets updated, has new features released and reacts to reported bugs.

damyot
Sticky Note
Expectations?
Page 19: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

11

3. Methodology We will use a decision matrix to score and compare two BI tools. Each tool will be scored for evaluation

criteria listed above, from 0-3 (0-Not Available, 1-Minimal, 2-Proper, 3-Exceptional), as shown in Table 3-

1, and each criterion is assigned a relative weight (1-Nice to Have, 5-Important, 10-Crucial), based on the

view of the importance of each factor in our evaluation.

Table 3-1 Criteria Evaluation Score

Evaluation Score Score Description

Not Available 0 This element is not available in the product.

Minimal 1 This element is present but with limited features, which cannot fully satisfied the user’s requirements.

Proper 2 This element is present with comprehensive features, which satisfies the user’s requirements

Exceptional 3 This element is present with outstanding features, which is far beyond the user’s requirements.

A sample retailer database will be used to develop BI application using both tools, on which we can run

tests with regards to selected criteria. The results of the tests will be collected using a spreadsheet that

represents the evaluation matrix. The tool with a higher and satisfactory overall score will be the one

recommended for Ivy Technologies management team. For a close contest, the one scores high on

crucial and important criteria will be selected (a draw on crucial and important criteria indicates that

they are homogeneous tools, thus we need to include extra considerations such as availability of

technical support, potential features offered in the future, etc). If both tools exhibits poor performance,

alternatives will be considered and discussed.

Page 20: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

12

4. Evaluation of Pentaho Report Designer This section will perform the evaluation of Pentaho Report Designer (PRD) based on the evaluation

criteria and methodology we introduced in Section 3.

Pentaho is a complete BI suite, consisted of several open source projects, such as reporting, analysis,

dashboards and data mining. This evaluation report focuses on the reporting capabilities which the

project named Pentaho Report Designer. The version under evaluation is Pentaho Report Designer 5.2

which is free and open source software available under the terms of the GNU Lesser General Public

License (LGPL) version 2.1 [3].

Pentaho Report Designer provides a graphical user interface to help user design and publish any style of

report easily, which is totally different from the traditional way of creating reports programmatically or

by creating and manipulating XML. In addition, Pentaho Report Designer runs report designs against

data sources and renders report output in HTML, PDF, Excel and other formats.

When users start the journey from PRD, the first view is the welcome screen as shown in Figure 4-1. The

purpose of this window is to provide a simple and quick process for new user by clicking one of the

buttons named Report Wizard or New Report. In addition, this screen also shows a lists of sample

reports on the right side of screen, giving the user an intuitive view as to how to use a certain PRD

feature.

Figure 4-1 PRD UI with Welcome Window

Normally, the process of creating a report in PRD will include the following process [2]:

Page 21: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

13

Connect to a data source;

Write a query to select the data according to your needs;

Organize data elements in the PRD workspace;

Adding aggregates based on your needs; and

Finally, publish the report, either to the Pentaho BI Server or supported output format.

Figure4-2 shows the preview report in html output format.

Figure 4-2 Preview report in html output format

4.1. Report Designer Components Criteria Evaluation Whenever using a new product, the first thing user need to learn is how to navigate the user interface,

before they move on to more complex tasks. PRD consists of several components as follows [2]:

Page 22: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

14

The Welcome Screen: provides a quick, four-step process that walks the user through creating a

new report through the Report Design Wizard.

The Report Designer Main Toolbar: makes some of the most frequently used features more

accessible to users, such as: file, data source, publishing report and cut-and-paste operations.

Report Designer's Tabbed Views: views each report or sub-report, similar to text editors and

web browsers.

The Report Workspace: the place to design and customize the report. This component

dominated by the layout bands which define each individual section of the report.

The Structure Pane: shows the hierarchy of every element included in a report.

The Data Pane: enables the user to add data sources and view the individual queries, functions,

and parameters in each report.

The Style Pane: displays all of the visual and positional style options for any given item in the

Structure pane.

The Attributes Pane: displays all of the low-level properties, and input and output options for

any given item in the Structure pane.

The Palette: contains all of the elements that users can add to build a report.

Table 4-1 contains the evaluation scores, observations and comments of the report designer

components criteria for PRD.

The evaluation of this product resulted in a final weighted score of 40 out of 45 for the report designer

components criteria.

Table 4-1 Report Designer Components Criteria Evaluation

Report Designer Components Criteria

Evaluation Score Observations and Comments

Common Report Designer Components

Exceptional (3)

As listed above, Pentaho Report Designer has included all basic user interface components, in order to help users design and publish any style of report easily. We have tested a couple of sample reports, such as product catalogue, revenue by product line, top n customers, html action report, which all of them can be successfully created and generated in the web browser.

Advanced Components Proper (2)

PRD does have some advanced components to provide a wide variety of choice, such as: Geometric shapes, Barcodes, Table of Contents, Sub-reports, Side-by-Side report, Hyperlinks within a report, etc. Sub-report is an important component in PRD, as its use sub-report to support reports with multiple queries, multiple group sections and multiple data sources.

Page 23: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

15

Report Designer Components Criteria

Evaluation Score Observations and Comments

But some components like cross-tabs, horizontal panning and multi-column layout are not available in Pentaho.

4.2. Data Source Criteria Evaluation The second step of creating a report is to setup a data source connection. Figure 4-3 displays the

database connection wizard in PRD, which the user need to first choose a connection type, then provide

all setting information and run a test to verify the status of connection. Once the connection is valid, the

user can write a query or use a SQL query editor to refine that data source such that it only contains the

information specified by the user.

PRD supports the following data sources types [2]:

JDBC: 47 JDBC-compliant database are included in PRD, however users will probably provide the

appropriate JDBC JAR file to the Pentaho directory.

Metadata: A Pentaho Metadata XMI file.

Pentaho Data Integration (kettle): Kettle KTR files can act as a data source, but extra JAR files are

required.

OLAP: only support OLAP source created by Pentaho Analysis (Mondrian) application.

XML.

XLS: a data table which imported from excel spreadsheet file.

MangoDB: use data stored in this document-oriented non-relational database.

Page 24: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

16

Figure 4-3 Data source connection in PRD

Table 4-2 contains the evaluation scores, observations and comments of the data source criteria for PRD.

The evaluation of this product resulted in a final weighted score of 55 out of 60 for the data source

criteria.

Table 4-2 Data Source Criteria Evaluation

Data Source Criteria Evaluation Score Observations and Comments

Support Multiple Data Sources Exceptional (3)

Pentaho can connect with 47 data source through JDBC and other types of data source as discussed above which fully covers the databases used by the clients of Ivy Technologies. Multiple data source and queries can achieved by using sub-report. By testing the connection with our sample database, we found that the tutorial or the data source connection wizard is not straightforward.

damyot
Sticky Note
Does it have the ones you need?
Page 25: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

17

Data Source Criteria Evaluation Score Observations and Comments

Support for Joining Multiple Data Sources in the Designer

Exceptional (3)

Pentaho Data Integration (Kettle) is another tool in Pentaho BI Suite, which can provide the data source to PRD. Using Kettle with PRD allows users to specialize on either the data acquisition or presentation [4].

Support Non-JDBC Data Sources Proper (2)

Again, PRD can support most of non JDBC data source by itself, but still requires extra support from Pentaho Data Integration to provide data source as Excel, Hbase and JSON [4].

4.3. Charting Criteria Evaluation By testing PRD, the process of creating a chart is seen as the most difficult task in the whole procedure

of designing a report, which the user need to fill required property value in a large dialog box as shown

in Figure 4-4. Furthermore, no wizard to guide users through the whole process, which makes it even

more difficult. During our testing, we also found that, whenever the user fill in some values under each

category in property dialog and decide to change a chart type by clicking the type illustrated along the

top, they are likely to lose every inputs in the editor without any warning message pop-up.

Figure 4-4 Chart editor in PRD

damyot
Sticky Note
Tested it? Not obvious.
Page 26: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

18

Table 4-3 contains the evaluation scores, observations and comments of the charting criteria for PRD.

The evaluation of this product resulted in a final weighted score of 55 out of 78 for the charting criteria.

Table 4-3 Charting Criteria Evaluation

Charting Criteria Evaluation Score Observations and Comments

Support for Common Chart Types

Proper (2)

PRD supports 14 common chart types, includes: 2D, 3D, Pie, Multi-pie, Bar, Stacked Bar, Bar XY Line, Line XY Area, Area XY, Stacked Area, Bar Line, Bubble, Scatter Plot, Multi-Axis, which satisfied what we are looking for. However, one weak points of charting in PRD is does not have a charting wizard to provide step-by-step guidance to create a chart. In addition, PRD provide property editor to set value and actions to each property while BIRT is providing dialog screens with guidance and structure to the property settings. This editor is also difficult to use as it does not offer users the pick-lists instead of a pop-up window that doesn’t provide any guidance on default inputs. This makes charting become more difficult in PRD.

Chart Interactivity Proper (2)

PRD does support a few html actions such as: on-click, double-click, on-mouse-down, on-mouse-up and hyperlinks. But, the following actions are not available in PRD: drill-through, hide/show series.

Advanced Charts Proper (2)

PRD supports 11 advanced chart types as follows: Ring, Waterfall, Step Area, Step, Difference, Radar, Thermometer, Survey Scale and Bar/Line/Pie Sparkline. However, some advanced charts are not included in PRD, such as: Study, Tube, Cone, Pyramid, Time Series, Meter and Gantt chart, which some of them are required by our clients.

Variety of Chart Theme Not Available (0)

PRD do not provide any chart theme.

Support for Scripting Exceptional (3)

The script category in PRD enable users to directly type in code from a supported scripting language as follows [2]:

JavaScript

Page 27: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

19

Charting Criteria Evaluation Score Observations and Comments

Groovy

Bean Shell

Netrexx

XSLT

JACL or Jython

4.4. Parameterization and Output Format Criteria Evaluation In PRD, parameters are included in data pane that enables users to add data sources and view queries,

functions and parameters in each report (see Figure 4-5). Once the query is well formed, we can also

add a parameter to the report, which enables report readers to customize the content of the output.

Right click the parameter icon in data pane, a parameter editor will pop-up. In parameter editor, users

can choose the value type from Number, Date, Time, Integer, Float, Double etc., and create a formula in

a formula editor.

Figure 4-5 Parameter editor in PRD

Table 4-4 contains the evaluation scores, observations and comments of the parameterization &output

format criteria for PRD.

The evaluation of this product resulted in a final weighted score of 55 out of 60 for the parameterization

&output format criteria.

Page 28: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

20

Table 4-4 Parameterization and Output Format Criteria Evaluation

Parameterization & Output Format Criteria

Evaluation Score Observations and Comments

Static Parameters Selection Exceptional (3)

PRD supports static parameters select parameter values from a hard-coded list of values During our testing, we can have an option in the report to change data values in a drop-down list. These kind of parameter are added after the data structure has been defined through a query [2].

Dynamic Parameters Selection Exceptional (3)

PRD supports dynamic parameters users select parameters from a list of values that came from a database.

One example is that we can add dynamic interactivity to a published report, which a user can specify how to constrain certain parts of the query data while viewing it. However, this requires a Pentaho Analysis (Mondrain) data source type, in order to create an OLAP-based report [2].

Multiple Output Formats Proper (2)

By using PRD, users can publish the report in a wide variety of output formats, includes: HTML, PDF, Excel, Plain Text, Rich Text and CSV. However, some common output format, such as: PowerPoint (PPT), Flash, Postscript and Open Office report types, does not available in PRD. Another reason that PRD is not excellent under this criteria is due to the “pixel positioning” of report design, which means that the report cannot adapt to different-sized displays.

4.5. Analytical Functionality Criteria Evaluation In order to perform analytical functionality in PRD, it requires the data source should arrived in an

appropriate format.

Table 4-5 contains the evaluation scores, observations and comments of the analytical functionality

criteria for PRD.

The evaluation of this product resulted in a final weighted score of 65 out of 90 for the analytical

functionality criteria.

Table 4-5 Analytical Functionality Criteria Evaluation

Analytical Functionality Criteria Evaluation Score Observations and Comments

Common Aggregations Exceptional PRD provide 27 types of mathematic

damyot
Sticky Note
Ok
Page 29: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

21

Analytical Functionality Criteria Evaluation Score Observations and Comments

(3) aggregations which fully covered the minimum requirement of clients. You can define the function to use to aggregate the attribute in attributes pane.

User-defined Aggregations Exceptional (3)

Similar like BIRT, users are allowed to define expressions based on filter conditions.

Slice & Dice Minimal (1)

Pentaho has a broad vision to provide a comprehensive solution for Data Integration and Business Analytics. This explains why Pentaho BI Suite was included a whole set of applications from reporting to data mining, in order to provide mutual support between each other. For this reason, the analytical functionality, such as: slice and dice, drill-down and roll-up, requires support from another application named Pentaho Analysis in BI Suite in order to perform. PRD needs to set up a Pentaho Analysis data source and add an MDX query to retrieve a data set from a multidimensional database[2]. This also means that when using only Pentaho Report Designer, it can be more difficult to create complex reports. That is why it only scored 1 out of 3 here.

Drill-down & Roll-up Minimal (1)

Sorting & Ranking Proper (2)

4.6. Usability Criteria Evaluation As mentioned above, PRD has a nice and user-friendly report wizard to get started with, which can be

very helpful when users are new to the product or when they want to initialize a layout of a report

quickly.

Table 4-6 contains the evaluation scores, observations and comments of the usability criteria for PRD.

The evaluation of this product resulted in a final weighted score of 22 out of 33 for the usability criteria.

Table 4-6 Usability Criteria Evaluation

Usability Criteria Evaluation Score Observations and Comments

User Efficiency Proper (2)

Pentaho Report Designer is easy and simple to learn how to create common reports. Its UI is concise and offers most commonly used functionality in a good layout. Nevertheless, charting function was cluttered with sophisticated attribute setting dialog box, which makes charting is more difficult. Also, there is no chart preview, i.e. users cannot see

damyot
Highlight
Don't discuss BIRT here; not covered yet!
damyot
Sticky Note
Then maybe a fairer comparaison would have been BIRT vs Pentaho Report Designer AND Analysis!
Page 30: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

22

Usability Criteria Evaluation Score Observations and Comments

the chart result until they run the report. In addition, the setup is very simple too, for example on setup in windows, download the zip distribution, extract PRD into any directory but recommended to "C:\Program Files\report-designer" directory, then start the application by executing (or double-clicking) either the "report-designer.bat" file or the launcher.jar file.

Product Documentation Proper (2)

Pentaho provide documentation for current and previous releases of all Pentaho products in two delivery formats: the Info Center web-based HTML and.zip archives that contain PDF files. The documentations and manuals are includes: user-guide, walk-through tutorials, tech tips, etc. However, all these materials are not categorised by product and not well organized, which means that it is takes time to dig out what users are looking for. In addition, the Javadoc HTML pages for the latest release of Pentaho Reporting are also available on web. One book named "Pentaho Reporting 3.5 for Java Developers" written by Will Gorman is also available online but it costs $16.99, which documented the reporting engine and most of the features of the report designer.

Support Proper (2)

Pentaho provides a community forum to answer all software related queries and provide a platform for users to exchange ideas. There are dozens of new posts are issued every day which indicates that the forum is quite active. Obviously, all the questions are answered by volunteers, so there is no guaranteed response time or level of service.

4.7. Installation and Implementation Criteria Evaluation

Table 4-7 contains the evaluation scores, observations and comments of the installation and

Implementation criteria for PRD.

Page 31: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

23

The evaluation of this product resulted in a final weighted score of 53 out of 66 for the installation and

Implementation criteria.

Table 4-7 Installation and Implementation Criteria Evaluation

Installation and Implementation Criteria

Evaluation Score Observations and Comments

Supports Multiple System Platforms

Exceptional (3)

PRD is able to run on Windows, Linux and iOS. Pre-requisites: Java 1.6 or higher Minimum hardware requirements [3]:

192MB of allocated heap-space to process reports

Additional memory and/or adjustments to the global configuration parameters of the reporting engine is required when reports with more than 400 pages or about 50 rows of data.

PRD requires a Java Runtime environment that is fully compatible to the Java Platform Specification 5.0 (JSR-176)

Deploying Report Onto an Application Server

Proper (2)

The report can be run by the Classic Engine or the Pentaho BI Server which is another application in Pentaho BI Suite [2]. The Pentaho BI Server is a Java EE web application that provides all services to manage reports in a Web-2.0 environment and also is free to use.

Support IDE Plug-in Not Available (0)

The Pentaho Report Designer is a stand-alone, desktop-installed client tool, and is not available as an Eclipse or NetBeans plug-in.

No Compilation Before Running Report

Exceptional (3)

Pentaho does not require that reports be compiled prior to running which makes publishing a report easier.

4.8. Miscellaneous Criteria Evaluation

Table 4-8 contains the evaluation scores, observations and comments of the miscellaneous criteria for

PRD.

The evaluation of this product resulted in a final weighted score of 13 out of 18 for the miscellaneous

criteria.

Page 32: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

24

Table 4-8 Miscellaneous Criteria Evaluation

Miscellaneous Criteria Evaluation Score Observations and Comments

Product Cost Exceptional (3)

PRD is an open source tool, however the enterprise edition with more advanced features, such as: analysis services, dashboard designer, interactive reporting, will approximate costs $11,000 per year for unlimited users.

Product Stability Proper (2)

Pentaho is performing well with none crushing during our test. Reporting bugs can be handled through discussing in the forum or reporting to JIRA system, which requires creating a JIRA login account but it is free [3]. One thing should mention here is, over the past a few years, Pentaho has shown the consistent improvement in its open source reporting tool.

4.9. Evaluation Summary PRD has much of the same functionality as the other report designers like BIRT. However, PRD does

have some strengths and weaknesses over the other report designers in the following ways:

Strengths:

PRD takes a great advantage on user experience and usability, especially provides the best build-

in report design wizard.

PRD reports are in XML format, which makes it possible to put it under revision control.

Pentaho provide a BI suite which contains several client applications for different tasks. If some

business are looking for a feature-rich BI solution, Pentaho could be a good choice.

Weakness:

The process of creating charts is one of the weak points in PRD, due to the lack of step-by-step

guidance and a user-friendly property editor.

PRD itself has limited functionality in terms of interactivity reporting. It requires extra support

from other applications in its BI Suite, such as: Pentaho Analysis and Pentaho Data Integration,

to provide data in a proper format in order to achieve analytic functionality.

In conclusion, based on our evaluation, PRD would be a good choice for creating a simple or a moderate

complexity report with less requirement on charting. Moreover, PRD integrates with other applications

of Pentaho BI Suite, in order to fulfil the requirement on data mining, data integration, OLAP analysis.

Final Score: 358

Satisfaction Percentage: 80%

Page 33: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

25

5. Evaluation of BIRT

BIRT, which is short for Business Intelligence and Reporting Tools, is a top-level Eclipse project. BIRT was

first proposed and sponsored by Actuate Corporation in 2004 and has been receiving contributions from

IBM, Innovent Solutions and the community ever since [5]. It provides a platform to create data

visualizations and reports that can be embedded into rich client and web applications, especially those

based on Java and Java EE.

The version we used for evaluation is Version 4.4.2 (can be downloaded from

http://download.eclipse.org/birt/downloads), an open source community version licensed under the

Eclipse Public License. It can be either installed by All-in-One package, or using the plug-ins for the

existing Eclipse environment, or through the Update Manager [6].

Developers who are familiar with Eclipse IDE will be comfortable with BIRT Report Designer since it is

part of the IDE. Figure 5-1 displays the standard UI of BIRT Report Designer, which integrates seamlessly

with Eclipse's project management features. In the upper left corner report items in the Palette pane

can be added to report using drag and drop. Next to the Palette is Data Explorer that shows data sources

(connections), data sets (queries) and report parameters. It is worth mentioning that BIRT allows the

reuse of report objects, such as tables, data sources and styles. These reusable objects are stored in a

library file under the Library Explorer.

Figure 5-1 BIRT Report Designer UI

BIRT uses data source elements to connect to database and uses data sets to retrieve data. Common

data sources like JDBC data source can be created by specifying the JAR files, database URL, username

and password, as shown in Figure 5-2.

Page 34: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

26

Figure 5-2 Create a new Data source

To build a data set, users can simply open the data set editor and define SQL queries either by typing the

whole clause or using drag & drop functions to select column names from Available Items on the left

hand side. The Preview Results function allows users to verify the retrieved data set before proceeding

to report layout design. Figure 5-3 illustrates a common SQL query in the editor.

Figure 5-3 Define a new Data set

Page 35: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

27

Once basic queries are defined, users will be able to add report items such as tables, grid, chart, images,

etc. Figure 5-4 displays a sample report that shows a bar chart with the top N customers by revenue,

followed by a listing of the top M% customers by revenue sorted by customer name. To verify if the

report is presenting the data as expected, users can preview the report by clicking on the View Report

button in the navigation bar. This will prompt the web browser to open the report, displayed in HTML

format. For example, Figure 5-4 is the preview of report designed in Figure 5-1.

Figure 5-4 Preview report in web browser

5.1. Report Designer Components Criteria Evaluation

BIRT consists of several components [7]:

BIRT Report Designer: It is an Eclipse perspective used to create BIRT report designs stored in an

open XML format.

Design Engine: It is responsible for producing concrete report designs and it allow modifications

to be made on these designs. The Report Designer uses Design Engine internally to construct the

XML designs.

Report Engine: The BIRT Web View uses Report Engine to generate and render reports. It is

basically a series of Java class files that can be deployed in any Java/J2EE environment. When it

is called in the context of an application, a report design is passed to it. Then it will access the

data source, retrieve the data, do the required operations such as aggregation, sorting, etc, and

then format the report for presentation as HTM or PDF or any supported output format.

Charting Engine: It is used to design charts that either can be integrated into BIRT designer or

into an application. The Design and Report Engines make use of this API to deliver Charts.

Page 36: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

28

BIRT Viewer: Although BIRT lacks an open source server component, it uses BIRT viewer on a

Java Application server preview reports within Eclipse.

Figure 5-5 [7] illustrates an architecture overview of BIRT. Basically it has two main components: a visual

report designer for creating BIRT Designs, and a runtime component for generating those designs that

can be deployed to any Java environment. BIRT also includes a charting engine that is both fully

integrated into the BIRT designer and can be used standalone to integrate charts into an application.

Figure 5-5 Architecture Overview of BIRT

Table 5-1 contains the evaluation scores, observations and comments of the report designer

components criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 40 out of 45 for the report designer

components criteria.

Table 5-1 Report Designer Components Criteria Evaluation

Report Designer Components Criteria

Evaluation Score Observations and Comments

Common Report Designer Components

Exceptional (3)

It includes all common report designer components, making it easy to design and generating common reports. Testing reports such as product catalogue, revenue by product line, top n customers, crosstab report can be successfully created and generated in the web browser.

Advanced Components Proper (2)

We tested some advanced report functionality and BIRT proved to be capable of carrying out most of these jobs. The tool supports Barcodes/sub-reports/side-by-side/Cross-tabs functions, but not includes Survey

Page 37: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

29

Report Designer Components Criteria

Evaluation Score Observations and Comments

Scale/Horizontal Panning/Multi-column layout

5.2. Data Source Criteria Evaluation For our business context, clients can have a wide variety of databases, therefore we would like the

product to support as many types of data sources as possible. The community version supports multiple

data sources, including JDBC, scripted, flat file, XML, POJO, EJB. It also provides Joint Data Set function

that combines two data sets from different sources.

Table 5-2 contains the evaluation scores, observations and comments of the data source criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 55 out of 60 for the data source

criteria.

Table 5-2 Data Source Criteria Evaluation

Data Source Criteria Evaluation Score Observations and Comments

Support Multiple Data Sources Exceptional (3)

Apart from JDBC connection, BIRT also supports EJB, XML, scripted and flat file data sources, which covers the majority databases used by the clients of Ivy Technologies. It should be noted that BIRT requires users to locate and install JDBC driver. In our testing we noticed that connecting to MS SQL Server Database is not very straightforward.

Support for Joining Multiple Data Sources in the Designer

Exceptional (3)

The tool supports both heterogeneous and homogeneous data set joins. This feature is very similar to SQL table joins, except that each joint data set can only involve two tables. But it is still possible to produce multiple dataset joins by chaining joint data sets, since a joint data set may be used to define another joint data set.

Support Non-JDBC Data Sources Proper (2)

The tool supports non-JDBC data sources, including XML, Web Service, Flat Files, EJB, Scripted. However it needs Source Forge Java Excel API to use excel. Currently the community version does not support JSON, Hadoop Hive.(some clients may use them as data source)

5.3. Charting Criteria Evaluation

BIRT supports a rich type of charts, such as bar, tube, pie, that comes in both 2D and 3D effects. Charts

can be easily created using the Chart Wizard, which takes users through the process of creating a report

Page 38: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

30

step-by-step. BIRT provides some advanced charting abilities with a few lines of custom code. Examples

include allowing the chart to further filter the data, use report parameters in the context of creating the

chart, and specify all kinds of interactivity for chart elements like data points, data series, and axes.

BIRT uses JavaScript for expressions, business logic, and integration with application-specific Java classes.

Chart interaction can be created by defining triggers (event-action pair) and add scripts. When the user

does a mouse click, changes focus, presses a key, and many other operations, it can trigger an action.

Actions can include going to a hyperlink, running a script, displaying a tooltip, highlighting the chart

element, or toggling the chart elements’ visibility. Figure 5-6 shows an example of defining an action

that invokes user-defined script when there's mouse clicking the chart.

Figure 5-6 Interactive chart by adding script

Table 5-3 contains the evaluation scores, observations and comments of the charting criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 66 out of 78 for the charting criteria.

Table 5-3 Charting Criteria Evaluation

Charting Criteria Evaluation Score Observations and Comments

Support for Common Chart Types

Exceptional (3)

Common charts are available in BIRT and can be created easily using Chart Wizard.

Chart Interactivity Exceptional (3)

The tool supports the following events: mouse-over tool tips drill-through

Page 39: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

31

Charting Criteria Evaluation Score Observations and Comments

hyperlinks hide/show series

Advanced Charts Minimal (1)

The community does support dynamic parameter selection and interactive charting, however, advanced charts like flash charts/gadgets/maps are not provided. These advanced functions are accessible through Actuate BIRT Designer Professional, which will cost approximately $400 per developer [8].

Variety of Chart Theme Minimal (1)

Same as above. The community version only provides limited selections of chart themes. Commercial version like BIRT Designer Professional provides developers with support for Flash Objects and produce rich, highly interactive and navigable reports with formatting and graphics.

Support for Scripting Exceptional (3)

Despite the limitations listed above, users will still able to leverage the power of JavaScript to add custom logic to the report. BIRT provides a rich scripting model based on the Mozilla Rhino engine. Script events are defined for Report Object, Report Elements and Data Sets. Users can create event handlers to override these script events, thus altering the report content. Selecting the Palette view while in the Script editor will display functions and variables that are available in the given event for the selected report element.

5.4. Parameterization and Output Format Criteria Evaluation BIRT provides an API for working with parameters and a default UI that prompts the user to enter

parameter values. The values will be available within the report where they can be displayed, passed to

a query, used in expressions, etc. Figure 5-7 shows a cascading parameter group allows a group of

parameters to be interlinked, where selecting a value for the first parameter affects the choices

available in the subsequent parameters.

BIRT stores a report design in XML format and has a ".rptdesign" suffix. The XML design file is created by

the BIRT designer and directly executed by the BIRT engine. Currently BIRT supports HTML, Paginated

HTML, PDF, WORD, XLS, PostScript, etc.

Table 5-4 contains the evaluation scores, observations and comments of the parameterization &output

format criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 55 out of 60 for the parameterization

&output format criteria.

Page 40: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

32

Figure 5-7 Cascading parameter group in BIRT

Table 5-4 Parameterization and Output Format Criteria Evaluation

Parameters & Output Format Criteria

Evaluation Score Observations and Comments

Static Parameters Selection Exceptional (3)

BIRT allows parameter values to be selected from a hard-coded list of values. Using parameters to filter the data also applies to the charts, and specify all kinds of interactivity for chart elements.

Dynamic Parameters Selection Exceptional (3)

BIRT allows parameters to be selected from a list of values that came from a database. An example is to build a master report and a detail report. The master report lists orders and allows the report user to click on a particular order number to drill into a detail report for the selected order. The hyperlink property within the master report is generated dynamically based on results retrieved from a query.

Page 41: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

33

Parameters & Output Format Criteria

Evaluation Score Observations and Comments

Multiple Output Formats Proper (2)

No pixel positioning: Although reports can be rendered in versatile formats, sometimes developers cannot precisely control the placement of each element in the report [12]. For reports that need to be printed with precise placement of each report element on the page, a report designer with a “pixel positioning” approach to report layout is required.

5.5. Analytical Functionality Criteria Evaluation

As long as data set returns data in tabular format, BIRT can perform sorting, aggregation, grouping

operations.

Crosstab functionality within BIRT can include data cube from defined queries. By editing the Crosstab’s

properties, users will be able to add subtotals and grand totals, specify sorting, format the individual

elements of the crosstab. The data cube element within BIRT provides some basic analytical

functionality which allows the developer to build cubes based on existing data sets. These cubes are

constructed using dimensions and measures as illustrated in Figure 5-8.

Page 42: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

34

Figure 5-8 Define Data Cube in BIRT

Table 5-5 contains the evaluation scores, observations and comments of the analytical functionality

criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 60 out of 90 for the analytical

functionality criteria.

Table 5-5 Analytical Functionality Criteria Evaluation

Analytical Functionality Criteria Evaluation Score Observations and Comments

Common Aggregations Exceptional (3)

BIRT supports a wide variety of totals: sum, running sum, average, minimum, maximum, count, count distinct, and many more. Common aggregation operation comes in an easy way. User can drag and drop an aggregation element from Quick Tools and specify the function and expression needed for a specified aggregation.

Page 43: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

35

Analytical Functionality Criteria Evaluation Score Observations and Comments

User-defined Aggregations Exceptional (3)

Apart from common aggregations, users are allowed to define expressions based on filter conditions. This can be done in the Expression Builder using JavaScript.

Slice & Dice Not available (0)

Despite the fact that it is possible to perform data mining by adding Data Cubes, the analytical functionality is rather limited. Currently the community version do not support advanced BI functions such as slice and dice, ad hoc query, forecasting. These extensions are available in the commercial version, Actuate OLAP tool BIRT Analytics, which is able to perform advanced and predictive analytics.

Drill-down & Roll-up Minimal (1)

Sub-reports, master & detail report, and crosstab report functionalities enable users to conduct drill-down on reports. However the function is still limited in that user cannot define dimensions or choose measures on-the-fly. Again these extensions are available in Actuate OLAP tool BIRT Analytics, the commercial version.

Sorting & Ranking Proper (2)

BIRT provides mapping and sorting features, which can be used in many places combined with each other, and turn out to be very flexible. Still these features have their limitation in the community version, since they are achieved through pre-defined parameters instead of on-the-fly.

5.6. Usability Criteria Evaluation

We give BIRT high marks on usability as users can easily design common reports through UI and Wizards.

We created a series of sample reports following the BIRT tutorial and came to have the general

impression that the tool is fairly simple to start with. We can usually figure out what to do without

resorting to the manuals. Upon mastering a few basic BIRT concepts, such as Datasets, Table Binding,

Expressions, most developers will be able to build complex reports.

Table 5-6 contains the evaluation scores, observations and comments of the usability criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 27 out of 33 for the usability criteria.

Table 5-6 Usability Criteria Evaluation

Page 44: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

36

Usability Criteria Evaluation Score Observations and Comments

User Efficiency Exceptional (3)

The tool runs in Eclipse environment. During our testing, the set up of this environment takes 20-30 seconds on average. Since developers at Ivy Technologies are very familiar with Eclipse IDE, the tool should be fairly easy to use. The preview of the report is very convenient, with just a click and the user will instantly see the render effect in web browser.

Product Documentation Proper (2)

Users can always resort to BIRT documentation: http://eclipse.org/birt/documentation/ BIRT documentation is consisted of a great deal of repository of samples, tutorials and FAQs. The documentation is complete, well organized and effective. It available for free as part of the product package. However we do noticed that some documentation only applies to older version. We encountered some issues when trying to applying these documentation to our designs.

Support Proper (2)

BIRT has abundant external resources for developers to find guides, share ideas, as well as download examples. Also there is a forum area to search and ask questions on the BIRT Developer Center. As an open source project, however, the support is not 7/24.

5.7. Installation and Implementation Criteria Evaluation

BIRT can be downloaded and installed through BIRT homepage, with the All-in-One package being the

fastest way to get started. As mentioned in previous section, BIRT does not have an open source server

component, instead it uses report viewer for previews and deployment.

Table 5-7 contains the evaluation scores, observations and comments of the installation and

Implementation criteria for BIRT.

The evaluation of this product resulted in a final weighted score of 56 out of 66 for the installation and

Implementation criteria.

Table 5-7 Installation and Implementation Criteria Evaluation

Installation and Implementation Criteria

Evaluation Score Observations and Comments

Supports Multiple System Platforms

Exceptional (3)

BIRT work on any platform on which Eclipse itself works. However, the product is built on

Page 45: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

37

Installation and Implementation Criteria

Evaluation Score Observations and Comments

Windows, which is the best-tested platform. The package includes everything needed to start creating BIRT designs. Pre-requisites: Java 1.6 JDK/JRE. Minimum hardware requirements [10]:

1GHz processor or faster;

512 MB RAM (1 GB recommended);

352 MB of disk space is required for product package and additional disk space is required for designing reports.

Deploying Report Onto an Application Server

Proper (2)

BIRT Viewer can be deployed onto the following application server: Tomcat, Jboss, WebSphere, WebLogic, Oracle Application Server, Sun's Application Server, etc [11]. In addition, starting from release 3.7, BIRT offers POJO Runtime to deploy POJO Viewer to different application servers.

Support IDE Plug-in Exceptional (3)

The BIRT designer is a set of Eclipse plug-ins that lets users to build reports as a perspective from within Eclipse.

No Compilation Before Running Report

Exceptional (3)

The BIRT Report Engine directly executes the report design XML file. This is one reason that BIRT uses interpreted JavaScript instead of compiled Java as its scripting language.

5.8. Miscellaneous Criteria Evaluation

BIRT stores report design in a "design file." This file is in XML format and has a ".rptdesign" suffix. Like

Pentaho, since reports are in XML format, they can be effectively put under revision control.

Table 5-8 contains the evaluation scores, observations and comments of the miscellaneous criteria for

BIRT.

The evaluation of this product resulted in a final weighted score of 13 out of 18 for the miscellaneous

criteria.

Table 5-8 Miscellaneous Criteria Evaluation

Miscellaneous Criteria Evaluation Score Observations and Comments

Product Cost Exceptional Community Version is free, however if switching

damyot
Sticky Note
Nice!
damyot
Highlight
Page 46: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

38

Miscellaneous Criteria Evaluation Score Observations and Comments

(3) to commercial version to get access to advanced functionalities, the cost is approximately $400 per developer.

Product Stability Proper (2)

BIRT uses Bugzilla to track bugs and enhancements. Users can find BIRT-related bugs using the Bugzilla search facility. We noticed there are some reported stability issue from Eclipse Community Forums. The tool crashed once in our testing. With that said, overall the stability performance is satisfactory. It should also be noted that since 2012 BIRT has shown signs of slowdown in terms of adding new features [4]. No major changes or improvements has been made in recent two years.

5.9. Evaluation Summary

BIRT is an Eclipse plug-in that allows users to design report templates and publish them in web-ready

formats. As a BI tool, BIRT provides most of the functionalities for creating and deploying reports to

enterprise applications. It satisfies our need to deliver enterprise portal applications that offer reporting

framework, and partially fulfills analytical functionality. Here is a summary of advantage and drawbacks

of this tool:

Pros

1. BIRT Report Designer is a robust, powerful tool for creating reports. It exhibits outstanding

charting capability.

2. Since Ivy Technologies does all its web application developments in Eclipse IDE, BIRT is fairly

easy to use (although it is occasionally too complex).

3. It supports reuse of reporting components: users can create libraries of shared resources that

can be dynamically used by multiple reports.

4. It can create complex reports with minimal custom code.

5. It presents flexible and extensive interactive charting function.

Cons

1. BIRT primarily focuses on reporting instead of analytics and therefore it is not appropriate to

work with OLAP data. To address this issue, Ivy Technologies may need to consider purchasing

the commercial version if clients require fully functional analytical capability.

damyot
Highlight
Page 47: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

39

2. It lacks a true report server, instead it uses the Viewer on a Java application server [12]. For

many applications, a report server provides functionality such as user management, report

management, security, notifications, and more.

Final Score: 372

Satisfaction Percentage: 83%

Page 48: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

40

6. Conclusions and Recommendations Table 6-1 Evaluation Scores Comparison

Criteria Weighted Evaluation Score

Pentaho BIRT

Report Designer Components 40 out of 45 40 out of 45

Data Source 55 out of 60 55 out of 60

Charting 55 out of 78 66 out of 78

Parameterization & Output Format 55 out of 60 55 out of 60

Analytical Functionality 65 out of 90 60 out of 90

Usability 22 out of 33 27 out of 33

Installation &Implementation 53 out of 66 56 out of 66

Miscellaneous 13 out of 18 13 out of 18

Total 358 out of 450 372 out of 450

The final results show that Pentaho is at par with BIRT, with BIRT scores a bit higher than Pentaho by 6%.

Neither of the tools scores zero on any crucial criteria. Of all the criteria, Pentaho scores higher on

analytical functionality while BIRT has better performance on charting, usability, installation and

deployment.

To summarize, Pentaho and BIRT prove to be excellent candidates and both satisfy our requirements.

Pentaho has stronger analytical capability and BIRT is easier to use, providing more interactivity. As open

source tools, however, some advanced features like ad-hoc queries, data exploration, interactive

dashboards are not available.

For our business context, we recommend BIRT because it is more user-friendly and it provides deepest

access to reporting functionality, with better chart implementation and rich extensions. Since our

evaluation does not include scalability testing, further investigation needs to be conducted if the client

adds new requirements such as the ability to handle large heterogeneous databases.

damyot
Highlight
good, not excellent!
Page 49: Business Intelligence and Reporting Tools Evaluation: BIRT ...site.uottawa.ca/~damyot/csi5112/project/BIRT-Pentaho.pdf · Business Intelligence and Reporting Tools Evaluation: BIRT

CSI 5112 Project Report Business Intelligence and Reporting Tools Evaluation

41

Reference [1] “ISO/IEC 9126,” Wikipedia, the free encyclopedia. 09-Mar-2015. [2] Design Print-Quality Reports with Report Designer, Pentaho Report Designer User Guide, Version 5.2,

30, September 2014, https://help.pentaho.com/Documentation/5.2/Version_5.2 [3] Pentaho Report Designer Version 5.2 Readme, 26, June 2014, https://github.com/pentaho/pentaho-

reporting/blob/master/README.md

[4] Open Source Reporting Review - BIRT, Jaspersoft, Pentaho, Innovent Solutions, February 2015, http://www.innoventsolutions.com/open-source-reporting-review-birt-jasper-pentaho.html

[5] About BIRT: http://eclipse.org/birt/about/ [6] Getting started with BIRT: http://eclipse.org/birt/getting-started/ [7] Architecture Overview of BIRT: http://eclipse.org/birt/about/architecture.php [8] About Actuate BIRT Report Designer Professional: http://www.actuate.com/be/download/Actuate-

BIRT-Report-Designer-Professional.pdf [9] Open Source Business Intelligence: http://www.innoventsolutions.com/ [10] BIRT FAQs: https://wiki.eclipse.org/BIRT/FAQ [11] BIRT Documentation: http://eclipse.org/birt/documentation

[12] Showdown of BI Tools: Pentaho vs Others: http://www.edureka.co/blog/pentaho-vs-jaspersoft-vs-birt/