Qlikviewforninjas Sample Preview

8/10/2019 Qlikviewforninjas Sample Preview

1/35


2/35

QlikView for Ninjas

The first and ultimate step to the exciting world of QlikView

Rajesh Pillai

2014 Rajesh Pillai


3/35

This book is dedicated to my son Rohan, my nephew Tanuv who being just 9.5 years has started

with CSS3, HTML and JavaScript, my wife Radhika and my parents and well wishers. This book isalso dedicated to all the participants who attended my QlikView training and has given some really

constructive feedback based on which this series of books have been structured.


4/35

Contents

Chapter 0 - Agenda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Case study - Adventure Works Cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1 - Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 - QlikView Development Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3 - Developer Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4 - Data and Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12QlikView Data Modelling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 - Connect, Select and Load . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

7 - Green, White and Gray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

SECTION 2 - Programming for QlikView Ninjas. . . . . . . . . . . . . . . . . . . . . . . . 24

Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25For..Next . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26If..then..elseif..else..end if . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29


5/35

Chapter 0 - AgendaLets quickly glance through the table of content in a nutshell.

SECTION 1 - QlikView Developer

Introduction

QlikView Development Methodology

Data and Scripting

Connect, Select and Load

Loading Data from SQL Server

Executing Stored Procedures Reading XML Data from SQL Server

Green, White and Gray

The Magic of Preceding Load

Associations

Loading Data from XML file

Loading Data from Excel and CSV files

Data Modelling Issues

Referencing external files

Basic Data Transformation

Data generation in the QlikView script

Scripting Best Practices

Master Calendar

Mapping Tables

Data Model Optimization

Link Tables

Interval Match

Cross Tables

Advanced Calculations

Alternate States Set Analysis

Metadata

Generating QVDs

QlikView Security

Additional Load Types

How to use QlikView Extensions


6/35

Chapter 0 - Agenda 2

Performance Tuning

SECTION 2 - Programming for QlikView Ninjas

SECTION 3 - Advanced DevelopmentSECTION 4 - QlikView Designer

SECTION 5 - Basic Server Administration


7/35

Case study - Adventure Works CyclesThis case study is adapted from the Microsoft Sample database files. The detailed description can befound here.

Adventure Works Cycles, the fictitious company on which the AdventureWorks sample databasesare based, is a large, multinational manufacturing company. The company manufactures and sellsmetal and composite bicycles to North American, European and Asian commercial markets. Whileits base operation is located in Bothell, Washington with 290 employees, several regional sales teamsare located throughout their market base.

In 2000, Adventure Works Cycles bought a small manufacturing plant, Importadores Neptuno,

located in Mexico. Importadores Neptuno manufactures several critical subcomponents for theAdventure Works Cycles product line. These subcomponents are shipped to the Bothell locationfor final product assembly. In 2001, Importadores Neptuno, became the sole manufacturer anddistributor of the touring bicycle product group.

Coming off a successful fiscal year, Adventure Works Cycles is looking to broaden its market share bytargeting their sales to their best customers, extending their product availability through an externalWeb site, and reducing their cost of sales through lower production costs.

In all the subsequent chapters, we will cover various scenarios which will address some ofAdventureWorks concern. We will build our datamodel for them and also build some visualizationaround this area.

The reason I have taken AdventureWorks as a case study is because it is widely available andaccessible, the domain is pretty much familiar to most of the people and the data structure is prettysignificantly complex to understand various issues when we will transform from OLTP to OLAPkind of schema.


8/35

1 - IntroductionWelcome to the QlikView for Ninja series of book. This book will be a handy step by stepintroduction to most of the QlikView features that will help you become a better QlikViewDeveloper. This book is in early access mode and chapters will be published every week. Also,the readers constructive feedback, will help form the structure of the book as well apart from thecore agenda.

QlikView is a Guided Analytical BI tool developed by QlikTech(now Qlik).

QlikView is more of a business discovery platform. It helps you find data quickly and get out meaningfrom it to make efficient and quick decisions with minimum support from the developer team.

Though mostly the business user or the end user can directly work with QlikView but for efficientadministration and management the support of the IT team is required.

Also, QlikView is one of the foremost in memory analytics, i.e. everything, all data, including filterseveything is stored in RAM once the document is loaded. This makes it very fast from user experienceperspective. Also, with more power comes more responsibilities(remember the Spiderman movie).

The application needs to be tuned as it scales and many aspects of performance needs to be takeninto account beginning from chart optimization, data model design, server configurations, user loadsetc.

Associative Technology

How QlikView refers from other BI products the associative user experience. Traditional BI toolshas a fixed navigation path to explore data, but in QlikView, you can start any where, any columnor field and this gives your mind more opportunity to explore and deduce meanings from data.

The below diagram visualizes the two experiences.


9/35

1 - Introduction 5

Some of the benefits of associative technology is outlined below

Works the way the mind worksThe user is not limited to follow a predefined path to access the data. The business user cansee hidden trends and make discoveries like with no other BI platform on the market.

Delivers direct and indirect searchThe user can conduct both direct and indirect searches. For e.g. if a user wanted to identifya sales rep but cant remember his/her name, he can just use some other attributes to do thesearch. If the business user remembers that the said sales rep sells laptop in APAC region, ecan search on the Sales Rep list box and search for APAC and Laptop to get the names ofsales reps who meet those criteria.

Delivers answers as fast as users an think up questionsThere are many ways in QlikView in which a user can asks question, such as putting data incharts and graphs, maps, tables, sliders, calenders etc. The user can quickly see relationshipsand find meaning in the data.

Puts in the meaning to the grayIn QlikView unassociated data is represented in gray color. The user can easily see the datanot associated and this gives them additional insight into their data. Sometimes the Ahamoment comes up by looking at data that is not directly associated.

** Components of the QlikView Business Discovery platform **


10/35

1 - Introduction 6

The below figure depicts a simplified view of QlikView deployment diagram containing the productsthat take part in the deployment.


11/35

1 - Introduction 7

QLIKVIEW DESKTOP

The QlikView Desktop is a windows based desktop tool that is used by developers and business

analysts to create data models and to build the graphical user interface for QlikView apps. This isthe swiss army knife for the QlikView developer and Designer who can use all the features to createrobust data model and a very efficient user interface for the clients. All the scripting is done throughthis application.

The file type that is created using the QlikView Desktop is known as QVW (.qvw or QlikView file).One can also create a readonly QVD (QlikView data) file, which is a format which QlikView uses tostore its compressed data.

QLIKVIEW SERVER (QVS)

The QVS is a server product that contains the in-memory analytics engine. The QVS handles all

client/server communication between a QlikView client (i.e. desktop, AJAX, IE plugin or Mobile) andthe server. It includes a management environment i.e QlikView Management Console for providingadministrator access to control all aspects of the server deployments including security, distribution,clustering, authorization etc) and also included a web server to provide front-end access to thedocuments within.

The web servers user portal is known as Access Point. Also, its important to note that whileQVS contains its own web server, once can also utilize Microsoft IIS (Internet Information Sever) for


12/35

1 - Introduction 8

this purpose as well. The QVS handles client authorization against existing directory providers likeMicrosoft Active Directory, LDAP etc and also performs read and write to ACLs (Access Controllists) for QVW documents.

QLIKVIEW PUBLISHERThe QlikView Publisher is also a server-side product that performs two important functions:

1. It is used to load data directly from data sources defined via connection strings in the QVWfiles.

2. It is also used as a distribution service to reduce data and applications from source QVW filebased on various rules, such as user authorization or data access privileges (these rules arebased on fields from the data model) and to distribute these newly-created documents to theappropriate QlikView servers or static PDF reports via email.

QlikView publisher is not a mandatory server component though, but is useful for large scaleenterprises. When QlikView Server is installed the publisher component is also installed, but it isonly activated when the publisher license is enabled in the QlikView Management Console.

** How QlikVIew Works: A peek under the covers **

When a QlikView document is published to a QlikView Server, the content it contains becomeavailable for consumption by any user with the required privileges to access it. The flow is outlinedbelow:

When a user first opens a QlikView document, data is loaded in memory (server memory).The compressed and unaggregated dataset is loaded from the disk into QlikView ServersRAM. This in-memory repository serves as the based dataset for this initial user and all otherusers requesting the same document. This repository stays in memory until no user activityhas occurred within a defined time-out period.

Users explore data via selections. The concept of user defined selection state is central toQlikVIew. As user clicks around in a QlikView document trying to demystify the maze ofdata, they indicate which subsets of data they are interested in analyzing and which subsetsshould be ignored. QlikView takes advantage of the highly indexed nature of the unaggregateddataset. QlikView dynamically presents a subset of all the data available to the QlikViewdocument based on the selection state. This happens in real time.

Upon selection, aggregates render instantly. On the fly, QlikView renders aggregates as

intuitive and interactive user interface objects via, charts, graphs, tables, listbox etc. Usersinteract with objects in QlikView documents through any supported client. Also users cancreate their own objects using collaboration features of QlikView.

We will have a look at how data is structured in QlikView memory model in laterchapters.


13/35

2 - QlikView DevelopmentMethodology

Each and every company and may be individuals have their own development methodology.Now QlikView doesnt put a hard and fast rule as to what kind of methodology works, but itsrecommended to go with any kind of agile methodology to get the most from the project in timeand budget.

Irrespective of the selection of any specific methodology, some coding practics and developmentguidelines needs to be strictly followed to get optimum result These practices are spread across the

following major activities.

Scripting Guidelines

Requirements Gathering,

Understanding the key metrics and KPIs

Source control integrations

Understanding the source data et el.

We will cover these topics incrementally over the next series of chapters.


14/35

3 - Developer RolesFor any successful QlikView implementation it is very essential to have the correct team structure.This team structure could compose of single person to start with but eventually for all different rolesthere could be specialist appointed.

Lets look at roles available in any QlikView implementation.

The QlikView team structure consist primary of a backend developer, a designer, a visualizationexpert and an administrator. Each and every role has its own responsibility, but as I said, when youare starting up with QlikView implementation, may be a single person could be playing all the roles.But as you move forward into the project, a dedicated team needs to be setup to deal with the chores

of each roles.A successful implementation consist of a fine mixture of the above roles. The responsibilities of eachrole is outlined below.

The various roles are summarized the the below table.


15/35


16/35

4 - Data and ScriptingAll the data files used with this book will be available shortly. In the meantime you can grab theprimary source from CodePlex. We will be using the AdventureWorks 2012 OLTP database for allour demo code. The required files can be found in the leanpub QlikView for Ninja home page.

QlikView Data Modelling

A data model is a conceptual model that describes how data are related and accessed in a system.The two main concepts in a data modelling is that of dimensions and facts.

** Dimensions and facts **

A dimension is used to categorize data, such as products, customers, region or ara etc. It consists ofone of more tables containing keys and attributes that describe the data values.

A fact table on the other hand generally contains the foreign keys of dimension tables along withmeasures (numeric data). Each row of the fact table is defined by the set of dimensional keys thatcontribute the measurements or measures. An example of a fact would be sales revenue, number ofissues resolved etc.

** Star schemas and snowflake schemas **

A schema is a way to graphically represent a model through a diagram. A data model with mergeddimensions can be represented by a star schema. The name derives from the fact that, a full fledgedstar schema resembles a start, with a central point and other other nodes erupting from it. Thisstructure contains a single fact table surrounded by a set of dimension tables.


17/35

4 - Data and Scripting 13

Snowflakes on the other hand more of a relational strucute, in which the dimension tables are notfully merged.


18/35


The general rule of thumb is to use star schema for analysis purpose and snowflake schema fortransactional database.

We will be build a data model that initially represents the first figure below and then through datamodelling activities we will make the model look as in the second figure.

Figure : 1


19/35


Figure : 2

Wishing you best in your data modelling adventures and we will use many of the ninja like tools toachieve our end objective.

At any point you can press Ctrl+T to bring up the table viewer in QlikView. Tableviewer is a handy tool that shows the struture of tables laid out in QlikViews memory.


20/35


It shows both the source structure and the internal table structure. We will use theinternal table structure view for troubleshooting most of our data modelling issues.

AdventureWorks Sample Database

https://msftdbprodsamples.codeplex.com/releases/view/55330
https://msftdbprodsamples.codeplex.com/releases/view/55330https://msftdbprodsamples.codeplex.com/releases/view/55330


21/35

5 - Connect, Select and LoadConnect, Select and Load is the basic mechanism through which data can be loaded into QlikView.Data can be loaded from various sources like flat files (csv, excel, tab delimited), RDBMS or anyother systems.You can also get/buy or write custom providers for data sources which QlikView doesnt providethe connectors.

Exercise 1. Load an excel file. This will be the only most elaborate step, starting with creating a newdoucment. Rest of the exercise will contain on the required screenshots for understanding.

1. Fire up your QlikView and click on New Document.

2. You will be greeted with the Getting started wizard. Close this by clicking on th X buttonor by pressing the ESC key.


22/35


23/35

5 - Connect, Select and Load 19

6. Click on the Relative Path check box and then click on the Table Files and select thespecial_offer.xlsx.

7. You will be presented with the File Wizard:Type dialog. Look at the File Type section onleft side. The file type Excel(xlsx) should be automatically selected.


24/35


8. Click on the Finish button for now and the following screen should be presented to you.

9. Click on the Reload toolbar menu to load the data into QlikView. You will get a progressdialog, which will be automatically closed or you have to manually close based on User


25/35


settings. We will see these settings in later chapters.

10. On the subsequent screen click on Add All and then click on OK.


26/35


11. The following will be the output of your hard labour. (Dont despair, this will get better). Clickon Layout menu and then click onRearrange Sheet Objects.


27/35

7 - Green, White and GrayThe color green, white and gray, has significant meaning within QlikView. Green color representsthe current selection or filter. White color represents records matching the current selections andGray color represents records not matching the selections.

Dont worry about how things are related. We will talk about relations in detail. For nowunderstand that if the name of the field matches, QlikView creates an association between the tablesautomatically.


28/35

SECTION 2 - Programming forQlikView Ninjas

Programming is a way to control the computer or rather in this case your QlikView applications.Programming is nothing more than a series of step by step instructions indication the computer todo a specific task.

The following figure depicts the basic element that constitutes any given program.

Lets dissect each of the above elements. In any way the above elements doesnt represent the entirething, but

they form the most important part from the whole.


29/35

SECTION 2 - Programming for QlikView Ninjas 25

Variables

Variables are placeholders in memory. They hold whatever value (number, strings, date) you put in

them. You canset a variable to a specific value and also read values from them. In QlikView there are two ways todefine a variable, by SET and LET statement.

SET statement can be used to define variables for lazy evaluation, i.e. variables assigned with SETis not immediately evaluated and is stored as is. This is useful for substituting strings, paths, drives,evaluating formulas

etc.

SET x = 3 + 2;

In the above statement the variable is evaluated as 3 + 2.

The LET statement on the other hand evaluates whatever is on the right hand side and store theresult in the variable.

For e.g.

SET y = 2 + 2;

In the above statement y contains the value 4.

Variables are a very powerful construct in QlikView (rather any programming environment). Itis variable that makes the system dynamic. You can change the behaviour of the application bychanging the variables. We will be heavily using variables when creating master calendars, qvd

generators, in reusing expressions, even multilingual design etc.

Loop

A loop represents repetition in the program. Loop is yet another powerful construct in anyprogramming environment. This helps us maintain our program/script and keep it small by avoidingrepetition. A loop executes so long as the condition defining the loop is met. There are various waysin which loops can be defined. Lets examine one at a time.

The do..loop control statement is a script iteration construct which executes one or several statementsuntil a logical condition is met. The syntax is:

do[ ( while | until ) condition ] [statements][exit do [ ( when | unless ) condition ] [statements]loop[ ( while | until ) condition ]

Where:

condition


30/35


31/35


1 [statements]

2

3 [exit for [ ( when \| unless ) condition ]

4

5 [statements]

next[counter]

Where:

counter is a variable name. If counter is specified after next it must be the same variable name asthe one found after the corresponding for.

expr1 is an expression which determines the first value of the counter variable for which the loopshould be executed.

expr2 is an expression which determines the last value of the counter variable for which the loopshould be executed.

expr3 is an expression which determines the value indicating the increment of the counter variableeach time the loop has been executed.

condition is a logical expression evaluating to true or false.

statements is any group of one or more QlikView script statements.

The expressions expr1, expr2 and expr3 are only evaluated the first time the loop is entered. Thevalue of the counter variable may be changed by statements inside the loop, but this is not goodprogramming practice.

If an exit for clause is encountered inside the loop, the execution of the script will be transferred tothe first statement after the next clause denoting the end of the loop. An exit for clause can be madeconditional by the optional use of a when or unless suffix.

Since the for..next statement is a control statement and as such is ended with either a semicolon orend-of-line, each of its three possible clauses (for..to..step, exit for and next) must not cross a lineboundary.

** Exercise: Loading variables defined in excel**

Lets create variables dynamically which is defined in excel. Later on we will see how to extend thisfunctionality.

For this use the file variables.xmls in the datasource folder.

Look at the figure below for the code. The full code in text format is given at the end of this section.


32/35


Heres is a line by line explanation.

Line 1 indicates that the file that is loaded uses relative path. This is optional, QlikView adds thiswhenever you check the Relative Paths checkbox.

Line 2 gives a new name to our table, i.e. VariableTable

Line 3 to 7 is the load statement

Line 10 is a comment statement. Comments are ignored by QlikView and the only reason for their

existence is for documentation purpose.Line 12 starts a for loop with variable i which starts with 0 and continues upto the count of rowspresent in the excel sheet. The NoOfRows(tablename) funtion takes a table name as a parameterand returns the count of rows. So, in our case there are 2 rows in the variables.xlsx file.

Line 13 creates a variable called vname that contains the name of the variable form the excelsheet. The peek() function in QlikView is used to read records from already loaded table. The firstparameter is the name of the field to read, the second parameter is the record number (record no.starts with 0 in QlikView) and the third parameter is the name of the table.

Line 14 creats a dynamic variable using $ sign expansion, which contains the value of the variable.So first the when the loop is executed the following will be the state of the program.

1 Let vName = "vSourceQvdPath";

2

3 \$(vName) = "c:\\qlikview\\apps\\qvd\\";

Line 15 will continue the loop.


33/35


Line 18 and 19 removes the unwanted variable.

Line 21 drops the table (as it is no longer needed and the variables are already created).

When you reload this script and inspect the variable (by clicking on Settings->Variable Overview)

you can see the variables that you created in the script.

Here is full code.

1 Directory;

2 VariableTable:

3 LOAD VariableName,

4 VariableValue

5 FROM

6 DataSources\variables.xlsx

7 (ooxml, embedded labels, table is Variables);

8

9

10 // Set variables for each row in the sheet

11

12 for i = 0 to NoOfRows('VariableTable') - 1

13 LET vname = peek('VariableName', i, 'VariableTable');

14 LET $(vname) = peek('VariableValue',i,'VariableTable');

15 next i

16

17 DROP Table VariableTable;

If..then..elseif..else..end if

The if..then control statement is a script selection construct forcing the script execution to followdifferent paths depending on one or several logical conditions. The syntax is:

1 if condition then

2

3 [ statements ]

4

5 { elseif condition then

6

7 [ statements ] }

8

9 [ else

10

11 [ statements ] ]


34/35


12

13 end if

1 Where:

2

3 condition is a logical expression which can be evaluated as true or

4 false.

5

6 statements is any group of one or more QlikView script statements.

Since the if..then statement is a control statement and as such is ended with either a semicolon orend-of-line, each of its four possible clauses (if..then, elseif..then, else and end if) must not cross aline boundary.

** Exercise: Lets only do an actual reload of the document only if the reload parameter is set inreloadflag.txt file **

The reloadflag.txt file can be found in the DataSources folder. In actual case, this file may be storedin a remote accessible shared location.

Please go through the line by line explanation of the above script.

Line 1 assigns a tablename to the load statement.

Line 2 Since the text file doesnt have any header we are indicating to QlikView that we only needto load the First 1 record. QlikView will automatically assign a column name [@@] 1 if no columnheaders is present.

Line 15 creates a varaible named reload, which contains the value from the text field.


35/35


Line 17 Trace simply outputs whatever is feeded to it. This can be viewed in the Script ExecutionProgress Dialog.

Line 19 check whether the reload value of the reload variable after coverting it to upper case is equal

to FALSE.Line 20 Will exit the script if Line 19 statement is evaluated to true (reload is false)

NOTE: Sometimes the Script Execution Progress Dialog just closes of automatically after the reload.You can change this behaviour by setting the Keep Progress Open After Reload value checked inthe Settings->User Preferences menu.

Here is the full code.

1 [ReloadFlag]:

2

3 First 1

4 LOAD *

5 FROM

6 DataSources\reloadflag.txt

7 (txt, codepage is 1252, no labels, delimiter is '\t', msq);

8

9 LET reload = Peek('@1',0,'ReloadFlag');

10

11 // Just for troubleshooting. This value will be shown in the Script Execution P\

12 rogress Dialog

13

14 TRACE reload;15

16 if upper(reload) = 'FALSE' then

17 exit Script

18 end if

Documents

Qlikviewforninjas Sample Preview