Seminar: Data Modeling for Optimization with MPL - Oct 2012

1Copyright © 2012 Maximal Software, Inc. All rights reserved

Data Modeling for Optimisation

The Database Connection in MPL

The database connection in MPL has the ability to access data from many different sources, such as:

• Relational Databases• Excel Spreadsheets• External Text Files• Internet

This gives the model developer the flexibility to choose the most efficient and convenient way to incorporate the data into the model. Among the data formats that are supported by MPL are:

• Microsoft Access and Excel• ODBC• FoxPro/DBase• SQL Server• Oracle• SAP



Import Indexes from Database

MPL allows you to import the elements for an index directly from a database. In the INDEX section, where you define the index, enter the keyword DATABASE after the assignment symbol (:=) followed by parentheses containing the table name and the column/field name you want to import from.

INDEX depot := DATABASE("Depots","DepotID");

In the above example, MPL will open the database table Depots, locate the column DepotID, and then read in the entries for the index depot.


DepotID Capacity

AtlantaChicagoNew YorkDallas

4000005000070000

100000

The Depots Table


Import Indexes from Database

In most cases the imported indexes are the key fields for the table which are underlined in the following examples:

The column name defaults to the name of the index so if it is the same you do not have to specify it. In the example below the column name in

the database table is DepotID which is the same as the index DepotID. INDEX DepotID := DATABASE("Depots");



Import Indexes from Database (Cont.)

MPL can import from more than one database in the same run. In the example below, MPL will read in the factory index from Access, instead of the default database:

INDEX

factory := DATABASE(Access, "Factory", "FactID");



MPL also allows you to import subset indexes from database tables. INDEX FactoryDepot[factory,depot] := DATABASE("FactDep");

This statement will open the database table FactDep and locate the columns for factory and depot and then read in the entries for FactoryDepot. The FactDep Table:

Notice that MPL automatically uses the same name as default for the columns FactID and DepotID, as in the original tables the indexes were defined from.

FactID DepotID TrCost Shipment

HoustonHoustonSeattleSeattleSeattleSeattle

ChicagoDallasAtlantaChicagoNew YorkDallas

320051002800680047005400

000000





If an index column does have a different name than in the original table, you can specify it following the table name by first entering the index name followed by an equal sign and the column name.

INDEX FactoryDepot[factory,depot] := DATABASE("FactDep", factory="Factory", depot="Depot");

This means if you are consistent in naming the columns in different tables you do not have to specify them each time you refer to them in MPL.





320051002800680047005400

000000


Import Data Vectors from Database

MPL allows you import the elements for a data vector directly from a database. In the DATA section, where you define the data vector, enter the keyword DATABASE after the assignment symbol (:=), followed by parentheses containing the table name and the column/field name you want to import from.

DATA FactDepCost[factory,depot] := DATABASE("FactDep", "TrCost");

In the above example, MPL will open the database table FactDep, locate the columns TRCost, FactID, and DepotID, and then read in the entries for the data vector FactDepCost.

The FactDep Table:



Import Data Vectors from Database (Cont.)

If an index column does have a different name than in the original table, you can specify it following the table name by first entering the index name, followed by an equal sign and the column name.

DATA FactDepCost[factory,depot] :=

DATABASE("FactDep","TrCost",factory="Factory",depot="Depot");

The column name defaults to the name of the data vector, so if it is the same you do not have to specify it. DATA TrCost := DATABASE("FactDep");



Import Data Vectors from Database (Cont.)

MPL can import from more than one database in the same run. The default database is specified in the Database Options dialog box in the Options menu. If you need to import a data vector from table in a database other than the default, you can do so by specifying the database name before the name of the table.

In the example below, MPL will read in the data vector DepotCustCost from Access instead of the default database. DATA DepotCustCost := DATABASE(Access, "DepCust", "TrCost");



Export Variable Values to Database

After optimizing the problem, MPL can export the variable values to the database where it can be used to report the solution back to the user. In the DECISION VARIABLES section, where you define the variable vector, enter the keyword EXPORT TO after the defined variable, followed by the keyword DATABASE and parentheses containing the table name and the column/field name you want to export to.

DECISION VARIABLES FactDepShip[factory,depot]

EXPORT TO DATABASE("FactDep", "Shipment");

In the above example, MPL will open the database table FactDep, locate the columns Shipment, FactID, and DepotID, and then export the solution values for the variable vector FactDepShip.


Export Variable Values to Database (Cont.)

Here is an example of the FactDep table after the solution values have been exported.

The FactDep Table:

Notice that MPL automatically uses the same name as the default for the index columns FactID and DepotID, as in the original tables the indexes were defined from.




320051002800680047005400

2100

4550

328189





MPL also allows you to export variable values other than the activity. You can export the reduced costs, the upper and lower ranges for the objective function, as well as the objective function coefficient values. You can change which values will be exported by entering one of the following keywords directly after the keyword EXPORT: Activity, ReducedCost, ObjectCoeff, ObjectLower, ObjectUpper.

For example, if you want to export the reduced cost for a variable enter the following:


EXPORT ReducedCost TO DATABASE("FactDep", "ReducedCost");

If you need to export more than one value for a variable vector you can do

so by entering multiple export statements after the variable definition.




MPL offers three different options on how the database table is updated by the EXPORT statement.

• UPDATE, which is the default, searches through the database and for each record locates the corresponding value in the solver solution and updates the database entry with it. This option minimizes the changes done to the database table since only the existing values are updated, but can sometimes be slow especially on SQL type databases.

• REFILL keyword right after the EXPORT keyword is used to specify that the whole database table should be emptied and then refilled with the entries from the solver solution. Since this takes out the necessity to search the table this can often lead to faster export times for larger tables.

• CREATE keyword right after the EXPORT keyword is used to specify that the database table should be created and then filled with the entries from the solver solution. This option is mainly useful when exporting to the database table for the first time.



Export Constraint Values to Database

Just as exporting variable values, MPL can also export the constraint values to the database. You will need to define constraint vectors in the CONSTRAINTS section with the keyword EXPORT TO followed by the keyword DATABASE and parentheses containing the table name and the column/field name you want to export to.

SUBJECT TO FactoryCapacity[factory]

EXPORT ShadowPrice TO DATABASE("Factory", "ShadowPrice");

In the above example, MPL will open the database table Factory, locate the columns FactID and ShadowPrice, and then export in the shadow price

values for the constraint FactoryCapacity.


Export Constraint Values to Database (Cont.)

Here is an example of the Factory table after the shadow price values have been exported.

Notice that MPL automatically uses the same name as the default for the index column FactID, as in the original tables the factory index was defined from. If an index column does have a different name than in the original table you can specify it by first entering the index name followed by an equal sign and the column name.

SUBJECT TO

FactoryCapacity[factory]

EXPORT ShadowPrice TO DATABASE("Factory", "ShadowPrice”, factory="Factory”);

This means, if you are consistent in naming the columns in different tables, you do not have to specify them each time you refer to them in MPL.

FactID Capacity ShadowPrice

HoustonSeattle

32000073000

120.9384

0.0000




Export Constraint Values to Database (Cont.)

In the example below, MPL will export the shadow price of the constraint vector FactoryCapacity from FoxPro instead of the default database.

SUBJECT TO FactoryCapacity[factory]

EXPORT ShadowPrice TO DATABASE(FoxPro,"Factory","ShadowPrice");

MPL also allows you to export constraint values other than the shadow price. You can export the slack values, the upper and lower ranges for the right-hand-side as well as the right-hand-side values. You change which values will be exported by entering one of the following keywords directly after the keyword EXPORT: Activity, Slack, ShadowPrice, RhsValue, RhsLower, RhsUpper. For example, if you want to export the slack for a constraint enter the following: SUBJECT TO FactoryCapacity[factory]

EXPORT Slack TO DATABASE("Factory","Slack");

If you need to export more than one value for a variable vector you can do so by entering multiple export statements after the variable definition.



SpreadSheet Modeling in MPL

• Spreadsheet optimisation allows users to create models that are easy to use, enabling the user to quickly update the data and solve the model.

• Spreadsheets are efficient at handling and managing 2-Dimensional dense data and single scalar values.



Import Indexes from Excel

MPL allows you to import the elements for an index directly from an EXCELRANGE. In the INDEX section, where you define the index, enter the keyword EXCELRANGE after the assignment symbol (:=) followed by parentheses containing the table name and the column/field name you want to import from.

INDEX depot := EXCELRANGE("Depots");

In the above example, MPL will read in the entries for the index depot from the Excel range called Depots.



HoustonHoustonSeattleSeattleSeattle

ChicagoDallasAtlantaNew YorkDallas

32005100280047005400

00000


Import Data Vectors from ExcelMPL allows you import the elements for a data vector directly from EXCEL. One can read in data in tabular format using the keyword EXCELRANGE or in sparse format using the keyword EXCELSPARSE, below is an example of how to import a data vector from EXCEL.

DATA FactDepCost[factory,depot] := EXCELSPARSE("FactDep", "3");

In the above example, MPL will read in the data vector FactDepCost from the range called FactDep, as we have specified the data is in sparse format it will take the first column to be entries for the Factory index, the second column to be entries for the Depot index and 3 specifies the value of the data parameter to be taken from the third column.

The FactDep Table:



Export Variable Values to Excel

After optimizing the problem, MPL can export the variable values to the EXCEL where it can be used to report the solution back to the user. In the DECISION VARIABLES section, where you define the variable vector, enter the keyword EXPORT TO after the defined variable, followed by the keyword EXCELRANGE/ EXCELSPARSE and parentheses containing the range name and the column number if you are using EXCELSPARSE.


EXPORT TO EXCELRANGE("Shipping");

In the above example, MPL will export the solution values for the variable vector FactDepShip to the range called Shipping.



Sparse Data vs. Dense Data

• Dense Data

• Standard representation in row and column format

• Row and column format preferably when data is highly dense, containing none or low percentage of null or zero values

• Can be difficult to deal with for large data sets.

• Sparse Data

• Includes only non zero values

• Highly efficient representation if small percentage of the values are nonzero

• Efficient. Generating and Solving models


Data Modelling for Optimisation

Sparse and Dense Data Examples

Dense Data

In the following example shows the datavector Shipcost. The parameter is indexed over 4 plants, 2 warehouses and 3 products.

ShipCost[plant, warehouse, product] := [

! p1 p2 p3 {pl1} 70, 52, 48, !w1 65, 40, 74, !w2 {pl2} 79, 52, 65, !w1 75, 50, 53, !w2 {pl3} 47, 52, 51, !w1 78, 52, 52, !w2

{pl4} 75, 80, 50, !w1 90, 52, 10]; !w2



Sparse and Dense Data Examples

Sparse Data

The first entry of each row specifies the first index. In our example this means plant, the second and third entries refers to the machine and product indexes and the fourth to the actual data value for ProdCost.

ProdCost[plant, machine, product] := [

p1, m11, A1, 73.30, p1, m11, A2, 52.90, p1, m12, A3, 65.40, p1, m13, A3, 47.60,

p2, m21, A1, 79.00, p2, m21, A3, 66.80, p2, m22, A2, 52.00,

p3, m31, A1, 75.80, p3, m31, A3, 50.90, p3, m32, A1, 79.90, p3, m32, A2, 52.10,

p4, m41, A1, 82.70, p4, m41, A2, 63.30, p4, m41, A3, 53.80];



Text vs. Database vs. Spreadsheet

Data can be read in at different speeds depending on the data source.

• Text Files - Are compact and simple and by far the quickest way to read in data.

• Databases - Are built for data management pretty quick to import and export data.

• Spreadsheets - Not designed to handle or manage large data sets, inefficient in importing data and is the slowest method to read in and export data.

Documents

Seminar: Data Modeling for Optimization with MPL - Oct 2012