Upload
hoangdung
View
228
Download
0
Embed Size (px)
Citation preview
IBM TM1 ServerFaster than the speed of light and more flexible than a Yogi
Hubert HeijkersSTSM, Chief Architect TM1 Server and OData [email protected]
IBM Cloud University 2017 | October
Please Note
IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.
Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.
The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract.
The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming inthe user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
TM1 ServerPlanning Analytics, powered by TM1 Server
Recent TM1 Server Features and Improvements
§ Hierarchies!
§ Multi-Threaded Feeders and Cube Load
§ Transitory View and Set creation in TI Processes
§ Locking, Contention, Performance and other Improvements
§ Encryption for 'Data at Rest'
§ Sandbox Compare
§ TI Debugger
6
Hierarchies
§ Introducing the concept of having multiple distinct hierarchies as part of one dimension– All hierarchies sharing the overall set of leaf level elements– Whereas each hierarchy has its own unique set of consolidated elements
§ Alternate hierarchies typically based off attribute values associated to the leaf elements – describing an aggregation structure for the dimension’s leaf elements
§ Use of alternate hierarchies diminishes the need for extra dimensions resulting in:– Smaller cubes – less memory consumption– Better performance!
§ Alternate hierarchies can, in an analysis, be used as just any other dimension
§ Hierarchies can be added and removed from dimensions ‘on the fly’
7
Hierarchies – The important things
§ Do I have to use Hierarchies?– You do not, they are opt-in and disabled by default, but really why wouldn’t you?
§ What clients can use and or see Hierarchies?– Our more modern clients. Planning Analytics Workspace, Planning Analytics for Excel
§ What will old clients see?– They will only see the same named hierarchy, hence it will look like it always did
§ What about Cognos Analytics?– Cognos Analytics, as of version 11.0.6, supports our REST API and supports hierarchies
§ If I add a new hierarchy will my reports (Perspectives/BI) break?– No, we’ve been really conscious of not breaking backwards compatibility. The old
reports will reference the same named hierarchy and continue to work as expected
8
Hierarchies
9
Query Performance
RAM Savings
Flexibility Standards Conformance
Primary Benefits
Primary Use Cases
Single Dimension Multiple AxesIntroduce
Hierarchies to Queries as Needed Dimension Versioning
MDX
Unique Levels per Hierarchy
Cubes with Less Dimensions
Hierarchies as ‘Virtual Dimensions’
Adopting Hierarchies
§ Upon Upgrade, a ‘Same Named Hierarchy’ is created
§ All Dimension Members are transitioned to that Hierarchy
§ Everything Operates the Same
§ Creation of the ‘next’ Hierarchy triggers new Hierarchy Behavior
= Dimension = Hierarchy
All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All SportsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All SportsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
Adopting HierarchiesAll Products
BallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All SportsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All SportsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All ProductsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
Levels
0 – All1 – Product Type2 - Product
Levels
0 – All1 – Season2 – Sport3 - Product
Adopting Hierarchies
All LeavesBaseballsLacrosse ballsSoccer ballsFootballsFielding GlovesSticksNetsShoulder Pads
All ProductsBins
BaseballsLacrosse balls
BoxesSoccer BallsFootballsNetsShoulder Pads
LooseFielding GlovesSticks
Levels
0 – All1 – Product
Levels
0 – All1 – Package Type2 – Product
All ProductsBallsBaseballsLacrosse ballsSoccer ballsFootballs
EquipmentFielding GlovesSticksNetsShoulder Pads
All ProductsSpring
BaseballBaseballsFielding Gloves
LacrosseLacrosse ballsSticks
FallFootball
FootballsShoulder Pads
SoccerSoccer BallsNets
Levels
0 – All1 – Product Type2 - Product
Levels
0 – All1 – Season2 – Sport3 - Product
Leaves Category Seasonal Packaging
Adopting Hierarchies with Planning Analytics
Intersect multiple hierarchies from the SAME dimension in a single ‘view’
13
Product Category: Balls
Time: Quarters
Seasonal: Spring Packaging: Bins
Q1 Q2 Q3 Q4
BaseballsLacrosse Balls
1
2 3Dim X Dim Y
More Precise Planning
Hierarchies – Business Logic
§ You can add Hierarchies to any Dimension– and Rules for those Cubes will continue to work
§ Unique Consolidations are fine. Same named consolidations must be qualified
§ All member references can be qualified to a Hierarchy in Rules
§ DB Statements can be references with more then one Hierarchy for a Dimension
§ New TIs to create/maintain Hierarchies
§ CellPut / CellGet can write to any Hierarchy combo Cell
14
Alternate Hierarchies
Two hierarchies ofthe same dimension
Multiple Alternate Hierarchies in the Same View
16
4 Hierarchies from the sameDimension
Multiple Alternate Hierarchies in the Same View
MTx – Multi-Threaded Feeders and Cube Load
§ Applies the MTQ framework to cube loads and feeder creation
§ Multi-Threaded Feeders are triggered– at startup,– by changing feeder definitions of cube rules, – when executing the CubeProcessFeeders() TI function
§ Single Cube, many cores!
§ Configured using new parameters in the server configuration– MTCL only kicking in over a, defined, minimum cub file size (MTCubeLoad.MinFileSize)
§ Be aware: these might (read: will) take up to more memory during processing
Caution: Issues have been identified with conditional feeders and MTCL, check support!17
Transitory objects TI Processes
§ Allows for Transitory Views and Sets to be created in TI Processes
§ No registration typically implies less lock contention
§ No need to clean up, they get deleted at the end of the process/chore
§ Simple to implement– SubsetCreate( ‘DimName’, ‘SubName’, 1 );– SubsetCreateByMDX( ‘SubName’, ‘MDX’, 1 );– ViewCreate( ‘CubeName’, ‘ViewName’, 1 );
Be aware: They exist for the duration of the ‘transaction scope’ but their use is scoped to the process they are created in so in some cases you might want to clean them up anyway.
18
Locking and Contention Improvements
§ PullInvalidationSubsets– Remove contentious RO dimension lock from creation/deletion– Also eliminates private subset login contention– Open and work with views when running dim updates
§ VersionedListControlDimensions– Remove contention on }Cubes, }Dimensions, }Groups, }Clients, etc…– Allows creation of new objects without IX locking the dimension
§ LoadPrivateSubsetsOnStartup– Avoid contention between dimension edits and client logins– Usually not a significant memory consumer
19
Locking and Contention Improvements
§ ReduceCubeLockingOnDimensionUpdate– Avoid cube IX locks when dimension edits don’t affect existing rule
§ Adding a new element that’s not referenced by any rules– Allows simultaneous edits to multiple dimensions sharing a cube– Allows writing to the cube during dimension edits
§ CreateNewCAMClients– Don’t like the automatic creation of clients at first login?– Have a process in place that assures that all CAM users with access rights are
represented as clients in your TM1 Server’s security?– You can now manage when locks associated to the creation of these clients occur
20
MTQ Improved
§ Multi-Threaded Queries (MTQ)– Dramatic performance improvement– Using, when available, many threads instead of one
§ MTQ View Datasource in TI (view extract)– MTQQuery=T– DisableMTQViewConstruct() in prolog disables for that process
21
Rule Profiling
§ Are rule slowing you down?
§ What rule line is slowest?
§ How the heck do I find that out?
22
Turbo Integrator related Improvements
§ Tm1runti.exe chore support-chore ‘ChoreName’
§ CubeDimensionCountGet( )No more looping using TabDim
§ SubsetElementGetIndex( ‘Dim’, ‘Subset’, ‘ElementName’, [StartIndex] )
§ LogOutput( Severity, Message )– Write directly to message log– Control output with tm1s-log.properties
23
Retries on File Operations
§ Introduced in 10.2.2.FP7
§ Help improve stability in cases where the file system occasionally is ‘unavailable’
§ Configured through these configuration settings:– FileRetry.FileSpec
§ Specifies the directory path(s) of the affected log files.§ String value, default value is <empty>
– FileRetry.Count§ Specifies the number of retry attempts.§ Integer value, default value: 5
– FileRetry.Delay§ Specifies the delay (in milliseconds) between retry attempts.§ Integer value, default value: 2000
24
Cancelled Thread Logger
§ Thread cancellation logger– TM1.Context.Cancel = DEBUG
Traditional:3512 [3] DEBUG 2015-10-23 15:08:24.175 TM1.Context.Cancel Cancelling function "ViewArrayConstruct" with parameters:3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel Argument #0: Object - Major(0000000020000006) : Minor(0000000000000000) : Type(0000000000000009) - Name "NULL"3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel Beginning View Array Construction for view: NULL (abstraction 0x0000000033C27E58, index 0x20000006)3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel cube: [SalesCube]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel user: [Admin]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel There are 3 title elements:3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [actvsbud].[Actual]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [region].[Argentina]3512 [3] DEBUG 2015-10-23 15:08:24.176 TM1.Context.Cancel [model].[S Series 1.8 L Sedan]3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel There is 1 row subset:3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel [account1].[NULL], element count: 53512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel There is 1 column subset:3512 [3] DEBUG 2015-10-23 15:08:24.177 TM1.Context.Cancel [month].[NULL], element count: 12
REST:10896 [101] DEBUG 2015-10-23 17:09:25.966 TM1.Context.Cancel Cancelling REST request: Method "POST" URI "/api/v1/Processes('foo')/tm1.Execute"
Lock Exception Logger
§ Lock exception logger– TM1.Lock.Exception = DEBUG
Encryption for 'Data at Rest'
§ What’s more to say…
§ Oh, you won’t be able to do fiddle around with your files on disk any longer
§ Once encrypted always encrypted (otherwise that would defeat the purpose)
§ (Data)Administrator can ask the server to ‘save’ an unencrypted ‘snap-shot’ somewhere
§ Look for an EncryptData option in your tm1s.cfg
27
Sandbox Compare
§ Implicit ‘virtual’ dimension
§ One element per available sandbox
§ Side by side comparison of sandboxes
§ Calculations across sandboxes
28
TI Debugger
§ A TI Debugger? The TI Debugger? What/Where?
§ Well, we added support for TI Debugging in the Server’s REST API, allowing anybody and everybody to build their own TI Debugger for their favorite development IDE
§ And, just for fun, we’ve provided a stand-alone one, downloadable from DeveloperWorks– https://developer.ibm.com/recipes/tutorials/ibm-tm1-turbointegrator-debugger/
§ Configure your TM1 Server (tm1s.cfg) by:– Making sure the HTTP port for the REST API is enabled i.e.:HTTPPortNumber=8000
– Enable TI Debugging:EnableTIDebugging=True
§ Requires TM1 Server v11 or the, more recent, v10.2.2.FP7
29
TI Debugger
30
Prolog,Metadata,Data,andEpilogtabsidenticaltothatofArchitect
ßAlleditableTIprocesses
Listofbreakpoints
VariablesusedinexecutionObjectlocksontheTM1serverStepOver,StepIn,StepOutcapability
What’s next in TM1 Server?TM1 Server features to be expected in future versions
Attribute Based, alternate, Hierarchies
§ Now that we have support for having, potentially many, alternate hierarchies
§ Hierarchy structures, more often then not, being driven by attribute values
§ Allow the system to auto maintain a hierarchy based on an, attribute based, specification
§ Resulting in hierarchy updates whenever a source attribute value is being update
§ Attribute Based Hierarchy specification would:– Refer to the attributes on which the hierarchy is build– Specify how to order members– How to create unique names for members– How to deal with null values– Maybe introducing the notion of skipping levels and raggedness
32
Sandbox sharing
§ Wouldn’t sandbox compare be even more powerful if one could share each others work?
§ Building on the new sandbox compare feature
§ Extend the list of sandboxes available to the user with those shared with him/her
§ While still allowing side by side comparison of any and all sandboxes
§ And allowing calculations across sandboxes
33
Feeder-less Rules, yes, you heard it right;-!
§ Brand new calc engine that spans both rules and MDX member expressions
§ Eliminates the need for feeders altogether
§ Feeder-less rule computation over sparse data– driven solely by the rule definition– feeder markers are not required
§ Faster– batch computation of rule results (as in stargate vs. per cell calculations)– incremental result set cache
§ What this is NOT is an engine that generates feeders, we are NOT writing feeders under the covers, feeders go away completely and forever.
34
Could live without feeders be any better?
§ No more memory used to store feeders– If the feeder doesn’t exist, nor does the
memory space it used to occupy
§ Real time modelling– Model changes don’t require lengthy
feeder updates
§ Simpler Modeling Experience– No need for feeder section in rules
§ Less model errors– Feeder definitions are error prone
§ Faster cell updates– No need to trigger feeder value
updates
§ Faster server load times– No need to load/generate feeders
§ And that’s not all!
35
Yes, it can get even better then that!
§ Better support for Virtual Cubes– Without feeders no ‘underfed’ tricks need to be applied to make ‘virtual’ cubes
perform, ignoring the restrictions those tricks came with
§ Slices/Partitions can come from other servers– Now that we minimize the ‘queries’ for data from other cubes we could envision,
especially in virtual cubes, data to come from different server instances that might reside on different machines (self-serve horizontal scalability?)
§ ‘Time’ dependent hierarchies– Not depending on feeders makes it much more feasible to accomplish this
36
Feeder-less Rules, where are we at?
§ Seriously, you wonder where we are at?– 99% of TM1 10.2.2 rule functionality covered without need of feeders
§ What we need to do?– Support for alternate hierarchies!– Additional functional coverage primarily focussing on supporting MDX natively– Expand test cases– Identify required improvements– Make some hard decisions as to how/what/where/when
37
What’s brewing in the lab?Interested to know what’s cooking wrt TM1 Server in the labs?
What’s brewing in the lab?
Well, that’s beyond the scope of this presentation but…
Don’t hesitate to reach out to me if you want to know more about a next generation, deployable as-a-service, highly available TM1 server, and want to know more about:
§ Source Control Integration
§ Containerization
§ Clustering, High Availability and Fail-Over
Or ideas around
§ Model/modeling improvement
§ A new scripting language to ‘replace’ TI perhaps
39
Questions
?Think of any more questions later?Don’t hesitate to contact me or send an e-mail to [email protected]
Thank you!
IBM Cloud University 2017 | October
Notices and disclaimersCopyright © 2017 by International Business Machines Corporation (IBM). No part of this document may be reproduced or transmitted in any form without written permission from IBM.
U.S. Government Users Restricted Rights — use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM.
Information in these presentations (including information relating to products that have not yet been announced by IBM) has been reviewed for accuracy as of the date of initial publication and could include unintentional technical or typographical errors. IBM shall have no responsibility to update this information. This document is distributed “as is” without any warranty, either express or implied. In no event shall IBM be liable for any damage arising from the use of this information, including but not limited to, loss of data, business interruption, loss of profit or loss of opportunity. IBM products and services are warranted according to the terms and conditions of the agreements under which they are provided.
IBM products are manufactured from new parts or new and used parts. In some cases, a product may not be new and may have been previously installed. Regardless, our warranty terms apply.”
Any statements regarding IBM's future direction, intent or product plans are subject to change or withdrawal without notice.Performance data contained herein was generally obtained in a controlled, isolated environments. Customer examples are presented as illustrations of how those customers have used IBM products and
the results they may have achieved. Actual performance, cost, savings or other results in other operating environments may vary.
References in this document to IBM products, programs, or services does not imply that IBM intends to make such products, programs or services available in all countries in which IBM operates or does business.
Workshops, sessions and associated materials may have been prepared by independent session speakers, and do not necessarily reflect the views of IBM. All materials and discussions are provided for informational purposes only, and are neither intended to, nor shall constitute legal or other guidance or advice to any individual participant or their specific situation.
It is the customer’s responsibility to insure its own compliance with legal requirements and to obtain advice of competent legal counsel as to the identification and interpretation of any relevant laws and regulatory requirements that may affect the customer’s business and any actionsthe customer may need to take to comply with such laws. IBM does not provide legal advice or represent or warrant that its services or products will ensure that the customer is in compliance with any law.
IBM Cloud University 2017 | October
Notices and disclaimers continuedInformation concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products in connection with this publication and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. IBM does not warrant the quality of any third-party products, or the ability of any such third-party products to interoperate with IBM’s products. IBM expressly disclaims all warranties, expressed or implied, including but not limited to, the implied warranties of merchantability and fitness for a particular, purpose.
The provision of the information contained herein is not intended to, and does not, grant any right or license under any IBM patents, copyrights, trademarks or other intellectual property right.
IBM, the IBM logo, ibm.com, Aspera®, Bluemix, Blueworks Live, CICS, Clearcase, Cognos®, DOORS®, Emptoris®, Enterprise Document Management System™, FASP®, FileNet®, Global Business Services®,Global Technology Services®, IBM ExperienceOne™, IBM SmartCloud®, IBM Social Business®, Information on Demand, ILOG, Maximo®, MQIntegrator®, MQSeries®, Netcool®, OMEGAMON, OpenPower, PureAnalytics™, PureApplication®, pureCluster™, PureCoverage®, PureData®, PureExperience®, PureFlex®, pureQuery®, pureScale®, PureSystems®, QRadar®, Rational®, Rhapsody®, Smarter Commerce®, SoDA, SPSS, Sterling Commerce®, StoredIQ, Tealeaf®, Tivoli® Trusteer®, Unica®, urban{code}®, Watson, WebSphere®, Worklight®, X-Force® and System z® Z/OS, are trademarks of International Business Machines Corporation, registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the Web at "Copyright and trademark information" at: www.ibm.com/legal/copytrade.shtml.