What is parallel loading?
TurboIntegrator processes (TIs) is the built-in ETL tool of IBM TM1 and Planning Analytics. TIs are very powerful and fast — with just a few seconds, millions of records can be loaded into your cubes. You can make them even faster by running them in parallel.
Instead of running one process loading one year of data, you could run 12 processes in parallel (each loading one month of data).
Why running processes in parallel?
The main reason is to improve user experience. Especially if some TIs need to run on a daily basis, you need them to be as fast as possible.
In this article we go through a common use case — copying one year of data across two cubes. Our process takes around 80 seconds. We will see how to reduce it to 20 seconds by running processes per month in parallel.
How to run processes in parallel
This article covers four ways: TM1RunTI, Hustle, RushTI and RunProcess.
1. TM1RunTI
TM1RunTI is a command line tool that can initiate a TI process or chore. The executable tm1runti.exe is in the bin directory of a TM1 server install.
Create a config file: Use TM1Crypt to encrypt your password, then create a tm1runti.config file with your credentials.
Executing TM1RunTI from a TI: Use ExecuteCommand(sCMD, 0) where 0 means don't wait for completion:
When to use TM1RunTI? It's easy but doesn't manage threads. The number of processes should never exceed the CPU count.
2. Hustle
Hustle manages threads when executing TM1RunTI commands. You specify the number of concurrent processes and pass a batch of commands.
Create a tasks file using ASCIIOutput, then execute Hustle with two arguments — the tasks file path and the max thread count:
When to use Hustle? Hustle removes the overhead of managing threads. Limitations: must be on TM1 server, and each TM1RunTI command needs separate authentication.
3. RushTI
RushTI is a Python script that executes TM1 processes in parallel using only one connection via the TM1 REST API.
RushTI vs Hustle
RushTI has three main advantages:
Uses one connection only — Hustle authenticates per thread; RushTI connects once and reuses the connection
Can execute processes on different instances — RushTI uses REST API and doesn't have to sit on the TM1 server
Ordering the threads — RushTI enables you to set predecessors, so processes wait for dependencies to complete before executing
4. RunProcess
IBM Planning Analytics introduced RunProcess — it runs TI processes in parallel, each on its own thread managed by TM1 Server. No additional settings required:
ExecuteProcess vs RunProcess
Sub-processes using ExecuteProcess are part of the same transaction — data changes aren't visible externally until the final commit. Processes using RunProcess are NOT part of the same transaction — they see the data model state BEFORE the master process started.
Optimizing parallel processing
How many threads? Should not exceed available CPUs. There's a tipping point where more splitting doesn't help.
Which dimension to split on? In our example: 12 months in parallel took 20 seconds, by departments took 52 seconds, by months AND departments took 40 seconds. The only way to find out is testing.
Reordering dimensions: Putting the splitting dimension first should improve calculation time.