Processing raw (source) data through Data Pipeline (Data Transformers and Filters)
Data may need to be processed to support advanced use-cases with the data, such as:
-
Sort / Filter data based on the defined criteria
-
Calculate min/max/average values
-
Convert data format to a defined one
To do so, Data Pipeline must be applied to a Widget. Data Pipeline consists of a single or multiple pipes: Transformers and/or Filters. The output of one pipe is the input for the subsequent pipe, so the order of pipes matters to get the required pipeline output.
Transformers are used to adjust/change the data by applying the selected operation to the data.
Filters are used to select the required data according to the criteria specified.


Adding a Transformer
# | Step | Details | Screenshot |
1 |
Add a Data Source for a Widget |
Please find the instructions on setting up a Data Source at Report Builder - Adding a Data Source for a Widget |
|
2 |
Select the widget |
Left-click on a Widget put on the Canvas |
![]() |
3 |
Go to ‘Properties’ tab |
Left-click on a ‘Properties’ tab of the selected Widget |
|
4 |
Click on ‘Add Transformer’ button inside Data Pipeline section |
Left-click on ‘Add Transformer’ |
![]() ![]() |
5 |
Select the required Data Transformer and set up its parameters if available |
Please find the list of available transformers and their description at Report Builder - Adding a Data Source for a Widget |
![]() ![]()
|
6 |
Click on ‘Save’ button |
Left-click on ‘Save’ |
List of Available Transformers
# | Data Transformer | Workflow | Parameters | Application Example |
1 |
Advanced Transformer |
Supports custom JS code to transforms the data |
|
/** |
2 |
Arithmetic Operation |
Returns the result of arithmetic operation applied to an attribute |
|
![]() ![]() Transformer Configuration: Value is multiplied by 5
![]() ![]() Raw Data
![]() ![]() Transformed Data
|
3 |
Convert Time |
Returns the converted time value |
|
![]() ![]() Hours to Seconds conversion for ‘timestamp’
![]() ![]() Timestamp in Hours
![]() ![]() Timestamp in Seconds
|
4 |
Count |
Returns the number of data strings |
None |
|
5 |
Cumulative delta |
Returns the difference in target attribute of each data row versus the next one
|
|
Example 1: Calculating delta in absolute value Cumilative Delta with the following parameters
will return: row 109 = row 109 - row 108 = 51-55= -4 ![]() ![]() Before Applying Cumulative Delta
![]() ![]() After Applying Cumulative Delta
Example 2: Calculating delta in % Cumilative Delta with the following parameters
will return: row 360 = row 360 / row 359 *100% = 9 / 6.1 * 100%= 1,475 * 100% = 147,5% (+48%) ![]() ![]() Before Applying Cumulative Delta
![]() ![]() After Applying Cumulative Delta
|
6 |
Extract attribute value |
Retrieve value of an attribute from data |
|
Getting attributes from a Data String:
|
7 |
Format Number |
Returns locale-based formatted number |
|
Formatting example for different locales
|
8 |
Format Time |
Returns Time in human-readable format |
|
Seconds to Week conversion for 'timestamp' attribute
Default Time (seconds)
![]() ![]() Formatted Time (weeks)
|
9 |
Format Relative data |
Formats relative date time strings in natural language ex: Last week, This month etc; to absolute date strings. |
None |
|
10 |
ISO formatter |
Formats ISO dates into user specified format |
|
|
11 |
Localize Data & Time |
Localizes the date and time to users's timezone |
|
Current User’s Timezone is UTC+3 Date&Time format before applying Localize Data & Time: ![]() ![]() Date&Time format after applying Localize Data & Time ![]() ![]() |
12 |
Parse URI |
Extracts attributes from Data String’s URI ("uri": "/client_id/site_name/properties/equip_name/ptype/point_name"). Example: ![]()
|
None |
Example 1: Table view Table widget applies Parse URI piplene by default to represent data from URI in multiple columns ![]() ![]() Example 2: String view
|
13 |
Reverse list |
Reverses the items in the list |
None |
![]() ![]() Original List
![]() ![]() Reversed List
|
14 |
Rollup data |
Rolls up history data based on the given rollup frequency and digest |
|
![]() ![]() Original Data (None Rollup)
![]() ![]() Daily Mean Rollup
![]() ![]() Daily Variance Rollup
|
15 |
Set precision |
Sets the precision of floating point values |
|
![]() ![]() Original Value
![]() ![]() After setting precision to 2 decimal places for Value
|
16 |
Sort data |
Sorts data by the specified attribute |
|
![]() ![]() Original List
![]() ![]() List sorted by Value (min to max)
![]() ![]() List sorted by Point Name (a-z)
|
17 |
Transform data to scatter |
|
Transforms data to support Scatter Chart widget. Will be always applied for that widget to see any data visualized. |
![]() ![]() Scatter Plot
|
Adding a Filter
# | Step | Details | Screenshot |
1 |
Add a Data Source for a Widget |
Please find the instructions on setting up a Data Source at Report Builder - Adding a Data Source for a Widget |
|
2 |
Select the widget |
Left-click on a Widget put on the Canvas |
![]() ![]() |
3 |
Go to ‘Properties’ tab |
Left-click on a ‘Properties’ tab of the selected Widget |
|
4 |
Click on ‘Add Filter’ button inside Data Pipeline section |
Left-click on ‘Add Transformer’ |
![]() ![]() |
5 |
Select the required Data Filter and set up its parameters if availbale |
Please find the list of available Filters and their description at Report Builder - Filtering and Transforming Data through Data Pipeline |
![]() ![]() |
6 |
Click on ‘Save’ button |
Left-click on ‘Save’ |
List of Available Filters
Data Transformer | Workflow | Attributes | Application Example | |
1 |
Contains |
Filters rows of data whose attr's value is contained in the specified list of of values |
|
![]() ![]() Original List
![]() Filtered List (Contains Attr='point_name', Values='Dewpoint', ‘Temperature’)
|
2 |
Equal to (==) |
Filters rows of data whose value is equal to the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Equal To Attr='value', Value='30.11')
|
3 |
First Item |
Retrieves the first item in the data |
None |
![]() ![]() Original Data Massive
![]() ![]() First Item
|
4 |
Greater than (>) |
Filters rows of data whose value is greater than the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Greater Than Attr='value', Value='31')
|
5 |
Greater than / equal to (>=) |
Filters rows of data whose value is greater than or equal to the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Greater Than or Equal TO Attr='value', Value='35')
|
6 |
Last Item |
Retrieves the last item in the data |
|
![]() ![]() Original Data Massive
![]() ![]() Last Item
|
7 |
Lesser than (<) |
Filters rows of data whose value is lesser than the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Lesser Than Attr='value', Value='7')
|
8 |
Lesser than/equal to (<=) |
Filters rows of data whose value is lesser than or equal to the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Lesser Than/Equal to Attr='value', Value='7')
|
9 |
Limit |
Limits/Truncates data to the no of items specified from the first item in the list |
|
![]() ![]() Original List
![]() ![]() Filtered List (Limit = ‘1', ‘From Bottom’ = 'No')
![]() ![]() Filtered List (Limit = ‘2', ‘From Bottom’ = 'Yes')
|
10 |
Not equal to (!=) |
Filters rows of data whose value is not equal to the specified value |
|
![]() ![]() Original List
![]() ![]() Filtered List (Not Equal To Attr='point_name', Value='Dewpoint')
|