Skip to main content

Graphics Builder - Filtering and Transforming Data through Data Pipeline

  • Updated

Processing raw (source) data through Data Pipeline (Data Transformers and Filters)

Data may need to be processed to support advanced use-cases with the data, such as:

  • Sort / Filter data based on the defined criteria

  • Calculate min/max/average values

  • Convert data format to a defined one

To do so, Data Pipeline must be applied to a Widget. Data Pipeline consists of a single or multiple pipes: Transformers and/or Filters. The output of one pipe is the input for the subsequent pipe, so the order of pipes matters to get the required pipeline output.

Transformers are used to adjust/change the data by applying the selected operation to the data.

Filters are used to select the required data according to the criteria specified.

data-pipeline.png
Data Pipeline
 
 

Adding a Transformer

 
# Step Details Screenshot

1

Add a Data Source for a Widget

Please find the instructions on setting up a Data Source at Graphics Builder - Adding a Data Source for a Widget | Adding-a-Data-Source-for-a-Widget

2

Select the widget

Left-click on a Widget put on the Canvas

steps-2-and-3.png

3

Go to ‘Properties’ tab

Left-click on a ‘Properties’ tab of the selected Widget

4

Click on ‘Add Transformer’ button inside Data Pipeline section

Left-click on ‘Add Transformer’
step-4.png

5

Select the required Data Transformer and set up its parameters if availbale

Please find the list of available transformers and their description at Graphics Builder - Adding a Data Source for a Widget | List-of-available-Transformers
step-5.png

 

6

Click on ‘Save’ button

Left-click on ‘Save’

 

List of Available Transformers

  Data Transformer Workflow Parameters Application Example
1

Converts value to string

TBD

None

TBD
2

Count

Returns the number of data strings

None

Count will return 3 according to the number of data strings:

step-2-count.png
 

Cumulative delta

Returns the difference in target attribute of each data row versus the next one

 

 
  1. Attr - the attribute name on whose value delta has to be calculated. Possible values are:
    1. ‘delta’ (difference between data rows returned by a previous pipe)
    2. ‘value’ (Point’s value)
    3. any custom attribute (variable) defined by a previous pipe
  2. Target Attr - the attribute to which the delta needs to be written. Possible values are:
    1. ‘delta’ (difference between data rows returned by a previous pipe)
    2. ‘value’ (Point’s value)
    3. any custom attribute (variable) defined by a previous pipe
  3. Calculate Percentage - if enabled, the delta is calculated in %. Otherwise, it’s calculated in absolute value

Example 1: Calculating delta in absolute value

Cumilative Delta with the following parameters

  1. Attr = value
  2. Target Attr = value
  3. Calculate Percentage = No

will return:

row 109 = row 109 - row 108 = 51-55= -4

row_109.png
Before applying Cumulative Delta on Value
after-applying-cumulative-delta.png
After applying Cumulative Delta on Value

Example 2: Calculating delta in %

Cumulative Delta with the following parameters

  1. Attr = value

  2. Target Attr = value

  3. Calculate Percentage = Yes

will return:

row 360 = row 360 / row 359 *100% = 9 / 6.1 * 100%= 1,475 * 100% = 147,5% (+48%)

before-applying-cumulative-data.png
Before applying Cumulative Delta on Value
after.png
4

Extract attribute value

Retrieve value of an attribute from data

Attr - attribute name whose value needs to be retrieved from data. Possible values are:

  1. ‘delta’ (difference between data rows returned by a previous pipe)
  2. ‘value’ (Point’s value)
  3. any custom attribute (variable) defined by a previous pipe

Getting attributes from a Data String (see more details about Data String in Graphics Builder - Fetching Point Data | Data-String-Structure):

  1. Add ‘Value’ widget to get the single value of a single attribute, specify any data source with existing data:

    step-1-add-value.png

  2. Add ‘Last item’ pipeline for Value widget to get the latest data string:

    add-last-item.png 

  3. Add ‘Extract Attribute value’ pipeline, try out different ‘Attr’ parameters to see their values:

    add-extract-attribute.png

    1. ‘Attr’ = ‘timestamp’ returns ‘2021-12-19T21:00:00+00:00’

      timestamp.png

    2. ‘Attr’ = ‘value’ returns '9'

      value.png

    3. ‘Attr’ = ‘uuid’ returns 'e96a3498-b332-4d4c-bd41-2d68dfe6c388'

      uuid.png

    4. ‘Attr’ = ‘uri’ returns '/TurntideTest/Denver Lab/Lab Equipment/Wattnode/properties/AI/Power Sum'

      uuid.png
5

Format Relative data

Formats relative date time strings in natural language ex: Last week, This month etc; to absolute date strings.

None

  1. Add a Text Widget

  2. Add ‘Format Relative Data’ pipeline

  3. Try out different ‘Text’ values with relative data and review the result:text.png

    1. ‘Text’ = ‘Today’ will return ‘Dec 20, 2021’

    2. ‘Text’ = ‘Yesterday’ will return 'Dec 19, 2021

    3. ‘Text’ = ‘Last Week’ wil return ‘Dec 13, 2021 - Dec 19, 2021’
6

ISO formatter

Formats ISO dates into user specified format

  1. Attr - name of the attribute whose value should be formatted. Ex: timestamp.

  2. Source Format - the source datetime format

  3. Format - the format to which the value must be converted to.

  1. Add a Table Widget, specify any data source with existing data:

  2. Select Table Columns:

    1. Timestamp

    2. Any other columns, e.g. Value, Parameter

  3. Add ISO formatter pipeline:

    1. The Source Data format is YYYY-MM-DDTHH:mm:ss (2021-12-17T00:00:00)

      source-data.png

  4. Try out different ‘Format’ values with and review the result:

    1. ‘Format’ = ‘YYYY-MM-DD’ will convert source data format to ‘2021-12-17’

      timestamp.png

    2. ‘Format’ = ‘DD-MM-YYYY’ will will convert source data format to ‘17-12-2021’

      timestamp-b.png

    3. ‘Format’ = ‘HH:MM’ will will convert source data format to ‘00:00’

      timestamp-c.png
7

Localize Data & Time

Localizes the date and time to users's timezone

  1. Attr - name of the attribute whose value should be formatted. Ex: timestamp.

  2. Enabled - if enabled, disables localizing, uses UTC

  1. Add a Table Widget, specify any data source with existing data:

  2. Select Table Columns:

    1. Timestamp

    2. Any other columns, e.g. Value, Parameter

  3. Add 'Localize Data & Time' pipeline

Current User’s Timezone is UTC+3

Date&Time format before applying Localize Data & Time:

date-time-before.png

Date&Time format after applying Localize Data & Time

date-time-after.png
8

Parse URI

Extracts attributes from Data String’s URI ("uri": "/client_id/site_name/properties/equip_name/ptype/point_name").

Example:

data-string.png

See more details on attributes in Graphics Builder - Fetching Point Data | Data-String-Structure

None

Example 1: Table view

Table widget applies Parse URI piplene by default to represent data from URI in multiple columns

table-view.png

Example 2: String view

  1. Add ‘Value’ widget to get the single value of a single attribute, specify any data source with existing data

    string-view.png

  2. Add ‘Last item’ pipeline for Value widget to get the latest data string

  3. Add ‘Parse URI' pipeline for Value widget

    example-2-step-3.png

  4. Add ‘Extract Attribute value’ pipeline, try out different URI’s parameters as ‘Attr’ to get their values:

    example-2-step-4.png

     

  1. ‘Attr’ = ‘client_id’ returns 'TurntideTest'

    attr-1.png

  2. ‘Attr’ = ‘site_name’ returns 'Denver Lab'

    attr-2.png

  3. ‘Attr’ = ‘equip_name’ returns 'Weather'

    attr-3.png

  4. ‘Attr’ = ‘ptype’ returns 'AI'

    attr-4.png

  5. ‘Attr’ = ‘point_name’ returns 'Dewpoint'attr-5.png
9

Reverse list

Reverses the items in the list

None
original-list.png
Original List
reversed-list.png
Reversed List
10

Rollup data

Rolls up history data based on the given rollup frequency and digest

  1. Digest - digest function to be run on the data

    1. First - use the first value available inside the given time frame

    2. Last - use the last value available inside the given time frame

    3. Max - use the max value available inside the given time frame

    4. Mean - use the first value available inside the given time frame

    5. Median - use the average value available inside the given time frame

    6. Min - use the middle number of a group of values available inside the given time frame. That is, half the numbers return values that are greater than the median, and half the numbers return values that are less than the median.

    7. Mode - use the the most commonly occurring value (the mode value) in the aggregated value available inside the given time frame

    8. None - shows the historical value change (raw data)  available inside the given time frame

    9. Standard Deviation - use the statistical standard deviation of all values in the set based on a sample of the population available inside the given time frame

    10. Sum - use the sum of values available inside the given time frame

    11. Variance - use the variance value available inside the given time frame

  2. Frequency - the frequency at which history data needs to be grouped and digest applied

  3. Group By - the attribute on which the data should be grouped by, the digest is run on each group. If data needs to be aggregated, leave this empty. Possible values:

    1. uuid
original-data.pngOriginal Data (non-rollup)
 
 
 
daily-mean-rollup.png
 
Daily Mean Rollup
 
daily-variance-rollup.png
Daily Variance Rollup
11

Set precision

Sets the precision of floating point values

  1. Attr - name of the attribute whose value should be transformed, ex. ‘value’

  2. Precision - the precision to round up to
original-value.png
Original Value
after-set-precision.png
After set precision to 2 decimal places for Value
12

Sort data

Sorts data by the specified attribute

  1. Attr - name of the attribute whose value should be transformed, ex. ‘value’

  2. Is Timestamp - defines whether a value will be sorted as a string or as a timestamp. Will be set to ‘true’ if the field is a timestamp value. Otherwise, will be set to ‘false’.
original-list.png
Original List
list-sorted-by-value.png
List sorted by Value (min to max)
list-sorted-by-point-name.png
List sorted by Point Name (a-z)
13

Transform data to scatter

  1. Xpoint - point to be displayed on X axis of a Scatter Chart

  2. Ypoint - point to be displayed on Y axis of a Scatter Chart

  3. Xrollup - rollup to be applied to X axis of a Scatter Chart (see Ref.10 Rollup data for rollups description)

  4. Yrollup - rollup to be applied to Y axis of a Scatter Chart (see Ref.10 Rollup data for rollups description)

Transforms data to support Scatter Chart widget. Will be always applied for that widget to see any data visualized.
scatter-chart.png
Scatter Chart

 

Adding a Filter

# Step Details Screenshot

1

Add a Data Source for a Widget

Please find the instructions on setting up a Data Source at Graphics Builder - Adding a Data Source for a Widget | Adding-a-Data-Source-for-a-Widget

2

Select the widget

Left-click on a Widget put on the Canvas
step-2_3.png

 

3

Go to ‘Properties’ tab

Left-click on a ‘Properties’ tab of the selected Widget

4

Click on ‘Add Filter’ button inside Data Pipeline section

Left-click on ‘Add Transformer’
step-4.png

 

5

Select the required Data Filter and set up its parameters if availbale

Please find the list of available Filters and their description at Graphics Builder - Adding a Data Source for a Widget | List-of-available-Filters
step-5.png

 

6

Click on ‘Save’ button

Left-click on ‘Save’

 

List of Available Filters

  Digital Transformer Workflow Attributes Application Example
1

Contains

Filters rows of data whose attr's value is contained in the specified list of of values

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Values - list of values that need to be considered for filtering (OR condition is applied in case of multiple Values specified)
original-list.png
Original List

 

filtered-list.png
Filtered List (Contains Attr='point_name', Values='Dewpoint', ‘Temperature’)
2

Equal to (==)

Filters rows of data whose value is equal to the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Equal To Attr='value', Value='30.11')

 
3

First Item

Retrieves the first item in the data

None
original-list-massive.png
Original Data Massive
first-item.png
First Item
4

Greater than (>)

Filters rows of data whose value is greater than the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Greater Than Attr='value', Value='31')
5

Greater than / equal to (>=)

Filters rows of data whose value is greater than or equal to the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Greater Than or Equal TO Attr='value', Value='35')
6

Last Item

Retrieves the last item in the data

 
original-data-massive.png
Original Data Massive
last-item.png
Last Item
7

Lesser than (<)

Filters rows of data whose value is lesser than the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Lesser Than Attr='value', Value='7')
8

Lesser than/equal to (<=)

Filters rows of data whose value is lesser than or equal to the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Lesser Than/Equal to Attr='value', Value='7')
9

Limit

Limits/Truncates data to the no of items specified from the first item in the list

  1. Limit - number of items to limit starting from the first item

  2. From Bottom - slice the list from bottom
original-list.png
Original List
filtered-list.png
Filtered List (Limit = ‘1', ‘From Bottom’ = 'No')
filtered-list-2.png
Filtered List (Limit = ‘2', ‘From Bottom’ = 'Yes')

 
10

Not equal to (!=)

Filters rows of data whose value is not equal to the specified value

  1. Attr - name of the attribute whose value should be compared. Ex: value, point_name etc.

  2. Value - list of values that need to be considered for filtering
original-list.png
Original List
filtered-list.png
Filtered List (Not Equal To Attr='point_name', Value='Dewpoint')

Related articles