Introduction
This article describes changes to how scripts and data functions are trusted by authorized users in Spotfire® release 10.3 and later. It is intended for Spotfire® Administrators that plan to upgrade to Spotfire® 10.3, and other users that want to understand more about the trust mechanism used to ensure scripts and data functions are safe to execute.
For more information about the trust mechanism for client users in general, see the Spotfire® Analyst User's Guide.
In short, Spotfire® 10.3 adds the trust mechanism to data functions, as well as improving it by introducing SHA-512 for calculating the trust stamp. This means that there are additional steps required during the upgrade to Spotfire® 10.3, to make sure data functions continue to work as before. The exact steps to take will depend on your use case and environment and some examples are described in a separate article referenced below.
Managing trust for data functions during the upgrade to Spotfire® 10.3
The administrator will need to perform some more steps than usual during the upgrade process. Since data functions are now covered by the trust mechanism, the admin will need to consider how to deal with that, for example, deciding whether all existing data functions should be automatically trusted or whether they should first be individually reviewed. In addition, all trust stamps for existing scripts (Iron Python and Java Script) and data connection custom queries should be updated with the new trust algorithm (SHA-512). Note that already trusted scripts and data connection custom queries will continue to work in Spotfire® 10.3, but that their trust stamps should still be updated for improved security and for continued operation in future releases.
To assist with the upgrade process for the trust stamps, Spotfire® 10.3 includes a new command in the server Command Line Interface (CLI) that helps administrators upgrade the trust stamp of existing scripts based on IronPython and JavaScript, and custom queries in data connections. This command is called find-analysis-scripts and in addition to upgrading the trust stamp, the command finds all data functions and scripts in all DXP files of the library and lists them in a .csv file that can be opened in Spotfire® or Excel. Optionally, the administrator may also use the find-analysis-scripts command to automatically assign trust to all data functions it finds.
Data functions, TERR inline expressions and TERR expression functions
There are three flavors of using TERR and R in Spotfire® analysis files. For the purposes of trust, data functions and TERR expression functions are the same. However TERR inline expressions are different in that they cannot be trusted. It should be noted that even if a data function, expression function or inline TERR expression is not trusted, Spotfire® will attempt to execute it using TERR's so called restricted execution mode. If this is successful, the function just runs and the end user does not notice any issue.
An inline TERR expression is when you write R code directly on the axis of a visualization such as in this example:
<TERR_Integer("output <- kmeans(data.frame(input1, input2, input3), 3)$cluster",[column 1],[column 2],[column 3])>
TERR inline expressions always use functions beginning with TERR_ (see the statistical functions for all options). As mentioned, they cannot be trusted and will therefore always be executed in TERRs restricted execution mode. This means that if an inline TERR expression would use functionality not available in TERRs restricted execution mode, the inline TERR Expression must be rewritten to a TERR Expression function.
Example of rewriting a TERR inline expression to an TERR Expression function.
Guidelines for upgrading and other questions
For guidance on things to consider during an upgrade, please refer to the Community article Guidelines on how to introduce the improved trust mechanism in Spotfire® 10.3 (and later) when upgrading
Also check the FAQ about data function trust.
Introduction to the find-analysis-scripts command
As mentioned, the find-analysis-scripts command can help the Spotfire® server administrator to obtain a list of scripts and data functions in the Spotfire® library and to update the trust stamp of already trusted scripts. In order to do this, the command needs to search the entire library, or part of the library (as per the supplied options), and load all files it encounters, analyze them for scripts and possibly trust data functions it encounters, if this option has been specified.
Practically the find-analysis-scripts command will load each file from the library (to the configurable location as set in the java.io.tmpdir environment variable), analyse it to find scripts and data functions, compute the stamp for the script or data function and then store the stamp back to the library. When the loaded file has been analyzed the command will dispose of it - in this way only one or a few files are downloaded at a time. The command also records information about the scripts it finds in each file and stores in a .csv file (see a later section inthis article for more details). Note: If the library is large (contains many and/or large files) the analysis process may take significant time, cpu and memory resources so make sure to read through the various options for running the command on a subset of the library as described below.
The find-analysis-scripts command logs its progress when working, so if it is interrupted for any reason, the next time it gets started it will start from where it was interrupted.
The options available for executing the find-analysis-scripts command include automatically trusting all data functions, running on a specified library folder, running with less memory and cpu usage and instead talking longer time, running on files that are above or below a certain size or that have or have not been accessed for a certain time etc. You can read everything about the CLI and the find-analysis-scripts command in the Spotfire® Server Installation and Administration Help, or by starting the command with the help parameter:
>config help find-analysis-scripts
Example usage
Run on the entire library
>config find-analysis-scripts
This will start the find-analysis-scripts command on the entire library (which may take significant time, cpu and memory resources as discussed above - please consider the below alternatives before deciding how to start). The trust stamp of all existing and trusted IronPython, JavaScripts and data connection custom queries will be updated. A .csv file called report_yyyy-mm-dd.csv will be generated. The csv file contains an entry for each script in each file of the library (including scripts and data functions within analysis files, custom queries in saved data connections and separate data functions), and the file will be generated and stored in the folder find-analysis-scripts.
Run single threaded
>config find-analysis-scripts -i
The -i switch tells the tool to run single-threaded, meaning that the command will use less CPU (without the switch the command will use several threads) and memory resources on the machine running the command but it will take longer time to complete. In all other respects this will have the same effect as the previous example.
Run on a selected folder
>config find-analysis-scripts -p /library-path
If you do not wish to run the tool on the entire library at once, for example, when preparing for the upgrade, you may supply a path to the directory on which to run. The command will also include all sub-folders of the specified folder.
>config find-analysis-scripts -p "/myfolder/data function trust"
Running the command on a specified folder with just a few DXPs in will take less time, and thus allows you to try and practice using the CLI without taking extensive time or CPU resources. It is also an option if you want to use a folder per folder approach rather than running on the entire library at once.
Run on small files only
>config find-analysis-scripts --library-search-expression "content_size:<2000MB"
The -library-search-expression allows you to specify an expression to run the command on the set of files that match the expression. This example specifies that the find-analysis-script command will only run on files that are less than 2000MB in size, which is handy if you have files that are very large and you want to avoid loading them with the command. The syntax and available options in library search expressions are described in the Spotfire® Analyst help.
Run on files accessed after/before a certain date
>config find-analysis-scripts --library-search-expression "accessed:>""2019-01-13T18:27:59CEST""" >config find-analysis-scripts --library-search-expression "accessed:>""1 weeks ago"""
The above two examples of using the -library-search-expression shows two different options for selecting files that were last accessed before or after a certain date or time.
Reviewing the find-analysis-scripts report
As described earlier, the find-analysis-script command generates a .csv report of scripts, data functions and TERR expression functions that it identifies in the library. You can open the report to view it in Spotfire® Analyst (version 10.3 or higher is recommended for the best experience) or Excel (see footnote in the end about regional settings).
The report helps you answer questions such as:
- Which analysis files has scripts or data functions, and are they trusted or not?
- When were the analysis files with untrusted scripts last accessed by any user?
- Who created or were the last to modify an analysis file with untrusted data functions?
The report contains one line per script, per library file. Together with other information, the report contains the following data columns:
Library file ID: A unique identifier for a DXP file in the library.
Library file title: The name of the DXP file.
Library file created by user display name: The user display name of the user that created the file.
Library file modified by user display name: The user display name of the user that last modified the file.
Name: The name of the script, data function or custom query.
Type: Specifies whether it is a script, custom query, expression function or data function.
Function type display name: More details about the type of the script or data function.
May contain inline scripts: True if the tool detected that the file may contain so called "inline TERR expressions". These cannot be trusted by the tool and will require user intervention.
Trusted: Indicates whether the script or data function that this row refers to is trusted or not.
Source of trust: This contains a set of library IDs where the script/data function is trusted. If the script is used in many analysis files, then this is a list of all analysis files where the script/data function is used.
Automatically trusted: True for those scripts that were trusted during this run of the tool, if any.
Below is an example of how the data from the report can be visualized in Spotfire®, to get an overview of the untrusted scripts and data functions in the library. The table below the bar chart is a details visualization.
Note that an example of a report .csv file (zipped), also in the form of a Spotfire® dxp file, is attached and available at the bottom of this page.
Note: Depending on regional settings Excel may not parse the report correctly. To resolve this either change the list separator character in Microsoft Windows to semicolon or specify "sep=;" on a new first row in the report file before opening it. Note that if you change the file then you need to tell Spotfire® to ignore the first row when importing.
More information
For more information please read the article about considerations when you are planning to upgrade and the FAQ, and if you like, view this video.
Downloads
report_example_2019-03-18_0.zip
Recommended Comments
There are no comments to display.