What's New in Spotfire® 10.3 LTS

Spotfire® 10.3 LTS is the second Spotfire® Long-Term Support release and also introduces a set of new capabilities such as data connectivity to Snowflake and MongoDB, and integration with the TIBCO Live Apps low code business application development platform. Spotfire® 10.3 LTS also improves security and governance for scripts and data functions, geo-coding coverage, APIs, data wrangling and provides a set of other small but important improvements.

NOTE: Administrators should be aware that Spotfire® 10.3 LTS requires additional steps during the upgrade due to the introduction of the data function trust feature. Before upgrading to Spotfire® 10.3 please read about data function trust and upgrading in these articles.

Improved governance and security for scripts and data functions

The demand for governance and security in Business Intelligence is ever growing. Spotfire® is known for its powerful and flexible automation, as well as for statistical and predictive capabilities. However, with power comes responsibility: Spotfire® 10.3 improves the Administrator capabilities in terms of governance and security for advanced applications using scripts and data functions. Scripts based on IronPython, JavaScript, and data connection custom queries have already in earlier versions of Spotfire® used a trust mechanism in order to control which users should be allowed to write scripts that can be executed by other users without explicitly asking for permission. This mechanism has now been extended to cover also data functions and TERR Expression functions. In addition, a new capability to find and list all scripts in the Spotfire® library has been added. The report provided contains information about all analysis files which contain scripts or data functions, whether the scripts or data functions are trusted or not, the author of the file, who last modified it, etc. Also, Spotfire® 10.3 updates the algorithm used to calculate the trust checksum for scripts to SHA-512.

Note for administrators: The above changes means that there are a few additional steps required during the upgrade to Spotfire® 10.3. The exact steps to take will depend on your use case and environment. Read more about the steps you need to consider in this community article.

Trust for data functions and TERR expression functions

Data functions and TERR expression functions are now covered by the same trust mechanism as IronPython scripts, JavaScripts and data connection custom queries. This means that a user writing a data function must be in the Script Author group if other users should be able to execute the data function without being asked for permission. When using analyses in the Spotfire® web clients, only scripts and data functions written and/or trusted by users in the Script Author group will be executed.

Restricted mode for executing TERR based data functions and expression functions

The integrated R engine TIBCO Enterprise Runtime for R (TERR) provides a "restricted" mode where only features that are considered always secure are allowed. If a data function or TERR expression function that is not trusted needs to be executed, Spotfire® will attempt to execute it using the restricted mode in TERR. The safe mode is restricted to a subset of features of the TERR engine and also does not allow importing other packages. Examples of operations not allowed in the safe mode are access to files or communicating over the internet. Read more about the restricted execution mode in the TERR documentation.

Inline TERR expressions

Spotfire® allows writing expressions that contain TERR functions on the axes of visualizations. This is done by using functions starting with TERR_. These functions will always be executed in the safe mode. If such an expression uses a function that is not allowed in the safe mode, the expression will not work. If that happens, the inline expression should be converted to a TERR expression function and trusted to allow it to run.

SHA-512 for determining trust

Spotfire® now uses the SHA-512 algorithm to generate the checksum for ensuring that scripts and data functions are identical to when they were trusted. This is done to eliminate the possibility that the script or data function has been tampered with by a user not authorized for trusting scripts and data functions.

Reviewing and trusting scripts in Spotfire® Analyst

When a user opens an analysis file containing untrusted scripts or data functions in Spotfire® Analyst, the user will be notified that there are scripts in the analysis that are not trusted. The user may review each script in detail and trust it, or use the Trust All button in case he/she is sure that the scripts are from a trustworthy source.

The find-analysis-scripts server CLI command

A new Command Line Interface (CLI) command called find-analysis-scripts is available in the Spotfire® Servers CLI. With this command, the Spotfire® administrator can get a report showing all scripts and data functions present in the library, review their type, trust status, who created and last modified the file etc. In addition, the command allows trusting all or selected types of scripts in selected locations, or the entire library.

The script report

As mentioned in the above section, the find-analysis script command generates a report in the form of a .csv file. The report contains one line per script, per library file. The information about the scripts includes among other information:

Library file ID: A unique identifier for a DXP file in the library.
Library file title: The name of the DXP file.
Library file created by user display name: The display name of the user that created the file.
Library file modified by user display name: The display name of the user that last modified the file.
Name: The name of the script, data function or custom query.
Type: Specifies whether it is a script, custom query, expression function or data function.
Function type display name: More details about the type of the script or data function.
May contain inline scripts: True if the tool detected that the file may contain so called "inline TERR expressions". These cannot be trusted by the tool and will require user intervention.
Trusted: Indicates whether the script or data function that this row refers to is trusted or not.
Source of trust: This contains a set of library IDs where the script/data function is trusted. If the script is used in many analysis files, then this is a list of all analysis files where the script/data function is used.
Automatically trusted: True for those scripts that were trusted during this run of the tool, if any.

Reviewing the script report in Spotfire® (screenshot of analysis).

Improvements when browsing the library

You have asked and we have listened! Several Ideas related to browsing the library are implemented to make it easier and quicker to find the analyses you are looking for. The enhancements are:

It is now possible to widen the content browser (the Files and data flyout) by dragging with the mouse, in order to see really long names.
The tooltip when hovering over a Spotfire® Analysis now includes the description, if any is available.
It is now possible to sort the library contents according to the last modified date.
Performance improvements: browsing the library is now faster.

GeoAnalytics

Improved recommendations for columns with relations to latitude and/or longitude

When the AI-powered recommendation engine detects relationships between a selected column and latitude or longitude, a map recommendation is now displayed, in addition to other visualizations including the selected column and latitude or longitude.

Geocoding coverage updates

The following data has been updated:

Over 4,000 new internal administrative boundaries (Admin 1, States, Provinces, Regions...) are now available worldwide.
French region names are updated to the latest naming scheme.

Hide geocoding warnings

Sometimes it is acceptable to not have columns matches or to have multiple entries for a feature on a map so we make it possible to hide the geocoding warnings that appear in the map chart title.

Layers control state

Whether the map layers control is in an expanded or collapsed state, it is now saved in the analysis.

Custom header in web clients

The custom header capability has been reintroduced in the Spotfire® web clients through the white-labeling/co-branding mechanism. This means it is possible to insert your custom header in order to brand the web client, for example in an OEM use case.

Data access

Oracle MySQL support via ODBC

The connector for Oracle MySQL has been migrated from using an ADO .Net driver to using ODBC drivers. Switching to ODBC enabled support for push down queries into a number of other MySQL compliant data sources, see more information further down.

Native MariaDB support

The SQL interface of MariaDB is similar to MySQL. After the switch from the ADO .Net driver to ODBC, the MySQL connector now also supports MariaDB.

When you have entered the server address, make sure to select MariaDB as database system.

Native MemSQL support

The SQL interface of MemSQL is similar to MySQL. After the switch from the ADO .Net driver to ODBC, the MySQL connector now also supports MemSQL.

When you have entered the server address, make sure to select MemSQL as database system.

Native MongoDB support

The SQL interface of the MongoDB BI Connector is similar to MySQL. After the switch from the ADO .Net driver to ODBC, the MySQL connector now also supports the MongoDB BI Connector. The MongoDB BI Connector lets you use MongoDB as a Spotfire® data source even though MongoDB is capable of storing multi-structured data. For more information about the MongoDB BI Connector please visit this page.

Native Snowflake support

This release has native Snowflake Data Warehouse support. Once connected, you can push live queries into Snowflake and optionally combine these queries with on-demand retrieval of rows for in-memory analytics using the powerful in-memory engine of Spotfire®. The connector is built on top of the new ODBC framework introduced in Spotfire® 10.0 and supports DSN.

The first step of connecting Spotfire® to Snowflake is to use Windows ODBC Data Source Administrator to create a DSN.

You can then select Snowflake in the Select data dialog.

From there, select the DSN you would like to import connection string arguments from.

Once connected you select tables, define relations, create optional custom queries, preview data tables, rename columns and everything else you are used to.

Native TIBCO Cloud Live Apps support

You can now connect directly to your TIBCO Cloud™ Live Apps data. This works from your own Spotfire® platform and from TIBCO Cloud Spotfire®.

Remember that you must have a TIBCO Cloud™ account, on which you have access to a Live Apps subscription. You do not have to install a driver to connect to Live Apps.

The TIBCO Cloud™ Live Apps data source is listed in the Connect to list.

To access your Live Apps case data, you need the following information:

The credentials of your TIBCO Cloud account.
The organization on TIBCO Cloud that owns the Live Apps subscription.
The region on TIBCO Cloud that the subscription is in.

Note: You can only access data from Live Apps applications that have the status 'published'. Other applications are not visible in Spotfire®.

If the data from a Live Apps application is hierarchical and includes lists, the lists are displayed as separate database tables. If there are further levels to the hierarchy, meaning that it contains sub-lists, those are also listed as database tables.

You can relate and join lists with their parent data to create a single data table in Spotfire®. To do this, you use the Add related tables functionality. Note that you can only join related tables by selecting a list and then adding the parent data as related tables.

In the list of Available tables in database, double-click to add the list that you want to add and join with its parent data.
In the Views in connection list, right-click the list that you added, and select Add related tables.

Result: All data from the Live Apps application that is located "above" the list in the hierarchy is added and joined as related tables.

Tip: When you select data from Live Apps, there is no functional difference between Add related tables and Add all related tables.

When you add and join data with lists from Live Apps this way, Spotfire® flattens the final data table. The result is a data table with repeated values in columns. This is illustrated in this image:

You are now ready to start analyzing your TIBCO Cloud™ Live Apps data.

Native support for PostgreSQL 11

You can now access data from PostgreSQL 11 with the data connector for PostgreSQL.

Data management

Edit and remove calculated columns from the data canvas

It is now possible to edit and remove top-level entities such as calculated columns, binned columns and group-from-marked columns directly from the data canvas.

You can also remove predicted columns, result columns from other types of calculations, custom hierarchies, tags and mask columns in the data canvas.

Server and administration

Manage licenses in the administration web UI

You can now set and change licenses in the Spotfire® administration web UI. Licenses determine the features that members of a particular group can access. For more information, see Groups and licenses in the server help.

Easier upgrades

Upgrading the Spotfire® Server is now easier.

The structure of scripts, configuration files and custom extensions has been remodeled. More of your settings are kept during server upgrade.
The upgrade tool can now be used for upgrades between service packs.
The performance for upgrading and deleting services has been improved

These improvements have led to important changes in server directories and files; for details, see "Changes in Functionality" in the Spotfire® server release notes.

Custom display names for Spotfire® servers

The Spotfire® server now supports display names as an alternative to the server URL. This makes it easier for users to find the server they should connect to.

The display name will be visible in the login dialog and if you hover over the globe in the tool bar.

Search for group members

In Users & Groups, it is now possible to search for group members.

Monitoring & Diagnostics

It is now possible to configure a web player to capture either small or large dumps when it is non-responsive.

There is a new action log category for licenses: exclude_license. It indicates that the license feature was removed from the group's enabled license.

Cobranding

The custom header capability has been reintroduced for Spotfire® web clients. For more information, see the Cobranding help.

Developer

For a summary of all API changes since Spotfire® 7.11 LTS, see API Changes Between Spotfire® 7.11 LTS and Spotfire® 10.3 LTS.

Calling the Library REST API on behalf of end users

In Spotfire® Server 10.2 we introduced a new Library REST API that allows external applications to upload SBDF-formatted data to the Spotfire® library. In version 10.3, the Spotfire® Server now supports OAuth2 Authorization Code Grant for API clients using the Library REST API. This means that the API will be called on behalf of the end users rather than the API client itself. For example, when uploading a file to the Spotfire® library, the library privileges of the end user will determine what library folders the API client can upload the file to.

The authorization flow can optionally be configured to prompt for end user consent.

The OAuth2 Authorization Code Grant flow is available for all REST and SOAP based services. You can read more about the Library REST API and the other web services in on the Spotfire® Community.

Improved API for reloading data tables

The C# API to reload data tables in an analysis has been redesigned for improved ease of use and robustness. This corresponds to the Reload all and Reload linked data options in the data canvas UI. You can also reload specific data sources with the API introduced in version 10.1.

Legacy SOAP Web Service API

The legacy SOAP Web Service API has been removed in 10.3. It is replaced by the OAuth 2.0 based SOAP Web Service API (introduced in 7.13). Instructions on how to migrate existing solutions can be found here.