What's New in Spotfire® 7.9

TIBCO Spotfire® 7.9

The main highlights in Spotfire® 7.9 are significant new inline data wrangling features.

Spotfire® 7.9 On Demand Webinar 

Inline data wrangling

Edit data transformations

Spotfire 7.6 introduced the Source View which provides an overview of your data transformations, calculations and how your data tables are derived from rows and columns combined from multiple data sources. Spotfire 7.7 made add rows (unions) editable and smart by usage of the Spotfire recommendations engine.

With Spotfire 7.9, one of the most anticipated new features of all times is now available; the ability to change data transformation settings. This saves you a lot of time, for example, when a recently added data transformation needs further editing, or if an existing data transformation needs to be adapted to changes in the data source.

Access points for editing data transformations

The image below shows an example of details in the Source View. There are two access points for editing data transformations; one for editing data transformations that are part of a data source, and the other access point is used to edit data transformations inserted as separate steps.

The image below shows the dialog for working with data transformations and how to gain access to the settings dialogs for each data transformation.

Available edit features

The following editing features are available from the Source View:

1. Edit a data transformation. (Edit...)
2. Delete a complete data transformation group. (The waste basket icon.)
3. Delete a data transformation from a group (including deletion of a data transformation in data source step). (Remove)
4. Insert a data transformation into an existing transformation group before or after existing data transformations. (Insert menu).
5. Change the order in which data transformations are applied. (Move Up/Move Down)

Certain non-editable use cases

In some cases, the ability to edit a data transformation will not be possible. In summary, if a data source (column producer) cannot be refreshed, it cannot be edited. There are two cases when this is the happens:

If the final data table is (top) embedded.

If a data source includes data transformations but its data is not linked or cached, but stored (embedded).

A stored data table with a disabled access point for edit data transformation:

A linked data table with an available access point for edit data transformation:

Indications when something goes wrong in data transformations

With Spotfire 7.9, you will be notified when a data preparation step cannot be applied as expected, or, if a data transformation is no longer necessary.

The image below shows an example of the three levels of indications, depending on severity.

An Error indication

An Error indication is displayed if a data transformation cannot be applied.

For example, if a column is missing for a calculation (if it has changed or has been removed in the data source, or, if it has been removed when editing a previous data preparation step in Spotfire), you will see an error. With Spotfire 7.9, and the ability to edit data transformations, many errors can be resolved in Spotfire. Once fixed, the error indication will be reevaluated, and hopefully disappear.

A Warning indication

A Warning indication is displayed if, for example, a defined value formatting step no longer can be applied.

For example, this happens if a column's data type (Real) and formatting (Percentage) has been changed using Spotfire's Data panel.

Now, if the data type changes to Real in the data source, Spotfire will not apply the data type change and thus cannot apply the Percentage formatting. A Warning highlights that you need to redefine the formatting on the column again.

An Information indication

An Information indication is displayed, for example, if a data type is changed to the same data type that the column already has, using a data transformation. This can happen if the data type has been wrong before, but now has been corrected in the data source. The data transformation in Spotfire is then no longer necessary, and this is highlighted using the Information indication.

Inline data cleaning

Spotfire now provides an easy way to clean up issues in your data, right when you see them. It is when you visualize data that you spot errors, so why not fix them right there and then? The new Replace value feature lets you change incorrect data values by double-clicking in a table, in the Details-on-Demand, or in the expanded Data panel. There are two flavors to the replace value feature; the ability to replace a single value only, or to replace all occurrences of that value in the column. 

Replace all occurrences of the value

For some types of data issues, the natural way to fix it is to replace all occurrences of the incorrect value. This helps you solve issues caused by alternative (mis)spellings like Tomatoes|Tomatos, Color|Colour or even if some rows of data use acronyms such as CA and some rows use the full name California.  It can also be used to group categorical values into different "buckets", such as grouping states into arbitrary regions.

Replace a single data value

Replacing a single data value is useful, for example, when you find issues in numerical data. Perhaps the decimal point is in the wrong place, or some other type of error. 

Replace specific value from a table details visualization

Replace specific value from the Details-on-Demand

Replacing the single value only requires that there is a defined key that can be used to identify this specific row of data. In the above screenshot, you can see a link to "Select key columns". The link leads to the below dialog that lets you define one or more columns to uniquely define each row of the data table.

How does it work?

Underneath the surface, the changes are implemented using two new data transformations, Replace value and Replace specific value. This means that no data is changed in the original data source. Instead, the value is replaced when the data is brought into Spotfire. It also means that when data is reloaded, the same corrections are applied again, and for the Replace value case new instances of the value in question are also being replaced.

The logic in the Replace specific value case is to replace the value only if it is the same value as when the transformation was created. Thus, if the value is changed in the data source after the transformation was defined, the transformation will no longer have any effect.

Review all changes

The visual Data source view lets you inspect and, if needed, remove the Replace value transformations.

Above, you can see how replace value transformations are shown in the source view.

Recommendations for add rows prefix and postfix support

Before Spotfire 7.9, Spotfire's recommendation engine would automatically detect if new data should be added as rows to existing data. With Spotfire 7.9, the recommendation engine for add rows also automatically matches columns with common names but different prefixes and/or postfixes. For example, the new column 'Sales (2016)' will match the existing column 'Sales (2015)'.

Columns that have the same prefix/postfix will have the prefix/postfix removed from the column name. In the example above, the column name will be 'Sales'.

The prefix/postfix will automatically be entered on all rows in the origin column. In the example above, the origin column will contain '2016' and '2015' for the respective data sources.

Access Amazon Redshift data from Spotfire Cloud web clients

Amazon Redshift is now supported in Spotfire Cloud Business Author and Consumer. This means that when you open an analysis file with data from Amazon Redshift in Spotfire Cloud Business Author and Consumer, you can now load data directly from your Amazon Redshift instance. Both in-database live queries and in-memory data import are supported.

Analysis files with Amazon Redshift connections are authored in Spotfire Cloud Analyst, saved to the Spotfire Cloud Library and are then available for Spotfire Cloud Business Author and Consumer users.

You can manually refresh data from individual data sources from Business Author's Source View.

Note: You might have to allow the Spotfire Cloud servers to access your Amazon Redshift data by whitelisting the servers' IP addresses. More information is available in the TIBCO Cloud Spotfire help.

Access Azure SQL data from Spotfire Cloud web clients

Azure SQL is now supported in Spotfire Cloud Business Author and Consumer. This means that when you open an analysis file with data from Azure SQL in Spotfire Cloud Business Author and Consumer, you can now load data directly from your Azure SQL instance. Both in-database live queries and in-memory data import are supported.

Analysis files with Azure SQL connections are authored in Spotfire Cloud Analyst, saved to the Spotfire Cloud Library and are then available for Spotfire Cloud Business Author and Consumer users.

You can manually refresh data from individual data sources from Business Author's Source View.

Note: You might have to allow the Spotfire Cloud servers to access your Azure SQL data by white listing the servers' IP addresses. More information is available in the TIBCO Cloud Spotfire help.

Access OData provider data from Spotfire Cloud web clients

Tutorial: https://community.spotfire.com/wiki/access-odata-provider-data-spotfire-clo...

OData is now supported in Spotfire Cloud Business Author and Consumer. This means that when you open an analysis file with data from OData in Spotfire Cloud Business Author and Consumer, you can now load data directly from your OData instance. The OData connector supports in-memory data import.

Analysis files with OData connections are authored in Spotfire Cloud Analyst, saved to the Spotfire Cloud Library and are then available for Spotfire Cloud Business Author and Consumer users.

You can manually refresh data from individual data sources from Business Author's Source View.

Note: You might have to allow the Spotfire Cloud servers to access your Odata providers by white listing the servers' IP addresses. More information is available in the TIBCO Cloud Spotfire help.

Connectors and live query data tables

Microsoft Azure HDInsight is now supported

Starting with Spotfire 7.9, the Hortonworks Hive connector now supports Microsoft Azure HDInsight.

For more information about Microsoft Azure HDInsight, see: https://azure.microsoft.com/en-us/services/hdinsight/

Apache KNOX is now supported

Starting with Spotfire 7.9, the Hortonworks Hive connector now supports Apache KNOX, with or without Kerberos.

For more details about Apache KNOX, see: https://knox.apache.org

SAP SSO is now supported with the SAP BW connector

It is common that SAP BW deployments use SAP's SSO solution. Spotfire's SAP BW integration now supports this authentication method in all clients and servers. This enables Spotfire users to analyze SAP BW data without entering their SAP BW credentials manually. It also provides a central location for users and roles administration for SAP BW administrators.

Instructions for how to configure Spotfire for SAP BW SSO is available here: https://community.spotfire.com/wiki/single-sign-tibco-spotfire-sap-bw-conne...

Configurable maximum allowed number of rows in live query results

Spotfire 7.9 introduces a new safety setting which allows system administrators to set a limit for how large the data tables loaded using live queries (in-database tables) can be. This is a protection against, for example, ad hoc analysts splitting a bar chart on a fact table's ID column, which could result in a gigabyte data table being loaded into client and Web Player memory.

Google Analytics system web browser authentication

Spotfire's Google Analytics connector now supports Google's new modernized OAuth implementation. The system web browser is now used for user authorization, instead of a built in Spotfire dialog. This means that if a user is already logged into Google in the system web browser, the login step will be performed automatically.

For more details about the reason for this change, see: https://developers.googleblog.com/2016/08/modernizing-oauth-interactions...

New data source versions support

Analysis Services 2016 is now supported

Spotfire 7.9 (and later) now supports Analysis Services 2016.

For details, see the system requirements page here: http://support.spotfire.com/sr_spotfire_dataconnectors.asp#ssas

PostgreSQL 9.5 and 9.6 is now supported

Spotfire 7.9 (and later) now supports PostgreSQL 9.5 and 9.6.

For details, see the system requirements page here: http://support.spotfire.com/sr_spotfire_dataconnectors.asp#postgresql

MySQL 5.7 is now supported

Spotfire 7.9 (and later) now supports MySQL 5.7.

For details, see the system requirements page here: http://support.spotfire.com/sr_spotfire_dataconnectors.asp#oraclemysql

SAP BW 7.5 is now supported

Spotfire 7.5 (and later) now supports SAP BW 7.5.

For details, see the system requirements page here: http://support.spotfire.com/sr_spotfire_dataconnectors.asp#sapnetweaver

Apache Spark SQL 2.0 is now supported

Spotfire's Spark SQL connector now supports Spark 1.6.0 to 2.0.2.

NOTE: The latest TIBCO ODBC Driver for Apache Spark SQL must be used in combination with the connector.

For details, see the system requirements page here: http://support.spotfire.com/sr_spotfire_dataconnectors.asp#apachesparksql

Information Services now supports constrained Kerberos delegation

Spotfire Information Services now supports constrained Kerberos delegation in combination with compatible JDBC drivers.

Location Analytics

Nautical Miles unit (new feature)

Nautical Miles is added as a unit of measurement in addition to existing imperial and metrics units, when using radius and rectangle selection.

Get the coordinates of a location (new feature)

You can now right-click anywhere on a map and get geographic coordinates (latitude and longitude) for a location.

Easier access to map layer (enhancement)

It is much easier to enable access to the map layer when Spotfire cannot access the Internet or is on a restricted environment. Now, only one unique domain needs to be allowed.

Advanced Analytics

• Continued work towards broader R compatibility, to enable more and more potential applications to be run on TERR. As of this release, 99% of packages on CRAN, almost 10,000 community packages, can be loaded in TERR. (Well done, TERR Team!). Full details on compatibility are available on the TERR Documentation site.
• Significant improvements to TERR performance in many areas.
• TERR can now be used in RStudio to create interactive R Markdown notebooks. R Notebooks allow for direct interaction with R while producing a reproducible document with publication-quality output.
• A new Guide to Graphics in TERR, which provides tips and examples on using Javascript-enabled packages, certain open-source R packages, and the TERR RinR package to create graphics from TERR.

Server

Log4J2

For Spotfire Server 7.9, the logging framework has been upgraded from Log4j to Log4j2. The benefits of upgrading to Log4j2 include the following:

• You can manage logging from the UI. For example, you can start debug logging during runtime, without having to manually edit configuration files.
• Log4J2 is garbage-free, which reduces the pressure on the garbage collector.
• Java 8 feature sets are fully supported, including lazy logging.

If you used a custom-modified log4j.properties file in any Spotfire Server version between 7.5 and 7.8, you must manually add these modifications to the new log4j2.xml file. 

Sites

You can now create multiple Spotfire environments that share the same Spotfire database, including the library and user directory. These environments, which are called sites, can be configured to reduce latency for multi-geographic deployments. Sites also enable the use of a variety of authentication methods, along with different user directories, within the same deployment. 

Each site includes one or more Spotfire Servers along with their connected nodes and services. A site's servers, nodes, and services can only communicate within the site, but because the Spotfire database is shared among the sites, all of the sites have access to the users, groups, and library in your Spotfire implementation.

The benefits of using sites include the following:

1. You can route user requests from a particular office to the servers and nodes that are physically closest to that office. This reduces the impact of network latency between servers that are located in different geographic regions. 
2. You can enable different authentication methods for different sets of users who share a Spotfire implementation. For example, internal users can be authenticated with Kerberos authentication while external users, such as customers and partners, can be authenticated with a username and password method.