What's New in Spotfire® 7.11 LTS

Spotfire^® 7.11 LTS

Spotfire® 7.11 brings highly requested improvements in data wrangling, cross tables, tables and maps, and it also makes the life of the Spotfire® Administrator easier through improvements in scheduled updates and management of multiple Sites. Developers and application builders will enjoy the upgraded IronPython engine that now supports the latest (2.7.7) IronPython version.

In addition to the new features, Spotfire® 7.11 has been designated as a Long Term Support (LTS) version.  LTS versions are typically supported for up to 36 months from release. For LTS versions, defect corrections will typically be delivered as hotfixes or service packs while for regular releases they will be delivered in subsequent releases.

Visual Analytics

Calculate the subtotals and grand totals in cross tables based on the aggregated values displayed in the cells

It is now possible to configure the cross table to calculate subtotals and grand totals based on the aggregated values visualized in the table, as an option to calculating it using the underlying row level data. This is useful, for example, when you want to visualize the sum of the absolute values of the categories displayed in the table.

In the screenshot above, you can see the Properties dialog where you can select, for each column, whether to calculate the subtotal and grand total on the underlying row values, or as the sum of the values displayed in the cross table cells. This is useful, for example, when one wants to compare the sum of absolute values in the subtotals.

Conditional color of the text in tables and cross tables

It is now possible to color the text in tables and cross tables through color rules, as an alternative to coloring the cell background. This provides more freedom in the visual expression of the tables.

Search and zoom to a location

You can now search for a geographic location on the map and quickly zoom in to its geographic area. When you start typing a location name, Spotfire® suggests locations you can select to zoom to on the map.

Switch data table now keeps the visualization configuration

Visualizations will now keep their configuration when switching to another data table, provided that the new and the old data table include the same columns. This saves time when switching back and forth between identical tables. 

Data wrangling

Replace Data source

As a Spotfire® user, you are used to working with multiple data sources mashed together, to provide more answers from your data. With this release of Spotfire®, you can easily replace one of those data sources with another data source, without compromising the data wrangling and data mashup you have done.

Example: Going from test to production

The picture below shows the source view in an analysis file. Three data sources are used and mashed together using Insert Columns (joins).

The first data source is a linked data table containing sales sample data, stored in a local Spotfire® Binary Data Format file (SalesOrderDetailSample.sbdf).

By working with an alternative and local data source you can develop an analysis file without access to the production data source. This is convenient, for example, when working off-site, or, when you have work in progress that you do not want to introduce in your production environment (for performance reasons or for other reasons).

Once you are ready to switch to the production data source, you can access the new replace data source feature from the data source menu in the source view:

The picture below shows the new Replace Data Source dialog. In this example, we select to switch to the corresponding data table in Microsoft SQL Server.

In the image below, the sample data source has been replaced. The data source type is now a data connection instead of an sbdf file.

Add transformation to existing data source

In addition to the capability to replace data sources, this release of Spotfire® also enables you to add data transformations to existing data sources. Previously, data transformations could only be added when creating a new data source or when editing data transformations already part of the data source.

There are certain situations when it's beneficial to attach transformations to data sources. The benefits are based on the fact that Spotfire® doesn't save the original data in the analysis file, only the transformed result.

Let's assume you prefer to store a copy of your data in your analysis file for it to be available offline and for you to be able to decide when a reload is needed. Let's also assume that you are loading 200M rows into Spotfire®, and then defines an pivot data transformation to reduce the size of the data table. Having the pivot data transformation as part of the data source will only store the pivoted result table and discard the 200M original rows. This dramatically reduces the size of the analysis file. If the transformation was performed as a separate step, the original 200M rows would be stored.

This will also reduce the loading time when opening the analysis, since the pivoted table is already available. If the transformation was performed as a separate step, the pivot operation would have had to be performed as part of loading the analysis file.

Custom data transformations may also benefit from being performed as part of the data source.

The image below shows the new access point to insert a transformation on a data source.

Edit replace value transformations

It is now possible to edit replaced values without creating additional transformation steps within the analysis. This means that you can go back and modify previously added replacement operations, if they are no longer applicable. By editing already created operations, you can avoid having a large number of transformations for replacing the same value over time, and make the analysis cleaner.

The image below shows the entry point for editing two replace specific value transformations. Click the Edit button to open the new edit dialog.

The image below shows the new edit dialog for Replace Specific Value:

Since we have replaced a specific value we have defined both a new value and a primary key column (PermitNumber). You can add more key columns and you can replace the currently used key column and/or value in the dialog.

You can also insert a new replace value transformation (using the new Replace Value and Replace Specific Value dialogs) into an existing transformation group by clicking Insert in the Edit Transformations dialog:

Edit relational data connection data sources from the source view

Previously, Spotfire® users had source view access to make quick changes to data connection configurations. This made it possible to add and remove tables and columns, add or modify custom queries, modify prompts, change column names and other settings that are part of data connections.

With this release, it is just as easy to make changes to the data source used by the data connection. The data source holds information regarding source IP, authentication method, time-outs and database, which are all easy to modify now.

For example, it has never been as easy to move from a test database to a production database. With a few clicks from the source view, you can now point the data source to another database, maybe even to a database with another type of authentication method. If different table names are used in the databases, for example, 'dbo.test.transaction' in the test database and 'dbo.prod.transaction' in the production database, Spotfire® highlights these differences in the data connection and makes it easy to select the corresponding table in the production database.

The image below shows a data connection data source being displayed in the source view. Click on the settings button (the gear icon) on the data source node to edit the data connection. 

The image below shows the Views in Connection dialog reached from the settings button (the gear icon). From here, you can enable full editing of the data connection by clicking the button in the lower left corner of the dialog.

The image below shows the new Edit Data Source Settings button. This is a new feature in 7.11 and provides a shortcut to editing your data source.

The image below shows the Microsoft SQL Server Connection dialog which contains the settings for the connection data source. From here, it is easy to, for example, switch from a test to production server or database. You can also switch authentication method.

Data Access

Option to query SAP BW directly towards the SAP BAPI API

The SAP BW connector now has the option to query SAP BW using the native SAP BAPI API without going through the ODBO API used until today. If you choose to enable the BAPI API integration you can expect a boost in performance and more detailed messages from SAP BW should something go wrong. If you choose not to enable the BAPI API, the SAP BW connector will use the ODBO API as before.

We are convinced that the BAPI API will provide a better user experience and allow us to develop new features over time. We have therefore decided to deprecate support for the ODBO API in a future Spotfire® release. However, both APIs will be available for a period of time, to allow you to upgrade your SAP BW client driver installation to the BAPI API at your own pace.

The image below shows the title of the SAP BW Connection dialog, where it is indicated that the SAP BAPI API is being used.

Load more than one million SAP BW data cells

Note: This feature becomes available when you have enabled the SAP BW's BAPI API on Spotfire® clients and servers. Please see the "Option to query SAP BW directly towards the SAP BAPI API" feature above for more details.

SAP BW limits the number of non-empty cells that can be retrieved in metadata and in result data sets. This limit is configurable in SAP BW, and common limits are between 500k and 1M non-empty cells. By leveraging the SAP BW BAPI API, Spotfire® is no longer dependent on this limitation and allows you to analyze more data than set by the limit. This means that you can connect to BEx queries representing more data, and thus extending the number of use cases you can implement with Spotfire®.

Only Spotfire® administrators can enable this capability in the Spotfire® platform.

Specify SAP BW operation timeout

It is common for SAP BW BEx queries to represent very large amounts of data. This means that Spotfire® data import queries towards BEx queries sometimes need some extra time to complete. You can now increase the default 10 minute timeout as part of the SAP BW data connection. This allows you to import and analyze larger data volumes without queries timing out before your data is available.

The images below show the SAP BW Connection dialog. Click the new Advanced tab to reach the operation timeout setting.

Increased SAP HANA function support

Spotfire®'sSAP HANA connector now supports the following additional functions: 

Median

Stddev_Pop

Stddev_Samp

Var_Pop

Var_Samp

Bitcount

Months_Between

Years_Between

The image below shows a few of the new functions in Spotfire®'s Custom Expression user interface. Note that the details for how to use these functions are documented by SAP and are subject to change over time.

Support for new thrift transport modes in Apache Spark SQL

Spotfire®'s connector for Apache Spark SQL now supports the thrift transport modes Binary, SASL and HTTP. Having the TLS security settings on the first page and turned on by default for new connections makes it quicker to configure your data connections in a secure way, for example, to Databricks data sources.

Support for Teradata 16

The Teradata connector and Information Services now support Teradata 16.

The images below show the different tabs available in the updated Teradata Connection dialog.

Spotfire® Cloud access to data from Spotfire® Data Catalog in Spotfire® Cloud Business Author and Consumer

Analysis files opened in Spotfire® Cloud Business Author and Consumer can now load data directly from publicly available Spotfire® Data Catalogs.

Analysis files are authored in Spotfire® Analyst, saved to the Spotfire® Cloud Library and are instantly available for Business Author and Consumer users.

As a Business Author and Consumer user, you will receive fresh data when the analysis is opened. You can manually refresh data from individual data sources in the source view of Spotfire® Business Author.

The image below shows the library browser of the Spotfire® Cloud web client.

When you open an analysis based on data from Spotfire® Data Catalog, it is now possible to refresh the data directly from the source view:

Spotfire® Server

LDAP and Spotfire® Authentication

Spotfire® 7.11 allows users to access Spotfire® even though they are not part of the external user directory.

If you configure authentication towards an external user directory such as an LDAP directory, or a Windows NT Domain, you can combine this with adding users manually to the Spotfire® database so you do not have to add them to the LDAP directory.

To see more on this feature,

Scheduling & Routing

Spotfire® 7.11 provides three new features for scheduling and routing, to help our administrators more easily manage files that are not cached and rules.

1. You can now prevent users from opening analysis files that are not cached by scheduled updates. This is useful because if there are certain analysis files that take a significant amount of resources to initially load, you can prevent that from happening by not allowing the user to open an uncached analysis file. To see more on how to do this, 
2. You can now recover a rule if it was automatically disabled. When an analysis file is deleted from the library, the routing rule associated to it will fail and the rule will become disabled. Now, if the analysis file is imported back to its previous location the rule is recovered and can automatically be reenabled by updating a setting in the server configuration file, enable-recovered-rules-automatically. To see more on this feature, .
3. You can now copy routing rules and schedules from one site to another. For details on how to use this feature, .

Update to the Library Browser Page

The Spotfire® library browser now provides a left hand navigation section that allows you to view recent files you have just opened as well as quickly browse for other files of interest.

Developer

IronPython support updated to version 2.7.7

Spotfire® 7.11 supports the latest version of IronPython (2.7.7), enabling more powerful language features and libraries.

IronPython is an implementation of the Python programming language which is tightly integrated with the .NET Framework. Using IronPython scripts with Spotfire®, you can utilize not only .Net Framework and Python libraries, but also the full Spotfire® C# API. This makes IronPython scripting a powerful tool when creating advanced analytic applications in Spotfire®. If there is a need to run certain scripts in the older version of Iron Python, this is still supported by selecting the older version in the drop-down list shown in the image below.

For tutorials and examples, see https://community.spotfire.com/articles/spotfire/ironpython-scripting-in-tibco-spotfire/