What's New in Spotfire Statistica®

Spotfire Statistica® 14.2

Spotfire Statistica® 14.2 is out now! 14.2 introduces new branding, external authentication for Statistica Data Entry, new parameterization options for Spotfire® data functions, and more! See below for further detail:

New Branding: A new name, but the same product you know and love. The Spotfire Statistica® name reflects our aim to offer predictive analytics and governance for Spotfire software. You will see the new name updated in the product, our documentation site, new community articles, and eDelivery. Older versions of Statistica® will still be named TIBCO® Statistica.

External Authentication in Statistica Data Entry: Spotfire Statistica 14.2 introduces external authentication for Web Data Entry users. External authentication configurations can now be configured in Statistica Data Entry via OpenID Connect (OIDC). Administrators can enable External Authentication by selecting “System Options” →  “External Authentications” in Statistica Enterprise Manager.

New parameterization options available for Spotfire Data Functions:

• You can now use the By Group option in Spotfire Statistica workspace nodes when using Spotfire Data functions.
• Spotfire Statistica® 14.2 now supports parameterization for the following Statistica workspace nodes from Spotfire Analyst and Spotfire Web Player:
  • Export Publishing Predictive Model Markup Language (PMML)
  • Sort
  • Process missing data
  • Recode Outliers
  • Transformations of variable

Exposed Data Miner Command-Line Interface Options: All Data Miner files are now available through the Command-Line interface.

Ability to search for deleted samples in Statistica Data Entry: A Global Search option is added to Data Entry to search for deleted samples only.

Enhanced installer logic: Users will now be required to specify a database during the Statistica Enterprise installation process.

New list separator function: A new list separator function has been implemented to accommodate non-comma delimiters and can be used in SVB scripts.

For more detailed information about additional features, fixes, and what's new in this release, please see the Documentation page which includes release notes.

TIBCO Statistica® 14.1

Statistica 14.1 builds on its best-in-class capabilities for data scientists and Enterprise business users alike by providing support for the latest cutting-edge technology.

A few highlights include:

• Support for the latest versions of Python and R 
• Support for StartTLS protocols
• Introduction of our new Session Viewer

For more detailed information about additional features, fixes, and what's new in this release, please see the TIBCO Documentation which includes release notes.

TIBCO Statistica® 14.0.1

Statistica 14.0.1 updated a number of lower-level system support libraries to their respective latest versions. 

TIBCO Statistica 14.0 

Statistica 14.0 solidifies and builds on its best-in-class capabilities for statistical and machine learning (ML/AI) analytics. In particular, Statistica 14.0 introduces many new and improved capabilities for validated manufacturing processes (GxP) and regulated manufacturing environments, such as the manufacture of pharmaceutical and medical devices. For example, through improved integration with TIBCO Spotfire, Statistica can provide even more effective ML/AI and comprehensive analytics for those applications: Analytics can be delivered through interactive Spotfire UI/UX to empower business and process stakeholders, engineers, and "citizen data scientists" exploring and applying leading-edge analytics for leading-edge results.

Here are just a few highlights:

• Numerous improvements and features to improve and simplify the integration of Statistica Workspaces with TIBCO Spotfire® via the Spotfire Data Function
• Support for TIBCO Spotfire® Statistics Services for TIBCO Spotfire Server LTS 10.10 and greater, enabling Statistica Workspace-based Data Functions for Spotfire Consumer and Web Client
• Comprehensive support for all OSI PI® Event Frame Attributes and other significant enhancements to OSI PI support 
• Generation of PMML 4.x for most Prediction Models; efficient scoring of PMML 4.x models generated by Statistica or open-source (Python, R) or commercial modeling platforms
• Numerous enhancements to the Data Entry application, including versioning of Labels and Characteristics in Enterprise Manager, further refinements to support double-blind validated data entry, a new permission/role of "Data Entry Auditor", and many other enhancements critical to GxP manufacturing
• Numerous additions to statistical analyses and graphs, such as confidence limits for probability plots, display of variance ratios used in 2D orthogonal fit, the new parallel coordinate plot in Data Miner Workspaces, predictor importance statistics for downstream documents from the MARSplines Node (operator), various enhancements to manufacturing and quality control statistics, etc.
• Significant performance and scalability improvements, in particular for large enterprise installations requiring version control and approval processes for model artifacts and data configurations

Updated Architecture for Version Control and GxP Compliance; 21 CFR Part 11

Statistica 14.0 implements a simpler and more efficient architecture, eliminating the separate SDMS (Statistica Document Management Server) component. Prior to the V14.0 release, this component (service) was responsible for the versioning of managed artifacts -- i.e., Data Configurations, Analysis Configurations, etc. -- in the Statistica Server (Enterprise) system. This functionality is primarily in use in so-called "validated implementations", where the analyses, analytic reporting, process monitoring dashboards, data-entry forms and rules, and other analytic artifacts must be "locked-down" and version, consistent with Standard Operating Procedures (SOP's) guiding the respective process. GxP processes are commonly required in the manufacture of pharmaceutical products and medical devices, for example.

In Version 14.0, all versioning of documents is now managed natively within the Statistica Enterprise application/database rather than with a separate SDMS application/service and documents saved on disk.  The revision and approval history of existing Enterprise objects that were versioned using SDMS will remain part of the Enterprise object's version history, along with new changes and approvals made after the upgrade to Version 14.0. Most important and as was the case with prior releases, Statistica V14 was designed to be used in validated GxP applications, and to support FDA 21 CFR Part 11 and equivalent international ICH guidelines, and the product was designed and manufactured consistent with the TIBCO ISO 9001 certificate and requirements. For more information on how to upgrade from previous versions of Statistica to Version 14.0 (e.g., how to migrate the governed artifacts to the new version of Statistica), see the TIBCO Support articles on How to run the Migration Utility to transfer SDMS content before the version 14 upgrade and TIBCO Data Science Author +Operations: Statistica server architecture of a typical Enterprise deployment.

For more detailed information about what's new in this release, please see TIBCO Documentation which includes release notes

TIBCO Statistica 13.6

Statistica 13.6 does not contain any changes to the Statistica application. 

The purpose of this release is to improve the Statistica/Spotfire integration. The new features are within the component named TIBCO® Data Science for TIBCO Spotfire® Analyst which is uploaded to a Spotfire Server. Spotfire Analysts use this component to create a TIBCO Statistica® Data Function in Spotfire.

TIBCO Statistica 13.5

Missing Data Processing in Spreadsheet Formulas (IMPORTANT)

Missing data (MD = missing data, empty cell) processing has changed in spreadsheet formulas. To revert the behavior used prior to 13.5, select Home ribbon -> Options -> Spreadsheets -> Use legacy MD comparisons in formulas. After changing this selection, quit and restart Statistica.

Why is this change being made? The goal was to simplify writing formulas.

Prior to 13.5:

 V1 = 1, V2 = MD  =iif(V1=V2, "match", "not match"), returned MD 

 V1 = MD, V2 = MD  =iif(V1=V2, "match", "not match"), returned MD

Now:

 V1 = 1, V2 = MD  =iif(V1=V2, "match", "not match"), returns "not match" 

 V1 = MD, V2 = MD  =iif(V1=V2, "match", "not match"), returns "match"

Spotfire Integration - More Parameterization Options

The Spotfire Analyst creating a Statistica data function can now parameterize the connected Statistica workspace. When the user registers a new Statistica Data Function by selecting a workspace, input parameters of value-type are created to expose node-level parameters. This is in addition to the input (table-type and value-type) and output (table-type) parameters created, based on the "input" nodes, "output" nodes, and workspace dictionary variables that are created. For example, the Spotfire dashboard could be connected to the Select Predictors node mentioned below. 

This gives the analyst greater control over the analytic options. It is possible to reduce the number of Statistica workspace parameters shown to the Spotfire Analyst. While editing a workspace within Statistica, select the Designer View button. You can select or unselect nodes and parameters within nodes. 

Spotfire Integration - Reporting Documents node

The Reporting Documents node within a Statistica workspace could not be used prior to 13.5. Now any spreadsheet-type documents (tables) generated and stored within Reporting Documents node will show up and can be used in the Spotfire Data Function.

Spotfire Integration - Share with Other Spotfire Analysts

... call Statistica Workspace (no code analytics) via a Spotfire Data Function

... embed Statistica Workspace within Spotfire .dxp file and share with other Spotfire Analysts

It is now optional to install Statistica on the same computer as Spotfire Analyst. The Spotfire Analyst only needs to have the Statistica Extension packages installed. 

Note: The Spotfire Dashboard can only be executed by Spotfire Analysts locally.  It is useful for Ad-hoc analysis, Exploring, Model Building, and Feature Selection. This functionality cannot be used by the Spotfire Consumer.

Variable Selection

When you work with large datasets and select long intricate lists of variables for analyses with different variable categories (example: dependent, categorical, continuous), if some variables in these lists are overlapping, it's hard to determine which variables are affected to make a correction. The Lists overlap error message has been improved. It states which variables overlap. And it provides the following three options:

• remove duplicates from the first list
• remove duplicates from the second list
• edit variable selections manually

This feature decreases the amount of time spent on variable selection within interactive modules and workspace nodes. 

Workspace

Alternative Least Squares Deployment node

Alternative Least Squares Deployment node has been updated. After selecting the Deploy to Enterprise button,  the user can now select a new None option instead of selecting a Data Configuration.

Customize Output for Workspace

The Customize Output feature is most commonly used to set the number of decimal places for a statistic, bold text, and format graphics. This feature is accessed by right-clicking on any node within a workspace.  You will see the menu to select and open the Customize Options dialog box.

A new option, Suppress output check box, was added. This allows the workspace designer to decide which output, per node, shows up in Reporting Documents node. All the graph outputs from all nodes or a specific node can be suppressed. All the spreadsheet (table) outputs can be suppressed for all nodes or a specific node. Individual graphs or spreadsheets can also be suppressed.

This feature provides granular control over what the Workspace's consumer sees. 

Data Health Check node

The use previous input description check box was added to the Data Health Check node. This was added to help develop a template. For example, this can be combined with the new Select Predictors node.

Elasticsearch Text Analytics node

On the Specifications -> Quick tab, select Files. Examine Browse For Folder dialog box. The selected directory is now displayed on this dialog box.

Filter and Process Data nodes

Filter Duplicate Cases, Filter Sparse Data, MD Imputation, Process Invariant Variables, Process MD and Rank nodes now support wildcards in variable selection. Wild card variable selections are important for building templates that can be reused. Other nodes already have this functionality.

For example, a variable can be selected:

• *CAT*
• 1*
• *A

General linear (GLM) node

The general linear (GLM) node is validated and released in 13.5. This was released as "beta" in prior releases. 

ITrees CHAID nodes

The Always split on minimum p option is added to ITrees CHAID Classification and Regression nodes.

K-Means Clustering node

The K-Means Clustering node is validated and released in 13.5. This was released as "beta" in prior releases. 

Lasso Regression node

The use previous input description check box was added to the Lasso Regression node. This was added to help develop a template. For example, this can be combined with the new Select Predictors node.

Normality tests node

The Normality tests node computes tests of normality (Kolmogorov-Smirnov test statistic, Kolmogorov-Smirnov p-value, Lilliefors p-value, Anderson-Darling test statistic, Anderson-Darling p-value, Shapiro-Wilks p-value) for each selected variable.  If two or more variables are selected, then you can compute the following tests of multivariate

normality:

• Mardia's test of multivariate skewness
• Mardia's test of multivariate kurtosis

Why test for normality? The normal distribution is a foundation for many algorithms. And verifying the normality of a data set can be critical to getting a reasonable result. This can be viewed as model selection for algorithms that have a hypothesis. For example:

PI nodes

Calendar control is added to the start time and end time fields for all the PI nodes. 

Reporting Tables node

You can now reorder the elements in the ∑ Placement group box of the Reporting Tables node. This change also applies to the interactive module for Reporting Tables. 

The node now generates a downstream dataset for further analysis. Prior to 13.5, this option was off by default.

Select Predictors node

Select Predictors node connects to a single data source. It can be very useful with Spotfire Integration. The user can select one target variable for predictive analytics problems. The node classifies and selects the remaining variables as continuous or categorical predictors. Then it passes the variable selection downstream. 

Connect the Select Predictors node to Advanced Classification Trees (C&RT), Advanced Regression Trees (C&RT), Advanced Classification CHAID, Advanced Regression CHAID, Boosted Trees, Data Health Check, K-Nearest Neighbors, Lasso Regression, Feature Selection, MARSplines, Random Forest, SANN Classification, SANN Clustering, SANN Regression, Support Vector Machines and SVB nodes that use dependent/predictor variable selection. 

TIBCO Statistica 13.4 

TIBCO Statistica 13.4 has some great new features which are all summarized and documented in detail in this section of the TIBCO product documentation site.

This upgrade also demonstrates how TIBCO Analytics products are working together. For one of the key new features of being able to trigger a Statistica data function from Spotfire and rendering it in Spotfire, we have created a video that you can view on the TIBCO Youtube channel: 

The new version has several "quick start" Workspace templates (accessible after opening a new Workspace document). We have posted 2 of these templates on the TIBCO Community Exchange for you to use with Statistica 13.3 if you have not yet upgraded to version 13.4:

• Data Preparation Quick Start Template for Spotfire Statistica® - This is the TIBCO Statistica® template guiding users through the process of data preparation steps. It is meant to be a quick start template allowing users to build their own data preparation process quicker. Users will go through the workflow, set, connect and use various nodes in a sequence in order to prepare and clean the data for further analyses.

• Classification Model Building Template for Spotfire Statistica® - This template features a typical analytical workflow for building predictive classification data mining models with TIBCO Statistica®. In this template, the user can simply change the input data source and run the whole modeling process on the new data with one click.

Some other important features that are added are: PI Event Frames, PI Asset Framework,  Dynamic Time Wrapping, Batch Synchronization, Elasticsearch for Text Analytics, and Publish PMML models to AMS (TIBCO Artifact Management Server), Lasso Regression Workspace Node, Improved Import and Export with Spotfire.

Statistica's workspace now has the ability to deploy a model into Streambase. This new feature in combination with Spotfire Statistica® Monitoring & Alerting Server (MAS) could be used to automatically refresh a model in production. MAS could monitor "prediction vs actual outcome" and if the variance was higher than X, trigger an alarm. And the alarm's job would be to rebuild models against current data, compare/select the best model and then publish the new model into production. 

Your upgrade should run smoothly but if you do run into any issues upgrading to 13.4 please submit a Spotfire support ticket. Or search for answers or post your question through the Forum section of the Spotfire Community.

For a lot more information on Spotfire Statistica, Spotfire Statistica® Enablement Hub or if you are just getting started with Spotfire Statistica and are in need of training videos and useful tips & tricks Getting Started with Spotfire Statistica®.