How to import data from multiple csv files into a Statistica Workspace?

Description

This page is a tutorial on how to use the "Folder CSV Export" node for merging multiple csv files. To access this functionality, you can download the node from our TIBCO Community Exchange: Folder CSV Import Node for Spotfire® Data Science - Workbench. 

Details

Sometimes data with the same format is stored in multiple csv files in a folder. It would be ideal to import all these files at once into a Statistica spreadsheet instead of importing these individually and appending the data. Consider data that looks like this that needs to be used for analysis:

The above-illustrated example files are zipped together with the Folder CSV Import Node for Spotfire® Data Science - Workbench. 

There are multiple ways to import these files with Statistica workspaces. One of the ways is to use multiple Import Text nodes and a Query Spreadsheets node or a Merge node(s) to append the data into one file and then proceed with analyses.

However, this is less than ideal when the number of files is variant or is a number greater than 5 files. This article and Folder CSV Import node provides an alternate way that will make the above workflow defined obsolete. A single node can import and merge files from a folder as long as they have the same data format.

Remark: more info about merging functionalities can be found in this article.

Resolution

STEP 1: Importing the node.

Download the attached zip folder and extract the files to a folder. Import the node as detailed in: https://support.tibco.com/s/article/How-to-import-a-custom-node-into-Statistica-Workspace-s-Node-Browser

You may alternatively copy the node from the existing Workspace (.sdm) file.

STEP 2: Using the node.

For the purpose of this article, lets use the example workspace with the file structure that is detailed above. Open the workspace with the node or double-click on the node in the node browser to import the node into the workspace.

Click on the gear icon of the node to fill in the parameters. Select the folder where the csv files to be imported are located. Note that you may select the parent folder here if you are enabling "Open Subfolders of Parent to check for the presence of csv files"

File Name Format - Enter the name of your file name format that Statistica should parse. In this case, we are mentioning *.csv but let's assume the folder had other csv files like pHMeasure.csv and we only wanted files that started with CellCount the format can be specified as "CellCount*.csv".

Format String: This is the format of the variables within the file. Note that for the node to import with no errors, all files in the folder need to have the same format. R represents Real numbers, D represents Date and T? represents Text. These can be manually entered or you may record a master macro for opening one of the files as shown below and copy the .FormatString from the macro :

Reference: https://support.tibco.com/s/article/How-to-record-Statistica-master-macro

NOTE: Although the above gif shows a more complex format string for the purpose of illustrating how the format string can be recorded and copied, for the example workspace and file this is simply 4 real numbers.

Log file for Import: Select the folder where you would like to create a log of the import. After Statistica imports the files a log named "FImpCsv.log" is created in the folder selected.

Enable Get Variables from the first row of the file if you would like to get the variable names from the first row of the file. If the Variable names are not consistent, Statistica will use the variable names from the first file it imports from the folder.

Reference links

• Knowledge base article: How to import node into Statistica
• Knowledge base article: How to record a macro of user's steps
• Merging Tables in Statistica
• Statistica Workspace - a graphical UI
• Spotfire Statistica Enablement Hub