Data stream connection - what message format

[Thomas Burke 2]

Hi,

I'm starting to look at using TIBCO data streams as a visual managment as part of our manufacturing processing. Before I start I've got a couple of questions:

1) Can we pull the live data from a file

2) What message formats it expects or how we send live data to it

Any help would be greatly appciated.

Thanks.

Tom

[Steve Barber]

Hi Thomas,

 

Well, TIBCO has two products roughly called Data Streams:

- TIBCO Cloud Data Streams (TCDS)

- TIBCO Spotfire Data Streams (TSDS)

TIBCO Cloud Data Streams is a managed service that runs in the TIBCO Connected Intelligence Cloud (CIC). TIBCO Spotfire Data Streams is an add-on to TIBCO Spotfire and can be installed on-premise, or in a public, private, or hybrid cloud of your choosing.

 

For both of these Data Streams products, the visualization client that people usually use it TIBCO Spotfire, whether that's a locally installed version of Spotfire (Analyst, Desktop, etc.), the Spotfire Web Player, or TIBCO Cloud Spotfire (available on CIC).

 

With that as background, to your questions:

 

> 1) Can we pull the live data from a file

 

I'm not sure what you mean exactly by "live" because reading data from a file is, well, reading data from a file whether it is considered live data or not. So, I guess the answer is yes, surely.

 

The reason I'm dickering here is that reading data from a file isn't really stream processing; it's batch processing. You may or may not want to use a Data Streams product to do that. The specifics of what you are doing and why will determine whether it's a good idea or not, as well as where you are in the journey from batch processing to stream processing and where you want to end up. (The range from batch processing to stream processing is more of a continuum rather than a dichotomy -- people often start out "batchy" and move towards "streamy," depending on a bunch of things.)

> 2) What message formats it expects

 

TIBCO Cloud Data Streams will read from CSV files (and simulate a stream by creating a single event out of each line in the file). TIBCO Cloud Data Streams will stream events from Apache Kafka, MQTT, or TIBCO Cloud Messaging (eFTL) -- those events must be in either JSON or Avro formats, I think.

 

TIBCO Spotfire Data Streams is much more flexible, though you may have to do a little bit of integration programming to get everything flowing. TSDS can read from anywhere that TCDS can, plus there are about 80 different input adapters TIBCO provides. A list of the adapters that TSDS supports is here: https://docs.tibco.com/pub/str/latest/doc/html/welcome/adapter-restrictions.html.

 

In terms of what data formats can be read and converted so that the resulting events can be placed in what we call a LiveView data table -- there are a number of built-in converter operators and functions-- JSON, XML, Avro, CSV come to mind -- and also a good number of string parsing functions for more generalized conversions. And some of the adapters on the list support their own formats, as well.

 

> or how we send live data to it

 

For TCDS, you provide connectivity information (or the CSV file), and run a Connectivity Wizard that automatically detects the format, makes the connection, and creates events from the input data.

 

For TSDS, there are a few ways:

 

1) There is a connectivity wizard that will generate the right thing for Kafka, EMS, FTL, and CSV files, and then perhaps some parsing logic will have to be specified.

 

2) Use one of the 80 or so input adapters and create an embedded publisher application to connect to your data source and convert the message payload data into a LiveView table row.

 

3) Use the LiveView Client API from Java, REST, .NET, or JavaScript to publish your data to the LiveView table. So you have a lot of flexibility there to do data conversion.

 

I hope this answer gives you an overview, even if the answer is perhaps not as specific as you may have been expecting.

[Thomas Burke 2]

Thank you for the quick feedback - it has been really helpful.

 

One comment - on point 3 you mensioned an API for Spotfire which we think would be the easiest way to get the data 'live' if we configure our end. 

 

How do we acess the API And what is the licening agreement (is it free) 

 

Thanks.