How to create custom SpotFire connector, for external data-sources

[D Devienne]

I work mostly in Data-Management, with scientific and geometric data from the Oil and Gas industry, developping RDBMS-based data-models and application code accessing these models, either directly via a C++ SDK, or indirectly via Corba (legacy case) and WebSocket client-server custom protocols.

Some of our clients are SpotFire users, and consultants outside the development organization have succesfully experimented directly accessing an Oracle-instantiated schema, via the existing Oracle connector, thus accessing one of our existing data-models. But accessing the raw schema is error prone and difficult for most users, and does not work with Corba and WebSocket servers.

Additionally, some of the data of interest to SpotFire users are large numerical arrays, which are stored on disk as files, or in database (as blobs) in HDF5 format, so not readily or easily accessible from SQL. So in short, we'd like to expose both structure data in tables/rows, but also bulk data, mostly numeric, stored in entity-specific blob columns.

Some projects/studies have thousands and sometimes even millions of rows (across many tables). And some can have megas (MB) and gigas (GB) of those numerical opaque-to-SQL data, in two diametrically opposed use-cases, i.e. many (millions) of small arrays, (e.g. 800,000x 1KB to 32KB arrays), and/or a few multi-MB to multi-GB large arrays (typically 3D cubes of floating points quantities, multiplied by the multiple attributes/properties at these cells locations, with some of those properties being time-dependent, i.e. the evolution of the same property across time).

I'm therefore considering what it would take to develop a custom connection, and I have a lotof questions regarding SpotFire, including how to declare the structure of the data exposed by the data-source(s), what language the connector must be developped in, whether it's request-response or async, etc...

Beside technical questions like above, I also wonder at the cost involved, if any, since to develop a connector one I guess must also have one or more licenses of SpotFire to test, and possibly also some kind of SDK, which may be additonal cost too possibly. As an example, the Oracle server is quite expensive to run in production, but OTOH one can use an Oracle server and its SDKs "for-free", as long as this is strictly for development and testing purposes, allowing ISVs to economically use Oracle tech. Does the same exist for SpotFire

I did findhttps://community.spotfire.com/wiki/extending-tibco-spotfirewhich seems to have a lot of good information, but a quick glance does not yield a lot of "guidance" specifically for the use-case I've tried to outline above, so any such guidance would be greatly appreciated.

Thanks, --DD

[Neeraj Khatri]

Hi DD,

Please refer to the following link to get started with creating a custom datasource in Spotfire:

https://community.spotfire.com/wiki/create-custom-data-source-tibco-spotfire

You would need to download the Spotfire Developer software which includes the SDK. The link for this along with the detailed documentation is here:

https://docs.tibco.com/pub/sfire_dev/area/doc/html/GUID-9E16BAAA-7891-456E-A146-782D310DE755.html

The developer package contains many custom-extension examples include one for creating a custom data source. All the class files are implemented, you would just need to extend / add in your functionality to the source code.

Regarding your questions:

- What language the connector must be developped in:

The connector should be developed in C# .Net.

- How to declare the structure of the data exposed by the data-source(s)

Any C# data structure like datatables, datasets, datareaders could potential be used for this.

- Whether it's request-response or async

Generally speaking it should be request-response as the data is first retrieved and brought into Spotfire.

- License

Please search for Spotfire Server onhttps://edelivery.tibco.com/. The spotfire developer package will be named as follows -TIB_sfire_dev__win.zip. If you dont have access, please contact your AE.

Thanks,

Neeraj