Limitations - Custom Connector for TIBCO Spotfire® to Connect to Azure SQL Server DB -

[Srinivas Guddanti]

Hi Team,

Recently we have deployed the Custom Connector (custom_azure_sql_connector_v8) for TIBCO Spotfire® to Connect to Azure SQL Server DB. we have successfully made the connection through Azure AD token authentication and trying to load the data in different modes (Import vs External Data Source).

Import Mode (In-memory) :

  --> I am trying to load 700k records data from Azure SQL Server. Seems it is very slow and taking more time (aprx 1hr 30 mins).

  

is there any limitations in-terms of data loading with this custom connector? Also, we tried to tune the custom connection (for example to add the MultipleActiveResultSets, packesize to the connection), but it is failing to connect. Also, there is no advanced options to make the connection to allow some performance metrics to add.

Please let us know if there are any such limitations or any steps to overcome to load the large data sets in this scenario?

External Data Source Option:

  --> Table visualization is loading only 10000 rows.

  --> Other visualizations are failing, connection timed out.

Any limitations this data loading mode? Any steps to resolve this issue?

Let me know if we can connect and discuss for better understanding?

Regards,

Srinivas G

[Srinivas Guddanti]

Hi Dave,

Thanks for your response.

Please be informed that, this is not related to Azure Databricks custom connector. We are working on the Azure SQL connector (custom_azure_sql_connector_v8) which we are trying to be connecting the Azure SQL Server DB.

Please find the link below connector for - Azure SQL and Azure Synapse

Custom Connectors for TIBCO Spotfire®

Please share me your thoughts on this.

Regards,

Srinivas G

[Dave Leigh]

Hi Srinivas

You are correct, I deleted my previous response so as not to mislead others and apologize for getting my wires crossed.

Unfortunately there are no additional options for the Azure SQL Custom Connector as it is based on the Spotfire SQL Server Connector which only allows Timeout and Connection Timeout settings. We could look at adding the additional options similar to the way the Databricks Custom Connector works, but this would take some time.

In the meantime, I would recommend trying a simple command line program to verify whether the download bottleneck is inside or outside of Spotfire. If you reach out to me directly, I can share with you the code for a .NET test program we have used in the past to isolate these kind of issues.

If it turns out that adding some of the additional connection parameters does indeed help then we can prioritize adding these in to the Custom Connector.

Cheers

Dave