"Big Data" at production site

[Martin Baumgartner]

Hi all together,

in our factory we handle test data from our products delivery from several test equipments in our production line as well as data from the equipments itself like messages and status.

During the last years we connect a lot of production line to our MS SQL server and more will come in the future. Our most used dashboards are analysis of the test results and visualisations of the error and messages of the production lines. Why we use Spotfire for that, well it is the only BI-tool which can handle the amount of data we are producing. But for some reasons we have problems with the performance of the data loading.

The test results were loaded on demand, because we don't know what the consumer will be viewing. Will he only load that for product A or B or C to H? We don't know the range of time in advance? 
I did a row count and came to the result of 10 Mio. rows for one product in one month, and unfortunately we aren't producing at normal level.

The other visualisation needed some preparation and calculations, which we did within Spotfire before. Due to the fact that in the past it was ok only to get the data from yesterday to the last few weeks we could use scheduled updates to get in quickly loaded. But now the consumer also wants to see today's data. Well, we switched the data preparation for the past data to Knime and only get the today's data directly loaded from the SQL server. Which is loading in an acceptable time. ;-)

What I'm interested today is, which data and the amount are handling? Which data sources are you using? Our It will sell us the ADX for some data, because SQL server isn't good for us.

Thanks in advance and I'm curious ....

Regards

Martin

[Alain Martens]

Hi Martin,

Thanks for sharing insights into your factory's data handling processes. I'm part of the Spotfire team and work with a number of High Tech Manufacturing clients that often deal with large datasets as well.

I completely agree that performance issues can be a major hurdle when it comes to deriving insights from your data, especially when dealing with on-demand loading and real-time updates. It's great to hear that you've found Spotfire to be the most suitable BI tool for handling your data volume, but I understand the frustration that comes with encountering performance bottlenecks.

Improving dashboard performance is a big topic, and I'd love to connect with you to explore different strategies and best practices that could help optimize your dashboards. 

Having worked with various High Tech Manufacturers, I've encountered similar scenarios and have gathered insights into improving performance and efficiency.

From what you've described, it seems like you've already tried some options like on-demand loading, scheduled updates and preprocessing data using Knime which all make a lot of sense. In my experience, optimizing dashboard performance often involves a combination of evaluating which data needs to be loaded in memory, assess data loading strategies (leveraging a combination loading data externally, on-demand and in memory), choosing the right database, caching strategies, and hardware considerations. 

It would be helpful to dive deeper into your specific setup and requirements to tailor recommendations accordingly. Would you be available for a call to discuss this further? I'd be happy to share any insights that might help you. 

Thanks,
Alain