Webscrapping on StreamBase

[Yasemin Alpay]

Hi,

I am using TIBCO StreamBase application.

I want to work on a project where I will need the live currency price data from a website instantly.

Is it possible to do webscrapping on StreamBase?

Thank you in advanced.

[Glenn Hoskins]

Hi Yasemin, yes, theoretically that should be possible. You will need to use the HTTP Client operator to read the contents of the web page. If the page requires authentication then that's going to make things more complicated, but there are options for passing auth parameters through in the request. This will then need to be setup to poll on a sensible interval.

The hard part will be parsing the contents of the page, and there won't be any out of the box way to do that. Often pages aren't even well-formed XHTML/XML so parsing using that operator might not be possible. So it's likely going to need a fair bit of custom processing to get the information you need.

[Eddie Galvez]

Try to identify if that live currency price data is available from a proper data feed of sorts - for example, https://fixer.io/ (note: we do not test, support or endorse that particular example), and using either a built-in adapter or a custom adapter, you would have a better more resilient application.