Externalization in StreamBase Using Substitution Variables, Container Parameters, Container Parameter Settings, and Module Parameters Together

Overview

Putting all your deployment environment-dependent values together in a substitutions file, and using those values to set module parameters by way of container parameters that are declared and set using containerParameterSettings is, as of TIBCO Streaming 10.5 and higher, the recommended way to do value externalization.

However, it's hard to find an example of using these things together from end to end.

So now there is one attached to this page (containerparamtest.zip).

As a bonus, this example also shows an easy way to specify absolute pathnames on Windows using substitution variables (hint: use forward slashes, not backslashes).

It's worth pointing out that this example uses module parameters and container parameters together, but there's no need to always use a module parameter to "chain" the container parameter value through, as this example shows. If there is only going to be one instance of a module at runtime, then that module can directly use a container parameter reference rather than a module parameter reference. If there is going to be more than one instance of a parameterized module, then using module parameters and the module parameter on each module instantiation with separate container parameters is worth considering.

There are three different kinds of parameters in StreamBase: module parameters, container parameters, and operator parameters. So, it's worth considering what kinds of parameters to use and why.

• Operator parameters: First off, operator parameters are deprecated as of TIBCO Streaming 10.5.0, so we probably should not be using them in new applications. Using them gives a typecheck warning, though they still work as 10.6.x, but the TIBCO Streaming product team advises using container parameters instead. There are still product samples, etc., that use operator parameters, so you may see them used for a while. But for now, good software engineering practice recommends we not use features that we know are subject to removal in future versions. Operator parameters are scoped at the level of the entire engine process and this is what makes them different than container parameters.
• Container parameters: As the name implies, the scope of a container parameter is a single StreamBase container. Container parameters are set using the containerParameterSettings property of the container's object in an EventFlowDeployment root object of the com.tibco.ep.streambase.configuration.sbengine configuration for that engine. They are set when the StreamBase container on which they are defined is instantiated, and their values can not be changed after the container is initialized. A reference to a container parameter can be in any StreamBase expression in any module that is instantiated in the container, but will cause a typecheck error if a container parameter reference doesn't find a value for that parameter. Therefore, some value has to be provided at typecheck time, whether or not that's the value that will be provided at container instantiation time.
• Module parameters: Module parameters are scoped to the module in which they are declared, and are declared on the Parameters tab of the module itself. A default value must be provided, though that default value can have a reference to a container parameter, if that is useful.

   

  If a module parameter has the same name as a container parameter, then setting the container parameter will override (set) the module parameter's value.

Example Walk-through

This example uses Windows file pathnames only. Linux/MacOS pathnames are a bit easier, anyway.Import the zip into StreamBase Studio (File > Import. . .  > General > Existing Projects into Workspace > Next > Archive File)

Before running this example, let's take a walk through it to see how each of the parameterization mechanisms are being used and relate to each other.

This application has a single EventFlow fragment that consists of a single EventFlow module named com.tibco.example.containerparamtest.ContainerParamTest. This module has a CSV File Reader adapter instances named CSVReader. We want to parameterize and then externalize the value of the adapter instance's File Name property so that the input CSV can be in different locations in different runtime and deployment environments without having to change any code in the application.

Here is a look at the relevant part of the ContainerParameterTest module:

Here the File Name property is parameterized by a parameter name FILE_PARAM. Note the parameter reference syntax is that we enclose the name of the parameter in ${} characters. This parameter reference syntax is the same for all three parameter types in StreamBase and EventFlow. We can't tell just by looking at the parameter reference what type of parameter this is. We have to examine the parameter's declaration elsewhere to figure that out.

Here is the Parameters tab for the ContainerParamTest module:

All the parameters declared on the Parameters tab of a module are, by definition, module parameters. So, this module declares two module parameters named THRESHOLD_PARAM and FILE_PARAM, and sets their default values. Notice that parameters have no declared data type, and that it isn't necessary to enclose values that are going to be of data string in quotes as we would ordinarily do with string literals in StreamBase expressions. The Default Values can be arbitrary StreamBase expressions, though -- but they are not evaluated in isolation as they are written in the Default Values expression -- they are evaluated in the context of each expressions that references the parameter at typecheck time.

Since we are focusing of FILE_PARAM throughout this example, let's note that the default value is s.csv, the name of a CSV file. This name is a relative path -- relative to where is something specific to the CSV File Reader's implementation and not of great concern for this example. (But just to be clear, we have set the Read As Resource property on this adapter instance, which means that the adapter will search for the file s.csv in any folder of the engine's resource path, in the order those folders occur on the resource path, and use the first s.csv file it finds.)

There is an s.csv file in the src/main/resources folder of the fragment project, so this a reference to the FILE_PARAM module parameter will always resolve successfully when the module is typechecked. But maybe that s.csv is just a placeholder so that typecheck succeeds -- at runtime in various deployment environments, we want the adapter to read a file that we specify for that deployment environment.

So, to do that, we are going to override the Default Value of the module parameter by using a container parameter with the same name of FILE_PARAM. As mentioned above, this is done in the EventFlowDeployment configuration for the fragment, in this case held in a configuration file named efdeploy.conf:

Here we have set the value of the FILE_PARAM container parameter on the default container for this fragment. Notice that we had to specify the top-level module of the default container even though there is only one .sbapp file in the fragment project.

The value of the FILE_PARAM container parameter is the expression "${S_FILE_PARAM:-s2.csv}". This particular expression is reference to a substitution variable. Substitution variables are the primary way we externalize values for Streaming applications, and it is quite common to use the values of substitution variable to set parameters, as we are doing here. Substitution variables and parameters are not the same things, and they exist in idependent name spaces. The scope of a substitution variable is a node. A substitution variable whose value is referenced in an sbengine configuration has to be set a node install time in order to have that value affect the engine at runtime. (There are other times substitution variables can be set, such as when loading a configuration, but that is only effective for configuration types that can be overridden after the node has been started, which is not true for an engine configuration.)

A substitution variable reference uses the same ${} syntax as a parameter reference, but may also contain a default value for the substitution variable -- anything after the :- (colon-hyphen) character sequence after the variable's name becomes the value of the reference expression if the variable isn't otherwise set. Here, they default value is s2.csv. A file named s2.csv is also provided in the fragment project, should we ever need to reliably resolve that name at typecheck time in all typecheck contexts.

Then, as shown in the command sequence below, we provide the runtime values for substitution variables using either the --substitutions or --substitutionfile options to the install node command.

Substitution files are very similar to, say, Java properties files -- one line per substitution variable, and the syntax is name=value.

Running the example

Copy app_csv/s3.csv to c:\tmp

From the Studio Project Explorer, right click and select StreamBase > Open StreamBase Command Prompt Here

From the StreamBase Command Prompt, imitate the following, making an changes necessary due to your machine environment:

TIBCO Streaming 10.6.1 2008190131

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\ContainerParamTest>cd ..\app_csv

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin install node --nodename=A.CSV --application=target\app_csv-0.0.1-SNAPSHOT-ep-application.zip --nodedirectory=C:\tmp\nodes
[A.CSV]         Installing node
[A.CSV]                 PRODUCTION executables
[A.CSV]                 File shared memory
[A.CSV]                 7 concurrent allocation segments
[A.CSV]                 Host name SBARBER-T480
[A.CSV]                 Starting node services
[A.CSV]                 Loading node configuration
[A.CSV]                 Auditing node security
[A.CSV]                 Deploying application
[A.CSV]                         Engine default-engine-for-com.tibco.example.ContainerParamTest
[A.CSV]                 Application deployed
[A.CSV]                 Administration port is 57010
[A.CSV]                 Service name is A.CSV
[A.CSV]         Node installed

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin install node --nodename=A.CSV --application=target\app_csv-0.0.1-SNAPSHOT-ep-application.zip --nodedirectory=C:\tmp\nodes --substitutions="S_FILE_PARAM=C:/tmp/s3.csv"
[A.CSV]         Installing node
[A.CSV]                 PRODUCTION executables
[A.CSV]                 File shared memory
[A.CSV]                 7 concurrent allocation segments
[A.CSV]                 Host name SBARBER-T480
[A.CSV]                 Starting node services
[A.CSV]                 Loading node configuration
[A.CSV]                 Auditing node security
[A.CSV]                 Deploying application
[A.CSV]                         Engine default-engine-for-com.tibco.example.ContainerParamTest
[A.CSV]                 Application deployed
[A.CSV]                 Administration port is 43572
[A.CSV]                 Service name is A.CSV
[A.CSV]         Node installed

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV start node
[A.CSV]         Starting node
[A.CSV]                 Engine application::default-engine-for-com.tibco.example.ContainerParamTest started
[A.CSV]                 Loading node configuration
[A.CSV]                 Auditing node security
[A.CSV]                 Host name SBARBER-T480
[A.CSV]                 Administration port is 43572
[A.CSV]                 Service name is A.CSV
[A.CSV]         Node started

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>copy con: substitutions.prop
S_FILE_PARAM=C:/tmp/s3.csv
^Z
        1 file(s) copied.

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV stop node
[A.CSV]         Stopping node
[A.CSV]                 Engine application::default-engine-for-com.tibco.example.ContainerParamTest stopped
[A.CSV]         Node stopped

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV remove node
[A.CSV]         Removing node
[A.CSV]                 Shutting down node services
[A.CSV]                 Engine System::administration stopped
[A.CSV]         Node removed

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin install node --nodename=A.CSV --application=target\app_csv-0.0.1-SNAPSHOT-ep-application.zip --nodedirectory=C:\tmp\nodes --substitutionfile=substitutions.prop
[A.CSV]         Installing node
[A.CSV]                 PRODUCTION executables
[A.CSV]                 File shared memory
[A.CSV]                 7 concurrent allocation segments
[A.CSV]                 Host name SBARBER-T480
[A.CSV]                 Starting node services
[A.CSV]                 Loading node configuration
[A.CSV]                 Auditing node security
[A.CSV]                 Deploying application
[A.CSV]                         Engine default-engine-for-com.tibco.example.ContainerParamTest
[A.CSV]                 Application deployed
[A.CSV]                 Administration port is 11716
[A.CSV]                 Service name is A.CSV
[A.CSV]         Node installed

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV start node
[A.CSV]         Starting node
[A.CSV]                 Engine application::default-engine-for-com.tibco.example.ContainerParamTest started
[A.CSV]                 Loading node configuration
[A.CSV]                 Auditing node security
[A.CSV]                 Host name SBARBER-T480
[A.CSV]                 Administration port is 11716
[A.CSV]                 Service name is A.CSV
[A.CSV]         Node started

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV stop node
[A.CSV]         Stopping node
[A.CSV]                 Engine application::default-engine-for-com.tibco.example.ContainerParamTest stopped
[A.CSV]         Node stopped

C:\Users\sbarber\Documents\StreamBase Studio 10.6.1 ContainerParam\app_csv>epadmin --servicename=A.CSV remove node
[A.CSV]         Removing node
[A.CSV]                 Shutting down node services
[A.CSV]                 Engine System::administration stopped
[A.CSV]         Node removed

After starting the node, have a look at the engine log file to validate the the s3.csv file is the one that was read, rather than the module parameter's default s.csv file or the substitution variable's default s2.csv:

2021-05-18 21:04:04.371000-0400 [31940:AsyncLogger] INFO  LogCSV: 3hello
2021-05-18 21:04:04.372000-0400 [31940:AsyncLogger] INFO  LogCSV: 3goodbye

See https://support.tibco.com/s/article/How-to-set-StreamBase-Properties-using-Parameters-in-TIBCO-Streaming for more explanation about the details of how module and container parameter references are evaluated in EventFlow component properties and expressions.

containerparamtest.zip