Using the Field Serializer for low latency transfer across a message bus

Typically the Field Serializer operator is used for simplifying a tuple's schema by storing a large group of unexamined fields into an opaque field to lessen latency within a single StreamBase EventFlow application. It does this by copying the binary representation of the tuple into a blob field. The operator can then unpack the blob at the end of the flow because it can overlay the same schema onto the blob, revealing the full tuple structure again.

When sending a tuple via a message bus such as TIBCO Rendezvous or EMS typically the tuple is converted to a native representation, JSON or XML, translating each individual field value to a text representation. This adds processing time and latency to the communication. For complex tuples with nested lists or other deep hierarchy this can add two milliseconds or more of additional latency.

Instead, by using the Field Serializer the tuple is simplified to one binary field. This has proven to be the fastest way to send a complex tuple across a message bus because the message representation is as simple as possible and no type-specific conversion is applied.

In this alternative use-case, to send a tuple over a message bus between two separate StreamBase Server instances, care must be used to make sure the same schema is available to the recipient server. This is done by making certain:

• The sbd versions sending and receiving are the same (specifically the sbclient.jar and sbserver.jar must be identical).
• The upstream Field Serializer must use a Named Schema which fully converts the input tuple to a single field (the output tuple must contain no unconverted fields).
• The downstream Field Serializer must use the exact same Named Schema to convert that field to an expanded tuple.

To make sure the Named Schema is identical in both instances, the applications should obtain this schema by importing it from the same application (sbapp) or interface (sbint) file artifact.

If any of the above criterion are not met, then the received blob is expected to be uninterpretable due to mismatches between it and the schema.