Server Communicaton Error / Java Heap Space

[Mehmed Aktas]

Hi,

I have migrated a project from 7.7 to 10.6 which was perfectly working in 7.7 debug mode without any memory leak.

However, when I tested it in 10.6 debug mode, it suddenly stops after an hour on average with the error I attached.

I inspected the log files and saw some java heap space errors. I tried to capture the leak with JVM, SB Manager and other Java monitoring tools, however, I don't see any leak or GC failure.

Since it was working perfectly in 7.7 I don't think there is a memory leak but the log files confused me. What would be the possible cause of the problem I attached some log files from EventFlow and LiveView apps (we run them simultaneously).

[Steve Barber]

As a quick response, what I'm seeing in what might be the StreamBase Studio (Eclipse) error log file is:

!ENTRY com.tibco.ep.sb.studio.liveview 4 0 2021-02-18 16:25:58.483

!MESSAGE Thread "Executing deploy com.example.myex.apps_main.MyEx" terminated due to uncaught exception; see thread tracebacks in "C:UsersMyUserNameAppDataRoamingStreamBaseLiveView Desktop Logs"

!STACK 0

java.lang.OutOfMemoryError: Java heap space

at java.base/java.util.Arrays.copyOf(Arrays.java:3745)

at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:172)

at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:538)

at java.base/java.lang.StringBuilder.append(StringBuilder.java:174)

at com.tibco.uep.studio.dtm.commands.DtmOutputTextGatherer.append(DtmOutputTextGatherer.java:128)

at com.tibco.uep.studio.dtm.commands.DtmOutputTextGatherer.appendLine(DtmOutputTextGatherer.java:124)

at com.tibco.uep.studio.dtm.commands.DtmOutputTextGatherer.error(DtmOutputTextGatherer.java:81)

at com.tibco.uep.studio.dtm.commands.ProgressBroadcaster.error(ProgressBroadcaster.java:60)

at com.tibco.ep.dtm.management.DtmExecute.reportFailure(DtmExecute.java:75)

at com.tibco.ep.dtm.management.DtmExecuteDeploy.run(DtmExecuteDeploy.java:59)Because it is showing up in the LiveViewDesktop log, unless you are actually running a standalone LiveView Desktop process and I'm guessing incorrectly that this is the Studio Error Log, then I'm wondering if the OutOfMemory error might be due to the use of the Studio LiveView Tables and LiveView Query views. These views are quite new in Studio, and didn't exist in the StreamBase 7.7/Live Datamart 2.1 line of releases. Did you maybe set the row limit very high or are you using LiveView queries that return a lot of results Or do you have many active queries Those things could potentially cause heap exhaustion in Studio. The solution to those conditions would be to either lower the row limit or make it so your queries are less result-intensive (consider using BETWEEN clauses rather than open-ended SELECT * FROM table queries -- the latter's result set will grow without bound if something continues to publish new inserted rows into the assocated table).

Another possibility is the DtmOutputTextGatherer is implicated in this stack trace. I previously reported an issue to the Streaming product team where Studio runs out of memory in 10.6 when left running for a long time. In environments where there are quite a few Streaming clusters whose cluster traffic is visible in the same subnet (perhaps 20 or 30). With that many active cluster, I see memory exhaustion in about 24 hours, but I speculate that the time to memory exhaustion will vary with the number of clusters that whose traffic is being processed. My guess is that this has to do with the way cluster traffic is being monitored by the current Studio Clusters View implementation. This issue first appeared in 10.6 and to my knowledge hasn't been resolved.

Of course, it may be something else entirely. I personally don't have the time or knowledge to analyze your GC logs in detail, but if you want someone to do that, I recommend submitting a Support Case to TIBCO Product Support, assuming you have an active support agreement.

I emphasize that we are talking about memory exhaustion in Streaming clients such as Studio and I don't see any evidence in what you've attached to support the idea the the server side (engines) are having any memory issues, which is good.

[Steve Barber]

OK, I've determined that the Error Log here is in-process with Studio, and that the LiveVIew Desktop folder is indeed used by the LiveView Perspective (LiveView Query and Table Views). The OutOfMemoryError in DtmOutputTextGatherer might indicate a lot of text output from the launched engine(s) or it might just be that's where the reported OOME happened, and other things exhausted the heap.

I'd think to better determine why memory is running, one could capture heap allocation information by adding JVM options to C:TIBCOsb-cep10.6libstudiosbstudio.ini, which is just a eclipse.ini file.

 

If you're a Java person comfortable with JVM debugging, you'll understand how to do what I just said. If not, please contact Product Support for more detailed guidance. In either case, you'll probably want to send the heap data to Support for analysis.

[Mehmed Aktas]

Hi steve, thanks for the response. As you said it would be better to transfer the case into the support. Could you please remove the directory path from the portion that you shared from my logs for privacy Thanks in advance!

[Steve Barber]

OK, I edited the file excerpt so that it has nothing specific to your environment.

I noticed that the file attachment with the logs is gone from this Answer page, too, which is fine, and I assume you did that.

Please be gently ever so gently reminded that absolutely everything you post on TIBCO Community is generally available to anyone who wants to look at it (except for messages sent to a specified Community user).

[Steve Barber]

Hmm, I had submitted another comment here that never got posted. I'll try again.

First, I want to gently challenge the idea that "ince it was working perfectly in 7.7 I don't think there is a memory leak." StreamBase 7.7 and StreamBase 10.6 have many, many, many differences. Assuming that a version migration wouldn't be the cause of vastly different behavior of Studio or the Streaming runtime is probably not the starting point for analysis that I would choose. I'd probably start by assuming nothing and just reading the evidence in front of me.

Second, I confirmed that the LiveView Desktop log folder indeed comes from the new LiveView Perspective views (Query and Table views).

Third, the DtmOutputTextGatherer itself as the name implies outputs text gathered from Studio-based launches of a node. It might be worth checking how much output your node (especially the engine(s) are generating, but in general theres's no reason to suspect that the method that throws an OOME is at all associated with the over-consumption of memory: it just might be that's when the heap was exhausted. It is indeed StreamBase Studio that is running out of memory, and not the launched node itself.

So, the way to try to locate what is consuming heap memory is gather and save heap usage statistics for StreamBase Studio. If you are a Java person, you already have some idea how to do this, probably, and if you are not, it's a discipline we're probably not going to teach here on TIBCO Community. For example, one could make use jvisualvm to make a heap dump and then have a look yourself to see if there's anything obvious that looks related to your application that you can take care of, and if not, send that information along to Product Support for further analysis.

The bit of knowledge that might be hard to find is that StreamBase Studio is launched using the arguments specified in your %TIBCO_HOME%sb-cep10.6libstudiosbstudio.ini file, and thus any specific JVM args for your favorite heap statistics gatherer can be specified there.