Alert Monitoring with 2 App servers

[Praveen Kumar Ramamoorthy]

Hi Team, 
Per the documentation, we have configured alert monitoring in 1 of the 2 application servers. But the jobs are running fine in 1 out of 5 executions (probably when run through the server where it is configured). If it is executing through the other app server then it fails. 
We have turned off one of the app servers and scheduled the alerting job, which was executed successfully all the time. 

Please provide your inputs on the same.

Below is the error message:

Task 2 'Apply Rules' failed with: Spotfire.AlertingFramework.DataManager.DataManagerConfigurationException: Unable to contact Alerting Server at https://spotfirephodev.wdc.com/alerting at Spotfire.AlertingFramework.DataManager.DataManager.checkConnectivity() at Spotfire.AlertingFramework.DataManager.DataManager.GetBaseSubscriptions(List`1 sids) at Spotfire.AlertingFramework.Tasks.RulesTask.ExecuteCore(TaskExecutionContext context)

[Olivier Keugue Tadaa]

Hi Praveen

There is the below statement in the Alerting documentation.

Please let me know what is your desired deployment requirement  (HA, load balancer, etc ) and we can help you with that.

 

[Praveen Kumar Ramamoorthy]

Hi @Olivier Keugue Tadaa, 
Thanks for your reply.
Yes, we have deployed it to one of the Spotfire nodes and it's based on the load balancer 

[Olivier Keugue Tadaa]

Sorry, I missed the information.
Since Spotfire is under a load balancer, you should use that specific server name  (not the load balancer name) for the alerting URL. Indeed, all traffic redirected by the load balancer to the wrong server will fail. If for some convenience reasons, you want to keep the load balancer, note that some load balancers allow the health check or other rules that can be used to  direct the traffic to a good app server

[Praveen Kumar Ramamoorthy]

@Olivier Keugue Tadaa I tried with the server name but its throwing ""/ui/error/error403

[David Boot-Olazabal]

Hi Praveen,

To be clear, when you have one application server, it works fine. But in a clustered setup, it fails whenever the 'wrong' application server is being used to run the schedule.

What does the alerting.log mentions about the error403?

Can you also share a screenshot of the alerting.properties and the setup in the Analyst?

Kind regards,

David

[Praveen Kumar Ramamoorthy]

Hi David, 
The server URL with one of the app server host names is not working and throws a 403 error. 

http://xxx/spotfire/ I am trying this to validate, but it throws "http://xxx/spotfire/ui/error/error403". 

Can we replace the load balancer URL with this in the alerting properties and try again? do you think it may work?

Edited September 5 by Dave Leigh

[David Boot-Olazabal]

Hi Praveen,

The alerting properties should hold the url to the spotfire server, in your case that would probably be (or https if you use that):
alerting.spotfire.url=http://pbt-md-sfssxd01.ad.shared/spotfire

In the Analyst, you have to refer to the alerting server url (yours should be http://pbt-md-sfssxd01.ad.shared/alerting):

This may help you out, but if not, please share your alerting.log file, your alerting.properties as well as a screenshot of the Preferences.

Kind regards,

David

[Praveen Kumar Ramamoorthy]

Hi David, 

I have updated the alerting.properties and the alerting service URL. Got below issues

 

Automation Job error message:
 

Task 2 'Apply Rules' failed with: Spotfire.AlertingFramework.DataManager.DataManagerConfigurationException: Unable to contact Alerting Server at https://XXXX.wdc.com/alerting at Spotfire.AlertingFramework.DataManager.DataManager.checkConnectivity() at Spotfire.AlertingFramework.DataManager.DataManager.GetBaseSubscriptions(List`1 sids) at Spotfire.AlertingFramework.Tasks.RulesTask.ExecuteCore(TaskExecutionContext context)

 

 

Alerting Log:

2024-09-06 12:51:34.171 ERROR 6276 --- [scheduling-1] c.t.p.s.a.scheduler.RunLogScheduler      : 

com.tibco.psg.spotfire.alerting.services.SubscriptionNotFoundException: Subscription with ID:39f5d1f0-a00d-47a2-b149-47a5f7f92cf4 is not found.
    at com.tibco.psg.spotfire.alerting.services.SubscriptionNotFoundException.createExceptionForId(SubscriptionNotFoundException.java:28) ~[classes/:1.1.0]
    at com.tibco.psg.spotfire.alerting.services.AlertsService.updateRunLog(AlertsService.java:672) ~[classes/:1.1.0]
    at com.tibco.psg.spotfire.alerting.services.AlertsService$$FastClassBySpringCGLIB$$ba5ab02c.invoke(<generated>) ~[classes/:1.1.0]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.31.jar:5.3.31]
    at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) ~[spring-aop-5.3.31.jar:5.3.31]
    at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) ~[spring-aop-5.3.31.jar:5.3.31]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) ~[spring-aop-5.3.31.jar:5.3.31]
    at com.tibco.psg.spotfire.alerting.services.AlertsService$$EnhancerBySpringCGLIB$$1ffab9ff.updateRunLog(<generated>) ~[classes/:1.1.0]
    at com.tibco.psg.spotfire.alerting.scheduler.RunLogScheduler.updateStatus(RunLogScheduler.java:63) ~[classes/:1.1.0]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]
    at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84) ~[spring-context-5.3.31.jar:5.3.31]
    at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-5.3.31.jar:5.3.31]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[na:na]
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) ~[na:na]
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[na:na]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[na:na]
    at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
 

[David Boot-Olazabal]

Hi Praveen,

Looking at the alerting log message, it seems that the subscription id is not found in the alerting database. Can you check that?
As the job fails at task two (the apply rules), it seems to me that the alerting server is working (otherwise it wouldn't have started at all).

Can you:
- create a new subscription, a simple one (just open an analysis file and send an email for instance);
- check in the database if the subscription is created properly (SUBSCRIPTIONS table);
- run the alert and check out if there are new entries in the RULE_RUN_LOG table;
- check the created .xml file for the alert and try to run it directly from the Automation Services job builder in the Analyst.

If this works, it might be that if you re-create the original subscription, it will then work.

Kind regards,

David

[Praveen Kumar Ramamoorthy]

Hi David, 

Even after updating the Alerting URL in the alerting properties and the analyst, still get the same error. 

So the alerting framework is working but something we need to check with the automation services. 

 

Thanks,
Praveen
 

[Praveen Kumar Ramamoorthy]

Hi David, 

I have updated the automation server,  nodemanager.properties with the app server where I configured the alerting as the primary, and now the jobs are executed successfully. 

I am testing a few other scenarios as well. Will keep you posted. 

Thanks, 
Praveen