Web Player says exhausted but no one using server

[Brandon Fenty]

I've set up a Spotfire server and it's great-except that once a web player becomes exhausted, it stays exhausted until I restart the instance. CPU usage on the server is

[Prashant Anand]

HeyBrandon. There could be multiple reasons why the web player is going exhusted. We will need to check the logs to see why that happened. You can check for errors in the web player logs located under "Node Manager installation directorytsnmversionnmlogs. Please check the debug web player log here. The web player debug log will be named like "Spotfire.Dxp.Worker.Host.Debug.7827c5b6-fa51-4aac-8d2c-46728ef8612b". Please check what error message you see there. If that still doesnt help, I would recommend opening a support ticket so we can look into this.

[Brandon Fenty]

This looked relevant:

 

	654970	WARN ;2019-04-22T18:34:48,898+00:00;2019-04-22 18:34:48,898;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;(null);BG Thread UpdateSomePerformanceCounters 4;;;Spotfire.Dxp.Web.Diagnostics.WebPlayerPerformance;"Failed to read the CPU average values used for strained and exhaustive instance levels."

 

 

[Prashant Anand]

Can you please search for "error" and check what errors you see

[Brandon Fenty]

Here are all errors listed:

 

	243998	ERROR;2019-04-15T01:16:46,903+00:00;2019-04-15 01:16:46,903;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;34;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244032 ERROR;2019-04-15T01:16:48,843+00:00;2019-04-15 01:16:48,843;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;32;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244083 ERROR;2019-04-15T01:16:58,999+00:00;2019-04-15 01:16:58,999;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;32;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244121 ERROR;2019-04-15T01:17:09,951+00:00;2019-04-15 01:17:09,951;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;35;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244158 ERROR;2019-04-15T01:17:20,823+00:00;2019-04-15 01:17:20,823;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;35;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244228 ERROR;2019-04-15T01:17:32,487+00:00;2019-04-15 01:17:32,487;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;4;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244296 ERROR;2019-04-15T01:18:11,672+00:00;2019-04-15 01:18:11,672;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;Web Server Worker Thread #2;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244341 ERROR;2019-04-15T01:18:14,142+00:00;2019-04-15 01:18:14,142;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;Web Server Worker Thread #2;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244429 ERROR;2019-04-15T01:18:21,820+00:00;2019-04-15 01:18:21,820;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244926 ERROR;2019-04-15T01:19:25,900+00:00;2019-04-15 01:19:25,900;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;33;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

244986 ERROR;2019-04-15T01:19:44,625+00:00;2019-04-15 01:19:44,625;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;35;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245020 ERROR;2019-04-15T01:19:47,300+00:00;2019-04-15 01:19:47,300;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245054 ERROR;2019-04-15T01:19:50,551+00:00;2019-04-15 01:19:50,551;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;4;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245109 ERROR;2019-04-15T01:19:55,856+00:00;2019-04-15 01:19:55,856;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;32;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245187 ERROR;2019-04-15T01:20:21,871+00:00;2019-04-15 01:20:21,871;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;33;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245243 ERROR;2019-04-15T01:20:43,247+00:00;2019-04-15 01:20:43,247;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;31;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245303 ERROR;2019-04-15T01:20:56,496+00:00;2019-04-15 01:20:56,496;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;32;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245354 ERROR;2019-04-15T01:21:09,604+00:00;2019-04-15 01:21:09,604;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;34;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245416 ERROR;2019-04-15T01:21:25,906+00:00;2019-04-15 01:21:25,906;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;35;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245463 ERROR;2019-04-15T01:21:32,815+00:00;2019-04-15 01:21:32,815;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;35;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245502 ERROR;2019-04-15T01:21:40,229+00:00;2019-04-15 01:21:40,229;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;32;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245545 ERROR;2019-04-15T01:21:48,853+00:00;2019-04-15 01:21:48,853;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;33;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245644 ERROR;2019-04-15T01:22:25,163+00:00;2019-04-15 01:22:25,163;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;34;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245684 ERROR;2019-04-15T01:22:30,403+00:00;2019-04-15 01:22:30,403;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;34;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245721 ERROR;2019-04-15T01:22:36,553+00:00;2019-04-15 01:22:36,553;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245766 ERROR;2019-04-15T01:22:54,538+00:00;2019-04-15 01:22:54,538;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245798 ERROR;2019-04-15T01:22:56,307+00:00;2019-04-15 01:22:56,307;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;68;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245861 ERROR;2019-04-15T01:23:10,981+00:00;2019-04-15 01:23:10,981;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;41;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

245958 ERROR;2019-04-15T01:24:18,865+00:00;2019-04-15 01:24:18,865;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;33;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

246551 ERROR;2019-04-15T01:28:51,237+00:00;2019-04-15 01:28:51,237;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

249073 ERROR;2019-04-15T02:36:44,604+00:00;2019-04-15 02:36:44,604;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;31;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

249248 ERROR;2019-04-15T02:37:04,177+00:00;2019-04-15 02:37:04,177;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;41;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

249309 ERROR;2019-04-15T02:37:19,882+00:00;2019-04-15 02:37:19,882;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;43;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

249348 ERROR;2019-04-15T02:37:23,316+00:00;2019-04-15 02:37:23,316;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;Web Server Worker Thread #4;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

249391 ERROR;2019-04-15T02:37:25,533+00:00;2019-04-15 02:37:25,533;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;Web Server Worker Thread #4;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

250076 ERROR;2019-04-15T02:54:50,569+00:00;2019-04-15 02:54:50,569;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;34;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

250112 ERROR;2019-04-15T02:54:55,278+00:00;2019-04-15 02:54:55,278;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;154;bfenty;5a0a3f4272d92dfd3a30caaa7bacd604;Spotfire.Dxp.Application.Visuals.Maps.FeatureLayerVisualization;"Failed to parse validation query."

252344 ERROR;2019-04-15T04:09:56,614+00:00;2019-04-15 04:09:56,614;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;150048d675DUxW;104

[Prashant Anand]

Brandon, it is not showing any exhausted error here. Do you have multiple Web Player instances It looks like this is not the instance that went exhausted. If you have multiple instances you will need to check the web player log for that specific instance that went exhausted. You can check the web player id from the Administrator UI and then check for "Spotfire.Dxp.Worker.Host.Debug.xxxx" where xxxx will be the web player instance ID.

[Brandon Fenty]

I can't paste everything so I pasted a sample of what's in the correct node log. Is there something specific I should filter for that would be useful

 

 

 

	59375	INFO ;2019-04-10T21:32:19,266+00:00;2019-04-10 21:32:19,266;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Data.Collections.DataBlockStorage;"Health status Ok->Strained, MemoryLoadExhaustedLimit: 98-93, MemoryLoadStrainedLimit 75-70, useProcessSizeForExhausted=False, currentValue=77, processSizeShare=7.75"

103777 WARN ;2019-04-12T02:09:05,126+00:00;2019-04-12 02:09:05,126;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating changed service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Strained). Update slots: 2 (2). Notifying server."

103778 DEBUG;2019-04-12T02:09:05,126+00:00;2019-04-12 02:09:05,126;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;85;monitoring@SPOTFIRESYSTEM;MonitoringUserSession;Spotfire.Dxp.Framework.Utilities.ServerLoggerManager;"Server log manager did not log Monitoring since category is disabled: 2019-04-12T02:09:05,126+00:00, 127.0.0.1, monitoring_wp, healthState, True, Exhausted, 091831d675qXPn, Exhausted, MemoryStatusChangedMonitoringEvent: Status Exhausted: MemoryAvailablePercentage 2, MemoryAvailableMb 157, MemoryStatus Exhausted., 091831d675qXPn, , e175c225-c095-4ef9-a1cc-fcc7294ca6a2."

103779 INFO ;2019-04-12T02:09:05,126+00:00;2019-04-12 02:09:05,126;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Data.Collections.DataBlockStorage;"Health status Strained->Exhausted, MemoryLoadExhaustedLimit: 98-93, MemoryLoadStrainedLimit 75-70, useProcessSizeForExhausted=False, currentValue=98, processSizeShare=7.72"

103780 DEBUG;2019-04-12T02:09:05,126+00:00;2019-04-12 02:09:05,126;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;64;monitoring@SPOTFIRESYSTEM;MonitoringUserSession;Spotfire.Dxp.LifeCycleEvents;"Send life cycle event synchronously 'MemoryStatusChangedMonitoringEvent: Status Exhausted, CpuStatus Ok, MemoryStatus Exhausted, PagingStatus Ok, WcfStatus Ok, DiskStatus Ok, AverageCpuUsage 11, AverageInstanceCpuUsage 0, AverageNodeCpuUsage 11, CurrentInstanceCpuUsage 0, CurrentNodeCpuUsage 17.72, MemoryAvailablePercentage 2, MemoryAvailableMb 157, CurrentInstanceMemoryUsage 546308096, CurrentInstanceMemoryUsagePercentage 6.6, AvailableDiskMb 41683, OutstandingWcfCalls 0, CachedDocuments 0, OpenDocuments 0, CurrentRunningQueries 0, AvailableScheduledUpdatesSlots 2, MaxScheduledUpdatesSlots 2, Explanation Health status is Exhausted and current RAM usage % is 98 which is above the limit 93/98..', to 'https://sawsserver:82/spotfire/nodemanager/messages'."

103781 WARN ;2019-04-12T02:09:10,131+00:00;2019-04-12 02:09:10,131;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103783 WARN ;2019-04-12T02:09:15,136+00:00;2019-04-12 02:09:15,136;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103790 WARN ;2019-04-12T02:09:20,142+00:00;2019-04-12 02:09:20,142;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103791 WARN ;2019-04-12T02:09:25,149+00:00;2019-04-12 02:09:25,149;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103792 WARN ;2019-04-12T02:09:30,154+00:00;2019-04-12 02:09:30,154;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103793 WARN ;2019-04-12T02:09:35,159+00:00;2019-04-12 02:09:35,159;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103794 WARN ;2019-04-12T02:09:40,166+00:00;2019-04-12 02:09:40,166;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103796 WARN ;2019-04-12T02:09:45,171+00:00;2019-04-12 02:09:45,171;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103797 WARN ;2019-04-12T02:09:50,177+00:00;2019-04-12 02:09:50,177;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103798 WARN ;2019-04-12T02:09:55,183+00:00;2019-04-12 02:09:55,183;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103809 WARN ;2019-04-12T02:10:00,188+00:00;2019-04-12 02:10:00,188;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2)."

103810 WARN ;2019-04-12T02:10:05,193+00:00;2019-04-12 02:10:05,193;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Exhausted). Update slots: 2 (2). Notifying server."

103811 DEBUG;2019-04-12T02:10:05,193+00:00;2019-04-12 02:10:05,193;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;85;monitoring@SPOTFIRESYSTEM;MonitoringUserSession;Spotfire.Dxp.Framework.Utilities.ServerLoggerManager;"Server log manager did not log Monitoring since category is disabled: 2019-04-12T02:10:05,193+00:00, 127.0.0.1, monitoring_wp, healthState, True, Exhausted, 091831d675qXPn, Exhausted, MemoryStatusChangedMonitoringEvent: Status Exhausted: MemoryAvailablePercentage 2, MemoryAvailableMb 157, MemoryStatus Exhausted., 091831d675qXPn, , e175c225-c095-4ef9-a1cc-fcc7294ca6a2."

103812 DEBUG;2019-04-12T02:10:05,193+00:00;2019-04-12 02:10:05,193;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;51;monitoring@SPOTFIRESYSTEM;MonitoringUserSession;Spotfire.Dxp.LifeCycleEvents;"Send life cycle event synchronously 'MemoryStatusChangedMonitoringEvent: Status Exhausted, CpuStatus Ok, MemoryStatus Exhausted, PagingStatus Ok, WcfStatus Ok, DiskStatus Ok, AverageCpuUsage 11, AverageInstanceCpuUsage 0, AverageNodeCpuUsage 11, CurrentInstanceCpuUsage 0, CurrentNodeCpuUsage 14.95, MemoryAvailablePercentage 2, MemoryAvailableMb 157, CurrentInstanceMemoryUsage 546308096, CurrentInstanceMemoryUsagePercentage 6.6, AvailableDiskMb 41683, OutstandingWcfCalls 0, CachedDocuments 0, OpenDocuments 0, CurrentRunningQueries 0, AvailableScheduledUpdatesSlots 2, MaxScheduledUpdatesSlots 2, Explanation Health status is Exhausted and current RAM usage % is 98 which is above the limit 93/98..', to 'https://sawsserver:82/spotfire/nodemanager/messages'."

103814 DEBUG;2019-04-12T02:10:07,696+00:00;2019-04-12 02:10:07,696;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;091831d675qXPn;BG Thread Cleanup CWUS 51;;;Spotfire.Dxp.Web.SiteManager;"TryRecoverMemory started. MemoryStatus=Exhausted, Action=Recycle"

104094 INFO ;2019-04-12T02:10:12,274+00:00;2019-04-12 02:10:12,274;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;(null);DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Data.Collections.DataBlockStorage;"Health status Ok->Strained, MemoryLoadExhaustedLimit: 98-93, MemoryLoadStrainedLimit 75-70, useProcessSizeForExhausted=False, currentValue=84, processSizeShare=1.65"

116870 WARN ;2019-04-12T12:08:56,687+00:00;2019-04-12 12:08:56,687;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;120210d67540tr;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Web.SiteManager;"Evaluating changed service instance health status. CPU: Ok, Memory: Exhausted, Paging Ok, TempDisk Ok, WCF Ok. Combined: Exhausted (Strained). Update slots: 2 (2). Notifying server."

116871 INFO ;2019-04-12T12:08:56,687+00:00;2019-04-12 12:08:56,687;e175c225-c095-4ef9-a1cc-fcc7294ca6a2;120210d67540tr;DataBlockStorage.MemoryStatusUpdater;;;Spotfire.Dxp.Data.Collections.DataBlockStorage;"Health status Strained->Exhausted, MemoryLoadExhaustedLimit: 98-93, MemoryLoadStrainedLimit 75-70, useProcessSizeForExhausted=False, currentValue=98, processSizeShare=4.36"

116872 DEBU

[Prashant Anand]

Brandon, it looks like this will need a detailed investigation as it is not evident what the web player went exhausted. This might need a thorough log inspection for node anager and server. I would recommend opening a support ticket and someone from Spotfire Support will surely look into it and find the root cause.

[Brandon Fenty]

Thank you for your support I will reach out to Spotfire.