Jump to content

CrashLoopBackOff on spotfire-server pod

Vanessa Sucre

By Vanessa Sucre
in Spotfire

 Share

Recommended Posts

Vanessa Sucre Explorer

Vanessa Sucre

Posted
Posted

Hi,
After installing Spotfire-server into AKS, we keep getting the following waring/error: 
  

vanessa [ ~ ]$ kubectl -n spotfire describe po spotfire-server-8b89895-nnd62   
Name:             spotfire-server-8b89895-nnd62
Namespace:        spotfire
Priority:         0
Service Account:  default
Node:             aks-spotfire-40725537-vmss000003/10.151.136.190
Start Time:       Thu, 28 Nov 2024 13:36:08 +0000
Labels:           app.kubernetes.io/component=server
                  app.kubernetes.io/instance=spotfire-server
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=spotfire-server
                  app.kubernetes.io/part-of=spotfire
                  app.kubernetes.io/version=14.0.5
                  helm.sh/chart=spotfire-server-0.0.61
                  pod-template-hash=8b89895
Annotations:      azure.workload.identity/client-id: 5a971272-bf6e-4251-aad6-1ac8b46f9241
                  azure.workload.identity/service-account: spotfire-service-account
                  azure.workload.identity/tenant-id: 6a98e456-3afc-4dae-813a-c28e4d491588
                  azure.workload.identity/use: true
Status:           Running
IP:               10.151.136.208
IPs:
  IP:           10.151.136.208
Controlled By:  ReplicaSet/spotfire-server-8b89895
Containers:
  spotfire-server:
    Container ID:  containerd://55b149e88267058a79af1a3b360ea7eed95edba7a4975fbe3819ea18df32313a
    Image:         crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0
    Image ID:      crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server@sha256:f4c4752d009132e247be7cf25e47222c67b01727c77993d8ea14827758e7b52b
    Ports:         8080/TCP, 9443/TCP, 9080/TCP, 9081/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      bash
      -c
    Args:
      source /azure/keyvault/secrets/app-config && env > /azure/keyvault/secrets/test; export AZURE_VAULT_NAME=kvsptidevwe001.vault.azure.net; for secret in ${AZURE_VAULT_NAME}/secrets/*; do export $secret; done; /opt/spotfire/entrypoint.sh /opt/spotfire/spotfireserver/tomcat/bin/catalina.sh run;
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 28 Nov 2024 13:39:06 +0000
      Finished:     Thu, 28 Nov 2024 13:39:06 +0000
    Ready:          False
    Restart Count:  5
    Environment:
      LOGGING_HTTPREQUESTS_ENABLED:  N
      ACCEPT_EUA:                    N
      SPOTFIREDB_PASSWORD:           <set to the key 'SPOTFIREDB_PASSWORD' in secret 'spotfire-server'>  Optional: false
      SPOTFIREDB_USERNAME:           <set to the key 'SPOTFIREDB_USERNAME' in secret 'spotfire-server'>  Optional: false
      POD_NAME:                      spotfire-server-8b89895-nnd62 (v1:metadata.name)
      SERVER_BACKEND_ADDRESS:        spotfire-server
      SERVER_IP_ADDRESS:              (v1:status.podIP)
      SITE_NAME:                     Default
      SITE_DISPLAY_NAME:              
      SITE_PUBLIC_ADDRESS:           http://spotfire.local
      SPOTFIREDB_URL:                sqlsptidevwe001.database.windows.net:1433;databaseName=dbsptidevwe001;encrypt=true;trustServerCertificate=false;logingTimeout=30;authentication=ActiveDirectoryDefault
      SPOTFIREDB_USERNAME:           ${SPOTFIREDB_USERNAME}
      SPOTFIREDB_CLASS:              com.microsoft.sqlserver.jdbc.SQLServerDriver
      LOGGING_LOGLEVEL:              debug
      JAVA_OPTS:                      -Dcom.sun.jndi.ldap.object.disableEndpointIdentification=true 
    Mounts:
      /opt/spotfire/spotfireserver/tomcat/application-data/library from library-import-export (rw)
      /opt/spotfire/spotfireserver/tomcat/logs from logs-volume (rw)
      /opt/spotfire/troubleshooting/ from troubleshooting-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v669n (ro)
  fluent-bit:
    Container ID:   containerd://be6128ebdfddad05bf832a645106bffcc7a8cd5fe67851e3e0fc072f07d4c0ae
    Image:          fluent/fluent-bit:2.0.5
    Image ID:       docker.io/fluent/fluent-bit@sha256:c62fe1b22d17eff9eaf74eb5c67058e7f41bd9deeae16a57009f35f24bda03b9
    Port:           2020/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Thu, 28 Nov 2024 13:36:09 +0000
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     2
      memory:  1Gi
    Requests:
      cpu:     100m
      memory:  128Mi
    Environment:
      POD_NAME:        spotfire-server-8b89895-nnd62 (v1:metadata.name)
      POD_NAMESPACE:   spotfire (v1:metadata.namespace)
      POD_ID:           (v1:metadata.uid)
      FILE_SEPARATOR:  /
      TSS_HOME:        /tss
      CONTAINER_NAME:  spotfire-server
    Mounts:
      /fluent-bit/etc/fluent-bit.conf from fluent-bit-config (rw,path="tss.stdout.fluentbit.conf")
      /fluent-bit/etc/tss.parsers.fluentbit.conf from fluent-bit-config (rw,path="tss.parsers.fluentbit.conf")
      /fluent-bit/etc/tss.podannotations.fluentbit.conf from fluent-bit-config (rw,path="tss.podannotations.fluentbit.conf")
      /fluent-bit/etc/tss.tail.fluentbit.conf from fluent-bit-config (rw,path="tss.tail.fluentbit.conf")
      /tss/logs from logs-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-v669n (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  logs-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  fluent-bit-config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      spotfire-server-spotfire-server-fluent-bit
    Optional:  false
  library-import-export:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  spotfire-server-library-import-export
    ReadOnly:   false
  troubleshooting-volume:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     
    SizeLimit:  <unset>
  kube-api-access-v669n:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  4m28s                  default-scheduler  Successfully assigned spotfire/spotfire-server-8b89895-nnd62 to aks-spotfire-40725537-vmss000003
  Normal   Pulled     4m27s                  kubelet            Successfully pulled image "crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0" in 235ms (235ms including waiting)
  Normal   Pulled     4m27s                  kubelet            Container image "fluent/fluent-bit:2.0.5" already present on machine
  Normal   Created    4m27s                  kubelet            Created container fluent-bit
  Normal   Started    4m27s                  kubelet            Started container fluent-bit
  Normal   Pulled     4m26s                  kubelet            Successfully pulled image "crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0" in 170ms (170ms including waiting)
  Normal   Pulled     4m11s                  kubelet            Successfully pulled image "crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0" in 237ms (237ms including waiting)
  Normal   Pulling    3m45s (x4 over 4m27s)  kubelet            Pulling image "crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0"
  Normal   Created    3m44s (x4 over 4m27s)  kubelet            Created container spotfire-server
  Normal   Started    3m44s (x4 over 4m27s)  kubelet            Started container spotfire-server
  Normal   Pulled     3m44s                  kubelet            Successfully pulled image "crgcfictemeacoreprodwe001.azurecr.io/spotfire/spotfire-server:14.0.0" in 1.219s (1.219s including waiting)
  Warning  BackOff    3m32s (x6 over 4m25s)  kubelet            Back-off restarting failed container spotfire-server in pod spotfire-server-8b89895-nnd62_spotfire(20b487f1-1662-42f1-9fdc-a3740d1893b8)


We don't know if we are trying to run startup scripts (such as cataline.sh) and don't have access to those folders since we are getting this on the spotfire-server log:
  

vanessa [ ~ ]$  kubectl --namespace spotfire logs spotfire-server-8b89895-nnd62  -c fluent-bit
Fluent Bit v2.0.5
* Copyright (C) 2015-2022 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2024/11/28 13:36:09] [error] [input:tail:tss.actionlog] read error, check permissions: /tss/logs/actionlogs/actionlog*.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.actionlog] error scanning path: /tss/logs/actionlogs/actionlog*.log*
[2024/11/28 13:36:09] [error] [input:tail:tss.informationservices.serverlog] read error, check permissions: /tss/logs/informationservices/server.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.informationservices.serverlog] error scanning path: /tss/logs/informationservices/server.log*
[2024/11/28 13:36:09] [error] [input:tail:tss.informationservices.performancemonitoring] read error, check permissions: /tss/logs/informationservices/performance.monitoring.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.informationservices.performancemonitoring] error scanning path: /tss/logs/informationservices/performance.monitoring.log*
[2024/11/28 13:36:09] [error] [input:tail:tss.informationservices.actionlog] read error, check permissions: /tss/logs/informationservices/actionlogs/actionlog*.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.informationservices.actionlog] error scanning path: /tss/logs/informationservices/actionlogs/actionlog*.log*
[2024/11/28 13:36:09] [error] [input:tail:tss.informationservices.sqllog] read error, check permissions: /tss/logs/informationservices/sql.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.informationservices.sqllog] error scanning path: /tss/logs/informationservices/sql.log*
[2024/11/28 13:36:09] [error] [input:tail:tss.informationservices.threaddump] read error, check permissions: /tss/logs/informationservices/thread_dump.log*
[2024/11/28 13:36:09] [ warn] [input:tail:tss.informationservices.threaddump] error scanning path: /tss/logs/informationservices/thread_dump.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.informationservices.serverlog] read error, check permissions: /tss/logs/informationservices/server.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.informationservices.serverlog] error scanning path: /tss/logs/informationservices/server.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.informationservices.performancemonitoring] read error, check permissions: /tss/logs/informationservices/performance.monitoring.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.informationservices.performancemonitoring] error scanning path: /tss/logs/informationservices/performance.monitoring.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.informationservices.actionlog] read error, check permissions: /tss/logs/informationservices/actionlogs/actionlog*.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.informationservices.actionlog] error scanning path: /tss/logs/informationservices/actionlogs/actionlog*.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.informationservices.threaddump] read error, check permissions: /tss/logs/informationservices/thread_dump.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.informationservices.threaddump] error scanning path: /tss/logs/informationservices/thread_dump.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.actionlog] read error, check permissions: /tss/logs/actionlogs/actionlog*.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.actionlog] error scanning path: /tss/logs/actionlogs/actionlog*.log*
[2024/11/28 13:37:08] [error] [input:tail:tss.informationservices.sqllog] read error, check permissions: /tss/logs/informationservices/sql.log*
[2024/11/28 13:37:08] [ warn] [input:tail:tss.informationservices.sqllog] error scanning path: /tss/logs/informationservices/sql.log*


 

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...