Splunk Operator: 3.0.0 is not working with new shiny change about GENERAL TERMS

[yaroslav-nakonechnikov]

Please select the type of request

Bug

Tell us more

Describe the request

2025-11-18T11:38:43.431291288Z  INFO    PublishEvent    publishing event        {"controller": "licensemanager", "controllerGroup": "enterprise.splunk.com", "controllerKind": "LicenseManager", "LicenseManager": {"name":"dev","namespace":"splunk-operator"}, "namespace": "splunk-operator", "name": "dev", "reconcileID": "e7479a14-6060-4bcf-9d8f-8495ef0c6752", "reason": "validateLicenseManagerSpec", "message": "validatelicensemanager spec failed license not accepted, please adjust SPLUNK_GENERAL_TERMS to indicate you have accepted the current/latest version of the license. See README file for additional information"}

while

[yn@ip-100-65-11-163 /]$ kubectl get licensemanagers.enterprise.splunk.com -n splunk-operator dev -o yaml
apiVersion: enterprise.splunk.com/v4
kind: LicenseManager
metadata:
  annotations:
    karpenter.sh/do-not-disrupt: "true"
  creationTimestamp: "2025-06-23T12:27:35Z"
  generation: 7
  name: dev
  namespace: splunk-operator
  resourceVersion: "92221268"
  uid: 9626540d-7342-4054-8648-a3a074d5809e
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nodegroup
            operator: In
            values:
            - prj-eks-dev-splunk-licensemanager
  appRepo:
    appInstallPeriodSeconds: 90
    appSources:
    - location: config-explorer/
      name: config-explorer
      scope: local
      volumeName: splunkapps
    appsRepoPollIntervalSeconds: 120
    installMaxRetries: 3
    volumes:
    - endpoint: https://s3-eu-central-1.amazonaws.com
      name: splunkapps
      path: 461225008865-prj-eks-dev/splunk-apps
      provider: aws
      region: eu-central-1
      storageType: s3
  clusterManagerRef:
    name: dev
  defaults: |
    splunk:
      multisite_master: splunk-dev-cluster-manager-service
      site: site0
      app_paths_install:
        default:
          - https://artifacts.prj/artifactory/config-explorer_1716.tgz
      conf:
        - key: server
          value:
            directory: /opt/splunk/etc/system/local
            content:
              imds:
                imds_version: v2
        - key: deploymentclient
          value:
            directory: /opt/splunk/etc/system/local
            content:
              deployment-client :
                disabled : false
              target-broker:deploymentServer :
                targetUri : ds.dev.prj-prj-dev.internal.prjgroup.cloud:8089
        - key: web
          value:
            directory: /opt/splunk/etc/system/local
            content:
              settings:
                enableSplunkWebSSL: true
        - key: authentication
          value:
            directory: /opt/splunk/etc/system/local
            content:
              authentication:
                authSettings : saml
                authType : SAML
              roleMap_SAML:
                admin : prj-aws-s-eng-admin;rbko-aws-s-eng-admin
        - key: authorize
          value:
            directory: /opt/splunk/etc/system/local
            content:
              role_admin:
                run_script_adhocremotesearchraw : enabled
                run_script_adhocremotesearch : enabled
                run_script_environmentpoller : enabled
                run_script_sleepy : enabled
  etcVolumeStorageConfig:
    ephemeralStorage: true
  extraEnv:
  - name: SPLUNK_GENERAL_TERMS
    value: --accept-sgt-current-at-splunk-com
  - name: SPLUNK_ANSIBLE_PRE_TASKS
    value: file:///mnt/ansible-tasks/pre_tasks.yml
  - name: SPLUNK_ANSIBLE_POST_TASKS
    value: file:///mnt/ansible-tasks/post_tasks_lm.yml
  imagePullPolicy: IfNotPresent
  licenseUrl: /mnt/licenses/enterprise.lic
  livenessProbe:
    failureThreshold: 20
    initialDelaySeconds: 300
    periodSeconds: 15
    timeoutSeconds: 15
  readinessProbe:
    failureThreshold: 20
    initialDelaySeconds: 300
    periodSeconds: 15
    timeoutSeconds: 15
  resources:
    limits:
      cpu: 0
      memory: 0
    requests:
      cpu: 0
      memory: 0
  serviceAccount: splunk-sa
  startupProbe:
    failureThreshold: 20
    initialDelaySeconds: 300
    periodSeconds: 15
    timeoutSeconds: 15
  tolerations:
  - effect: NoSchedule
    key: function
    operator: Equal
    value: splunk-licensemanager
  varVolumeStorageConfig:
    ephemeralStorage: true
  volumes:
  - configMap:
      name: splunk-license
    name: licenses
  - configMap:
      name: ansible-tasks-lm
    name: ansible-tasks
  - name: certs
    secret:
      optional: false
      secretName: certs
status:
  appContext:
    appRepo:
      appInstallPeriodSeconds: 90
      appSources:
      - location: config-explorer/
        name: config-explorer
        premiumAppsProps:
          esDefaults: {}
        scope: local
        volumeName: splunkapps
      appsRepoPollIntervalSeconds: 120
      defaults:
        premiumAppsProps:
          esDefaults: {}
      installMaxRetries: 3
      volumes:
      - endpoint: https://s3-eu-central-1.amazonaws.com
        name: splunkapps
        path: 461225008865-prj-eks-dev/splunk-apps
        provider: aws
        region: eu-central-1
        secretRef: ""
        storageType: s3
    appSrcDeployStatus:
      config-explorer: {}
    appsRepoStatusPollIntervalSeconds: 120
    appsStatusMaxConcurrentAppDownloads: 5
    bundlePushStatus: {}
    isDeploymentInProgress: false
    lastAppInfoCheckTime: 1759925961
    version: 1
  message: license not accepted, please adjust SPLUNK_GENERAL_TERMS to indicate you
    have accepted the current/latest version of the license. See README file for additional
    information
  phase: Error
  telAppInstalled: true

Expected behavior

• A clear and concise description of what you expected to happen.

Splunk setup on K8S

• Details of the Splunk setup on the K8s cluster.

Reproduction/Testing steps

• Steps to reproduce the bug. For an enhancement or feature request, please provide steps to test.

K8s environment

• Useful information about the K8S environment being used. Eg. version of K8s, kind of K8s cluster etc..

Proposed changes(optional)

• Proposed change, if any.

K8s collector data(optional)

• Please provide data collected from the K8s collectors, if any.

Additional context(optional)

• Add any other context about the problem here.

[patrykw-splunk]

Hi @yaroslav-nakonechnikov. I see that you're using extraEnv for your manifest. Can you check if it's properly templated/translated into pod's environment variable?

[yaroslav-nakonechnikov]

in your crds there is no "env", only "extraEnv".

splunk-operator/config/crd/bases/enterprise.splunk.com_licensemanagers.yaml

Line 1258 in 1b0ee23

extraEnv:

[rlieberman-splunk]

@yaroslav-nakonechnikov please add the SPLUNK_GENERAL_TERMS environment variable to the splunk operator deployment, not the individual CRs per the documentation. This should fix the problem you are seeing. We can take this as an enhancement in the future to check the variable set on individual CRs separately.

[yaroslav-nakonechnikov]

yes, it was a case.
but there is problem, that without this extraEnv nothing starts, so i had to add it as well.

and, as a user, i don't expect splunk-oparator be in running state, if terms are not applied. It has to fail on startup

[rlieberman-splunk]

It does take a few minutes for the environment variable from the operator to propagate to the CRs, so the extraEnv should still not be needed.

The SPLUNK_GENERAL_TERMS acceptance is required for the docker splunk image, which is only used in the CRs, not the operator image. This is why the splunk operator can start without acceptance to the terms, but the custom resources running docker splunk cannot.

[yaroslav-nakonechnikov]

this is strange because docker image 9.3.7 doesn't have such requirement to supply that variable.
it still uses old way with startup args.

but making splunk-operator in running state - adds a lot of confusion. Because it can be updated during running cluster, and from 1st point of view - nothing changes.
And only if there some activity to shutdown/run new pods - it will lead to the issues.

[gblues]

My thoughts:

• Since the new variable is only needed by splunk 10.x images, it should only throw an error if you try to deploy splunk 10 with it. RELATED_IMAGE_SPLUNK_ENTERPRISE is right there with the info you need.
• either way, if the variable isn't set, the controller should fail to enter READY state to indicate to other k8s mechanisms that there's a problem