[BUG] ISM policy converting local to remote indexes

[sashalarsoo]

Describe the bug

Hi i have got the following policy. I wanted to after 2 days pass the indexes in the remote storage (s3) and remove them from my persistent volume in k8s so i can manage my space for only recent indexes:

{
  "policy": {
    "policy_id": "archive-to-remote",
    "description": "Archive indices older than 2 days to searchable snapshots, delete after 730 days",
    "default_state": "hot",
    "states": [
      {
        "name": "hot",
        "actions": [],
        "transitions": [
          {
            "state_name": "archive",
            "conditions": {
              "min_index_age": "2d"
            }
          }
        ]
      },
      {
        "name": "archive",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "snapshot": {
              "repository": "snapshot-repo",
              "snapshot": "{{ctx.index}}"
            }
          },
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "convert_index_to_remote": {
              "repository": "snapshot-repo",
              "snapshot": "{{ctx.index}}"
            }
          }
        ],
        "transitions": [
          {
            "state_name": "delete",
            "conditions": {
              "min_index_age": "730d"
            }
          }
        ]
      },
      {
        "name": "delete",
        "actions": [
          {
            "retry": {
              "count": 3,
              "backoff": "exponential",
              "delay": "1m"
            },
            "delete": {}
          }
        ],
        "transitions": []
      }
    ],
    "ism_template": [
      {
        "index_patterns": ["logs-*"],
        "priority": 100
      },
      {
        "index_patterns": ["events-*"],
        "priority": 100
      },
      {
        "index_patterns": ["traces-*"],
        "priority": 100
      },
      {
        "index_patterns": ["oci-logs-*"],
        "priority": 100
      },
      {
        "index_patterns": ["security-auditlog-*"],
        "priority": 100
      }
    ]
  }
}

However after 2 days the policy has created new indexes. exemple: i wanted my index "oci-logs" to be moved to s3 after 2 days and the policy created the index "oci-logs_remote". The old index is still there and this new index is yellow and i cant query him without an error but i see the policy took a snapshot of the old indexes.
Is this the normal behaviour ? How could the old indexes but really deleted from local storage after 2 days and be passed to s3 and how query them ? Opus 4.6 is saying i dont have search nodes.

Related component

Indexing

To Reproduce

Try to recreate the policy.

Expected behavior

Indexes deleted from local storage after 2 days and accessible from s3.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.