Fix IllegalArgumentException when resolved indices are empty by Mmuzaf · Pull Request #5797 · opensearch-project/security

Mmuzaf · 2025-11-18T11:04:32Z

Description

This PR fixes an IllegalArgumentException that occured when resolved indices are empty in the PrivilegesEvaluator. This issue occurs when all indices are closed (e.g., during cluster maintenance) and operations like _cat/recovery are executed.

Why these changes are required?

When all indices in a cluster are closed, certain operations like _cat/recovery can still be valid and should be authorized. However, when getAllIndicesResolved() returns an empty set, the code was attempting to create a CheckTable with an empty collection, which throws an IllegalArgumentException.

This prevents valid operations from being authorized when indices are closed.

What is the old behavior before changes and new behavior after changes?

Old behavior:

When all indices were closed and getAllIndicesResolved() returned an empty set, the code would attempt to create a CheckTable with an empty collection and throw an exception.

New behavior:

When getAllIndicesResolved() returns an empty set allow operations to proceed
Operations like _cat/recovery now succeed when users have the appropriate permissions, even when all indices are closed

Issues Resolved

Resolves #5794

Is this a backport?

No

Do these changes introduce new permission(s) to be displayed in the static dropdown on the front-end?

No

Testing

Added unit test hasIndexPrivilegeEmptyResolvedIndices()
Added integration test IndexAuthorizationWithClosedIndicesIntTests

Manual Testing

Verified that _cat/recovery and similar operations work correctly when all indices are closed

Check List

New functionality includes testing
New functionality has been documented (CHANGELOG.md updated)
New Roles/Permissions have a corresponding security dashboards plugin PR (N/A - no new permissions)
API changes companion pull request created (N/A - no API changes)
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

nibix · 2025-11-18T11:38:33Z

That looks quite good to me, thank you!

Just a few things:

We should first target the main branch; we can then backport to 2.19
You need to sign off your commit (this will then fix the DCO check). See https://github.com/opensearch-project/.github/blob/main/CONTRIBUTING.md
It's not a blocker for me, but it might be worthwhile to also have an integration test for this (or at least a quick manual test), as the index resolution process actually has sometimes a few unexpected pecularities (we are working on fixes for these in Introduced explicit index resolution API #5399 ) Integration tests are in https://github.com/opensearch-project/security/blob/main/src/integrationTest/java/org/opensearch/security/privileges/int_tests/IndexAuthorizationReadOnlyIntTests.java (it is quite a complicated beast though, I can help if needed).

Mmuzaf · 2025-11-20T15:45:54Z

@nibix during the implementation of the integration test, I encountered another edge case: if there are no indices, and users do not have any indices priveleges for the action at all, such requests should be also forbidden. To check this edge case I've added - a new method hasAnyIndexPrivilegeForAction. Could you please take a look if this is correct?

You've also mentioned IndexAuthorizationReadOnlyIntTests.java to add a new ingegration test there. I've added new IndexAuthorizationWithClosedIndicesIntTests instead for the following reasons:

we close all indices (including hidden ones) in order to perform a test on our specific case. In my opinion, this operation does not align with the read-only test as closing indices affects the whole IndexAuthorizationReadOnlyIntTests suite state;
adding new users to the USERS for our specific case in IndexAuthorizationReadOnlyIntTests suite affects all the tests in this suite due to the whole suite is parametrized, I thinks it's better to avoid it in our case;
we may want to expand test coverage for another issues when all indexes are closed;

wdyt?

nibix · 2025-11-21T05:07:03Z

@Mmuzaf

Thanks for the update! On Friday, I will be quite busy with other stuff, so I will only have time to take a closer look on Monday.

if there are no indices, and users do not have any indices priveleges for the action at all, such requests should be also forbidden.

We need to take a closer look at this:

I believe that this behavior is the case also for all other index actions. If this is a fundamentally exposed behavior, we will likely have to mantain backwards compatiblity
For remote index operations, such a behavior is actually required. It is even possible that there are coordinator clusters with no local indices at all. In such cases it is expected that no local privileges are required.

Of course, this is a discussable topic - at the moment, we are working on some fundamental behavior changes (see #3905, #5367 and #5399 ; I also need to write an updated RFC soon). Feel invited to discuss there.

nibix · 2025-11-28T13:58:09Z

@Mmuzaf

Sorry for the late reply, was just too busy :-(

That looks generally very good to me! Yet, I think we need to briefly reflect on this:

if there are no indices, and users do not have any indices priveleges for the action at all, such requests should be also forbidden

If I see this correctly, this would have the following effect:

Doing a GET /a*,remote_cluster:b/_search would be allowed even if the local user has no local privileges for search if a* does not match any indices. That's due to the condition here:

security/src/main/java/org/opensearch/security/privileges/actionlevel/RuntimeOptimizedActionPrivileges.java

Line 94 in 91a7097

if (!resolvedIndices.isLocalAll() && resolvedIndices.getAllIndices().isEmpty()) {
Doing a GET /*,remote_cluster:b/_search would be then forbidden however as this goes through the new gate this.index.hasAnyIndexPrivilegeForAction(context, actions)

This is a kind of inconsistent behavior which we should avoid IMHO (even though it might be a case that seldomly happens in the real word).

For now, I would propose not to do the hasAnyIndexPrivilegeForAction() check to stay in sync with the existing behavior.

At the moment, we are busy on revising the index authorization behavior, see #5814. We could use that chance and implement the behavior change in that go. Feel invited to have a look there and comment :)

@cwperks wdyt?

Mmuzaf · 2025-12-02T17:35:57Z

@nibix thanks for the comment!

I agree with your suggestion. Let's keep the fix short and simple to avoid any problems with backwards compatibility, bearing in mind that the refactoring is still ongoing in other PRs. I've updated the PR to reflect our discussion and amended the description.

Is there anything else I need to do to get this into the next security plugin release (2.19.4.0)?

nibix · 2025-12-08T00:34:23Z

@Mmuzaf

Thanks for the update!

It seems that the new int tests are failing at the moment:

2025-12-08T00:12:53.5490290Z IndexAuthorizationWithClosedIndicesIntTests > cat_recovery_allIndicesClosed {legacy_system_index_perm, no index privileges} FAILED
2025-12-08T00:12:53.5491364Z     java.lang.AssertionError: 
2025-12-08T00:12:53.5495715Z     Expected: Response has status 403 Forbidden with /error/root_cause/0/reason no permissions for [indices:monitor/recovery]
2025-12-08T00:12:53.5496605Z          but: Status is not 403 Forbidden:
2025-12-08T00:12:53.5497607Z     <HttpResponse [inner=200 OK HTTP/1.1, body=index_b1 0 52ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.5kb 0 0 100.0%
2025-12-08T00:12:53.5498951Z     index_b2 0 53ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.2kb 0 0 100.0%
2025-12-08T00:12:53.5500081Z     index_a1 0 21ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 49 0b 0b 100.0% 189.6kb 0 0 100.0%
2025-12-08T00:12:53.5501225Z     index_a2 0 52ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 46 0b 0b 100.0% 174.2kb 0 0 100.0%
2025-12-08T00:12:53.5502869Z     , header=[X-OpenSearch-Version: OpenSearch/3.4.0-SNAPSHOT (opensearch), content-type: text/plain; charset=UTF-8, content-length: 504], statusCode=200, statusReason=OK]>
2025-12-08T00:12:53.5504237Z         at __randomizedtesting.SeedInfo.seed([72371A713E48A51E:61C6B8E3195A74BB]:0)
2025-12-08T00:12:53.5505097Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2025-12-08T00:12:53.5505840Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
2025-12-08T00:12:53.5507410Z         at org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests.cat_recovery_allIndicesClosed(IndexAuthorizationWithClosedIndicesIntTests.java:210)
2025-12-08T00:12:53.5542169Z 
2025-12-08T00:12:53.5542180Z 
2025-12-08T00:12:53.5542879Z Suite: Test class org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests
2025-12-08T00:12:53.5544023Z   2> WARNING: A restricted method in java.lang.foreign.Linker has been called
2025-12-08T00:12:53.5544984Z   2> WARNING: java.lang.foreign.Linker::downcallHandle has been called by the unnamed module
2025-12-08T00:12:53.5545946Z   2> WARNING: Use --enable-native-access=ALL-UNNAMED to avoid a warning for this module
2025-12-08T00:12:53.5546502Z 
2025-12-08T00:12:53.5546657Z   2> java.lang.AssertionError: 
2025-12-08T00:12:53.5547522Z     Expected: Response has status 403 Forbidden with /error/root_cause/0/reason no permissions for [indices:monitor/recovery]
2025-12-08T00:12:53.5548428Z          but: Status is not 403 Forbidden:
2025-12-08T00:12:53.5549458Z     <HttpResponse [inner=200 OK HTTP/1.1, body=index_b1 0 80ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.5kb 0 0 100.0%
2025-12-08T00:12:53.5550847Z     index_a1 0 33ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 52 0b 0b 100.0% 200.7kb 0 0 100.0%
2025-12-08T00:12:53.5552347Z     index_b2 0 66ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.2kb 0 0 100.0%
2025-12-08T00:12:53.5554332Z     index_a2 0 69ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 46 0b 0b 100.0% 174.2kb 0 0 100.0%
2025-12-08T00:12:53.5555870Z     , header=[X-OpenSearch-Version: OpenSearch/3.4.0-SNAPSHOT (opensearch), content-type: text/plain; charset=UTF-8, content-length: 504], statusCode=200, statusReason=OK]>
2025-12-08T00:12:53.5557233Z         at __randomizedtesting.SeedInfo.seed([72371A713E48A51E:61C6B8E3195A74BB]:0)
2025-12-08T00:12:53.5558160Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2025-12-08T00:12:53.5558960Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
2025-12-08T00:12:53.5560679Z         at org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests.cat_recovery_allIndicesClosed(IndexAuthorizationWithClosedIndicesIntTests.java:210)
2025-12-08T00:12:53.5562363Z   2> java.lang.AssertionError: 
2025-12-08T00:12:53.5563249Z     Expected: Response has status 403 Forbidden with /error/root_cause/0/reason no permissions for [indices:monitor/recovery]
2025-12-08T00:12:53.5564292Z          but: Status is not 403 Forbidden:
2025-12-08T00:12:53.5565331Z     <HttpResponse [inner=200 OK HTTP/1.1, body=index_b1 0 52ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.5kb 0 0 100.0%
2025-12-08T00:12:53.5566776Z     index_b2 0 53ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 37 0b 0b 100.0% 106.2kb 0 0 100.0%
2025-12-08T00:12:53.5567926Z     index_a1 0 21ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 49 0b 0b 100.0% 189.6kb 0 0 100.0%
2025-12-08T00:12:53.5569084Z     index_a2 0 52ms existing_store done n/a n/a 127.0.0.1 cluster_manager_0 n/a n/a 0 0 100.0% 46 0b 0b 100.0% 174.2kb 0 0 100.0%
2025-12-08T00:12:53.5570635Z     , header=[X-OpenSearch-Version: OpenSearch/3.4.0-SNAPSHOT (opensearch), content-type: text/plain; charset=UTF-8, content-length: 504], statusCode=200, statusReason=OK]>
2025-12-08T00:12:53.5572135Z         at __randomizedtesting.SeedInfo.seed([72371A713E48A51E:61C6B8E3195A74BB]:0)
2025-12-08T00:12:53.5572962Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
2025-12-08T00:12:53.5573701Z         at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
2025-12-08T00:12:53.5575294Z         at org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests.cat_recovery_allIndicesClosed(IndexAuthorizationWithClosedIndicesIntTests.java:210)
2025-12-08T00:12:54.0480186Z 
2025-12-08T00:12:54.0512520Z Tests with failures:
2025-12-08T00:12:54.0514080Z  - org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests.cat_recovery_allIndicesClosed {legacy, no index privileges}
2025-12-08T00:12:54.0516582Z  - org.opensearch.security.privileges.int_tests.IndexAuthorizationWithClosedIndicesIntTests.cat_recovery_allIndicesClosed {legacy_system_index_perm, no index privileges}

Maybe we also need to adapt these to expect a 200 response in case no indices are requested?

Signed-off-by: Maxim Muzafarov <mmuzaf@apache.org>

Mmuzaf · 2025-12-08T14:03:31Z

@nibix Sorry, my mistake. Not all the changes made locally have been pushed to the origin.
Now everything is fixed, and I also rebased the changes on top of the main.

codecov · 2025-12-09T00:26:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 73.71%. Comparing base (212c132) to head (f4969b7).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5797      +/-   ##
==========================================
+ Coverage   73.68%   73.71%   +0.02%     
==========================================
  Files         438      438              
  Lines       26644    26649       +5     
  Branches     3938     3939       +1     
==========================================
+ Hits        19633    19643      +10     
+ Misses       5138     5134       -4     
+ Partials     1873     1872       -1

Files with missing lines	Coverage Δ
.../actionlevel/RuntimeOptimizedActionPrivileges.java	`100.00% <100.00%> (ø)`

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

cwperks · 2025-12-16T20:55:45Z

...nTest/java/org/opensearch/security/privileges/actionlevel/RoleBasedActionPrivilegesTest.java

+        @Test
+        public void hasIndexPrivilegeEmptyResolvedIndices() throws Exception {
+            SecurityDynamicConfiguration<RoleV7> roles = SecurityDynamicConfiguration.fromYaml(
+                "test_role:\n"


nit: I think we can use java text blocks to define multi-line strings.

cwperks · 2025-12-16T20:59:41Z

@Mmuzaf Apologies for not getting back sooner. The changes look reasonable, but for my understanding what does _cat/recovery do when all indices are closed?

cwperks · 2025-12-16T21:09:23Z

When all indices are closed is it possible to call _cat/recovery?expand_wildcards=all?

I'm also wondering if it makes sense to respond with a warning telling the caller that it could not resolve to indices.

cwperks · 2025-12-16T21:15:15Z

@Mmuzaf @nibix @willyborankin do you think we should respond with a warning when we cannot resolve to indices? I believe https://github.com/opensearch-project/OpenSearch/blob/799fb9bf46b1eb823f26f72e368f70a5675bf88f/server/src/main/java/org/opensearch/cluster/metadata/MetadataIndexTemplateService.java#L651-L662 shows an example of how it can be done.

nibix · 2025-12-17T13:45:16Z

@cwperks

do you think we should respond with a warning when we cannot resolve to indices?

You mean the case when an index expression yields an empty set of indices?

For me that sounds like a rather normal case, there are many reasons why that might happen. IMHO, that does not need to be warned about.

But, generally, that should a core thing, wdyt?

cwperks · 2025-12-17T14:02:10Z

You mean the case when an index expression yields an empty set of indices?

I see, yea that makes sense esp in cases where a search is performed on an index pattern that does not resolve to any concrete indices.

Edit:

Searching on an index pattern which matches 0 indices works gracefully:

> curl -XGET https://admin:password@localhost:9200/does-not-exist\*/_search\?pretty -k
{
  "took" : 7,
  "timed_out" : false,
  "_shards" : {
    "total" : 0,
    "successful" : 0,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : 0.0,
    "hits" : [ ]
  }
}

Signed-off-by: Craig Perkins <cwperx@amazon.com>

cwperks · 2025-12-17T14:02:48Z

@nibix can you re-approve? I resolved the conflict in the CHANGELOG.

nibix

re-approved

Mmuzaf mentioned this pull request Nov 18, 2025

[BUG] java.lang.IllegalArgumentException: Must contain at least one column and at least one row (got []/[indices:monitor/recovery]) #5794

Closed

Mmuzaf force-pushed the fix-resolved-indices-empty-ex branch from 4692a36 to b9d0dec Compare November 18, 2025 15:20

Mmuzaf changed the base branch from 2.19 to main November 18, 2025 15:25

Mmuzaf force-pushed the fix-resolved-indices-empty-ex branch 5 times, most recently from 96c8e4e to 91a7097 Compare November 20, 2025 15:45

cwperks added the v3.4.0 label Nov 24, 2025

Mmuzaf force-pushed the fix-resolved-indices-empty-ex branch from 91a7097 to 1c46d7d Compare December 2, 2025 17:31

Mmuzaf marked this pull request as ready for review December 2, 2025 21:36

Mmuzaf requested review from DarshitChanpura, RyanL1997, cwperks, derek-ho, nibix, reta, shikharj05 and willyborankin as code owners December 2, 2025 21:36

Fix IllegalArgumentException when resolved indices are empty

5f3992e

Signed-off-by: Maxim Muzafarov <mmuzaf@apache.org>

Mmuzaf force-pushed the fix-resolved-indices-empty-ex branch from fc936e2 to 5f3992e Compare December 8, 2025 14:01

cwperks removed the v3.4.0 label Dec 16, 2025

cwperks added the v3.5.0 Issues targeting release v3.5.0 label Dec 16, 2025

cwperks reviewed Dec 16, 2025

View reviewed changes

nibix previously approved these changes Dec 17, 2025

View reviewed changes

Merge branch 'main' into fix-resolved-indices-empty-ex

f4969b7

Signed-off-by: Craig Perkins <cwperx@amazon.com>

cwperks dismissed nibix’s stale review via f4969b7 December 17, 2025 14:02

cwperks approved these changes Dec 17, 2025

View reviewed changes

willyborankin approved these changes Dec 17, 2025

View reviewed changes

nibix approved these changes Dec 17, 2025

View reviewed changes

cwperks merged commit 3226ec4 into opensearch-project:main Dec 17, 2025
65 of 66 checks passed

Conversation

Mmuzaf commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Category

Why these changes are required?

What is the old behavior before changes and new behavior after changes?

Issues Resolved

Is this a backport?

Do these changes introduce new permission(s) to be displayed in the static dropdown on the front-end?

Testing

Manual Testing

Check List

Uh oh!

nibix commented Nov 18, 2025

Uh oh!

Mmuzaf commented Nov 20, 2025

Uh oh!

nibix commented Nov 21, 2025

Uh oh!

nibix commented Nov 28, 2025

Uh oh!

Mmuzaf commented Dec 2, 2025

Uh oh!

nibix commented Dec 8, 2025

Uh oh!

Mmuzaf commented Dec 8, 2025

Uh oh!

codecov bot commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cwperks Dec 16, 2025

Choose a reason for hiding this comment

Uh oh!

cwperks commented Dec 16, 2025

Uh oh!

cwperks commented Dec 16, 2025

Uh oh!

cwperks commented Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nibix commented Dec 17, 2025

Uh oh!

cwperks commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cwperks commented Dec 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nibix left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Mmuzaf commented Nov 18, 2025 •

edited

Loading

codecov bot commented Dec 9, 2025 •

edited

Loading

cwperks commented Dec 16, 2025 •

edited

Loading

cwperks commented Dec 17, 2025 •

edited

Loading

cwperks commented Dec 17, 2025 •

edited

Loading