-
Notifications
You must be signed in to change notification settings - Fork 8.5k
[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring #247060
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring #247060
Conversation
… in the ESQL query for risk scoring
|
Pinging @elastic/security-entity-analytics (Team:Entity Analytics) |
tiansivive
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the fix @abhishekbhatia1710
Great work jumping on the SDH!
🚀 🚢
...ugins/security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.ts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR fixes JSON parsing failures in the risk scoring ESQL queries when field values contain special characters (quotes, backslashes, newlines, etc.). The solution encodes rule_name and category fields using Base64 in the ESQL query output, then decodes them when processing the results. The implementation maintains backward compatibility by falling back to plain text fields if Base64-encoded versions are not present.
Key Changes
- ESQL query now Base64-encodes
rule_nameandcategoryfields to prevent JSON string escaping issues - Decoding logic in
buildRiskScoreBuckethandles both new Base64-encoded format and legacy plain text format - Comprehensive test coverage for various special character scenarios (quotes, backslashes, newlines, Unicode, mixed formats)
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
calculate_esql_risk_scores.ts |
Modified ESQL query to Base64-encode rule_name and category fields; added decoding logic with backward compatibility fallback in buildRiskScoreBucket function |
calculate_esql_risk_scores.test.ts |
Added comprehensive test suite covering special characters (quotes, backslashes, newlines, Unicode), backward compatibility with old format, and mixed format handling |
calculate_esql_risk_scores.test.ts.snap |
Updated snapshot to reflect the modified ESQL query with Base64 encoding |
.../security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.test.ts
Outdated
Show resolved
Hide resolved
...ugins/security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.ts
Outdated
Show resolved
Hide resolved
...ugins/security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.ts
Outdated
Show resolved
Hide resolved
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
History
|
|
Starting backport for target branches: 9.1, 9.2, 9.3 |
…aracters in ESQL query for risk scoring (elastic#247060) ## Summary Fixes `json.parse()` failures when ES|QL risk score calculation query's output contain special characters (quotes, backslashes, newlines, etc.) by encoding field values with Base64 in queries. Fixes: elastic/sdh-security-team#1529 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. (cherry picked from commit 998226c)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
…aracters in ESQL query for risk scoring (elastic#247060) ## Summary Fixes `json.parse()` failures when ES|QL risk score calculation query's output contain special characters (quotes, backslashes, newlines, etc.) by encoding field values with Base64 in queries. Fixes: elastic/sdh-security-team#1529 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels. (cherry picked from commit 998226c) # Conflicts: # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/risk_score/__snapshots__/calculate_esql_risk_scores.test.ts.snap # x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.ts
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
…d_step_back * commit '51756d2722200a991607658d48ecda50aeb04a7d': (76 commits) [Synthetics] Fix SyncGlobalParamsSpaces flaky test (elastic#246487) [Synthetics] Fix useSyntheticsRules test (elastic#247259) [ES|QL] Fix index editor flaky test (elastic#247233) [ResponseOps][Reporting] Fix "failed to decrypt apiKey" error while disabling/enabling scheduled reports (elastic#247236) [Console] Update console definitions (main) (elastic#247214) [ES|QL] Esql indentation shortcut on the editor (elastic#247234) [Streams] Use original request for rules client when in default space (elastic#247014) Consolidate and improve unflattenObject (elastic#246725) [scout] use svl mode to run api-int tests (elastic#247223) SIEM Readiness V2 (elastic#245776) [ObsPresentation][A11y] Fix asset details flyout header announcement (elastic#246872) [Streams] Add abort support and silent mode for stream description generation (elastic#247082) [SLO] Add environment context to SLO feedback button (elastic#247221) Ignore the reason and retry systematically (elastic#246830) Update dependency @types/moment-duration-format to ^2.2.7 (main) (elastic#242221) [Streams 🌊 ] Add explicit waits for data grid rows before clicking expand button (elastic#246919) [Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (elastic#247060) [ML] Data frame analytics: Updates page headers (elastic#247097) [ES|QL] Build function arguments suggestions from hints (elastic#246736) Update dependency @hey-api/openapi-ts to v0.88.1 (main) (elastic#247210) ...
…ial characters in ESQL query for risk scoring (#247060) (#247244) # Backport This will backport the following commits from `main` to `9.3`: - [[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)](#247060) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Abhishek Bhatia","email":"117628830+abhishekbhatia1710@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-22T12:05:12Z","message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Entity Analytics","backport:version","v9.1.0","v9.2.0","v9.3.0","v9.4.0"],"title":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring","number":247060,"url":"https://github.com/elastic/kibana/pull/247060","mergeCommit":{"message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","9.2","9.3"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.3","label":"v9.3.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/247060","number":247060,"mergeCommit":{"message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}}]}] BACKPORT--> Co-authored-by: Abhishek Bhatia <117628830+abhishekbhatia1710@users.noreply.github.com>
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
4 similar comments
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
…ial characters in ESQL query for risk scoring (#247060) (#247247) # Backport This will backport the following commits from `main` to `9.2`: - [[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)](#247060) <!--- Backport version: 9.6.6 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Abhishek Bhatia","email":"117628830+abhishekbhatia1710@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-22T12:05:12Z","message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Entity Analytics","backport:version","v9.1.0","v9.2.0","v9.3.0","v9.4.0"],"title":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring","number":247060,"url":"https://github.com/elastic/kibana/pull/247060","mergeCommit":{"message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","9.2"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.3","label":"v9.3.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/247244","number":247244,"state":"OPEN"},{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/247060","number":247060,"mergeCommit":{"message":"[Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes `json.parse()` failures when ES|QL risk score calculation query's\noutput contain special characters (quotes, backslashes, newlines, etc.)\nby encoding field values with Base64 in queries.\n\nFixes: https://github.com/elastic/sdh-security-team/issues/1529\n\n\n### Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers should verify this PR satisfies this list as well.\n\n- [ ] [Flaky Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was\nused on any tests changed\n- [ ] The PR description includes the appropriate Release Notes section,\nand the correct `release_note:*` label is applied per the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n- [ ] Review the [backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand apply applicable `backport:*` labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}}]}] BACKPORT--> --------- Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
…aracters in ESQL query for risk scoring (elastic#247060) ## Summary Fixes `json.parse()` failures when ES|QL risk score calculation query's output contain special characters (quotes, backslashes, newlines, etc.) by encoding field values with Base64 in queries. Fixes: elastic/sdh-security-team#1529 ### Checklist Check the PR satisfies following conditions. Reviewers should verify this PR satisfies this list as well. - [ ] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ ] The PR description includes the appropriate Release Notes section, and the correct `release_note:*` label is applied per the [guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process) - [ ] Review the [backport guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing) and apply applicable `backport:*` labels.
Summary
Fixes
json.parse()failures when ES|QL risk score calculation query's output contain special characters (quotes, backslashes, newlines, etc.) by encoding field values with Base64 in queries.Checklist
Check the PR satisfies following conditions.
Reviewers should verify this PR satisfies this list as well.
release_note:*label is applied per the guidelinesbackport:*labels.