Skip to content

Conversation

@abhishekbhatia1710
Copy link
Contributor

@abhishekbhatia1710 abhishekbhatia1710 commented Dec 19, 2025

Summary

Fixes json.parse() failures when ES|QL risk score calculation query's output contain special characters (quotes, backslashes, newlines, etc.) by encoding field values with Base64 in queries.

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

  • Flaky Test Runner was used on any tests changed
  • The PR description includes the appropriate Release Notes section, and the correct release_note:* label is applied per the guidelines
  • Review the backport guidelines and apply applicable backport:* labels.

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-entity-analytics (Team:Entity Analytics)

Copy link
Contributor

@tiansivive tiansivive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix @abhishekbhatia1710
Great work jumping on the SDH!

🚀 🚢

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes JSON parsing failures in the risk scoring ESQL queries when field values contain special characters (quotes, backslashes, newlines, etc.). The solution encodes rule_name and category fields using Base64 in the ESQL query output, then decodes them when processing the results. The implementation maintains backward compatibility by falling back to plain text fields if Base64-encoded versions are not present.

Key Changes

  • ESQL query now Base64-encodes rule_name and category fields to prevent JSON string escaping issues
  • Decoding logic in buildRiskScoreBucket handles both new Base64-encoded format and legacy plain text format
  • Comprehensive test coverage for various special character scenarios (quotes, backslashes, newlines, Unicode, mixed formats)

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
calculate_esql_risk_scores.ts Modified ESQL query to Base64-encode rule_name and category fields; added decoding logic with backward compatibility fallback in buildRiskScoreBucket function
calculate_esql_risk_scores.test.ts Added comprehensive test suite covering special characters (quotes, backslashes, newlines, Unicode), backward compatibility with old format, and mixed format handling
calculate_esql_risk_scores.test.ts.snap Updated snapshot to reflect the modified ESQL query with Base64 encoding

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #45 / Cloud Security Posture Security Alerts Page - Graph visualization ECS fields only expanded flyout - entity enrichment for multiple generic targets - single target field
  • [job] [logs] FTR Configs #104 / serverless observability UI Dataset Quality Dataset quality details navigate to dataset details should navigate to details page from a main page

Metrics [docs]

✅ unchanged

History

cc @abhishekbhatia1710

@abhishekbhatia1710 abhishekbhatia1710 merged commit 998226c into elastic:main Dec 22, 2025
13 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 9.1, 9.2, 9.3

https://github.com/elastic/kibana/actions/runs/20431359651

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Dec 22, 2025
…aracters in ESQL query for risk scoring (elastic#247060)

## Summary

Fixes `json.parse()` failures when ES|QL risk score calculation query's
output contain special characters (quotes, backslashes, newlines, etc.)
by encoding field values with Base64 in queries.

Fixes: elastic/sdh-security-team#1529

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

(cherry picked from commit 998226c)
@kibanamachine
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
9.1 Backport failed because of merge conflicts

You might need to backport the following PRs to 9.1:
- [Security Solution] fix alert flyout threat intelligence section not showing multiple values (#245449)
- [Security Solution] Updates kibana MITRE data to v18.1 (#246770)
- Add refusal field to assistant conversations (#243423)
9.2 Backport failed because of merge conflicts
9.3

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 247060

Questions ?

Please refer to the Backport tool documentation

abhishekbhatia1710 added a commit to abhishekbhatia1710/kibana that referenced this pull request Dec 22, 2025
…aracters in ESQL query for risk scoring (elastic#247060)

## Summary

Fixes `json.parse()` failures when ES|QL risk score calculation query's
output contain special characters (quotes, backslashes, newlines, etc.)
by encoding field values with Base64 in queries.

Fixes: elastic/sdh-security-team#1529

### Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.

(cherry picked from commit 998226c)

# Conflicts:
#	x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/risk_score/__snapshots__/calculate_esql_risk_scores.test.ts.snap
#	x-pack/solutions/security/plugins/security_solution/server/lib/entity_analytics/risk_score/calculate_esql_risk_scores.ts
@abhishekbhatia1710
Copy link
Contributor Author

💔 Some backports could not be created

Status Branch Result
9.2
9.1 Conflict resolution was aborted by the user

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 247060

Questions ?

Please refer to the Backport tool documentation

mbondyra added a commit to mbondyra/kibana that referenced this pull request Dec 22, 2025
…d_step_back

* commit '51756d2722200a991607658d48ecda50aeb04a7d': (76 commits)
  [Synthetics] Fix SyncGlobalParamsSpaces flaky test (elastic#246487)
  [Synthetics] Fix useSyntheticsRules test (elastic#247259)
  [ES|QL] Fix index editor flaky test (elastic#247233)
  [ResponseOps][Reporting] Fix "failed to decrypt apiKey" error while disabling/enabling scheduled reports (elastic#247236)
  [Console] Update console definitions (main) (elastic#247214)
  [ES|QL] Esql indentation shortcut on the editor (elastic#247234)
  [Streams] Use original request for rules client when in default space (elastic#247014)
  Consolidate and improve unflattenObject (elastic#246725)
  [scout] use svl mode to run api-int tests (elastic#247223)
  SIEM Readiness V2 (elastic#245776)
  [ObsPresentation][A11y] Fix asset details flyout header announcement (elastic#246872)
  [Streams] Add abort support and silent mode for stream description generation (elastic#247082)
  [SLO] Add environment context to SLO feedback button (elastic#247221)
  Ignore the reason and retry systematically (elastic#246830)
  Update dependency @types/moment-duration-format to ^2.2.7 (main) (elastic#242221)
  [Streams 🌊 ] Add explicit waits for data grid rows before clicking expand button (elastic#246919)
  [Security Solution][Entity Analytics][Risk Scoring] Handle special characters in ESQL query for risk scoring (elastic#247060)
  [ML] Data frame analytics: Updates page headers (elastic#247097)
  [ES|QL] Build function arguments suggestions from hints (elastic#246736)
  Update dependency @hey-api/openapi-ts to v0.88.1 (main) (elastic#247210)
  ...
abhishekbhatia1710 added a commit to abhishekbhatia1710/kibana that referenced this pull request Dec 23, 2025
kibanamachine added a commit that referenced this pull request Dec 23, 2025
…ial characters in ESQL query for risk scoring (#247060) (#247244)

# Backport

This will backport the following commits from `main` to `9.3`:
- [[Security Solution][Entity Analytics][Risk Scoring] Handle special
characters in ESQL query for risk scoring
(#247060)](#247060)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Abhishek
Bhatia","email":"117628830+abhishekbhatia1710@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-22T12:05:12Z","message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Entity
Analytics","backport:version","v9.1.0","v9.2.0","v9.3.0","v9.4.0"],"title":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk
scoring","number":247060,"url":"https://github.com/elastic/kibana/pull/247060","mergeCommit":{"message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","9.2","9.3"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.3","label":"v9.3.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/247060","number":247060,"mergeCommit":{"message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}}]}]
BACKPORT-->

Co-authored-by: Abhishek Bhatia <117628830+abhishekbhatia1710@users.noreply.github.com>
@kibanamachine kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 23, 2025
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @abhishekbhatia1710

4 similar comments
@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @abhishekbhatia1710

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @abhishekbhatia1710

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @abhishekbhatia1710

@kibanamachine
Copy link
Contributor

Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync.
cc: @abhishekbhatia1710

abhishekbhatia1710 added a commit that referenced this pull request Dec 30, 2025
…ial characters in ESQL query for risk scoring (#247060) (#247247)

# Backport

This will backport the following commits from `main` to `9.2`:
- [[Security Solution][Entity Analytics][Risk Scoring] Handle special
characters in ESQL query for risk scoring
(#247060)](#247060)

<!--- Backport version: 9.6.6 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sorenlouv/backport)

<!--BACKPORT [{"author":{"name":"Abhishek
Bhatia","email":"117628830+abhishekbhatia1710@users.noreply.github.com"},"sourceCommit":{"committedDate":"2025-12-22T12:05:12Z","message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7","branchLabelMapping":{"^v9.4.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Entity
Analytics","backport:version","v9.1.0","v9.2.0","v9.3.0","v9.4.0"],"title":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk
scoring","number":247060,"url":"https://github.com/elastic/kibana/pull/247060","mergeCommit":{"message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}},"sourceBranch":"main","suggestedTargetBranches":["9.1","9.2"],"targetPullRequestStates":[{"branch":"9.1","label":"v9.1.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.2","label":"v9.2.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"9.3","label":"v9.3.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/247244","number":247244,"state":"OPEN"},{"branch":"main","label":"v9.4.0","branchLabelMappingKey":"^v9.4.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/247060","number":247060,"mergeCommit":{"message":"[Security
Solution][Entity Analytics][Risk Scoring] Handle special characters in
ESQL query for risk scoring (#247060)\n\n## Summary\n\nFixes
`json.parse()` failures when ES|QL risk score calculation
query's\noutput contain special characters (quotes, backslashes,
newlines, etc.)\nby encoding field values with Base64 in
queries.\n\nFixes:
https://github.com/elastic/sdh-security-team/issues/1529\n\n\n###
Checklist\n\nCheck the PR satisfies following conditions. \n\nReviewers
should verify this PR satisfies this list as well.\n\n- [ ] [Flaky
Test\nRunner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1)
was\nused on any tests changed\n- [ ] The PR description includes the
appropriate Release Notes section,\nand the correct `release_note:*`
label is applied per
the\n[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)\n-
[ ] Review the
[backport\nguidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)\nand
apply applicable `backport:*`
labels.","sha":"998226c69dd42c12dd551b1011d87518cd3115d7"}}]}]
BACKPORT-->

---------

Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
@kibanamachine kibanamachine added v9.2.4 and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Dec 30, 2025
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Jan 6, 2026
…aracters in ESQL query for risk scoring (elastic#247060)

## Summary

Fixes `json.parse()` failures when ES|QL risk score calculation query's
output contain special characters (quotes, backslashes, newlines, etc.)
by encoding field values with Base64 in queries.

Fixes: elastic/sdh-security-team#1529


### Checklist

Check the PR satisfies following conditions. 

Reviewers should verify this PR satisfies this list as well.

- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [ ] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)
- [ ] Review the [backport
guidelines](https://docs.google.com/document/d/1VyN5k91e5OVumlc0Gb9RPa3h1ewuPE705nRtioPiTvY/edit?usp=sharing)
and apply applicable `backport:*` labels.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants