Skip to content

Fix CockroachDB ingestion for tables with hidden shard columns #25089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
ulixius9 merged 4 commits into from
Jan 8, 2026
Merged

Fix CockroachDB ingestion for tables with hidden shard columns #25089

ulixius9 merged 4 commits into from
Jan 8, 2026
+217 −1

Conversation

Copy link
Contributor

Copilot AI commented Jan 6, 2026 •
edited
Loading

Describe your changes:

CockroachDB creates hidden virtual columns (NOT VISIBLE) for hash-sharded indexes with names like crdb_internal_id_shard_16. These columns are excluded from get_columns() but included in get_pk_constraint(), causing a mismatch that results in Invalid column name found in table constraint errors (400).

Changes:

  • Added HIDDEN_SHARD_COLUMN_PATTERN compiled regex to match ^crdb_internal_.*_shard_\d+$
  • Added _is_hidden_shard_column() static method for pattern detection
  • Overrode _get_columns_with_constraints() to filter hidden shard columns from PK constraints

Tests added:

  • test_is_hidden_shard_column - regex pattern validation
  • test_hidden_shard_columns_filtered_from_pk_constraints - single column filtering
  • test_all_hidden_shard_columns_filtered_results_in_no_pk - edge case handling
  • test_multi_column_pk_with_hidden_shard_column - composite key with mixed columns

Type of change:

  • Bug fix

Checklist:

  • I have read the CONTRIBUTING document.

  • My PR title is Fixes <issue-number>: <short explanation>

  • I have commented on my code, particularly in hard-to-understand areas.

  • For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

  • I have added a test that covers the exact scenario we are fixing. For complex issues, comment the issue number in the test for future reference.

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • central.sonatype.com
    • Triggering command: /home/REDACTED/.local/bin/antlr4 antlr4 -Dlanguage=Python3 -o ingestion/src/metadata/generated/antlr openmetadata-spec/src/main/antlr4/org/openmetadata/schema/EntityLink.g4 openmetadata-spec/src/main/antlr4/org/openmetadata/schema/Fqn.g4 openmetadata-spec/src/main/antlr4/org/openmetadata/schema/JdbcUri.g4 (dns block)
  • www.antlr.org
    • Triggering command: /usr/bin/curl curl REDACTED (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

Original prompt

This section details on the original issue you should resolve

<issue_title>Deal with metadata agent error messages for cockroachDB database service for hidden sharded indexes</issue_title>
<issue_description>Internal Bug Report: CockroachDB Ingestion Failure on Hidden Shard Columns. This technique is common to avoid hotspots in cockroachDB and is not limited to the system table. https://www.cockroachlabs.com/docs/stable/hash-sharded-indexes

Affected module Ingestion Framework (specifically the CockroachDB Connector/SQLAlchemy Dialect).

Describe the bug The CockroachDB metadata agent fails to ingest tables that utilize NOT VISIBLE virtual columns in their primary key constraints (common in system tables or tables using USING HASH).

The agent skips these "hidden" columns during the column discovery phase but includes them in the Table Constraint payload. This results in a 400 Bad Request from the OpenMetadata server because the constraint references a column name that was not defined in the table entity.

To Reproduce

Set up a CockroachDB cluster (v22.2+). I used 25.4.2 running local on my Mac

Add the CockroachDB Service to OpenMetadata v1.11.4.

Attempt to ingest the system.public.mvcc_statistics table (or any table with a hashed primary key).

Observe the log error: message='Invalid column name found in table constraint' status=400.

Run SHOW CREATE TABLE system.public.mvcc_statistics in CockroachDB to verify the presence of the crdb_internal_..._shard_16 column marked as NOT VISIBLE.

Expected behavior The Ingestion Framework should be aware of CockroachDB's NOT VISIBLE columns. It should either:

Filter out hidden columns from constraint definitions before sending the payload to the server.

Include the hidden column in the columns list of the Table Entity (perhaps marked as a system column) so that the constraint remains valid.

Version:

OS: macOS 26.2 (Apple Silicon).

Python version: 3.9+ (Containerized)

OpenMetadata version: 1.11.3

OpenMetadata Ingestion package version: openmetadata-ingestion==1.11.3.0

Additional context This bug makes it impossible to ingest the CockroachDB system database. While system tables are often excluded via filters, any user-defined table using Hash Sharded Primary Keys (a recommended practice for high-scale CockroachDB clusters) will trigger this same failure, preventing metadata discovery for mission-critical data.
</issue_description>

<agent_instructions>Issue description has details, we need to add pytests for the coverage</agent_instructions>

Comments on the Issue (you are @copilot in this section)

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

@harshach
Fix CockroachDB ingestion for tables with hidden shard columns
3c0aeab 
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix metadata agent error messages for CockroachDB ingestion Fix CockroachDB ingestion for tables with hidden shard columns Jan 6, 2026
Copilot AI requested a review from harshach January 6, 2026 20:22
@harshach harshach marked this pull request as ready for review January 6, 2026 20:33
@harshach harshach requested a review from a team as a code owner January 6, 2026 20:33
@harshach harshach added the safe to test Add this label to run secure Github workflows on PRs label Jan 6, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 6, 2026

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR.
You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 6, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion-base-slim:trivy (debian 12.12)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (4)

Package Vulnerability ID Severity Installed Version Fixed Version
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/extended_sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/lineage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_data_aut.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.json

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage.yaml

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /ingestion/pipelines/sample_usage_aut.yaml

No Vulnerabilities Found

@github-actions
Copy link
Contributor

github-actions bot commented Jan 6, 2026 •
edited
Loading

🛡️ TRIVY SCAN RESULT 🛡️

Target: openmetadata-ingestion:trivy (debian 12.12)

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Java

Vulnerabilities (33)

Package Vulnerability ID Severity Installed Version Fixed Version
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.12.7 2.15.0
com.fasterxml.jackson.core:jackson-core CVE-2025-52999 🚨 HIGH 2.13.4 2.15.0
com.fasterxml.jackson.core:jackson-databind CVE-2022-42003 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4.2
com.fasterxml.jackson.core:jackson-databind CVE-2022-42004 🚨 HIGH 2.12.7 2.12.7.1, 2.13.4
com.google.code.gson:gson CVE-2022-25647 🚨 HIGH 2.2.4 2.8.9
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.3.0 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.3.0 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.3.0 3.25.5, 4.27.5, 4.28.2
com.google.protobuf:protobuf-java CVE-2021-22569 🚨 HIGH 3.7.1 3.16.1, 3.18.2, 3.19.2
com.google.protobuf:protobuf-java CVE-2022-3509 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2022-3510 🚨 HIGH 3.7.1 3.16.3, 3.19.6, 3.20.3, 3.21.7
com.google.protobuf:protobuf-java CVE-2024-7254 🚨 HIGH 3.7.1 3.25.5, 4.27.5, 4.28.2
com.nimbusds:nimbus-jose-jwt CVE-2023-52428 🚨 HIGH 9.8.1 9.37.2
com.squareup.okhttp3:okhttp CVE-2021-0341 🚨 HIGH 3.12.12 4.9.2
commons-beanutils:commons-beanutils CVE-2025-48734 🚨 HIGH 1.9.4 1.11.0
commons-io:commons-io CVE-2024-47554 🚨 HIGH 2.8.0 2.14.0
dnsjava:dnsjava CVE-2024-25638 🚨 HIGH 2.1.7 3.6.0
io.netty:netty-codec-http2 CVE-2025-55163 🚨 HIGH 4.1.96.Final 4.2.4.Final, 4.1.124.Final
io.netty:netty-codec-http2 GHSA-xpw8-rcwv-8f8p 🚨 HIGH 4.1.96.Final 4.1.100.Final
io.netty:netty-handler CVE-2025-24970 🚨 HIGH 4.1.96.Final 4.1.118.Final
net.minidev:json-smart CVE-2021-31684 🚨 HIGH 1.3.2 1.3.3, 2.4.4
net.minidev:json-smart CVE-2023-1370 🚨 HIGH 1.3.2 2.4.9
org.apache.avro:avro CVE-2024-47561 🔥 CRITICAL 1.7.7 1.11.4
org.apache.avro:avro CVE-2023-39410 🚨 HIGH 1.7.7 1.11.3
org.apache.derby:derby CVE-2022-46337 🔥 CRITICAL 10.14.2.0 10.14.3, 10.15.2.1, 10.16.1.2, 10.17.1.0
org.apache.ivy:ivy CVE-2022-46751 🚨 HIGH 2.5.1 2.5.2
org.apache.mesos:mesos CVE-2018-1330 🚨 HIGH 1.4.3 1.6.0
org.apache.thrift:libthrift CVE-2019-0205 🚨 HIGH 0.12.0 0.13.0
org.apache.thrift:libthrift CVE-2020-13949 🚨 HIGH 0.12.0 0.14.0
org.apache.zookeeper:zookeeper CVE-2023-44981 🔥 CRITICAL 3.6.3 3.7.2, 3.8.3, 3.9.1
org.eclipse.jetty:jetty-server CVE-2024-13009 🚨 HIGH 9.4.56.v20240826 9.4.57.v20241219
org.lz4:lz4-java CVE-2025-12183 🚨 HIGH 1.8.0 1.8.1

🛡️ TRIVY SCAN RESULT 🛡️

Target: Node.js

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: Python

Vulnerabilities (9)

Package Vulnerability ID Severity Installed Version Fixed Version
Werkzeug CVE-2024-34069 🚨 HIGH 2.2.3 3.0.3
aiohttp CVE-2025-69223 🚨 HIGH 3.12.12 3.13.3
aiohttp CVE-2025-69223 🚨 HIGH 3.13.2 3.13.3
deepdiff CVE-2025-58367 🔥 CRITICAL 7.0.1 8.6.1
ray CVE-2025-62593 🔥 CRITICAL 2.47.1 2.52.0
starlette CVE-2025-62727 🚨 HIGH 0.48.0 0.49.1
urllib3 CVE-2025-66418 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2025-66471 🚨 HIGH 1.26.20 2.6.0
urllib3 CVE-2026-21441 🚨 HIGH 1.26.20 2.6.3

🛡️ TRIVY SCAN RESULT 🛡️

Target: /etc/ssl/private/ssl-cert-snakeoil.key

No Vulnerabilities Found

🛡️ TRIVY SCAN RESULT 🛡️

Target: /home/airflow/openmetadata-airflow-apis/openmetadata_managed_apis.egg-info/PKG-INFO

No Vulnerabilities Found

@ulixius9 ulixius9 self-assigned this Jan 8, 2026
@gitar-bot
Copy link

gitar-bot bot commented Jan 8, 2026 •
edited
Loading

Code Review ✅ Approved

Clean bug fix for CockroachDB hidden shard column handling with comprehensive test coverage.

What Works Well

Pre-compiled regex pattern at module level for efficient matching. Comprehensive test suite covering positive/negative pattern matching, single/multi-column PKs, and edge cases. Clear documentation explaining the CockroachDB-specific behavior and linking to official docs.

Options

Auto-apply is off Gitar will not commit updates to this branch.
Display: compact Hiding non-applicable rules.

Comment with these commands to change:

Auto-apply Compact 
gitar auto-apply:on         

gitar display:verbose

Was this helpful? React with 👍 / 👎 | This comment will update automatically (Docs)

@harshach harshach added the To release Will cherry-pick this PR into the release branch label Jan 8, 2026
@sonarqubecloud
Copy link

sonarqubecloud bot commented Jan 8, 2026

Quality Gate Passed Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

@ulixius9 ulixius9 merged commit 41d7d2b into main Jan 8, 2026
29 of 42 checks passed
@ulixius9 ulixius9 deleted the copilot/fix-metadata-agent-errors-cockroachdb branch January 8, 2026 20:26
@github-actions
Copy link
Contributor

github-actions bot commented Jan 8, 2026

Changes have been cherry-picked to the 1.11.5 branch.

github-actions bot pushed a commit that referenced this pull request Jan 8, 2026
@harshach @ulixius9
Fix CockroachDB ingestion for tables with hidden shard columns (#25089)
5756772 
* Initial plan

* Fix CockroachDB ingestion for tables with hidden shard columns

Co-authored-by: harshach <38649+harshach@users.noreply.github.com>

* Fix pyformat

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: harshach <38649+harshach@users.noreply.github.com>
Co-authored-by: Sriharsha Chintalapani <harshach@users.noreply.github.com>
Co-authored-by: ulixius9 <mayursingal9@gmail.com>
(cherry picked from commit 41d7d2b)
@gitar-bot gitar-bot bot mentioned this pull request Jan 9, 2026
Merged
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ulixius9 ulixius9 ulixius9 approved these changes

@harshach harshach Awaiting requested review from harshach

Assignees

@harshach harshach

@ulixius9 ulixius9

Copilot code review Copilot

Labels

safe to test Add this label to run secure Github workflows on PRs To release Will cherry-pick this PR into the release branch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Deal with metadata agent error messages for cockroachDB database service for hidden sharded indexes

3 participants

@ulixius9 @harshach