some first efforts at trying to document this repo #429

rdhyee · 2025-11-14T15:09:35Z

No description provided.

…plan This commit adds three new documentation files to improve repository orientation: 1. REPOSITORY_OVERVIEW.md - Comprehensive guide covering: - Project purpose and architecture overview - Key components and directory structure - Getting started instructions - Data flow diagrams - API endpoints and CLI tools - Development workflow and testing - Deployment considerations - Common tasks and examples 2. docs/geoparquet_export_findings.md - Investigation report on GeoParquet export code: - Detailed search methodology - Current export capabilities (CSV/JSONL only) - Analysis of export service architecture - Hypotheses about GeoParquet file creation - Recommendations for locating/recreating export code 3. docs/geoparquet_to_pqg_conversion_plan.md - Detailed conversion plan: - Overview of PQG (Property Graph in DuckDB) format - iSamples data model analysis - Graph model design with node types and relationships - Complete implementation plan with Python code examples - Phase-by-phase execution steps - Testing and validation strategy - Performance considerations and query examples These documents provide essential orientation for new developers and document the path forward for converting iSamples data to the PQG graph format.

MAJOR UPDATE: Found the GeoParquet export code in export_client repository! This commit adds complete documentation of the actual GeoParquet export implementation and updates all existing documentation to reference it. New file: - docs/geoparquet_export_code.md - Comprehensive documentation of: * Location: https://github.com/rdhyee/export_client * Implementation: geoparquet_utilities.py conversion code * Architecture: Server exports JSONL → Client converts to GeoParquet * Dependencies: pandas, geopandas, geoarrow-pyarrow * Usage examples with the 'isample' CLI tool * Complete code walkthrough and schema documentation * How the Zenodo file was created Updated files: - docs/geoparquet_export_findings.md: * Added prominent "CODE FOUND" notice at top * Confirmed original hypothesis was correct (external repository) * Added reference to new geoparquet_export_code.md documentation - docs/geoparquet_to_pqg_conversion_plan.md: * Added note about export_client at top of document * Clarified how GeoParquet files are actually created * Referenced geoparquet_utilities.py implementation - REPOSITORY_OVERVIEW.md: * Added "Related Repositories" section documenting export_client * Expanded export documentation with two options: 1. Export Client (recommended for GeoParquet) 2. Direct API (CSV/JSONL only) * Clarified that GeoParquet is only available via export_client * Added links to other iSamples repositories (metadata, vocabularies, pqg) Key findings: - GeoParquet export is client-side, not server-side - export_client provides CLI tool with GeoParquet, CSV, JSONL support - Uses pandas + geopandas to convert JSONL → GeoParquet - Automatically generates STAC metadata catalogs - Supports ORCID authentication and local web viewer This resolves the investigation and provides complete path forward for working with iSamples GeoParquet exports and PQG conversion.

Copilot

Pull Request Overview

This pull request adds comprehensive documentation to the iSamples in a Box repository, including a repository overview, conversion plans for GeoParquet to PQG format, and documentation of the GeoParquet export implementation.

Key changes:

Four new documentation files covering repository structure, data conversion strategies, and export implementations
Detailed technical documentation with code examples and architectural diagrams
Cross-referenced documentation linking related concepts across files

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 10 comments.

File	Description
REPOSITORY_OVERVIEW.md	Comprehensive repository guide covering architecture, setup, development workflow, and common tasks
docs/geoparquet_to_pqg_conversion_plan.md	Detailed conversion plan from GeoParquet format to PQG property graph format with implementation code
docs/geoparquet_export_findings.md	Investigation findings documenting the location of GeoParquet export code in a separate repository
docs/geoparquet_export_code.md	Technical documentation of the GeoParquet export implementation, architecture, and usage examples

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-14T15:13:03Z

docs/geoparquet_to_pqg_conversion_plan.md

+    python scripts/convert.py \\
+        --input isamples_export_2025_04_21_16_23_46_geo.parquet \\
+        --output isamples_graph.duckdb \\
+        --export-geojson samples.geojson \\


Inconsistent backslash usage in bash commands. Line 630 uses \\ for line continuation while lines 840, 849 show single \. In markdown code blocks, use single backslash \ for line continuation in bash commands.

Suggested change

python scripts/convert.py \\

--input isamples_export_2025_04_21_16_23_46_geo.parquet \\

--output isamples_graph.duckdb \\

--export-geojson samples.geojson \\

python scripts/convert.py \

--input isamples_export_2025_04_21_16_23_46_geo.parquet \

--output isamples_graph.duckdb \

--export-geojson samples.geojson \

Copilot · 2025-11-14T15:13:04Z

docs/geoparquet_to_pqg_conversion_plan.md

+    args = parser.parse_args()
+
+    # 1. Load GeoParquet
+    print(f"Loading GeoParquet from {args.input}...")


Typo in "GeoParqet" should be "GeoParquet".

Copilot · 2025-11-14T15:13:04Z

docs/geoparquet_to_pqg_conversion_plan.md

+python scripts/convert.py \\
+    --input isamples_export_2025_04_21_16_23_46_geo.parquet \\
+    --output test_graph.duckdb \\
+    --limit 100 \\
+    --verbose
+```
+
+4. **Run full conversion:**
+```bash
+python scripts/convert.py \\
+    --input isamples_export_2025_04_21_16_23_46_geo.parquet \\
+    --output isamples_full_graph.duckdb \\
+    --export-geojson isamples_sites.geojson \\


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

Suggested change

python scripts/convert.py \\

--input isamples_export_2025_04_21_16_23_46_geo.parquet \\

--output test_graph.duckdb \\

--limit 100 \\

--verbose

```

4. **Run full conversion:**

```bash

python scripts/convert.py \\

--input isamples_export_2025_04_21_16_23_46_geo.parquet \\

--output isamples_full_graph.duckdb \\

--export-geojson isamples_sites.geojson \\

python scripts/convert.py \

--input isamples_export_2025_04_21_16_23_46_geo.parquet \

--output test_graph.duckdb \

--limit 100 \

--verbose

Run full conversion:

python scripts/convert.py \ --input isamples_export_2025_04_21_16_23_46_geo.parquet \ --output isamples_full_graph.duckdb \ --export-geojson isamples_sites.geojson \

Copilot · 2025-11-14T15:13:04Z

docs/geoparquet_to_pqg_conversion_plan.md

+python scripts/convert.py \\
+    --input isamples_export_2025_04_21_16_23_46_geo.parquet \\
+    --output test_graph.duckdb \\
+    --limit 100 \\
+    --verbose
+```
+
+4. **Run full conversion:**
+```bash
+python scripts/convert.py \\
+    --input isamples_export_2025_04_21_16_23_46_geo.parquet \\
+    --output isamples_full_graph.duckdb \\
+    --export-geojson isamples_sites.geojson \\


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

Suggested change

python scripts/convert.py \\

--input isamples_export_2025_04_21_16_23_46_geo.parquet \\

--output test_graph.duckdb \\

--limit 100 \\

--verbose

```

4. **Run full conversion:**

```bash

python scripts/convert.py \\

--input isamples_export_2025_04_21_16_23_46_geo.parquet \\

--output isamples_full_graph.duckdb \\

--export-geojson isamples_sites.geojson \\

python scripts/convert.py \

--input isamples_export_2025_04_21_16_23_46_geo.parquet \

--output test_graph.duckdb \

--limit 100 \

--verbose

Run full conversion:

python scripts/convert.py \ --input isamples_export_2025_04_21_16_23_46_geo.parquet \ --output isamples_full_graph.duckdb \ --export-geojson isamples_sites.geojson \

Copilot · 2025-11-14T15:13:04Z

REPOSITORY_OVERVIEW.md

+curl -H "Authorization: Bearer <JWT>" \
+  "https://central.isample.xyz/isamples_central/export/create?q=source:SESAR&export_format=jsonl"


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

Copilot · 2025-11-14T15:13:05Z

REPOSITORY_OVERVIEW.md

+curl -H "Authorization: Bearer <JWT>" \
+  "https://central.isample.xyz/isamples_central/export/status?uuid=..."


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

Copilot · 2025-11-14T15:13:05Z

REPOSITORY_OVERVIEW.md

+curl "http://localhost:8000/solr/search?q=keywords:geology&rows=10"
+


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

Copilot · 2025-11-14T15:13:05Z

docs/geoparquet_to_pqg_conversion_plan.md

+                except:
+                    pass


Empty except block without any exception handling or logging. This bare except: will silently catch and ignore all exceptions, including system exits and keyboard interrupts, which can make debugging difficult. Consider either:

Catching specific exceptions (e.g., except json.JSONDecodeError:)

Adding logging to track when parsing fails

Re-raising the exception if it can't be handled

Copilot · 2025-11-14T15:13:05Z

docs/geoparquet_to_pqg_conversion_plan.md

+                except:
+                    pass


Empty except block without any exception handling or logging. This bare except: will silently catch and ignore all exceptions. Consider either:

Catching specific exceptions (e.g., except json.JSONDecodeError:)

Adding logging to track when parsing fails

Re-raising the exception if it can't be handled

Copilot · 2025-11-14T15:13:06Z

REPOSITORY_OVERVIEW.md

+curl -H "Authorization: Bearer <JWT>" \
+  "https://central.isample.xyz/isamples_central/export/download?uuid=..."


Inconsistent backslash usage in bash commands. Should use single backslash \ for line continuation in bash commands, not double backslash \\.

claude added 2 commits November 14, 2025 14:53

rdhyee requested a review from Copilot November 14, 2025 15:09

Copilot started reviewing on behalf of rdhyee November 14, 2025 15:10 View session

Copilot finished reviewing on behalf of rdhyee November 14, 2025 15:11

Copilot AI reviewed Nov 14, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

some first efforts at trying to document this repo #429

some first efforts at trying to document this repo #429

rdhyee commented Nov 14, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Copilot AI Nov 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		curl -H "Authorization: Bearer <JWT>" \
		"https://central.isample.xyz/isamples_central/export/create?q=source:SESAR&export_format=jsonl"

		curl "http://localhost:8000/solr/search?q=keywords:geology&rows=10"

some first efforts at trying to document this repo #429

Are you sure you want to change the base?

some first efforts at trying to document this repo #429

Conversation

rdhyee commented Nov 14, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants