From 82a7f790cb06fc6349d681f647221997423ea605 Mon Sep 17 00:00:00 2001 From: alltheseas Date: Thu, 23 Oct 2025 12:09:30 -0500 Subject: [PATCH] docs: document embedding workflow --- README.md | 236 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 235 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 89bdb318a..0c9b1bb0b 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,6 @@ The unfairly fast nostr database backed by lmdb. nostrdb stores nostr events as a custom in-memory representation that enables zero-copy and O(1) access to all note fields. This is similar to flatbuffers but it is custom built for nostr events. - These events are then memory-mapped inside lmdb, enabling insanely fast, zero-copy access and querying. @@ -18,12 +17,242 @@ as a C library into any application with full nostr query support. [1]: https://github.com/hoytech/strfry +## Getting Started (Embedding) + +``` +git clone https://github.com/damus-io/nostrdb.git +cd nostrdb +git submodule update --init --recursive +make lib ndb +``` + +The build generates `libnostrdb.a` alongside the `ndb` CLI, reusing headers under +`src/` and vendored dependencies in `deps/`. Link your application against the +static archive and the libraries nostrdb relies on: + +``` +cc app.c libnostrdb.a \ + -Isrc -Ideps/lmdb -Ideps/secp256k1/include \ + -llmdb -lsecp256k1 -lpthread -lzstd +``` + +`/var/lib/nostrdb` is a convenient default path for the LMDB environment, but any +empty directory will do: + +``` +mkdir -p /var/lib/nostrdb +``` ## API The API is *very* unstable. nostrdb is in heavy development mode so don't expect any of the interfaces to be stable at this time. +### Minimal C Example + +```c +#include "nostrdb.h" + +int main(void) { + struct ndb *db = NULL; + struct ndb_config cfg; + + ndb_default_config(&cfg); + ndb_config_set_flags(&cfg, NDB_FLAG_SKIP_NOTE_VERIFY); + + if (ndb_init(&db, "/var/lib/nostrdb", &cfg) != 0) + return 1; + + const char *ldjson = + "{\"id\":\"...\",\"pubkey\":\"...\",\"kind\":1,\"content\":\"hello\"," + "\"created_at\":1700000000,\"tags\":[],\"sig\":\"...\"}\n"; + + if (ndb_process_events(db, ldjson, strlen(ldjson)) != 0) + goto cleanup; + + struct ndb_txn txn; + if (ndb_begin_query(db, &txn) != 0) + goto cleanup; + + struct ndb_filter filter; + unsigned char filter_buf[4096]; + + ndb_filter_init(&filter); + ndb_filter_start_field(&filter, NDB_FILTER_KINDS); + ndb_filter_add_int_element(&filter, 1); + ndb_filter_end_field(&filter); + ndb_filter_end(&filter); + + struct ndb_query_result results[8]; + int count = 0; + + if (ndb_query(&txn, &filter, 1, results, 8, &count) == 0) { + for (int i = 0; i < count; i++) { + struct ndb_note *note = results[i].note; + printf("created_at=%u content=%.*s\n", + ndb_note_created_at(note), + ndb_note_content_length(note), + ndb_note_content(note)); + } + } + + ndb_filter_destroy(&filter); + ndb_end_query(&txn); + +cleanup: + ndb_destroy(db); + return 0; +} +``` + +### Configuration + +`struct ndb_config` lets you tune ingest throughput, disk usage, and extensions. + +| Setter | Purpose | +| --- | --- | +| `ndb_default_config` | Populate defaults (signature verification, fulltext, note blocks, 2MB writer scratch buffer, 16 GiB mapsize). | +| `ndb_config_set_flags` | Toggle runtime features: skip verification (`NDB_FLAG_SKIP_NOTE_VERIFY`), disable fulltext (`NDB_FLAG_NO_FULLTEXT`), disable parsed blocks (`NDB_FLAG_NO_NOTE_BLOCKS`), keep stats off (`NDB_FLAG_NO_STATS`), and more. | +| `ndb_config_set_ingest_threads` | Control the number of concurrent JSON ingest workers. | +| `ndb_config_set_writer_scratch_buffer_size` | Grow beyond 2MB when storing large events; shrink to save RAM when ingesting tiny notes. | +| `ndb_config_set_mapsize` | Increase the LMDB map when expecting many events (must be a multiple of the OS page size). | +| `ndb_config_set_ingest_filter` | Register a callback that accepts, rejects, or skips validation for each parsed note. | +| `ndb_config_set_subscription_callback` | Receive `ndb_subscribe` notifications when queried events arrive. | + +### Event Ingestion + +Use `ndb_process_event` for single JSON events or `ndb_process_events` for +line-delimited JSON (LDJSON) blobs. Both parse, verify (unless disabled), and +store notes into LMDB. `ndb_process_event_with` and +`ndb_process_events_with` accept an `ndb_ingest_meta` struct so you can track the +relay or client that supplied the data. On Unix-like systems, +`ndb_process_events_stream` streams from `FILE *` descriptors. + +When ingesting trusted data, `NDB_FLAG_SKIP_NOTE_VERIFY` avoids signature +checks. Provide `ndb_ingest_filter` hooks to veto malformed or undesired events +before they commit to disk. + +### Structured Queries + +Queries execute inside a read transaction (`struct ndb_txn`) that you open with +`ndb_begin_query` and close with `ndb_end_query`. Filters describe the NIP-01 +constraints to apply. + +```c +struct ndb_filter filter; +unsigned char buf[4096]; + +ndb_filter_init(&filter); +ndb_filter_start_field(&filter, NDB_FILTER_IDS); +ndb_filter_add_id_element(&filter, some_id32); +ndb_filter_end_field(&filter); + +ndb_filter_start_tag_field(&filter, 't'); +ndb_filter_add_str_element(&filter, "nosql"); +ndb_filter_end_field(&filter); + +ndb_filter_end(&filter); +``` + +Populate filters manually, reuse them across transactions, or call +`ndb_filter_from_json` to load raw JSON filter objects using a caller-supplied +buffer. Execute the query via `ndb_query(txn, filters, num_filters, results, +capacity, &count)`; results reference in-memory notes valid until the +transaction ends. + +### Fulltext Search + +nostrdb maintains an LMDB-backed inverted index (if fulltext is enabled) that +`ndb_text_search` and `ndb_text_search_with` query. Configure ordering and result +limits with `ndb_default_text_search_config`, `ndb_text_search_config_set_order`, +and `ndb_text_search_config_set_limit`. When needed, pass a filter to +`ndb_text_search_with` to combine keyword search with structured constraints. + +### Reading Data + +The note accessor helpers expose raw content, metadata, and parsed structures: + +* `ndb_note_content`, `ndb_note_content_length` +* `ndb_note_created_at`, `ndb_note_kind` +* `ndb_note_id`, `ndb_note_pubkey`, `ndb_note_sig` +* `ndb_note_tags`, `ndb_tags_iterate_start`, `ndb_tags_iterate_next`, + `ndb_iter_tag_str` +* `ndb_note_relay_iterate_start`, `ndb_note_relay_iterate_next`, + `ndb_note_relay_iterate_close` + +Parsed block data provides structured hashtags, mentions, URLs, and invoices: + +```c +struct ndb_blocks *blocks = + ndb_get_blocks_by_key(ndb, &txn, results[i].note_id); + +if (blocks) { + struct ndb_block_iterator it; + ndb_blocks_iterate_start(ndb_note_content(note), blocks, &it); + + struct ndb_block *block; + while ((block = ndb_blocks_iterate_next(&it))) { + if (ndb_get_block_type(block) == BLOCK_HASHTAG) { + struct ndb_str_block *tag = ndb_block_str(block); + printf("#%.*s\n", (int)ndb_str_block_len(tag), ndb_str_block_ptr(tag)); + } + } + + ndb_blocks_free(blocks); +} +``` + +Disable block extraction entirely with `NDB_FLAG_NO_NOTE_BLOCKS` when the feature +is unused. + +### Creating Events + +`struct ndb_builder` helps construct, sign, and ingest new notes: + +```c +unsigned char buf[2048]; +struct ndb_builder builder; +struct ndb_note *note = NULL; + +ndb_builder_init(&builder, buf, sizeof(buf)); +ndb_builder_set_pubkey(&builder, pubkey32); +ndb_builder_set_kind(&builder, 1); +ndb_builder_set_created_at(&builder, time(NULL)); +ndb_builder_set_content(&builder, "hello nostrdb", -1); + +ndb_builder_new_tag(&builder); +ndb_builder_push_tag_str(&builder, "p", -1); +ndb_builder_push_tag_id(&builder, other_pubkey32); + +struct ndb_keypair keypair; +ndb_decode_key("nsec1...", &keypair); +ndb_builder_finalize(&builder, ¬e, &keypair); + +ndb_process_event(db, (const char *)note, builder.note_cur.size); +``` + +Supporting functions include `ndb_calculate_id`, `ndb_sign_id`, +`ndb_create_keypair`, and `ndb_note_verify`, which wrap the bundled secp256k1 +implementation. + +### Subscriptions + +Long-lived consumers can call `ndb_subscribe` to register filters and receive +updates. Deliveries arrive through either `ndb_wait_for_notes` (blocking) or +`ndb_poll_for_notes` (non-blocking), returning note IDs for follow-up queries. +Couple this with `ndb_config_set_subscription_callback` to inject per-note hooks. + +### Language Bindings + +`make bindings` regenerates higher level bindings using `flatc` and `flatcc`: + +* `src/bindings/c/*` +* `src/bindings/rust/ndb_profile.rs`, `src/bindings/rust/ndb_meta.rs` +* `src/bindings/swift/NdbProfile.swift`, `src/bindings/swift/NdbMeta.swift` + +Include these generated files in your host project. Regenerate them whenever the +schemas in `schemas/` change. + ## CLI nostrdb comes with a handy `ndb` command line tool for interacting with nostrdb @@ -66,3 +295,8 @@ $ ndb search --limit 2 --oldest-first 'nosy ostrich' Q: What do you call a nosy ostrich? A: A nosTrich! ``` + +## Development + +Run `make test` for sanitizer-enabled tests. The repository vendors LMDB, +secp256k1, flatcc, and CCAN modules under `deps/` to keep reproducible builds.