SynapsD

A very simple, naive implementation of a JSON document store with some bitmap indexes in the mix. Module primarily but not exclusively for use with Canvas (https://github.com/canvas-ai/canvas-server)

Architecture

Components

• LMDB, to-be-replaced by pouchdb or rxdb as the main KV backend (https://www.npmjs.com/package/lmdb)
• Compressed (roaring) bitmaps (https://www.npmjs.com/package/roaring)
• FlexSearch for full-text search (https://www.npmjs.com/package/flexsearch)
• LanceDB (https://www.npmjs.com/package/@lancedb/lancedb)

JSON Document Store

• Simple LMDB KV store with enforced document schemas (See ./src/schemas for more details)
• Every data abstraction schema (File, Note, Browser tab, Email etc) defines its own set of indexing options

Index implementation

Hashmaps / Inverted indexes

• algorithm/checksum | docID
  Example: sha1/4e1243.. => document ID)
• timestamp | docID
  Example: 20250212082411.1234 => document ID
  We could use composite keys and LMDB range queries instead (timestamp/docID => document) but for now this way is more practical.

Bitmap indexes

The following bitmap index prefixes are enforced to organize and filter documents:

• internal/ - Internal bitmaps
• context/ - Context path bitmaps, used internally by Canvas (as context tree nodes, context/uuid)
• data/abstraction/<schema> - Schema type filters (incl subtrees like data/abstraction/file/ext/json)
• data/mime/<type>
• data/content/encoding/<encoding>
• client/os/<os>
• client/application/<application>
• client/device/<device-id>
• client/network/<network-id> -We support storing documents on multiple backends(StoreD), when a canvas application connects from a certain network, not all backends may be reachable(your home NAS from work for example)
• user/
• tag/ - Generic tag bitmaps
• custom/ - Throw what you need here

API Documentation

API Patterns

SynapsD follows a hybrid API pattern - for better or worse -

Single Document Operations

Single document operations:

try {
    const docId = await db.insertDocument(doc);
    // Success case
} catch (error) {
    // Error handling
}

Available single document operations:

• insertDocument(doc, contextSpec, featureBitmapArray)
• updateDocument(docId, updateData, contextSpec, featureBitmapArray)
• deleteDocument(docId)
• getDocument(docId)
• getDocumentById(id)
• getDocumentByChecksumString(checksumString)

Batch Operations

Batch operations return a result object:

const result = await db.insertDocumentArray(docs);
if (result.failed.length > 0) {
    // Handle partial failures
}
// Access successful operations
const successfulIds = result.successful.map(s => s.id);

// Using findDocuments
const result = await db.findDocuments('/some/path', ['feature1'], ['filter1']);
if (result.error) {
    console.error('Query failed:', result.error);
} else {
    console.log(`Found ${result.count} documents:`, result.data);
}

// Using query
const queryResult = await db.query('some query', ['context1'], ['feature1']);
if (queryResult.error) {
    console.error('Query failed:', queryResult.error);
} else {
    console.log(`Found ${queryResult.count} documents:`, queryResult.data);
}

Result object structure:

interface BatchResult {
    successful: Array<{
        index: number;    // Original array index
        id: number;       // Document ID
    }>;
    failed: Array<{
        index: number;    // Original array index
        error: string;    // Error message
        doc: any;        // Original document
    }>;
    total: number;       // Total number of operations
}

Available batch operations:

• insertDocumentArray(docs, contextSpec, featureBitmapArray)
• updateDocumentArray(docs, contextSpec, featureBitmapArray)
• deleteDocumentArray(docIds)
• getDocumentsByIdArray(ids)
• getDocumentsByChecksumStringArray(checksums)

Query Operations

Pagination is supported for all queries. Default page size is 100 documents.

Options:

• limit (number): page size (default 100)
• offset (number): starting index (default 0)
• page (number): 1-based page number (ignored if offset provided)
• parse (boolean): parse into schema instances (default true)

Usage:

// First page (implicit): limit=100, offset=0
const docs = await db.findDocuments(contextSpec, featureBitmapArray, [], { limit: 100 });
console.log(docs.length, docs.count); // docs has .count metadata

// Second page via page
const page2 = await db.findDocuments(contextSpec, featureBitmapArray, [], { page: 2, limit: 100 });

// Or using offset directly
const next100 = await db.findDocuments(contextSpec, featureBitmapArray, [], { offset: 100, limit: 100 });

Return shape:

type QueryResultArray = Array<any> & { count: number; error: string | null };

Available query operations:

• findDocuments(contextSpec, featureBitmapArray, filterArray, options)
• query(query, contextBitmapArray, featureBitmapArray, filterArray)
• ftsQuery(query, contextBitmapArray, featureBitmapArray, filterArray)

Error Handling

SynapsD uses standard JavaScript Error objects with specific error types:

• ValidationError: Document validation failed
• NotFoundError: Document not found
• DuplicateError: Document already exists
• DatabaseError: General database errors

Example:

try {
    await db.insertDocument(doc);
} catch (error) {
    if (error instanceof ValidationError) {
        // Handle validation error
    } else if (error instanceof DatabaseError) {
        // Handle database error
    }
}

References

• LMDB Documentation
• Node.js Crypto Documentation
• Roaring Bitmaps
• LlamaIndex
• FlexSearch
• LanceDB
• Why-not-indices

License

Licensed under AGPL-3.0-or-later. See main project LICENSE file.

---

This project is funded by Augmentd Labs