Skip to content

feat: Major CafeDB enhancement - production-ready with advanced features#1

Open
shubhayu-dev wants to merge 1 commit intoCrystallineCore:mainfrom
shubhayu-dev:main
Open

feat: Major CafeDB enhancement - production-ready with advanced features#1
shubhayu-dev wants to merge 1 commit intoCrystallineCore:mainfrom
shubhayu-dev:main

Conversation

@shubhayu-dev
Copy link
Copy Markdown

BREAKING CHANGES:

  • Custom exceptions replace generic ValueError (TableNotFoundError, TableExistsError, QueryError)
  • Automatic metadata injection (_inserted_at, _updated_at timestamps)

NEW FEATURES:

  • Thread-safe operations with RLock for concurrent access
  • Transaction context manager with automatic rollback on errors
  • Batch insert via insert_many() for efficient bulk operations
  • Advanced select() with sorting (order_by, reverse), pagination (limit, offset), and field projection
  • Automatic backup creation (.backup file) before writes (configurable)
  • New clear_table() to empty tables without dropping
  • New info() method for database-wide metadata and statistics
  • Enhanced stats() with field percentages, min/max/avg for numeric fields, and size estimation
  • Support for operator to check field presence

IMPROVEMENTS:

  • Atomic writes with crash-safe temp file mechanism
  • Conditional writes (only write if data changed) in update/delete
  • Unicode support with ensure_ascii=False in JSON serialization
  • Table name validation (no underscore prefix - reserved for internal)
  • Alphabetically sorted table lists
  • Comprehensive error messages with helpful suggestions and available options
  • Type validation on all inputs with descriptive error messages
  • Better operator validation with full list of valid operators in errors
  • Graceful handling of type mismatches in comparisons
  • Version tracking in metadata

DOCUMENTATION:

  • Comprehensive docstrings for all public methods
  • Type hints throughout
  • Detailed parameter descriptions and return values
  • Examples in docstrings

PERFORMANCE:

  • Single write per insert_many() vs N writes for N inserts
  • Skip disk writes when no rows affected in update/delete
  • Efficient sorting with fallback for mixed types

This update transforms CafeDB from a simple prototype into a production-ready database suitable for small to medium applications with proper error handling, thread safety, audit trails, and data protection mechanisms.

BREAKING CHANGES:
- Custom exceptions replace generic ValueError (TableNotFoundError, TableExistsError, QueryError)
- Automatic metadata injection (_inserted_at, _updated_at timestamps)

NEW FEATURES:
- Thread-safe operations with RLock for concurrent access
- Transaction context manager with automatic rollback on errors
- Batch insert via insert_many() for efficient bulk operations
- Advanced select() with sorting (order_by, reverse), pagination (limit, offset), and field projection
- Automatic backup creation (.backup file) before writes (configurable)
- New clear_table() to empty tables without dropping
- New info() method for database-wide metadata and statistics
- Enhanced stats() with field percentages, min/max/avg for numeric fields, and size estimation
- Support for  operator to check field presence

IMPROVEMENTS:
- Atomic writes with crash-safe temp file mechanism
- Conditional writes (only write if data changed) in update/delete
- Unicode support with ensure_ascii=False in JSON serialization
- Table name validation (no underscore prefix - reserved for internal)
- Alphabetically sorted table lists
- Comprehensive error messages with helpful suggestions and available options
- Type validation on all inputs with descriptive error messages
- Better operator validation with full list of valid operators in errors
- Graceful handling of type mismatches in comparisons
- Version tracking in metadata

DOCUMENTATION:
- Comprehensive docstrings for all public methods
- Type hints throughout
- Detailed parameter descriptions and return values
- Examples in docstrings

PERFORMANCE:
- Single write per insert_many() vs N writes for N inserts
- Skip disk writes when no rows affected in update/delete
- Efficient sorting with fallback for mixed types

This update transforms CafeDB from a simple prototype into a production-ready
database suitable for small to medium applications with proper error handling,
thread safety, audit trails, and data protection mechanisms.
@shubhayu-dev
Copy link
Copy Markdown
Author

Quick Overview of Changes by Function

Core Infrastructure

__init__

  • Added: backup parameter, _lock for threading, version tracking

_read_db

  • Added: try/except with custom exceptions instead of crashes

_write_db

  • Added: thread lock, automatic backup creation, ensure_ascii=False, last_modified tracking, error handling

_match_condition

  • Added: try/except around comparisons, $exists operator, better error messages

_build_condition_function

  • Added: special handling for $exists operator

New Functions

transaction() - NEW

  • Context manager for atomic operations with rollback

insert_many() - NEW

  • Batch insert, single disk write instead of N writes

clear_table() - NEW

  • Empty table without dropping it

info() - NEW

  • Database-wide metadata and statistics

CRUD Operations

create_table

  • Added: thread lock, custom exception, table name validation, sorted list, better errors

drop_table

  • Added: thread lock, custom exception, shows available tables, reports row count

insert

  • Added: thread lock, type validation, automatic _inserted_at timestamp, better errors

select

  • Added: thread lock, 5 new parameters (fields, limit, offset, order_by, reverse), better errors

update

  • Added: thread lock, automatic _updated_at timestamp, conditional write (only if changed)

delete

  • Added: thread lock, conditional write (only if changed)

Introspection

stats

  • Added: thread lock, percentage calculations, min/max/avg for numbers, size estimation in bytes/KB

list_tables, exists_table, count

  • No changes (already good)

Summary in 3 Points:

  1. Thread safety everywhere (all functions wrapped in locks)
  2. Better errors (custom exceptions, helpful messages)
  3. More features (sorting, pagination, projections, batch ops, transactions, timestamps, backups)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant