Feature/async storage implementation with SQLite / PostgreSQL

[fancyboi999]

Feature: Async Implementation for Persistent Storage Configuration

This PR implements asynchronous persistent storage functionality, addressing the TODO item in the codebase to make DeerFlow compatible with SQLite and PostgreSQL storage backends.

Changes Made

• Implemented support for SQLite and PostgreSQL storage options in src/graph/builder.py
• Added connection pool configuration for PostgreSQL to optimize performance
• Updated src/server/app.py to utilize async storage interfaces
• Enhanced documentation in docs/configuration_guide.md with storage configuration details

Testing

• All tests pass: make test ✓
• Code follows style guidelines: make lint and make format ✓
• Test coverage checked: make coverage ✓

Documentation

The configuration guide has been updated with detailed instructions for the new storage options:

STORAGE:
  # Memory storage (default)
  # type: memory
  
  # SQLite storage
  # type: sqlite
  # db_path: checkpoints.sqlite
  
  # PostgreSQL storage
  type: postgres
  db_uri: "host=localhost port=5432 dbname=deerflow user=postgres password=password"
  pool:
    min_size: 1     
    max_size: 5      
    timeout: 30      
    max_idle: 300    
    max_lifetime: 3600  
    name: "pgpool"   

Code Standards

• Code follows PEP 8 guidelines
• Added appropriate type hints
• Included descriptive docstrings
• Maintained single-purpose functions and methods
• Added comments for complex logic

Related TODO Item

This PR resolves the TODO item in src/graph/builder.py:

"""Build and return the agent workflow graph with memory."""
# use persistent memory to save conversation history
# TODO: be compatible with SQLite / PostgreSQL
memory = MemorySaver()

By implementing the requested storage backends and maintaining compatibility with the existing code.

[CLAassistant]

All committers have signed the CLA.

[hetaoBackend]

Hey @fancyboi999 , thanks for contributing the code, glad to see the persistence layer will be integrated. However, could u plz extract the storage implementation logic into a separate file, so the logic could be used in multiple graphs in Deerflow project. cc: @MagicCube

[fancyboi999]

Hi @hetaoBackend , @MagicCube
Thanks for the feedback! I've refactored the code and moved the storage implementation logic into a separate file as suggested. Please help review when you get a chance 🙏
All test already passed
Let me know if there's anything else that needs adjustment!

[chenxingwei-wayne]

e use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:06,793 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:06,808 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:08,770 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:08,819 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:12,734 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:12,853 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:20,643 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:20,904 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())'
2025-05-20 14:19:25,793 - src.config.storage - ERROR - Error setting up PostgreSQL connection: couldn't get a connection after 20.00 sec
2025-05-20 14:19:25,798 - src.config.storage - ERROR - Traceback: Traceback (most recent call last):
File "D:\code\deer-flow\src\config\storage.py", line 108, in _create_checkpointer_from_config
async with pool.connection() as conn:
File "C:\python3\Lib\contextlib.py", line 210, in aenter
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "D:\code\deer-flow.venv\Lib\site-packages\psycopg_pool\pool_async.py", line 195, in connection
conn = await self.getconn(timeout=timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\code\deer-flow.venv\Lib\site-packages\psycopg_pool\pool_async.py", line 230, in getconn
raise PoolTimeout(
psycopg_pool.PoolTimeout: couldn't get a connection after 20.00 sec 我自己测试有报错

[chenxingwei-wayne]

e use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:06,793 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:06,808 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:08,770 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:08,819 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:12,734 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:12,853 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:20,643 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:20,904 - psycopg.pool - WARNING - error connecting in 'pgpool': Psycopg cannot use the 'ProactorEventLoop' to run in async mode. Please use a compatible event loop, for instance by setting 'asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())' 2025-05-20 14:19:25,793 - src.config.storage - ERROR - Error setting up PostgreSQL connection: couldn't get a connection after 20.00 sec 2025-05-20 14:19:25,798 - src.config.storage - ERROR - Traceback: Traceback (most recent call last): File "D:\code\deer-flow\src\config\storage.py", line 108, in _create_checkpointer_from_config async with pool.connection() as conn: File "C:\python3\Lib\contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "D:\code\deer-flow.venv\Lib\site-packages\psycopg_pool\pool_async.py", line 195, in connection conn = await self.getconn(timeout=timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "D:\code\deer-flow.venv\Lib\site-packages\psycopg_pool\pool_async.py", line 230, in getconn raise PoolTimeout( psycopg_pool.PoolTimeout: couldn't get a connection after 20.00 sec 我自己测试有报错

@fancyboi999 有时间可以看下吗

[fancyboi999]

@chenxingwei-wayne 感谢反馈！我看了 pg 插件和程序的事件循环在 Windows 上的兼容性问题，确认在 Mac 上可以正常运行。为了解决 Windows 上的事件循环问题，建议在以下文件的首行添加以下代码：

在 server.py、storage.py 和 main.py 文件的首行加入：

import platform
import asyncio

# 为 Windows 平台设置正确的事件循环策略
if platform.system() == "Windows":
    from asyncio import WindowsSelectorEventLoopPolicy
    asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())

[chenxingwei-wayne]

@chenxingwei-wayne 感谢反馈！我看了 pg 插件和程序的事件循环在 Windows 上的兼容性问题，确认在 Mac 上可以正常运行。为了解决 Windows 上的事件循环问题，建议在以下文件的首行添加以下代码：

在 server.py、storage.py 和 main.py 文件的首行加入：

import platform
import asyncio

# 为 Windows 平台设置正确的事件循环策略
if platform.system() == "Windows":
    from asyncio import WindowsSelectorEventLoopPolicy
    asyncio.set_event_loop_policy(WindowsSelectorEventLoopPolicy())

@fancyboi999 根据你提供的方法已经解决了，多谢。