Cortex.cpp: API Server

[dan-menlo]

Goal

• Cortex.cpp runs in the background, exposing an Open-AI compatible API server at port ? (similar to Docker Daemon)
• Other programs can connect to it as an AI provider

Key Decisions

• Drogon or non-drogon?

Current Architecture

• Port number?

Related

• discussion: https library #1080
• feat: set up HTTPS for Custom Models #1097
• bug: Download models/engine failed cause api server stop #1010 (some timeout issue)
• feat: Cortex supports custom API Keys #1005
• https://github.com/janhq/cortex.cpp/issues/1059
• feat: Allow transforming sse chunk response #568
• idea: Cortex.cpp supports simple Authorization #551
• idea: Production Level Queue System #580

[dan-menlo]

@nguyenhoangthuan99 @vansangpfiev I recall you guys making some decisions on Drogon vs. non-Drogon a few weeks ago. Can you help me summarize:

• What was the decision about?
• Why did we eventually choose to stay with Drogon?

I can refactor them into the Spec above, once you guys provide me sufficient detail

[vansangpfiev]

@nguyenhoangthuan99 @vansangpfiev I recall you guys making some decisions on Drogon vs. non-Drogon a few weeks ago. Can you help me summarize:

• What was the decision about?
• Why did we eventually choose to stay with Drogon?

I can refactor them into the Spec above, once you guys provide me sufficient detail

We decided to stay with Drogon because:

• It supports 'non-blocking' socket I/O which can avoid the bottle-neck when the number of requests increase (blocking I/O fro httplib)
• It supports epoll which has better performance than select/poll (which httplib uses)
• We solved the openssl compilation issue