128 lines
3.8 KiB
Markdown
128 lines
3.8 KiB
Markdown
# RoleMesh Gateway
|
|
|
|
|
|
|
|
RoleMesh Gateway is a lightweight **OpenAI-compatible** API gateway for routing chat-completions requests to multiple
|
|
locally hosted LLM backends (e.g., `llama.cpp` `llama-server`) **by role** (planner, writer, coder, reviewer, …).
|
|
|
|
It is designed for **agentic workflows** that benefit from using different models for different steps, and for
|
|
deployments where **different machines host different models** (e.g., GPU box for fast inference, big RAM CPU box for large models).
|
|
|
|
## What you get
|
|
|
|
- OpenAI-compatible endpoints:
|
|
- `GET /v1/models`
|
|
- `POST /v1/chat/completions` (streaming and non-streaming)
|
|
- `GET /health` and `GET /ready`
|
|
- Model registry from `configs/models.yaml`
|
|
- Optional **node registration** so remote machines can announce role backends to the gateway
|
|
- Robust proxying with **explicit httpx timeouts** (no “hang forever”)
|
|
- Structured logging with request IDs
|
|
|
|
## Quick Start
|
|
|
|
This is the fastest path to a working local setup.
|
|
|
|
### 1. Install
|
|
|
|
```bash
|
|
python -m venv .venv
|
|
source .venv/bin/activate
|
|
pip install -e .
|
|
```
|
|
|
|
### 2. Start two OpenAI-compatible backends
|
|
|
|
Any backend that exposes `GET /v1/models` and `POST /v1/chat/completions` will work.
|
|
One practical option is `llamafile` in server mode:
|
|
|
|
```bash
|
|
llamafile --server -m /path/to/planner-model.gguf --host 127.0.0.1 --port 8011 --nobrowser
|
|
llamafile --server -m /path/to/writer-model.gguf --host 127.0.0.1 --port 8012 --nobrowser
|
|
```
|
|
|
|
### 3. Create a gateway config
|
|
|
|
```yaml
|
|
version: 1
|
|
default_model: planner
|
|
auth:
|
|
client_api_keys:
|
|
- "change-me-client-key"
|
|
models:
|
|
planner:
|
|
type: proxy
|
|
openai_model_name: planner
|
|
proxy_url: http://127.0.0.1:8011
|
|
defaults:
|
|
temperature: 0
|
|
max_tokens: 128
|
|
writer:
|
|
type: proxy
|
|
openai_model_name: writer
|
|
proxy_url: http://127.0.0.1:8012
|
|
defaults:
|
|
temperature: 0.6
|
|
max_tokens: 256
|
|
```
|
|
|
|
Save that as `configs/models.yaml`.
|
|
|
|
### 4. Run the gateway
|
|
|
|
```bash
|
|
ROLE_MESH_CONFIG=configs/models.yaml uvicorn rolemesh_gateway.main:app --host 127.0.0.1 --port 8000
|
|
```
|
|
|
|
### 5. Verify it
|
|
|
|
```bash
|
|
curl -sS http://127.0.0.1:8000/v1/models \
|
|
-H 'X-Api-Key: change-me-client-key'
|
|
```
|
|
|
|
```bash
|
|
curl -sS -X POST http://127.0.0.1:8000/v1/chat/completions \
|
|
-H 'Content-Type: application/json' \
|
|
-H 'X-Api-Key: change-me-client-key' \
|
|
-d '{
|
|
"model": "planner",
|
|
"messages": [{"role":"user","content":"Say hello in 3 words."}]
|
|
}'
|
|
```
|
|
|
|
If you prefer the provided example file, copy `configs/models.example.yaml` and adjust the `proxy_url` values.
|
|
|
|
## Multi-host (node registration)
|
|
|
|
If you want machines to host backends and “register” them dynamically, run a tiny node agent on each backend host
|
|
(or just call the registration endpoint from your own tooling).
|
|
|
|
- Gateway endpoint: `POST /v1/nodes/register`
|
|
- Node payload describes which **roles** it serves and the base URL to reach its OpenAI-compatible backend.
|
|
|
|
See: `docs/DEPLOYMENT.md` and `docs/CONFIG.md`.
|
|
|
|
## Status
|
|
|
|
This repository is a **preliminary scaffold**:
|
|
- Proxying to OpenAI-compatible upstreams works.
|
|
- Registration and load-selection are implemented (basic round-robin).
|
|
- API-key auth for clients and nodes is available.
|
|
- Persistence is basic JSON-backed state, not a full service registry.
|
|
|
|
## License
|
|
|
|
MIT. See `LICENSE`.
|
|
|
|
## Node Agent (per-host)
|
|
|
|
This repo also includes a **RoleMesh Node Agent** (`rolemesh-node-agent`) that can manage **persistent** `llama.cpp` servers (one per GPU) and report inventory/metrics back to the gateway.
|
|
|
|
- Sample config: `configs/node_agent.example.yaml`
|
|
- Docs: `docs/NODE_AGENT.md`
|
|
|
|
## Safe-by-default binding
|
|
|
|
Gateway and node-agent default to binding on `127.0.0.1` to avoid accidental exposure. Bind only to private/LAN or VPN interfaces and firewall ports if you need remote access.
|