105 lines
2.4 KiB
Markdown
105 lines
2.4 KiB
Markdown
# Configuration
|
|
|
|
RoleMesh Gateway loads configuration from a YAML file (default: `configs/models.yaml`).
|
|
Set `ROLE_MESH_CONFIG` to override.
|
|
|
|
## Top-level schema
|
|
|
|
```yaml
|
|
version: 1
|
|
default_model: writer
|
|
gateway:
|
|
host: 0.0.0.0
|
|
port: 8000
|
|
auth:
|
|
client_api_keys: ["..."]
|
|
node_api_keys: ["..."]
|
|
models:
|
|
<alias>:
|
|
type: proxy | discovered
|
|
openai_model_name: <string>
|
|
...
|
|
```
|
|
|
|
- `<alias>` is what clients pass as `model` in `/v1/chat/completions`.
|
|
- `openai_model_name` is the model id returned by `/v1/models` (usually same as alias).
|
|
|
|
## Proxy models
|
|
|
|
Route to a fixed upstream (any host reachable from the gateway):
|
|
|
|
```yaml
|
|
models:
|
|
writer:
|
|
type: proxy
|
|
openai_model_name: writer
|
|
proxy_url: http://127.0.0.1:8012
|
|
defaults:
|
|
temperature: 0.6
|
|
```
|
|
|
|
Notes:
|
|
- The model alias (`writer` above) is what the client sends in `model`.
|
|
- `openai_model_name` is what the gateway returns from `GET /v1/models`.
|
|
- `proxy_url` is the actual upstream backend to call.
|
|
- `defaults` are only applied when the incoming request does not already set those keys.
|
|
|
|
## Discovered models
|
|
|
|
Route to a dynamically registered node that claims the role:
|
|
|
|
```yaml
|
|
models:
|
|
reviewer:
|
|
type: discovered
|
|
openai_model_name: reviewer
|
|
role: reviewer
|
|
strategy: round_robin
|
|
```
|
|
|
|
### Registering nodes
|
|
|
|
Nodes register to `POST /v1/nodes/register`:
|
|
|
|
```json
|
|
{
|
|
"node_id": "gpu-box-1",
|
|
"base_url": "http://10.0.0.12:8014",
|
|
"roles": ["reviewer", "planner"],
|
|
"meta": {"gpu": "Tesla P40", "notes": "llama-server on GPU0"}
|
|
}
|
|
```
|
|
|
|
If `auth.client_api_keys` is set (non-empty), callers of `/v1/models` and `/v1/chat/completions` must provide an API key.
|
|
|
|
If `auth.node_api_keys` is set (non-empty), node agents calling `/v1/nodes/register` and `/v1/nodes/heartbeat` must provide a node key.
|
|
|
|
Supported headers:
|
|
- Clients: `Authorization: Bearer <key>` or `X-Api-Key: <key>`
|
|
- Nodes: `Authorization: Bearer <node_key>` or `X-RoleMesh-Node-Key: <node_key>`
|
|
|
|
## Quick example
|
|
|
|
```yaml
|
|
version: 1
|
|
default_model: planner
|
|
auth:
|
|
client_api_keys: ["change-me-client-key"]
|
|
node_api_keys: ["change-me-node-key"]
|
|
models:
|
|
planner:
|
|
type: proxy
|
|
openai_model_name: planner
|
|
proxy_url: http://127.0.0.1:8011
|
|
defaults:
|
|
temperature: 0
|
|
max_tokens: 128
|
|
writer:
|
|
type: proxy
|
|
openai_model_name: writer
|
|
proxy_url: http://127.0.0.1:8012
|
|
defaults:
|
|
temperature: 0.6
|
|
max_tokens: 256
|
|
```
|