Operational-Premise-Taxonomy/README.md

# OPT: Operational Premise Taxonomy for AI Systems

This repository collects:

- The LaTeX manuscript defining the **Operational Premise Taxonomy (OPT)** and the OPT‐Code convention.
- Prompt sets for classifying AI systems into OPT mechanisms using large language models (LLMs).
- A small Python library and scripts to run an end‐to‐end **Classifier → Evaluator → Adjudicator** pipeline.
- A hand‐annotated **gold test suite** of systems (backprop, GA, A*, rule‐based expert systems, PSO, AIS, etc.).
- Example JSONL/YAML **audit logs** for storing OPT classifications.

The core idea: classify AI implementations by their **operative mechanism** (learning, evolution, symbolic reasoning, probabilistic inference, search, control, swarm, or hybrids), while explicitly separating that from **execution details** (parallelism, pipelines, hardware).

---

## 1. Repository layout

```text
Operational-Premise-Taxonomy/
├── README.md              # This file
├── LICENSE
├── .gitignore
├── Makefile               # Top-level convenience targets
│
├── paper/                 # LaTeX sources for the OPT paper
│   ├── main.tex           # arXiv/general article format
│   ├── main_ieee.tex      # IEEE two-column wrapper
│   ├── main_acm.tex       # ACM-style wrapper
│   ├── main_kaobook.tex   # Book-style wrapper
│   ├── body_shared.tex    # Shared main content
│   ├── related-work.tex   # Related work section
│   ├── appendix_opt_prompts.tex
│   ├── appendix_prompt_minimal.tex
│   ├── appendix_prompt_maximal.tex
│   ├── appendix_prompt_evaluator.tex
│   ├── figures/           # TikZ/PGFPlots figures
│   │   ├── opt_radar_1.tikz
│   │   ├── opt_radar_2.tikz
│   │   └── opt_eval_pipeline.tikz
│   ├── Makefile           # Build main.pdf, IEEE/ACM variants
│   └── bib/
│       └── references.bib
│
├── prompts/               # Plain-text LLM prompts
│   ├── minimal_classifier_prompt.txt
│   ├── maximal_classifier_prompt.txt
│   ├── evaluator_prompt.txt
│   └── adjudicator_prompt.txt
│
├── opt_eval/              # Python library for OPT classification/evaluation
│   ├── __init__.py
│   ├── opt_prompts.py     # Utility to load prompt text
│   ├── opt_pipeline.py    # Data classes + run_pipeline + parsers
│   ├── model_client.py    # Abstraction over your local/remote LLM endpoint
│   ├── cli.py             # CLI entrypoint for simple use
│   └── tests/
│       ├── __init__.py
│       ├── test_parsers.py
│       ├── test_gold_suite.py
│       └── data/
│           ├── gold_opt.yaml
│           └── gold_opt.jsonl
│
├── data/
│   ├── gold/
│   │   ├── opt_gold.yaml   # Canonical gold test suite
│   │   └── opt_gold.jsonl
│   └── examples/
│       ├── opt_audit_example.jsonl
│       └── opt_audit_example.yaml
│
├── scripts/
│   ├── run_eval_pipeline.py
│   └── export_gold_to_jsonl.py
│
└── docs/
    ├── usage.md
    ├── schema_opt_audit.md
    └── model_notes_local_llm.md
````

---

## 2. Building the paper

The paper lives in `paper/` and is structured to support multiple venues (arXiv, IEEE, ACM, book‐style).

### Prerequisites

* A reasonably recent TeX Live (or MikTeX) with:

  * `pgfplots` (with `polar` library),
  * `newtxtext`, `newtxmath`,
  * `booktabs`, `longtable`, `framed`, `fancyvrb`, etc.
* `latexmk` and `make`.

### Typical build

From the repository root:

```bash
cd paper
make          # builds main.pdf by default
# Or explicitly:
make main.pdf

# For an IEEE variant:
make main_ieee.pdf

# For ACM:
make main_acm.pdf
```

If you run into font or pgfplots `compat` warnings, consult comments at the top of `main.tex` and `body_shared.tex` (we assume `\pgfplotsset{compat=1.18}` and `\usepackage{newtxtext,newtxmath}`).

---

## 3. Python OPT evaluation pipeline

The `opt_eval` package provides:

* Data classes for candidate classifications, evaluator results, and adjudications.
* Parsers for extracting OPT lines and rationales from LLM output.
* A `run_pipeline` function that wires together:

  * Classifier A and B,
  * Evaluator,
  * Adjudicator,
  * and returns a structured result suitable for JSONL/YAML logging.

### 3.1 Installation (local dev)

Option 1: editable install with `pip`:

```bash
python -m venv .venv
source .venv/bin/activate
pip install -U pip

pip install -e .
# or, if you don’t define setup.cfg/pyproject:
pip install pyyaml
```

Option 2: just use it in-place with `PYTHONPATH`:

```bash
export PYTHONPATH=$PWD
```

### 3.2 Configuring a local LLM

You must implement `opt_eval/model_client.py` to talk to your model(s). A typical pattern:

* For an OpenAI-compatible HTTP endpoint (local or remote), use `requests` or `openai` client.
* For **Ollama** or **llamafile**, call `http://localhost:11434` or similar.

`model_client.call_model(system_prompt, user_content, model="local-llm")` should:

1. Send `system_prompt` as the system role (if your API supports it).
2. Send `user_content` as the user content.
3. Return the raw text content of the model’s reply.

Once implemented, you can run the pipeline on a simple description.

---

## 4. Quickstart: Running the evaluation pipeline

Minimal example (from repo root, after configuring `model_client.py`):

```bash
python scripts/run_eval_pipeline.py << 'EOF'
This system trains a fully-connected neural network on MNIST using SGD and
cross-entropy loss, and then uses the trained weights for inference only.
EOF
```

A typical JSON-like output will include:

* `candidate_a`, `candidate_b`
* `eval_a`, `eval_b`
* `final` (final OPT-Code and rationale)
* `adjudication` (if performed)

You can adapt `run_eval_pipeline.py` to write JSONL to `data/examples/opt_audit_example.jsonl`.

---

## 5. Gold test suite and benchmarking

The directory `data/gold/` contains a small hand‐annotated test suite (`opt_gold.yaml` and `opt_gold.jsonl`) covering:

* Backprop MLP on MNIST (Lrn),
* GA for TSP (Evo),
* A* gridworld planner (Sch),
* Rule-based expert system like XCON (Sym),
* Bayesian network for fault diagnosis (Prb),
* Deep Q-Network for Atari (Lrn),
* PID + Kalman filter drone control (Ctl),
* PSO for hyperparameter tuning (Swm),
* Immune negative-selection anomaly detection (Evo/Sch+Prb),
* Three-stage hybrid: GA → rule pruning → Bayesian classifier (Evo/Sym/Prb).

To run tests (after you’ve wired up `model_client.py`):

```bash
pytest opt_eval/tests
```

`test_gold_suite.py` will:

* Call the classifier prompt(s) on each gold description.
* Compare predicted OPT roots against the gold OPT‐Code.
* Optionally compute partial-match metrics (Jaccard similarity of root sets) and simple accuracy.

---

## 6. JSONL/YAML audit logs

For large-scale use, we recommend JSONL or YAML for storing evaluations.

* Example JSONL audit: `data/examples/opt_audit_example.jsonl`
* Example YAML audit: `data/examples/opt_audit_example.yaml`

Each record includes:

* `id`, `description`
* `candidates` (A, B)
* `evaluations` (verdicts, scores)
* `adjudication`
* `final` (final OPT-Code)
* `meta` (timestamps, model IDs, etc.)

See `docs/schema_opt_audit.md` for field descriptions.

---

## 7. Using smaller local LLMs

OPT classification needs:

* Understanding of code/algorithm descriptions.
* Solid instruction-following.
* Ability to respect a fairly structured output format.

Models that are feasible to run locally and are good candidates:

* **LLaMA 3 8B Instruct**
  Good general reasoning and code understanding; works well as Classifier, Evaluator, and Adjudicator if VRAM allows.

* **Mistral 7B Instruct** (and compatible fine-tunes like Dolphin, OpenHermes)
  Strong general-purpose local model with solid coding and instruction-following; good as a classifier.

* **Qwen2 7B / 14B Instruct**
  7B is a capable all-rounder; 14B (if you can run it) is strong for the evaluator/adjudicator roles.

* **Phi-3-mini (3.8B) Instruct**
  Smaller footprint; may work as a classifier on simpler cases. For nuanced hybrid systems (Evo/Sym/Prb, Swm vs Evo, Ctl vs Prb), you may want a larger model as evaluator/adjudicator.

A reasonable starting configuration:

* Classifier A: `llama3-8b-instruct`
* Classifier B: `mistral-7b-instruct`
* Evaluator: `qwen2-14b-instruct` (if available) or `llama3-8b-instruct`
* Adjudicator: same as Evaluator

You can also run all roles on the same 7–8B model if resources are constrained; the explicit prompts and the evaluator rubric are designed to catch many misclassifications.

See `docs/model_notes_local_llm.md` for more detailed notes on deployment options (Ollama, llamafile, vLLM, etc.) and recommended quantization levels.

---

## 8. Citing

Once the OPT paper is on arXiv or accepted somewhere, include a BibTeX entry like:

```bibtex
@article{Elsberry_OPT_2025,
  author  = {Wesley R. Elsberry and N.~Collaborators},
  title   = {Operational Premise Taxonomy (OPT): Mechanism-Level Classification of AI Systems},
  journal = {arXiv preprint},
  year    = {2025},
  eprint  = {XXXX.YYYYY},
  archivePrefix = {arXiv}
}
```

(Replace with the actual venue and identifier when available.)

---

## 9. Contributing

* Extend the gold test suite (YAML + JSONL) with more systems and hybrids.
* Add additional prompts (e.g., language-specific variants for Python-only code, RL-specific prompts).
* Improve the parsing logic or add better metrics (confusion matrices, root-wise F1).
* Open issues for any misclassifications that recur: they can inform future revisions of prompts and possibly the taxonomy itself.

Pull requests that add well-documented examples, tests, or tooling around OPT are welcome.

```