EcoSpecies-Atlas/docs/roadmap.md

# EcoSpecies Modernization Roadmap

## Current Status

As of 2026-03-27, the repo is no longer at the pure planning stage. The following pieces are already implemented and working in the live stack:

- Docker Compose deployment with explicit `ecospecies-...` container names
- path-based hosting support for `/apps/ecospecies`
- in-repo-only source directory resolution with safe path validation
- legacy SLH ingest into PostgreSQL-backed species, sections, citations, audit, and document records
- editor/admin workflows for draft, review, publish, archive, and audit history
- contributor registration and draft-authoring workflow with token-based access
- structured Markdown document storage and editor/API round-trip
- persisted taxon identifier scaffolding with legacy identifiers separated from future-facing external identifiers
- citation extraction, review, enrichment, batch enrichment, candidate matching, and reviewed-candidate selection/addition
- citation persistence back into the structured Markdown source of truth

The roadmap below has been updated to reflect that actual state.

## Target Product

Create a Docker Compose-based, open-source EcoSpecies successor that:

- ingests legacy SLH text files and future species submissions
- exposes a stable API for species, sections, citations, and ecological linkages
- provides a responsive public web app
- supports researcher/editor workflows for curation and publishing
- generates exports aligned with legacy reporting needs and future FLELMR-style outputs

## Recommended Stack

### Core platform

- Backend: Python API service
- Primary datastore: PostgreSQL
- Search/indexing: PostgreSQL full-text initially, optional Meilisearch/OpenSearch later
- Frontend: static SPA or React-based client once requirements stabilize
- Deployment/runtime: Docker Compose for development and small-scale deployment

### Why this stack

- permissive licenses
- strong support for text ingestion, APIs, and data processing
- easy local development
- clear path from prototype to production

## Product Capabilities By Phase

### Phase 0: Discovery and migration planning

Status: completed

- Inventory legacy assets and user-facing capabilities.
- Capture the replacement architecture and ingestion strategy.
- Define acknowledgements, provenance, and licensing boundaries.

### Phase 1: Ingestion foundation

Status: substantially complete, with parser refinement ongoing

- Parse legacy `.txt` SLH inputs into structured JSON records.
- Normalize common metadata: title, scientific name, common name, FLELMR/EcoSpecies code, headings, references.
- Create ingest diagnostics to flag malformed files and missing metadata.
- Continue parser refinement for legacy edge cases in headings, citations, and historical bibliography formats.

### Phase 2: Public read experience

Status: implemented baseline

- Species listing and search.
- Species detail view with section navigation.
- Provenance and acknowledgement display.
- Summary metrics on corpus coverage.
- Path-based deployment under `/apps/ecospecies`.

### Phase 3: Structured persistence and editorial workflow

Status: implemented baseline, with editor UX still maturing

- PostgreSQL-backed persistence for species, sections, citations, documents, taxon identifiers, and audit history.
- Editor-safe import jobs and audit metadata.
- Raw-source preservation alongside normalized records.
- Authentication and role-based access for admin/editor/contributor workflows.
- Persisted editorial workflow state for draft, review, published, and archived records.
- Structured Markdown document storage and round-trip editing.
- Citation review, enrichment, candidate selection, and reviewed-candidate addition.
- Contributor draft creation and owner-scoped editing.

### Phase 4: Standards-aware identity and bibliography

Status: partially implemented

- Preserve legacy local identifiers as provenance.
- Persist taxon identifiers separately from legacy identifiers.
- Expose `legacy_identifiers`, `taxon_identifiers`, and `primary_taxon_*` API fields.
- Persist structured citation records with DOI/OpenAlex/DataCite-style enrichment fields.
- Continue toward multi-authority identifier review, richer citation entities, and CiteGeist-backed bibliography expansion.

### Phase 5: Editor ergonomics and advanced review

Status: in progress

- Structured Markdown editor is live.
- Citation match-review dialog is live.
- Remaining work:
  - CodeMirror-based Markdown editor with folding
  - inline parser diagnostics in the editor
  - richer citation diff/review affordances
  - clearer document-node and citation provenance in the UI

### Phase 6: Linkages and visualization

Status: not started

- Model predator/prey, habitat, and ecological association edges.
- Add graph endpoints and species-relationship views.
- Support public-friendly visual explanations and expert filters.

### Phase 7: Reports and export

Status: partially implemented

- JSON and Markdown exports exist through the API/document model.
- Structured Markdown is now the primary human-readable editor/export format.
- Remaining work:
  - recreate legacy-like text/RTF export
  - support export profiles for legacy compatibility and standards-forward outputs
  - improve citation/bibliography export fidelity

### Phase 8: Assisted research workflows

Status: planned

- Add local-LLM-assisted extraction and drafting in a human-review loop.
- Integrate bibliography tooling for citation consolidation and topic expansion.
- Support candidate-species intake for records not yet in the historical corpus.
- Restrict assisted drafting and publication actions to authenticated editorial roles.

## Data Model Direction

Initial core entities:

- `species`
- `source_document`
- `document_section`
- `citation`
- `taxon_identifier`
- `citation_identifier`
- `bibliography_topic`
- `taxon`
- `linkage`
- `media_asset`
- `ingest_run`

Key design rules:

- preserve raw source text
- retain provenance and import timestamps
- separate public published records from draft/editor states
- make sections addressable for citation and graph linking
- prefer a canonical document AST over direct projection from free-form source text

## LLM Extension Strategy

Use local models only for assistive tasks, never silent publication:

- extracting candidate structured fields from new SLH text
- suggesting missing headings or linkage labels
- clustering similar citations
- resolving bibliography entries toward DOI/OpenAlex/DataCite where available
- treating local legacy codes as provenance, not canonical identifiers
- drafting summaries for editor review

Guardrails:

- raw text remains authoritative
- all generated content is marked as draft
- every automated extraction stores source spans where possible

## Near-Term Priorities

1. Add CodeMirror-based folding and structure-aware editing to the Markdown document editor.
2. Expand taxon identifier review workflows for WoRMS, GBIF, Catalogue of Life, and related authorities.
3. Deepen citation quality controls, including better parsed-field visibility and stricter/manual review loops where resolver confidence is weak.
4. Add CiteGeist-style topic expansion and bibliography-suggestion review for under-cited species.
5. Improve document export fidelity so reviewed citations and standards-based identifiers are clearly represented in Markdown and downstream exports.
6. Begin the first ecological-linkage data model and API endpoints once citation/identifier workflows stabilize.

## Definition Of Done For The Initial Milestone

- `docker compose up` starts a working API and frontend.
- The system can enumerate the legacy corpus and show parsed species detail for real SLH files.
- Editors can curate structured Markdown documents and citations through authenticated workflows.
- Contributors can register, create drafts, and edit only their own submissions.
- Project docs describe both the implemented modernization state and the next phases.