AI Operational Intelligence Prototype — SRS pack

This page assembles sections relevant to a software requirements specification: context and problem through security, quality, and operations.

Contents

• Summary
• Context and Problem
• Goals, Requirements, and Constraints
• Role and Responsibilities
• System Model
• Architecture and Integrations
• Security, Quality, and Operations

Summary

Status

Technical PoC / working prototype, June 2026

Role

System Designer, AI-assisted Prototype Engineer

Stack

Type: Enterprise AI / controlled LLM execution prototype / evidence-backed analytics prototype

Python, FastAPI, LangGraph, Open WebUI, PostgreSQL, Qdrant, MinIO, Redis, Docker Compose

Project value

A technical PoC of controlled LLM execution for evidence-backed analytics. The prototype explores how an executive analytics assistant could be built: the LLM does not answer freely from memory and does not get direct access to data. It operates inside a backend-mediated tool environment over prepared synthetic scenarios.

The prototype validated the architectural idea on single-turn analytical requests. It is not a production platform, not a complete decision-support product, and not a multi-turn conversational agent.

What was implemented

• chat-like UI based on Open WebUI;
• experimental LangGraph-based execution flow;
• backend tools for accessing prepared synthetic data;
• initial tool registry / tool description concept;
• synthetic financial and cross-functional management scenarios;
• single-turn analytical requests;
• evidence-backed response pattern;
• basic execution trace / run details;
• architectural documentation and future direction.

Current limitations

The most important limitation: each new message in Open WebUI was effectively processed as a new independent request rather than continuation of the same analytical session.

What this demonstrates

• understanding of enterprise AI risks;
• controlled LLM execution instead of free chat;
• separation of chat UI and execution layer;
• tool-mediated analytics;
• evidence-backed response design;
• execution trace as a trust/debugging mechanism;
• ability to build a working prototype quickly;
• ability to honestly document limitations.

Context and Problem

Context

Enterprise AI can help with synthesis, but a free-form chat over corporate data is a weak and unsafe model for managerial analytics.

In executive questions, the answer text is not enough. It also matters:

• which data the conclusion rests on;
• which calculation was performed;
• which documents were used;
• which limitations the analysis has;
• what is a fact and what is a hypothesis;
• why this diagnostic path was chosen.

This prototype explores how LLM-based managerial analytics could be made controlled, traceable, and useful for enterprise discussion.

Problem

Typical executive analytics still requires manual assembly of a picture from BI reports, spreadsheets, task trackers, ITSM, PMO materials, documents, meeting notes, and domain experts.

LLMs can speed up synthesis, but unconstrained chat over corporate data creates risks:

• hallucination;
• wrong source selection;
• no reproducible calculation;
• no audit path for the conclusion;
• mixing facts and interpretation;
• unsafe model access to data.

The prototype was built to check a narrower idea: the LLM should not “answer from memory” and should not get direct access to business data. It should operate inside a controlled execution loop: select an allowed tool path, call backend tools, retrieve structured results, preserve a basic execution trace, and produce an evidence-backed answer.

The intended product direction remains an evidence-backed decision-support prototype for executives and domain owners. That direction was not delivered as a complete product in this work.

Goals, Requirements, and Constraints

Goals and Non-Goals

Primary project goals

• Check whether an LLM can work with prepared enterprise traces through controlled backend tools, not through direct data access.
• Build a reproducible lab stand with synthetic enterprise data.
• Implement an experimental vertical slice: user request → controlled execution flow → tool selection → backend tool call → structured result → evidence-backed answer → execution trace.
• Show an approach to managerial diagnostics where conclusions rest on facts, documents, calculations, and explicit analysis limits.
• Document an architectural direction for a future product: backend control plane, Tool Gateway, Tool Registry, playbook-based scenarios, semantic layer, audit trail.

What this PoC is not

• Not a real MVP.
• Not a production-ready enterprise platform.
• Not a complete multi-turn conversational agent.
• Not an autonomous "AI executive".
• Not a BI replacement.
• Not a complete process-mining product.
• Not a production authentication and authorization model.
• Not a set of real enterprise data connectors.
• Not a complete playbook engine.
• Not an evaluation pipeline for answer quality.
• Not a production deployment model.
• Not a full observability and audit model.

What was implemented

• chat-like UI based on Open WebUI;
• experimental LangGraph-based execution flow;
• backend tools for accessing prepared synthetic data;
• initial tool registry / tool description concept;
• synthetic financial and cross-functional management scenarios;
• single-turn analytical requests;
• evidence-backed response pattern;
• basic execution trace / run details;
• architectural documentation and future direction.

The prototype also explored a playbook-routing concept: a question could be directed into a bounded diagnostic path with a limited tool set. This was a prototype playbook concept, not a complete playbook engine.

What was not implemented

• full multi-turn session state;
• continuation of an analytical scenario after a clarification question;
• correct handling of the user’s answer to system clarification;
• durable conversation memory;
• persisted analytical run context;
• production authentication and authorization;
• real enterprise data connectors;
• complete playbook engine;
• evaluation pipeline for answer quality;
• production deployment model;
• full observability and audit model.

Current limitations

The most important limitation: each new message in Open WebUI was effectively processed as a new independent request rather than continuation of the same analytical session.

Even when the prototype asked a clarification question such as “which scenario did you mean?”, the next user answer was processed as a new independent request, not as continuation of the previous run.

The limitation was intentionally documented. The prototype validated a single-turn controlled execution idea; it did not deliver a conversational analytics product.

Requirements

The items below describe the PoC intent. They should not be read as a claim that every requirement was fully implemented.

• BR-001. Evidence-backed managerial analytics The prototype should investigate a managerial question and return a conclusion grounded in calculations, documents, tool results, and explicit analysis limits.

• BR-002. Faster first-pass analysis The prototype should reduce the time of a first-pass hypothesis check by routing the request, calling tools, collecting evidence, and forming a structured answer.

• BR-003. Transparency instead of opaque AI The user should see not only the final text, but also the basis of the conclusion: selected diagnostic path, tool calls, parameters, results, sources, and limits.

• BR-004. Safe data access model The LLM should not receive direct access to databases, documents, or arbitrary SQL.

• BR-005. Domain diagnostic paths Analysis should be framed as bounded diagnostic paths, not as free chat over the full tool catalog.

• BR-006. Demonstration without client data The PoC should demonstrate the approach on a synthetic enterprise dataset.

• BR-007. Extensibility direction New analysis domains should be addable through tools, tool descriptions, and diagnostic paths rather than one giant prompt.

• BR-008. Future enterprise constraints Architectural choices should leave room for later audit, RBAC/ACL, controlled integrations, private deployment, and workflow portability. These were not implemented in the PoC.

Functional requirements

• FR-001. Chat-like interface for an analytical question The user should be able to ask a free-form managerial question through a chat-like UI.

• FR-002. Diagnostic path selection The prototype should determine the domain of the question and select a bounded diagnostic path, or ask a clarification. Clarification was explored, but the user’s answer was not handled as continuation of the same run.

• FR-003. Tool restriction by diagnostic path A selected path should limit the available tools so the LLM does not work with the full catalog at once.

• FR-004. Data access only through the tool-server The LLM should receive data through controlled backend tools.

• FR-005. Structured tool results The tool-server should return structured JSON with result, metadata, status, and enough technical detail for a run log.

• FR-006. Financial metric calculation Financial metrics should be calculated from structured database data, not generated by the LLM.

• FR-007. RAG as document evidence direction Documents should be usable as additional evidence through a MinIO/Qdrant/RAG contour.

• FR-008. Structured analytical answer The final answer should include facts, interpretation, hypotheses, analysis limits, and recommended actions where possible.

• FR-009. Run transparency The prototype should expose the selected diagnostic path, tool calls, inputs, outputs, and basic run details.

• FR-010. Same-language response The prototype should answer in the same language as the user question.

• FR-012. Clarifying question If the question is incomplete, too broad, or missing critical parameters, the prototype may ask for the missing information. This was not a working multi-turn loop: the next message was treated as a new request.

• FR-013. Reproducible synthetic data import The lab stand should restore demo state from the repository: PostgreSQL seed, MinIO objects, Qdrant artifacts, manifests, and scripts.

• FR-014. Several connected synthetic domains The demo dataset should contain linked domains: finance, sales, products, documents, delivery, ITSM, PMO, meetings, and tasks.

Constraints

• CON-001. Verifiable conclusions A material conclusion should be linked to a tool result, document, calculation, or an explicit limit.

• CON-002. Controlled data access The LLM should not get direct access to PostgreSQL, Qdrant, MinIO, the filesystem, or arbitrary SQL.

• CON-003. Execution traceability A run should preserve the selected diagnostic path, tool calls, parameters, results, and the final answer at a basic level.

• CON-004. Synthetic data only The PoC uses synthetic data. Real client data is not connected.

• CON-005. Limited PoC scale The PoC is sized for demo scenarios and architectural validation, not production load, mass users, or SLA.

• CON-006. LLM usage A local LLM is a target for a future product. The PoC may use an external LLM and should remain adaptable to model replacement.

Assumptions

• The prototype works on a synthetic dataset, not on live client data.
• Some cross-domain scenarios are demonstrative and need further stabilization.
• The RAG contour shows a document-evidence direction; it is not a mature ingestion/lifecycle solution.
• Tool Registry and related factory ideas are early: part implemented, part described architecturally.
• The UI is a temporary chat-like harness, not a target executive cockpit.
• The prototype does not prove production readiness. It proves that a controlled LLM analytics loop can be assembled and demonstrated on synthetic data.

Role and Responsibilities

Role

System Designer, AI-assisted Prototype Engineer.

This was prototype architecture work: translating an enterprise AI idea into a constrained lab stand, not production architecture ownership.

Scope of work

• translated the product idea into a prototype architecture model;
• formulated the controlled LLM execution concept;
• chose a temporary lab stack for the PoC;
• designed the synthetic dataset;
• designed a prototype playbook / diagnostic-path approach;
• designed the controlled tool-access model (see Security and Access Model);
• implemented and evolved the initial Tool Registry concept (see Architecture);
• prepared synthetic enterprise demo data (see Domain Model);
• developed the evidence and transparency approach (see Integration principles);
• configured Open WebUI as a temporary chat-like interface (see Trade-offs);
• documented limitations, including the absence of a working multi-turn session loop;
• shaped a future direction toward reports, signal cards, evidence views, and backend-native orchestration. These items were not implemented.

Work performed

• designed the prototype architecture: chat-like harness, experimental execution flow, diagnostic-path routing concept, tool layer, synthetic data layer, document-evidence direction, and execution trace;
• built a laboratory FastAPI + LangGraph runtime for experimental single-request diagnostic flows;
• established the principle that the LLM does not query PostgreSQL, Qdrant, or MinIO directly;
• implemented / laid down controlled tool execution through a FastAPI tool-server;
• prepared a synthetic dataset for a fashion/retail/manufacturing company;
• expanded the dataset toward cross-domain diagnostics: finance, delivery, ITSM, PMO, meetings, documents;
• recorded the target direction without presenting it as delivered scope.

Use of AI

The project was developed with AI-assisted prototyping.

LLMs were used to speed up:

• draft code generation;
• synthetic data preparation;
• prompt and process-logic drafts;
• architecture option analysis;
• documentation;
• test scenario drafts.

Key decisions remained under manual control:

• architectural boundaries;
• data-access model;
• lab versus target runtime;
• verifiability requirements;
• synthetic dataset structure;
• meaning of diagnostic paths;
• PoC limits;
• result review;
• project positioning.

System Model

Domain Model

Main entities

Entity	Role
Executive User	Asks an analytical question and receives a prototype answer
Domain Owner	Intended reviewer of conclusions in a domain: finance, delivery, PMO, ITSM
Platform Admin	Target-system role for sources, access, diagnostic paths, and tools; not implemented in the PoC
Diagnostic path / playbook concept	Bounds the diagnostic process, available tools, and domain frame
Tool	Controlled operation over data: metric, RAG, aggregated result
Diagnostic Run	One analysis run with run details, selected path, tool history, and final answer
Evidence Item	Fact, calculation, document chunk, or limitation linked to a conclusion
Claim	Statement in the answer that should refer to evidence

A diagnostic run in this prototype is a single-turn analytical request. The model does not describe a durable multi-turn conversation.

Synthetic enterprise dataset

Prepared synthetic enterprise demo data for a fashion/retail/manufacturing company, conceptually extended toward FashionCo Group / fashionco-enterprise.

Company domain:

• premium apparel / small-to-medium manufacturing;
• B2B / B2B2C through distributors, boutiques, showrooms, and marketplace partners;
• data period: 2024-2025;
• business domains linked in one synthetic enterprise contour.

Connected business domains:

Domain	Content
core	customers, products, sales orders, order items
crm	companies, contacts, deals, activities, tasks
finance	invoices, payments, accounts receivable, COGS
production	production orders, operations, materials, supplier deliveries
documents	document objects, invoice files, metadata
rag	RAG documents and chunks
delivery	epics, tasks, transitions, rework, cycle time
itsm	incidents, SLA, affected services, business impact
pmo	roadmap items, milestones, slippage, status reports
meetings	decisions, action items, decision-to-action gaps
goals	KPI, objectives, ownership, conflicts — target/extension
semantic	metric definitions, business entities, calculation rules
system	dataset version, runtime metadata
eval	scenario truth, expected claims, forbidden claims — not exposed to normal tools

Data Model

Simplified ERD logic:

flowchart LR
    SalesOrder[core.sales_orders] --> SalesItem[core.sales_order_items]
    SalesOrder --> Invoice[finance.invoices]
    SalesOrder --> Payment[finance.payments]
    SalesOrder --> Deal[crm.deals]
    Deal --> Roadmap[pmo.roadmap_items]
    Roadmap --> Epic[delivery.epics]
    Epic --> Task[delivery.tasks]
    Task --> Transition[delivery.task_transitions]
    Roadmap --> Incident[itsm.incidents]
    Roadmap --> Decision[meetings.decisions]
    Decision --> Action[meetings.action_items]
    Document[documents.document_objects] --> Chunk[rag.rag_chunks]
    Chunk --> Roadmap
    Chunk --> Incident
    Chunk --> Decision

API Contracts

Agent endpoint

POST /agent/check-hypothesis
Content-Type: application/json

Example request:

{
  "question": "Why did gross margin drop in March 2025?",
  "hypothesis": "The margin drop is related to discounts",
  "context": {
    "period": "2025-03",
    "domain": "finance"
  }
}

Example response:

{
  "selected_playbook": "financial_operations",
  "verdict": "partially_supported",
  "final_answer": "...",
  "evidence": [
    {
      "type": "metric",
      "tool_id": "metric_gross_margin",
      "period": "2025-03",
      "summary": "gross margin decreased compared with baseline"
    }
  ],
  "tool_calls": [
    {
      "tool_id": "metric_gross_margin",
      "args": { "period": "2025-03" },
      "status": "ok"
    }
  ],
  "limitations": [
    "Analysis is based on synthetic dataset only"
  ]
}

The selected_playbook field reflects the prototype playbook-routing concept, not a complete playbook engine.

Tool-server health

GET /health

Purpose: tool-server availability check.

Gross margin tool

POST /tools/metric/gross-margin
Content-Type: application/json

Supported request forms:

{ "period": "2025-03" }

{ "start_date": "2025-02-01", "end_date": "2025-03-31", "group_by": ["month"] }

Calculation logic:

revenue = SUM(core.sales_orders.net_amount_rub)
cogs = SUM(core.sales_orders.cogs_amount_rub)
gross_margin = revenue - cogs
gross_margin_rate = gross_margin / revenue

RAG search tool

POST /tools/rag-search
Content-Type: application/json

Example request:

{
  "query": "Definition of Ready decision for the promo feature",
  "filters": {
    "domain": "delivery",
    "source_type": "meeting_minutes",
    "period": "2025-Q1"
  }
}

Expected response:

{
  "status": "ok",
  "results": [
    {
      "document_id": "DOC-PMO-2025-03-12",
      "chunk_id": "CHUNK-001",
      "title": "PMO weekly meeting notes",
      "score": 0.82,
      "object_key": "executive-demo-docs/pmo/2025-03-12.md",
      "snippet": "..."
    }
  ]
}

Architecture and Integrations

Architecture idea

The architecture describes a prototype flow, not a completed product architecture.

User request
→ controlled execution flow
→ tool selection
→ backend tool call
→ structured result
→ evidence-backed answer
→ execution trace

The LLM was intended to act inside a controlled backend-mediated tool environment, not as a free-form chatbot with direct arbitrary data access.

OpenWebUI
  → experimental LangGraph / FastAPI flow
  → tool selection / prototype playbook routing
  → Tool Registry concept
  → tool-server / Tool Gateway
  → PostgreSQL / Qdrant / MinIO
  → structured result
  → evidence-backed answer
  → execution trace

This is a single-request prototype loop. A second user message in Open WebUI was not treated as continuation of the same analytical session.

Context diagram

flowchart TB
    User[User]
    Prototype[AI Operational Intelligence Prototype]
    Synth[(Synthetic enterprise data)]
    Docs[Synthetic documents]
    LLM[OpenAI-compatible LLM]

    User --> Prototype
    Prototype --> Synth
    Prototype --> Docs
    Prototype --> LLM

The diagram shows the lab stand. Real ERP, ITSM, PMO, or document-system connectors were not implemented.

Tool Gateway pattern

All data access goes through controlled HTTP tools with explicit input contracts, validation, structured output, and metadata.

Prototype playbook concept

The prototype explored routing questions into domain-specific diagnostic paths instead of exposing all tools to the LLM at once.

Each domain was intended to expose a bounded set of allowed tools, diagnostic steps, constraints, and expected evidence. This was an experimental routing concept, not a complete playbook engine.

Tool Registry

Implemented and evolved an initial Tool Registry / tool-description concept as a machine-readable catalog of available tools, schemas, domains, and constraints.

One diagnostic run

sequenceDiagram
    autonumber
    participant U as User
    participant UI as OpenWebUI
    participant AG as experimental flow
    participant LLM as LLM
    participant TG as tool-server
    participant PG as PostgreSQL
    participant QD as Qdrant
    participant MN as MinIO

    U->>UI: Analytical question
    UI->>AG: POST /agent/check-hypothesis
    AG->>LLM: plan next diagnostic step
    LLM-->>AG: selected tool
    AG->>TG: controlled tool call
    TG->>PG: execute named query
    PG-->>TG: metric result
    TG-->>AG: structured result
    AG->>LLM: evaluate evidence
    LLM-->>AG: optional document evidence
    AG->>TG: rag_search
    TG->>QD: vector search with filters
    QD-->>TG: chunks + scores
    TG->>MN: resolve object refs
    MN-->>TG: source metadata
    TG-->>AG: document evidence
    AG->>LLM: synthesize answer with limitations
    AG-->>UI: final answer + run details
    UI-->>U: evidence-backed answer + execution trace

The sequence describes one request. It does not describe a durable conversational loop.

Integration principles

1. The LLM does not execute SQL.
2. The LLM does not read documents directly.
3. The LLM should not receive the full unrestricted tool list.
4. The backend / tool-server validates input parameters.
5. Tools return structured JSON, metadata, warnings, and status.
6. Evidence is linked to a tool call, document, period/entity, and claim where possible.
7. Debug visibility is available through run details and should not expose private chain-of-thought.

Evidence-first answers

Final responses are expected to rest on tool outputs, document evidence, calculations, or explicitly stated limitations.

Run trace as a trust layer

Each diagnostic run preserves, at a basic level, the selected diagnostic path, tool calls, parameters, outputs, and run details for debugging and discussion.

Evidence and transparency approach

The prototype explored an evidence and transparency pattern: selected diagnostic path, tool calls, parameters, tool results, execution timeline, run details, and JSON-level debug visibility.

Security, Quality, and Operations

Security and Access Model

Implemented in the PoC

• Only synthetic data is used.
• The LLM does not get direct access to PostgreSQL, Qdrant, or MinIO.
• Data access goes through controlled HTTP tools.
• Tools have explicit input contracts.
• Financial calculations are performed by named queries / backend logic, not by arbitrary SQL from the LLM.
• Run details can show the selected diagnostic path, tool calls, inputs, and outputs.

Target production model — not implemented

• SSO / IdP integration.
• RBAC / ABAC.
• Tenant isolation.
• ACL-aware RAG retrieval.
• Tool permissions by diagnostic path, user role, and domain.
• Read-only mode by default.
• Approval gates for write actions.
• Full audit log: run_id, user_id, tool_id, params hash, result hash, source refs.
• Secrets management.
• On-prem/private deployment option.

Controlled LLM execution

The LLM can plan the next step, but data access is delegated to controlled backend tools — not free chat over corporate data.

Controlled tool access model

The LLM never queries PostgreSQL, Qdrant, or MinIO directly.

Non-Functional Requirements

The PoC was not sized or proven against production NFR. The relevant lab constraints were:

• demonstrable single-request flow;
• traceability of tool calls;
• no direct model access to data stores;
• reproducible Docker Compose stand;
• synthetic data only.

Failure Modes

Failure mode	How it appeared	Notes
Wrong diagnostic-path routing	An operational question could go into a finance path	Clarification was explored, but the user’s answer was not continued as the same session
Unsupported question	Question outside dataset/tool coverage	Explicit limitation is preferable to a fabricated answer
Duplicate tool calls	Same tool called with the same parameters	Fingerprint tool_id + canonical_json(args), run-local cache
Stub/empty tool response	Tool returned no data or a stub	Status handling, warning, insufficient-evidence verdict
Missing RAG evidence	Document layer finds no confirmation	Explicit limitation: document evidence not found
Hallucinated conclusion	LLM states a conclusion without evidence	Evidence-first prompt; no evaluation pipeline was implemented
Incomplete cross-domain linkage	Metrics exist, but finance ↔ delivery ↔ ITSM is not proven	Needs a stronger semantic layer; not fully stabilized
External LLM unavailable	API down or rate-limited	Retry/backoff and local-model option are future work
Context overflow	Tool manifest/evidence too large	Context budget, summarization, retrieval filters — early
Data leakage risk	Model sees extra data	Tool-level permissions and no direct data access; no production ACL

Sizing and Cost Notes

The current PoC is sized for demonstration:

• 1-3 concurrent users;
• a few diagnostic runs during a demo;
• synthetic dataset for 2024-2025;
• tens/hundreds of thousands of rows at most in lab data;
• one run usually expected to stay within 1-10 tool calls;
• cost is driven by LLM API calls and Docker/VPS/local infrastructure;
• production sizing was not performed.

Production would need separate estimates for source volume, document corpus, ingestion frequency, users, RPS, SLA, LLM routing cost, and private deployment. That work was not done.