AOAMultimodal AI Support Agent… waking server (up to ~1 min)

PYTHON · FASTAPI · LANGGRAPH · VISION · SPEECH-TO-TEXT · RAG

Multimodal customer support

Send a message the way a real customer would — as text, a photo of the problem, a voice message, or all three at once. The agent (a Python backend) transcribes the audio, analyzes the image, fuses every channel into one view, classifies and validates it (Pydantic), finds the company policy in a vector database, calls tools/APIs, weighs the evidence, decides and — when needed — pauses for human approval. All live.

For recruiters — what this is and what you'll see

HOW TO TEST

  1. Pick an example or type your own ticket.
  2. Optionally attach a photo (e.g. a damaged product) and/or a voice message.
  3. Click “Send ticket” and watch the agent trace (the workflow nodes).
  4. See the transcript, the image analysis, the evidence check, the decision and the reply.
  5. For a large refund or suspected fraud the agent waits for approval — approve or reject.

WHAT IT DEMONSTRATES

  • Multimodal — text + image (vision) + audio (speech-to-text) fused into one decision
  • LangGraph — stateful workflow with branching and human-in-the-loop
  • Pydantic — validation of every input and every model output
  • RAG + vector database — company policies as the source of decisions
  • Tool calling — Order / Shipping / Customer API
  • FastAPI — this demo calls a real API (see /docs)
PythonPydanticLangGraphFastAPIVisionWhisperRAGQdrantSupabaseHugging FaceOpenAIDockerpytestobservability

API documentation (Swagger) ↗

⏳ Waking the agent server (free hosting sleeps the service — the first wake-up takes ~30–60 s). You can send a ticket right away — the first response just takes a moment.

Tip: for the “damaged product” example, attach any photo — the vision step will analyze it and the agent weighs it as evidence. With no API keys the backend runs in a deterministic offline mode.

ℹ️ For the demo to work, disable AdBlock / uBlock for this page and refresh — such extensions can block API requests.

Architecture

Client ─▶ FastAPI ─▶ LangGraph
  ingest_attachments (Supabase / local storage)
        ▼
  transcribe_audio (Whisper) ─ analyze_images (Vision) ─ fuse_multimodal
        ▼
  classify → extract → validate ─(missing data)─▶ request_information ─▶ END
                          ▼
                   retrieve_policy (RAG / Qdrant)
                          ▼
                   route_tools ▶ execute_tools (Order / Shipping / Customer)
                          ▼
                   evaluate_evidence (photo/voice vs. claim)
                          ▼
                       decide ─(refund≥threshold / fraud / critical / weak evidence)─▶ human_approval ─▶ END
                          │                                                              ▲
                     (auto / safe)                                             approve / reject
                          ▼
                  generate_response ─▶ END
                          ▼
              Supabase / PostgreSQL (audit + resume)

The backend starts with safe fallbacks (works deterministically with no API keys) and switches to OpenAI (LLM + Whisper + Vision) + Qdrant + Supabase when keys are set. Code: Python (FastAPI, LangGraph), pytest tests, an evaluation dataset, Docker.