This project is a backend system that answers multiple-choice questions (MCQs) using a Retrieval-Augmented Generation (RAG) pipeline. It was built as part of a technical assessment for an AI startup and achieved 80%+ accuracy across a real 80-question dataset.

• Goal: Answer financial planning MCQs using only PDF materials (estate planning modules).
• Method: Parse → Chunk → Embed → Retrieve → Prompt LLM → Predict answer (A/B/C/D).
• Result: 80%+ accuracy on real data from the Malaysian Financial Planning Council (MFPC).
• API: Built with FastAPI, fully testable via /quiz/answer and /quiz/batch-answer endpoints.

.
├── data/                     # Source PDFs and question sets
│   ├── Lessons/              # Chapter PDFs and parsed .json/.md
│   ├── Quiz/                 # Quiz question PDFs
│   └── processed/            # Final question dataset (.json)
├── rag_backend/              # Core backend: chunking, retrieval, RAG chain
├── chroma_db/                # Persisted vector store
├── validator_script.py       # Evaluates overall accuracy
├── main.py                   # Script-based pipeline interface
├── server.py                 # FastAPI backend server
├── pyproject.toml            # Dependencies
└── README.md

📄 Full log with all question-level details: Google Sheet – Evaluation Results

• 🔍 Context Retrieval: Embeds and retrieves PDF content using ChromaDB.
• 🧠 LLM Reasoning: Uses Google Gemini (via langchain-google-genai) to answer quiz questions.
• 📄 High-Fidelity PDF Parsing: Uses Meta’s llama-parser for reliable content structure.
• ⚙️ FastAPI Server: Easily testable endpoints for real-time prediction and batch evaluation.
• 📊 Auto Evaluation: Script to validate predictions against expected answers.

GET /

Returns welcome message.

POST /rag/build

{
  "pdf_directory": "data/5_estate_planning/Lessons",
  "rebuild_db": true
}

POST /quiz/answer

{
  "question": "Which of the following is not true about estate planning? A. ..., B. ..., C. ..., D. ..."
}

Returns predicted option (A, B, C, or D).

curl -X POST "http://localhost:8000/quiz/batch-answer" \
-H "Content-Type: application/json" \
-d '{
  "questions": [
    {
      "question": "...",
      "answer": "B"
    },
    {
      "question": "...",
      "answer": "D"
    }
  ]
}'

Returns accuracy, prediction, and correctness for each question.

• ✅ RAG Pipeline: Combines LangChain retriever with custom prompt for Gemini.
• ✅ Chunking Strategy: Optimized with llama-parser to preserve layout structure.
• ✅ Validation: Tracks accuracy and highlights failed questions for debugging.
• ✅ Stateless Server: API initializes vector DB and LLM on startup for quick querying.

This system was built for a real-world technical assessment by DocuAsk.ai, based on content from the Malaysian Financial Planning Council (MFPC).

💬 Contact & Credits

Original task and assessment design by Joseph Chin – DocuAsk.

Implementation, accuracy testing, and improvements by me.

Billy Mosis Priambodo 🔗 billymosis.com | GitHub | LinkedIn