I'm building an AI voice-interview application and I'm still getting comfortable with RAG. I'm unsure whether my current architecture is adding useful complexity or unnecessary latency.
Before the interview, I currently split the candidate's resume and the job description into chunks, generate embeddings, and store them in pgvector. I then send the complete resume and job description to an LLM to create a ranked topic plan. Before each question, I retrieve resume and job-description chunks related to the current topic and ask the LLM to generate the question.
My concern is that if both documents fit comfortably in the model's context window, I could simply send them once, generate a fixed set of questions, and avoid retrieval before every question. On the other hand, I want the interview to be dynamic: the next question should adapt to the candidate's answers, identify weak or interesting claims, and potentially ask follow-ups grounded in specific resume details.
Would it be reasonable to use the full documents once to create a ranked topic plan, then use the interview transcript and targeted retrieval only for dynamic follow-ups? If so, how should I reduce latency when generating the next question, and are there better places to apply RAG in this system?
4 Answers
If you want RAG to have a clearer role in the project, consider using it with a larger question bank, evaluation rubric, competency library, or collection of job descriptions. Those sources are more likely to exceed the context window and benefit from semantic search. For just one short resume and one short job description, direct prompting is usually simpler and faster.
If the resume and job description are only a few thousand tokens, RAG probably adds no value for the initial question-generation step. You can keep the documents, topic plan, and relevant interview state available during the session, then let the running transcript and current topic guide the next question. Re-querying a vector database over the same two small documents before every question may only add a round trip and latency.
A single LLM call is perfectly fine if you only need a fixed list of questions. RAG becomes useful when the interview needs to react to the candidate. For example, if someone gives a vague answer about a skill listed on their resume, retrieval can bring that exact resume bullet or the relevant job requirement back into context so the system can ask a focused follow-up. Without that step, the model is mostly limited to the plan it created at the beginning.
A sensible design would be to create the ranked topic plan before the interview, ideally from summarized or structured versions of the documents if they might be large. During the interview, pass the current topic, recent conversation, candidate answer, and only the evidence needed for the next decision. Use retrieval when you need to locate a specific resume claim, job requirement, scoring criterion, or follow-up context—not simply because a vector database is available.
You can also precompute embeddings, cache common retrieval results, stream the candidate’s answer into a fast analysis step, and generate or prepare likely follow-ups while the candidate is speaking. That keeps retrieval useful without forcing every question to wait on a heavyweight pipeline.
That clears things up. I do want the interview to be dynamic, so I’m thinking of using the complete resume and job description once to create a ranked topic plan, then using the current topic, transcript, and candidate answer for retrieval during the interview. Does that seem reasonable, or would the initial full-document call create too much delay for large documents?