I'm building an AI voice interview application and I'm still learning how to design the retrieval-augmented generation (RAG) workflow. Right now, I split the candidate's resume and the job description into chunks, generate embeddings, and store them in pgvector. Before the interview, I send the complete resume and job description to an LLM to create a ranked topic plan. Before each question, I retrieve relevant chunks for the current topic and ask the LLM to generate the question.
I'm unsure whether that second retrieval step is actually useful. If the documents fit comfortably in the model's context window, I could make one LLM call with the full resume and job description and generate ten questions in advance. That would reduce latency, but it would also make the interview static.
I want the interview to adapt to the candidate's answers, including asking follow-up questions when an answer is weak or when the candidate claims experience with something specific. Would it be reasonable to use the full documents once to create a ranked topic plan, then use the conversation history, current topic, and retrieved evidence during the interview to decide the next question? I'm also concerned that retrieving context and making another LLM call before every question could make the experience feel slow. What architecture would you recommend?
4 Answers
If you only need a fixed list of questions, one LLM call is perfectly reasonable. RAG becomes useful when the interview is adaptive. For example, if someone gives a vague answer about a skill, retrieval can bring back the exact resume bullet or job requirement so the system can ask a grounded follow-up. Otherwise, the first batch of questions is all the model has planned, and it cannot respond as specifically to what the candidate says.
A practical design would be: use one pre-interview call to create a structured topic tree with priorities, required skills, evidence to look for, and suggested questions. During the interview, keep the topic state and transcript, then make a smaller decision call after each answer to select a follow-up, move to another topic, or score the current one. Retrieve only when you need specific supporting evidence, such as a resume bullet, a job requirement, a scoring rubric, or a larger question bank.
For a normal resume and job description, RAG may not add much value because both documents are often only a few thousand tokens. You can keep the relevant documents, topic plan, and conversation state in the prompt and let the model choose the next question from the running transcript. Repeatedly embedding and retrieving from the same small documents can add a network round trip without improving the answer.
If you keep retrieval in the loop, optimize for latency rather than retrieving everything each time. Store resume and job-description chunks with metadata, filter by the active topic or skill, retrieve only a few chunks, and stream or precompute the next question when possible. You could also use a fast model for routing and scoring, reserving a larger model for complex follow-ups. A vector database is more justifiable if you later add large libraries of rubrics, question banks, company information, or many documents—not merely one small resume and one job description.
That makes sense. I specifically want a dynamic interview rather than a fixed list. I’m thinking of using the complete resume and job description once to create a ranked topic plan, then using the candidate’s answer, current topic, and targeted retrieval during the interview to choose the next topic and question. Does that seem reasonable? My main concern is the delay before each next question.