I'm using Claude 3.7 for extracting dates from documents, and I'm curious about how the max_output setting impacts my results. I have my temperature set to 0 for consistency, but when I change the max_output values (like 8192, 16384, 32768, and 64000), it seems to affect how many dates I extract. Sometimes, I end up with fewer dates extracted when I increase the max_output, which doesn't make sense to me since it should be deterministic with the same prompt. I'm wondering if adjusting max_output affects how the model prioritizes or structures the output, and if there are any internal mechanics at play that influence this. Has anyone seen similar results or have insights into this?
2 Answers
Have you tried testing with the same inputs at different max_output levels? It's tricky because for feeds requiring large outputs, like sonnets, the model might enter different modes above certain thresholds. If your inputs are large, it might help to break them down instead. Sometimes, working with smaller chunks can yield more consistent results.
I've had similar experiences! When using tools with the Anthropic API, I found that if the max_output was set too low, the responses could be empty, even when they were critical. It seems like the LLM does some planning based on the max_output to prevent responses from getting chopped off. Think about it: when you ask it to generate a short response (like 100 tokens), it might conclude that there's no valid answer without more space, while a larger limit (like 10,000 tokens) allows it to provide full context. I wonder if this helps explain the varied results you’re seeing as well.
Related Questions
xAI Grok Token Calculator
DeepSeek Token Calculator
Google Gemini Token Calculator
Meta LLaMA Token Calculator
OpenAI Token Calculator