I've been working with OpenSearch Serverless and I've run into a problem. I have a file containing about 3000 documents, but only 700 of them get synced before I'm hit with a timeout. Does the bulk API have a limit that I should be aware of?
2 Answers
Honestly, I find OpenSearch Serverless pretty problematic. It's expensive but doesn't scale well. In a previous job, we tried using it for logging OCPP data, but it just didn't work out. We eventually switched back to the non-serverless version, which performed much better.
Yes, there is definitely a limit when using the bulk API in OpenSearch, serverless or not. Based on my experience, if you try to ingest too many documents at once without using an ingestion queue, the system will refuse to process them. Surprisingly, the bulk API might not return errors, but single document calls do show errors. It's really important to implement your own ingestion queue and handle retries on failures. It feels a bit absurd that a system like this doesn’t provide more reliable data insertion methods.
It’s a common misunderstanding to think of search engines just like databases.
I recently tried indexing documents from Snowflake into OpenSearch. I transferred the files to S3 and buffered them through SQS. Out of 170k documents, only 130k were indexed, and there were no error messages in the logs. Sounds like one of those silent failures, right?
Thanks for the insight! I'm just beginning to learn this for an interview and find serverless options easier for setup.