I have roughly 1.3 million files distributed across folders on a network drive. At the moment, I manually upload batches of about 20 files to an AI assistant, compare them with a reference document, and export classification codes to Excel. This is too slow and limited by upload constraints.
I'm looking for a fully automated pipeline that can scan the network drive, extract text or other relevant content from each file type, compare the results with a reference rules document, assign a classification or reference code, and save the results in a structured format such as a database or spreadsheet. What architecture and tools would be practical for this volume?
4 Answers
The file count alone shouldn’t be the main concern; the important variables are file types, average size, OCR requirements, classification complexity, and service limits. If these are mostly standard Office files, built-in labeling or content extraction features may help. For scanned documents, you’ll need OCR. I’d begin with a small representative sample, define the rules and expected outputs, then build an idempotent processor that records each file’s status, code, confidence, and any error.
You probably don’t need to force everything through a manual AI upload workflow. A Python or similar service could process the files directly, maintain state in SQLite or another database, and export a final report when needed. Convert Office documents and other supported formats to text first, then classify in controlled batches. A database is generally safer than writing directly to one huge Excel file, especially if the job needs to run for days and recover from errors.
A low-code assistant by itself is unlikely to handle this reliably. Treat it as a data-processing application instead: inventory the files, deduplicate if possible, extract text, classify against a versioned rules document, validate the result, and store everything in a database. Excel can be generated as a reporting output, but it shouldn’t be the system of record for 1.3 million processing results.
A queue-based worker architecture would be a good fit. Start with a scanner that walks the network drive and places file paths or batches of paths onto a durable queue. Worker processes can then extract the content, apply the classification rules, and write the result plus processing status to a database. This lets you retry failures, resume interrupted work, and scale the workers independently. The network drive is the awkward part, so a small VM or hybrid worker may be needed to scan it and submit work to the queue. Be careful about throttling whichever extraction or AI service you use.
That makes sense. I’ll need to identify the file types first, since the extraction approach may be different for Office files, PDFs, scans, and other formats.