How should I automate classification for 1.3 million files?

The existing assistant workflow is not a good fit for this volume. A proper data-engineering solution should separate scanning, text extraction, classification, and output. It should also enforce concurrency limits for any external extraction or AI service, because sending thousands of requests in parallel can trigger throttling and become expensive. Start with a representative sample, measure accuracy and processing time, then scale the workers once the pipeline is reliable.

A Python or similar service could handle this if the file formats and classification logic are reasonably well defined. Process one file at a time or in controlled batches, record each file’s state in SQLite or a production database, and add logging plus retry handling. Before choosing the technology, confirm which file types are involved and whether the reference document contains clear rules or requires semantic judgment. Those details determine whether standard parsers and keyword rules are enough or whether an AI classifier is necessary.

You probably don’t need to load every file into an AI tool at once. Build a normal ingestion and extraction pipeline first, converting supported documents to text and storing the extracted content and metadata in a database. Apply deterministic rules locally wherever possible, and only send ambiguous cases to an AI model. Keep the structured results in a database and generate Excel exports afterward, since Excel is not ideal as the primary store for millions of records.

Treat it as a queue-based batch-processing system rather than uploading files manually. A small scanner can enumerate the network drive and place file paths or metadata onto a durable queue. Worker processes then retrieve each item, extract the content, apply the classification rules, and write the result to a database. Track status, retries, failures, and checksums so the process can resume safely without reprocessing everything. If you use cloud services, a VM or hybrid worker may be needed because accessing the network drive is usually the difficult part; the processing workers can then scale out independently.