We're experts in privacy and data law, but not IT specialists, and we're facing a challenging cleanup process for a client who has over 360,000 documents in a messy SharePoint migration. The original IT team executed a straightforward lift-and-shift from M365 to a single SharePoint site, leading to ineffective permissions and an unusable folder structure. Our client has virtually no budget, so we're trying to figure out how to address these problems without relying on expensive third-party tools.
We've planned a new folder structure based on our understanding of data purposes and workflows, set up new sites and folders, and compiled a manifest detailing the new paths for each file. However, we're hitting several roadblocks:
1. Throttling issues when trying to move 360,000 files via the Graph API/Power Automate or even through a web browser.
2. Loss of metadata, as it appears the Graph API and standard Move To/Copy To functions reset important information, which is problematic for compliance.
3. We initially started using Postgres for database architecture but shifted to Cloudflare durable objects since it stood as a clear tool for analysis. Now we have our manifest ready for folder creation, file migrations, and permissions mapped out in CSV files.
We're wondering:
- Can we trick the SharePoint Migration Tool into moving files from one SharePoint site to another?
- Does Migration Manager in the Admin Center allow for cross-site moves while keeping version history intact?
- Are there low-code methods to use our mapping CSVs with the SharePoint Migration API to avoid throttling issues?
Any insights would be hugely appreciated, especially from anyone who's handled similar regulated or audited migrations!
3 Answers
Honestly, it's going to be pretty tough to pull this off without any budget. When you’re working with regulated data and such a large volume of documents, a professional consultancy might be the best way to go. It’s risky to try and DIY this with compliance hanging over your head.
Yeah, you definitely need some tools and possibly expert help for this. If compliance is a priority, then avoiding expenses here could lead to bigger issues down the line.
It seems like the IT provider either didn’t fully understand what they were taking on, or the client expected too much for a very low budget. You usually get what you pay for in service.
I hear you! It’s frustrating that legal firms often want to cut costs but then expect the impossible when it comes to regulations. If they were serious about compliance, they should plan for the necessary budget.