How to Dynamically Redact PII in Multi-Stage Scripts?

Could you give a specific use case for your setup? We generally add a random primary key to keep the data anonymous while still traceable throughout the chain. Would this kind of approach work for you?

I've also seen it done through format-preserving encryption, but just a heads up, many available libraries aren’t very mature. It's crucial to consider your threat model—understanding your risks can help determine the best approach. How do you plan to express your threat models in schema annotations, or do you think it's more effective to handle it at the system level?

We’ve implemented something similar using protobuf annotations. By defining a schema that specifies the type of data (like IP addresses, names, etc.) and their privacy levels, we can annotate our proto definitions. That way, we can filter privacy levels dynamically based on requests.

For example, rather than pulling data from one source, you could have a different pipeline for each privacy level. This method not only provides flexibility but also enables good audit trails.