How to Dynamically Redact PII in Multi-Stage Scripts?
Asked by
CloudyPanda77
Hey everyone! I'm working on a privacy-focused solution to manage sensitive data, specifically PII, as it travels through multi-stage pipelines like scripts and APIs. I'm facing a significant challenge concerning scoped visibility. My goal is to figure out how to dynamically redact or expose fields based on the role of the script or agent and the stage of the workflow.
For example:
- Stage 1 gets the full input with all details.
- Stage 2 only sees fields that are non-sensitive.
- Stage 3 might need to temporarily access redacted data if necessary.
I'd love to hear if anyone has come across common design patterns or open-source solutions for this challenge. Would you suggest using middleware, decorators, metadata tags, or something else entirely?
Answers
We’ve implemented something similar using protobuf annotations. By defining a schema that specifies the type of data (like IP addresses, names, etc.) and their privacy levels, we can annotate our proto definitions. That way, we can filter privacy levels dynamically based on requests.
For example, rather than pulling data from one source, you could have a different pipeline for each privacy level. This method not only provides flexibility but also enables good audit trails.
Reply to this answer
I've also seen it done through format-preserving encryption, but just a heads up, many available libraries aren’t very mature. It's crucial to consider your threat model—understanding your risks can help determine the best approach. How do you plan to express your threat models in schema annotations, or do you think it's more effective to handle it at the system level?
Reply to this answer
Could you give a specific use case for your setup? We generally add a random primary key to keep the data anonymous while still traceable throughout the chain. Would this kind of approach work for you?
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions