I'm working on a machine learning and big data project in Python and losing track of where data comes from, how it gets parsed, which scripts use which helpers, and how everything is orchestrated. What tools or approaches would help me get an overview?
What do you use to visualize a Python project's structure?
Answers
Try diagrams.net for a manual flowchart. It's free and works well for sketching out data sources, processing steps, scripts, and outputs. There's also an extension for VS Code if you want the diagram close to your code.
Keep the first version small enough that you'll actually update it when the project changes. A detailed diagram that's out of date can be more confusing than no diagram at all.
Reply to this answer
For questions like which script imports a helper, your IDE might be quicker than a flowchart. Try go to definition and find usages to trace the actual code. That won't give you the whole data pipeline at a glance, but it avoids maintaining a separate map of every function.
Reply to this answer
Start with pen and paper. Draw the data sources on one side, the processing steps in the middle, and the outputs on the other. If that helps, you can turn the useful bits into a digital diagram later.
Reply to this answer
Whimsical is another option if you mainly want a straightforward flowchart. You could make one diagram for the data flow and a separate one for orchestration rather than trying to cram every import and transformation into the same picture.
Reply to this answer
Add your answer
Answers are reviewed before publication.
Related Discussions