Hey everyone,
I'm reaching out for some genuine advice on improving my coding skills. I've been working as a data scientist and, unfortunately, I don't have a formal background in programming. Over time, I've learned some ways to write cleaner code and create reusable components, but I often find myself facing challenges in my current role, where I'm responsible for a lot of data generation tasks.
Sometimes, I leave test snippets in my code, like using `[:1]` for testing and then wonder why my data output is incomplete. I get stuck trying to backfill data because of this. I realize that coding is an iterative process, but my iterations often feel unproductive. I generally check my notebooks or scripts fully to avoid obvious errors, especially when submitting pull requests, but introducing new logic leads to issues that didn't show up in my initial tests, often due to unforeseen data problems.
To make it tougher, my team operates under a fast-paced culture where everyone posts constant updates on their progress. I find myself stressed to keep up, and when it comes to scrum meetings, I usually lack meaningful updates.
Does anyone have similar experiences? I'm looking for advice on how to speed up my turnaround times while still producing quality work.
Oh, and just to add, I mainly work with dataframes (using pandas or PySpark) and do a lot of row-by-row function applications. I've looked into using thread pools to parallelize tasks, but I could use some practical tips!
5 Answers
You should consider writing each unique task in its own class or file. This way, you have a clean interface between them. If something isn’t right, you can easily fix it without messing up the whole system. Long functions can become a headache, so breaking them up makes everything more manageable.
One of the key things about writing clean code is to not force reusability everywhere. It can actually create more problems down the road. Try to keep your code simple and isolated. Minimize third-party dependencies and avoid deep nesting. This way, your code becomes much more testable!
Thanks for the advice!
Instead of testing directly in your code, create separate test cases. This helps clarify and isolate what you’re testing. If you’re struggling with filtering or customizing your data, it could indicate a design flaw.
This advice really resonates with me. I've been trying to keep track of variables throughout my scripts, but maybe I should move past that.
I've found that writing code in an object-oriented style can create the wrong encapsulation boundaries. When I shifted back to procedural programming and focused on datasets, a lot of complexities that held me back just vanished. Sometimes simpler is better.
If you're using a testing framework, keeping your test elements makes sense. Tools like doctest can be useful if your code complexity isn't too high.
That’s a great point! I tend to use classes only when I'm sure the logic is stable, but I might need to rethink that and use them from the start.