I'm new to using Kubernetes and currently working on data pipelines in a shared cluster. One of my jobs processes a large volume of data every hour and keeps getting stuck. I'm curious if other jobs running on the same cluster, even if not overlapping in schedule, could impact mine. I don't fully understand how pods operate behind the scenes, specifically how memory is allocated when multiple pods run on a single node, and whether previous jobs might still be holding onto resources when my job starts. Additionally, since my cluster is managed by a cloud orchestration tool, how can I verify what resources each pod is using? Is there a way to check real-time memory usage for each pod without direct kubectl access? Coming from a managed compute background, I'm eager to learn more about Kubernetes internals but I'm not sure where to start!
1 Answer
It’s crucial to grasp what a container is first. I recommend checking out a great video on the topic when you have time! As for your issue, are you and your team setting resource requests and limits for your pods? These, combined with cgroups, can help ensure that your pod gets the resources it needs. Also, do you have any monitoring tools set up for your clusters? That could offer valuable insights about resource usage.
Thanks for sharing that video! I’ll definitely give it a watch. I’m still getting a handle on setting resource requests and limits since that’s new to me. I think I might need to ask our admin for help on monitoring my pod's resource usage without bothering them too much.