I'm working with a standalone high-performance computing system that has 256 cores, 1TB RAM, and 512GB of swap space. Recently, I've been running into memory allocation issues that result in system-related errors, such as failing to save LLDP data, and SSH sessions being killed unexpectedly by the out-of-memory (OOM) manager. Specifically, a rogue Python process tends to consume excessive memory, leading to network and SSH failures. I'm attempting to use systemd's cgroups to set memory limits with the following commands:
- `systemctl set-property user-1000.slice MemoryMax=950G`
- `systemctl set-property user-1000.slice MemoryHigh=940G`
Will these settings help prevent my system from crashing due to high memory usage?
4 Answers
Have you considered using SLURM? It's designed to handle resource management for HPC environments, even if you're just running a workstation for a specific task. It can really ease the burden of managing memory allocation on its own.
You can limit each user’s resource usage quite effectively via systemd. Create a config file like `/etc/systemd/system/user-.slice.d/50-default-quotas.conf` and set parameters like these:
```
[Slice]
CPUQuota=400%
MemoryMax=8G
MemorySwapMax=1G
TasksMax=512
```
This will cap each user to 4 CPU cores, 8GB of memory, 1GB of swap, and 512 processes for forks, which can help mitigate issues like fork bombs.
Thanks for the tips! That solid quota setup should really help. I’ll give it a shot.
Have you thought about containerizing your workloads with Kubernetes? It's built around cgroups and can efficiently manage resources. However, keep in mind that Kubernetes isn't the typical choice for HPC setups since each workload's needs can be quite different.
I’ve read that Kubernetes isn't ideal for HPC environments. There are some resources that contrast it with SLURM effectively.
To really ensure that your memory limits are honored, set your memory properties like this:
```bash
systemctl set-property user-1000.slice MemoryMax=900G MemoryHigh=850G
```
Also, don't forget to tune your kernel to help manage out-of-memory situations better. You can create a file called `/etc/sysctl.d/99-hpc-oom-protection.conf` for settings like reducing swappiness and handling memory overcommit. This can significantly improve system stability during high memory usage scenarios.
Sounds promising! I'll definitely look into the kernel tuning aspect, too.
That makes a lot of sense! If you plan on scaling up or adding more systems in the future, getting users familiar with a job scheduler like SLURM will pay off.