Where should I start with building on-prem infrastructure?

Before picking tools, get a clear inventory: how many machines, what workloads they’ll run, how much downtime is acceptable, and who will maintain them. Also pin down the network, storage, identity, backups, and GPU requirements. The right setup for a few internal services can be very different from one serving customers.

Write down the requirements and constraints first. On-prem means your team now owns the pieces a cloud provider used to operate, including hardware failures, network paths, capacity planning, and recovery.

If you want a manageable first step, look at a hypervisor such as Proxmox and use virtual machines for the services that don’t need direct hardware access. It can be a gentler way to learn host management than starting with a full Kubernetes setup. GPU passthrough may work for some VM setups, but check compatibility with the exact hardware and workload before committing.

Keep the first deployment small. You can add more automation and orchestration after the basic network, storage, and recovery process are understood.

Ansible is a solid place to begin for repeatable server configuration. Use it to apply updates, install packages, configure services, and keep machine settings consistent. If you’re provisioning Linux hosts from scratch, pair it with an unattended installation method supported by your distribution.

Once the hosts are dependable, Kubernetes may make sense if you actually need to run and manage containerized workloads across machines. A lightweight distribution such as k3s is one option. GitOps tooling can help manage deployments later, but it adds another system to learn, so I wouldn’t make it step one.

Don’t treat storage and networking as afterthoughts. Decide how data will be protected, how services will be reached, and how you’ll restore things after a failed disk or host. Check subnet and address planning early, especially if this environment needs to connect to existing company networks.

For clustered storage or network load balancing, there are open-source options, but they bring operational complexity. Choose them only if the workload needs their features and someone can own troubleshooting and upgrades.

For the GPUs, validate the full path before building around them: operating system and driver support, access from the intended VM or container, and compatibility with the software that will use them. Don’t assume a general-purpose server setup will handle GPU scheduling or sharing automatically.

I’d make one machine a test environment, document the working configuration, and only then roll it out more broadly. If this is production infrastructure, also check what hardware and software support the company expects.