I'm really curious about how major cloud providers such as AWS, GCP, Azure, and DigitalOcean set up their infrastructure from scratch—right from the physical servers all the way to running a complete self-service cloud platform. My goal is to create a similar, but smaller-scale version where users can sign up, create virtual machines (VMs), or databases, and get billed hourly, like what they offer. Before diving in, I'd like to understand a few things:
- What software stack is typically used on bare metal by these big providers?
- How do they handle virtualization, networking, storage, and tenant isolation?
- Are there notable open-source tools (like OpenStack, Proxmox, Harvester, etc.) that I should look into?
- How is billing, metering, and provisioning automated in these systems?
- Lastly, are there any good resources, such as books, blogs, or courses, to learn this from the ground up?
I'd greatly appreciate any insights, advice, or learning paths from anyone who has built something similar or works in cloud engineering. Thanks in advance!
6 Answers
From my experience building a private cloud at my last job, it's quite a complex process. We spent about $30 million and managed two data centers with strong synchronization between them, so if one went down, we could still serve traffic. We used a mix of Dell hardware and Juniper switches, and ran Kubernetes directly on bare metal along with our databases.
If you're looking to do this on a smaller scale, starting with Proxmox is a great idea. It allows you to self-host and interact via its API, so you can scale up from just getting a VM running to developing a full UI for user interaction. Security and redundancy are also crucial, so think about those from the start!
Funny enough, we did a similar setup at my previous job too! Was it an internet company? We spent millions and it was still in the trial phase when I left.
OpenStack could be a good choice since it's one of the closest open-source solutions to what big providers do. Just keep in mind it's resource-intensive and has a steep learning curve, making it more suited for larger setups or educational environments.
Proxmox is easier to start with and offers solid features for smaller environments. Just remember, security should be a priority as you build out your infrastructure!
Thanks for the tip! I'll make sure to check out Proxmox.
I agree! You should plan for security from day one; retrofitting it later can be a hassle.
Honestly, replicating what big cloud providers do is super ambitious! They invest heavily in custom solutions and infrastructure. Everything is optimized for their scale, while open-source options are great but don't quite have the same level of support. To succeed, you need a dedicated team and a clear plan for your cloud objectives. Don't underestimate the man-hours and budget needed!
For a DIY cloud, you just need to break it down into layers. Start with bare metal provisioning using tools like Cobbler, add a hypervisor like KVM managed through Libvirt, and for networking, implement an SDN approach with Open vSwitch and VLAN overlays. Pick an orchestrator like Proxmox to manage VMs and something like Ceph for storage. The challenge of automating billing can often be tackled by combining OpenStack modules or custom scripts.
Some great resources to look into include OpenStack's detailed documentation and YouTube for tutorials to guide you through the process.
From what I've seen, DigitalOcean uses QEMU and Libvirt for VM management and stores data using Ceph. There’s a lot of behind-the-scenes work to tie everything together, like having agents on servers and a robust control plane managing it all. If you want to go deeper into building a mini cloud, consider technology stacks involving KVM, Ceph for storage, and automating processes with tools like Terraform.
Thank you so much for this info!
You could get started on a limited scale—like with two solid internet connections and a cabinet in a data center. If you have the right router setup and a basic cloud stack, you can create something decent without breaking the bank!
Thank you, this really helps me a lot!