Cloud GPU pricing varies widely, and more inference workloads are being moved closer to where data is collected. For people with hands-on experience, what are you actually running locally—training, fine-tuning, inference, image generation, or something else? How do power, cooling, electrical capacity, reliability, and maintenance hold up under sustained load? At what usage level does owning hardware become more cost-effective than renting cloud resources, and what unexpected hardware or operational problems have you encountered?
4 Answers
I use a couple of desktop systems and a PC with a 12 GB GPU for proofs of concept and internal experiments. Tools for running models locally and generating images are useful for testing ideas, such as combining clothing designs with different environments. In practical office use, staff generally prefer an AI service that handles the complexity for them. Comparing tools side by side was revealing: one model took several minutes and produced an average result, another responded almost immediately with the best output, and a locally run model finished in about half a minute with a result that was nearly as good. Local hardware can therefore be valuable for experimentation and privacy, even when hosted services remain easier for general users.
I operate several local machines for machine learning, graphics, rendering, and data processing. They run in a separate room with multiple air conditioners, KVM switches, and independent electrical circuits rather than relying on one breaker. For our usage, keeping the equipment locally is cheaper than using cloud resources continuously, though we still rent cloud capacity for jobs needing more memory or faster GPUs than our local cards provide. The difficult parts are reliability and maintenance: full-load systems can throttle or shut down because of cooling, power-supply, or stability problems. Power outages are especially inconvenient because large UPS systems are expensive and can be depleted very quickly, so machines sometimes need to be restarted manually. HDMI dummy plugs can help prevent systems from booting on integrated graphics when no monitor is attached. A generator with a short transfer time can also make sense for larger installations.
For a bigger site, sizing the UPS for only a few minutes and pairing it with a generator that can assume the load quickly is more practical than trying to run everything from batteries for a long period.
A GPU with around 24 GB of VRAM can run a surprisingly broad range of open-weight models, although performance varies a lot. My local use is mainly inference rather than training or fine-tuning; choosing a smaller model and supplying the right context and tools is often enough. Power and cooling are broadly comparable to running a high-end gaming PC. On-prem makes the most sense once your workload is predictable and you know that smaller models can handle most requests. Large accelerators such as the L40S, A100, H100, H200, and MI300X are much faster but expensive. Systems with large shared memory, such as higher-memory desktop platforms, can run larger models more cheaply, though with lower speed, and multiple systems can be networked together.
We moved image and video generation on-prem after cloud costs grew quickly. Users often needed several attempts to get a result they liked, so local hardware made unlimited experimentation much cheaper. We built a roughly 10,000 CAD system with consumer components instead of buying an enterprise AI server costing around 150,000 CAD. Since the workload was useful but not business-critical, we accepted occasional downtime and handled repairs ourselves. The trade-off is lower support and potentially longer outages, but the economics worked well for us.
Display dummy plugs are a simple fix for systems that fail to detect their dedicated GPU when they start without a monitor.