I'm an intern at a small startup. Over the past three months, I've been learning Java and Spring Boot and building small CRUD applications. I'd now like to move toward more advanced backend development by creating a media and video-streaming pipeline.
The project could allow users to upload videos, process them with FFmpeg, generate multiple resolutions, and serve them through HLS. I'm also interested in learning about object storage, CDNs such as Cloudflare, webhooks and events, distributed processing, queues, background workers, retries, idempotency, and designing systems that can scale.
The difficult part is figuring out where to begin. There are many connected concepts, and I'm not sure how to divide the project into manageable stages. Would it be better to design the complete architecture first, or build a simple end-to-end version and gradually introduce more advanced components?
How would you structure this project from the beginning? What should I build first, and which concepts should I understand before moving on to each stage?
3 Answers
Start with the smallest working end-to-end version: accept one video upload, run FFmpeg inside the Spring Boot application, save the generated HLS files locally, and play the result in a browser. Leave queues, webhooks, CDNs, and distributed workers out initially.
Once that works, add one new boundary at a time. Move the media files to object storage, then make transcoding asynchronous with a queue and worker, then add job status, retries, and idempotency. Add CDN delivery near the end, since it mainly addresses distribution and scale rather than the core pipeline.
This approach leaves you with a working project at every stage and makes it easier to identify which change introduced a problem.
Thanks, that gives me a much clearer starting point.
Treat the project like a debugging exercise: isolate the smallest unit that works before adding more moving parts. You could begin with a single endpoint that accepts a video and stores it on disk, without transcoding or queues. Then add processing, playback, persistence, and asynchronous jobs one step at a time.
You may also want to study an established streaming server such as SRS to understand how production-oriented media systems are structured, but don’t let that replace building a small version yourself.
Be careful about the scope of allowing arbitrary users to upload videos. A public service can create copyright, moderation, storage, and abuse-related responsibilities. For a learning project, it may be safer to keep uploads private or use test videos that you have permission to process.
That’s a good way to avoid overengineering. Build the thinnest working path first, because the real architectural requirements usually become clearer after you encounter practical problems. It’s also worth assuming that the first design will eventually be replaced or significantly revised.