How should I collect a large file in C when it arrives in chunks?

November 15, 2026 4 answers
Asked by 8s!gasketcase_315

A library gives me album art in chunks, and I don't know the total size in advance. If I need the complete image in memory, should I keep allocating and copying as chunks arrive, or is there a better way?

Answers

n418+clampcase_5

Use a growable buffer. Track the number of bytes stored separately from the allocated capacity. Start with a reasonable capacity, append each chunk, and double the capacity when the next chunk won't fit. Keep doubling until it's large enough for the incoming data.

realloc can extend the allocation or move the data to a new block for you. Assign its result to a temporary pointer first. If it returns NULL for a nonzero size, the original allocation is still yours and can be freed or otherwise handled.

Check for size_t overflow when adding the incoming length and when increasing capacity. Also consider a maximum image size so an unexpectedly large transfer can't consume unlimited memory.

q27__parcelgate+8:

Doubling also avoids copying the entire accumulated buffer on every chunk. Over many appends, the total copying grows roughly in proportion to the amount of data received.

Reply to this answer
BrambleSocks

Before assembling the whole file, check what the next step needs. If you're saving it, you can write each chunk to a file and reuse the same receive buffer. An image decoder that supports incremental input may let you feed it chunks too.

For a scrolling gallery, you could load the visible images and decide how much to cache as the user moves around. Decoded images can take substantially more memory than the compressed files, so that choice matters too.

Reply to this answer
7r!opledger_284

Separate allocations are fine if your consumer can work with a sequence of chunks. Keep their pointers and lengths, then free them when you're done. If an image API needs one contiguous buffer, allocate that once after you've received everything and copy the chunks into it.

That avoids repeated resizing, but it adds bookkeeping and temporarily needs space for both the chunks and the final buffer during the copy.

Reply to this answer
Tavlo-4j

Check the library's completion and size rules before writing the receive loop. Use the number of bytes actually returned, and stop according to the API's documented end condition rather than assuming every short read means the image is complete. Handle errors separately from successful completion.

Reply to this answer

Add your answer

Answers are reviewed before publication.

Related Discussions