Why is a text-to-image service a queue, not a token stream like a chatbot?

Because of what the reader can consume. An LLM emits one token at a time and a human reads them as they land, so a chatbot streams — the first word can appear in a few hundred milliseconds and the rest drip after it. A diffusion image is different: it is built by running the whole model tens of times over (each pass is one denoising step, a small cleanup of a noisy canvas), and no usable picture exists until the last step finishes. It is all-or-nothing for whole seconds. Streaming a half-denoised blur helps nobody, so the honest product is a job queue with a progress bar: submit the prompt, get a ticket, watch a bar advance from steps completed, and receive the finished image (or a push notification) when the fleet gets to it. The contrast with /llm-serving-platform is the whole design — same GPUs, opposite UX.

How long does it take to generate one image, and what does it cost?

It is steps times per-step cost. A denoising step runs the model once over the latent canvas; a typical run is 20–50 steps for a fast product and up to a couple hundred for maximum quality. The latent-diffusion paper reports that "producing 50k samples takes approximately 5 days on a single A100 GPU" at 250 DDIM steps — that works out to about 8.6 seconds per image, or roughly 34 milliseconds per step. Fewer steps is proportionally faster: 30 steps is about a second. Turn that into money with throughput: at ~8.6 s/image one A100 makes about 420 images an hour, so at a couple of dollars per GPU-hour a 250-step image costs around half a cent — and a 30-step image a fraction of that. The two levers a product actually turns are step count and resolution.

What is a LoRA adapter and how does a personalized-headshot product use it?

A LoRA (Low-Rank Adaptation) adapter is a tiny set of extra weights that teaches a shared base model one new thing — here, one person's face. Instead of copying and fine-tuning the whole multi-gigabyte model per user, LoRA freezes the base and learns a small low-rank delta: for a weight matrix W it stores the update as ΔW = BA with two skinny matrices, "where B ∈ R^(d×r), A ∈ R^(r×k), and the rank r ≪ min(d,k)." Because r is small (often 4, sometimes as low as 1), the file is small — the LoRA paper shows the same trick shrinking a GPT-3 checkpoint "from 350GB to 35MB," and diffusion LoRAs are "a few hundred MBs" at most, often single-digit. The headshot product uploads a handful of selfies, trains a per-user adapter in minutes-to-hours, and stores just those megabytes. At serving time one base model stays resident and each request hot-loads its user's adapter, so thousands of personalized models share a few GPUs.

Should the image-safety classifier fail open or fail closed?

This is the honest question, and the answer is fail closed — reject the request — for almost any consumer product. A text-to-image service runs two safety gates: a prompt filter before generation and an image classifier after it, so nothing reaches the user unscreened. If the post-generation classifier goes down and you fail open (ship the image unscreened), you save the request but risk publishing exactly the content the classifier exists to stop — a brand and legal exposure that dwarfs the value of one image. Fail closed and the cost is bounded: some users see "try again in a moment" and you refund or retry. The blast radius is a queue of frustrated users, not a headline. The rare exception is an internal or already-trusted-content pipeline where the classifier is defense-in-depth rather than the only gate.

Why is text-to-video so much more expensive than text-to-image?

Because time is a whole new dimension multiplying the compute. A video is not one image — it is many frames that must also be consistent with each other, so the model denoises across space and time at once. A five-second clip at 24 frames per second is 120 frames; even before accounting for the extra work of keeping motion coherent, that is two orders of magnitude more denoising than a single still. The serving skeleton is identical — prompt, encode, a denoising loop, decode, all behind a queue — but every number in the cost model gets multiplied: seconds per clip, GPU memory per job, fleet size for a given demand. That is why video products lean even harder on the async queue, longer honest wait-time quotes, and aggressive step-count and resolution limits.

Design a Text-to-Image Service

The image-generation sibling of the LLM serving platform — and it inverts almost everything. A chatbot streams tokens the reader consumes as they arrive; an image is all-or-nothing for whole seconds, so the product is a queue with a progress bar, not a stream. Build it up honestly: what a request really is (prompt → text encoder → N denoising steps → decode, with the diffusion loop as a black box), a commit-first estimate of seconds-and-cents per image, the three serving shapes deliberated interviewer-style, the GPU fleet alive under real queueing arithmetic, LoRA adapters for the personalized-headshot product, two-sided safety with an honest fail-open/fail-closed call, the failure sweep, and text-to-video as the same skeleton at brutal scale.

System design · AI / ML. The source ↗

A free, interactive, animated visual explainer of Design a Text-to-Image Service — built to be understood, not skimmed.

Questions

Why is a text-to-image service a queue, not a token stream like a chatbot?: Because of what the reader can consume. An LLM emits one token at a time and a human reads them as they land, so a chatbot streams — the first word can appear in a few hundred milliseconds and the rest drip after it. A diffusion image is different: it is built by running the whole model tens of times over (each pass is one denoising step, a small cleanup of a noisy canvas), and no usable picture exists until the last step finishes. It is all-or-nothing for whole seconds. Streaming a half-denoised blur helps nobody, so the honest product is a job queue with a progress bar: submit the prompt, get a ticket, watch a bar advance from steps completed, and receive the finished image (or a push notification) when the fleet gets to it. The contrast with /llm-serving-platform is the whole design — same GPUs, opposite UX.
How long does it take to generate one image, and what does it cost?: It is steps times per-step cost. A denoising step runs the model once over the latent canvas; a typical run is 20–50 steps for a fast product and up to a couple hundred for maximum quality. The latent-diffusion paper reports that "producing 50k samples takes approximately 5 days on a single A100 GPU" at 250 DDIM steps — that works out to about 8.6 seconds per image, or roughly 34 milliseconds per step. Fewer steps is proportionally faster: 30 steps is about a second. Turn that into money with throughput: at ~8.6 s/image one A100 makes about 420 images an hour, so at a couple of dollars per GPU-hour a 250-step image costs around half a cent — and a 30-step image a fraction of that. The two levers a product actually turns are step count and resolution.
What is a LoRA adapter and how does a personalized-headshot product use it?: A LoRA (Low-Rank Adaptation) adapter is a tiny set of extra weights that teaches a shared base model one new thing — here, one person's face. Instead of copying and fine-tuning the whole multi-gigabyte model per user, LoRA freezes the base and learns a small low-rank delta: for a weight matrix W it stores the update as ΔW = BA with two skinny matrices, "where B ∈ R^(d×r), A ∈ R^(r×k), and the rank r ≪ min(d,k)." Because r is small (often 4, sometimes as low as 1), the file is small — the LoRA paper shows the same trick shrinking a GPT-3 checkpoint "from 350GB to 35MB," and diffusion LoRAs are "a few hundred MBs" at most, often single-digit. The headshot product uploads a handful of selfies, trains a per-user adapter in minutes-to-hours, and stores just those megabytes. At serving time one base model stays resident and each request hot-loads its user's adapter, so thousands of personalized models share a few GPUs.
Should the image-safety classifier fail open or fail closed?: This is the honest question, and the answer is fail closed — reject the request — for almost any consumer product. A text-to-image service runs two safety gates: a prompt filter before generation and an image classifier after it, so nothing reaches the user unscreened. If the post-generation classifier goes down and you fail open (ship the image unscreened), you save the request but risk publishing exactly the content the classifier exists to stop — a brand and legal exposure that dwarfs the value of one image. Fail closed and the cost is bounded: some users see "try again in a moment" and you refund or retry. The blast radius is a queue of frustrated users, not a headline. The rare exception is an internal or already-trusted-content pipeline where the classifier is defense-in-depth rather than the only gate.
Why is text-to-video so much more expensive than text-to-image?: Because time is a whole new dimension multiplying the compute. A video is not one image — it is many frames that must also be consistent with each other, so the model denoises across space and time at once. A five-second clip at 24 frames per second is 120 frames; even before accounting for the extra work of keeping motion coherent, that is two orders of magnitude more denoising than a single still. The serving skeleton is identical — prompt, encode, a denoising loop, decode, all behind a queue — but every number in the cost model gets multiplied: seconds per clip, GPU memory per job, fleet size for a given demand. That is why video products lean even harder on the async queue, longer honest wait-time quotes, and aggressive step-count and resolution limits.

Design a Text-to-Image Service

Questions

Related explainers