An image-to-video model takes one photo and a line of text and returns a few seconds of video. On the server side that is one API call. Most of the bugs we hit while building one were not on the server, though. They came from the photo itself: its shape, its real file type, its size, and whether the browser could even read it. Each of them produced a clip that was wrong, a refusal from the provider, or a request that never went out, and none of them showed up as an error in our own code.
This article walks through the four checks we now run in the browser before a photo leaves the page, with the code for each.
In short: pick the aspect ratio from the photo instead of using a default, read the file type from its first bytes instead of its name, shrink anything big to a 2048-pixel copy before upload, and when the browser cannot read a file, ask the user to pick it again instead of reporting a failed upload.
1. Let the photo choose the aspect ratio
Video models render to a fixed canvas: 16:9, 9:16, 1:1 and a few others. If the photo has a different shape, many providers do not crop it for you. They stretch it to fit.
We found this the hard way. A 768 x 1024 portrait sent with our default 16:9 setting came back as an 864 x 480 clip, and the face in it had been pulled into a wide oval. The request succeeded, the file was valid, and the result was unusable.
The fix is to look at the photo's dimensions once it is loaded and pick the closest ratio the model offers, unless the user has already chosen one themselves:
// ratios: what this model accepts, e.g. ['16:9', '9:16', '1:1']
function ratioForPhoto(width, height, ratios) {
const r = width / height;
const wanted = r > 1.15 ? '16:9' : r < 0.87 ? '9:16' : '1:1';
return ratios.includes(wanted) ? wanted : null; // null: keep the current choice
}
const img = new Image();
img.src = URL.createObjectURL(file);
await img.decode();
if (!userPickedRatio) {
const next = ratioForPhoto(img.naturalWidth, img.naturalHeight, model.ratios);
if (next) setRatio(next);
}
Two details matter here. The thresholds leave a band around square photos, so a 4:5 Instagram crop does not get forced into 9:16. And a choice the user made by hand always wins: the photo only sets the ratio while nobody has touched the control.
2. Read the file type from the bytes, not the name
File names lie. A screenshot tool saves a JPEG as image.png, a messaging app strips the extension, a phone hands over a HEIC with a .jpg name. Your upload code usually trusts file.type, which the browser derives from the extension. The provider does not. It reads the bytes, sees a JPEG where you promised a PNG, and refuses the image. One provider we use returns REMOTE_IMAGE_SIGNATURE_INVALID for exactly this.
Reading the first few bytes is enough to know what you have:
const SIGNATURES = [
{ type: 'image/jpeg', at: 0, bytes: [0xff, 0xd8, 0xff] },
{ type: 'image/png', at: 0, bytes: [0x89, 0x50, 0x4e, 0x47] },
{ type: 'image/webp', at: 8, bytes: [0x57, 0x45, 0x42, 0x50] },
];
const BRANDS = { ftypheic: 'image/heic', ftypheix: 'image/heic', ftypmif1: 'image/heic', ftypavif: 'image/avif' };
async function sniffImageType(file) {
const head = new Uint8Array(await file.slice(0, 12).arrayBuffer());
const hit = SIGNATURES.find((s) => s.bytes.every((v, i) => head[s.at + i] === v));
if (hit) return hit.type;
const brand = new TextDecoder().decode(head.slice(4, 12));
return BRANDS[brand] || file.type || 'application/octet-stream';
}
Use the sniffed type for the upload's Content-Type. Then decide what to do with formats the model will not take. Most video models accept JPEG, PNG and WebP and nothing else, while your file input happily accepts HEIC, AVIF, BMP and TIFF. If the browser can decode the file, convert it to JPEG on the device before upload. Safari decodes HEIC natively; for other browsers a small decoder library does the job. The server usually cannot help here: decoding an iPhone HEIC in a serverless function is slow at best.
3. Send a 2048-pixel copy, not the 9 MB original
A phone photo is often 5 to 10 MB. A short image-to-video clip comes out at 480p or 768p. Sending the original buys nothing in the output and costs a lot on the way.
Upload time is the visible part. The hidden part is memory. Several video APIs want the image inline, as base64 inside the JSON body, rather than as a link. On our side that means the server reads the file, base64-encodes it (a third bigger), builds the request body around it, and may keep a copy so a retry sends the exact same bytes. Run a few of those at once in a 128 MB serverless worker and the worker dies, taking every other request it was serving with it.
So the browser now re-encodes anything over 1.5 MB:
async function modelCopy(file, maxBytes = 1.5 * 1024 * 1024, sides = [2048, 1536]) {
if (file.size <= maxBytes) return file;
const bitmap = await createImageBitmap(file, { imageOrientation: 'from-image' });
for (const side of sides) {
const scale = Math.min(1, side / Math.max(bitmap.width, bitmap.height));
const canvas = new OffscreenCanvas(
Math.round(bitmap.width * scale),
Math.round(bitmap.height * scale)
);
canvas.getContext('2d').drawImage(bitmap, 0, 0, canvas.width, canvas.height);
const blob = await canvas.convertToBlob({ type: 'image/jpeg', quality: 0.85 });
if (blob.size <= maxBytes) return new File([blob], 'photo.jpg', { type: 'image/jpeg' });
}
return file; // could not get under the limit: send the original
}
imageOrientation: 'from-image' applies the EXIF rotation while decoding, so a portrait taken on a phone does not arrive sideways. That also makes check 1 honest, because the width and height you measure are the ones the viewer sees.
If your product promises to keep users' photos untouched, upload the original separately for storage and use the copy only for generation. That is what we do for signed-in users: the copy goes to the model, the original goes next to it in the bucket.
4. A Load failed error is often the photo picker, not the network
This one took longest to understand. On Android, some uploads failed before a single byte was sent. The error was a generic Load failed message or a NotReadableError, and the same photo kept failing however many times the user tried.
The cause is the system photo picker. When the file input's accept attribute is image-only, Chrome on Android 13 and later opens the system Photo Picker. Photos that live only in a cloud album can come back as a File whose reported size does not match bytes the browser can actually read. Every read of that file fails.
Two changes fixed most of it. First, catch the read failure separately and say what is really going on:
async function readable(file) {
try {
await file.slice(0, 16).arrayBuffer();
return true;
} catch {
return false; // ask the user to pick it again or save it to the device first
}
}
Second, for the retry, use an accept value that includes one non-image type, for example image/*,application/x-myapp-upload. That sends Chrome to its ordinary file chooser, which hands over a real local file, while the list still shows images in practice. Keep the image-only accept for the first pick, where the photo grid is the nicer experience.
Putting it together
The order matters. Check that the file is readable, sniff its type, convert it if the model will not take it, shrink it, and only then measure it for the aspect ratio and upload. Every step is cheap compared with a generation that is refused, stretched or never sent.
We run exactly this pipeline in the browser for Free Image to Video, a free image-to-video generator, and the same rules apply to any model that animates a still. The server work starts after the photo is right: the clip that comes back from an image to video AI with sound carries its own audio track, and that file needs its own checks before you show it to anyone. But that is a separate article. If the input photo is wrong, nothing downstream can fix it.
