Extend the canvas and let a model continue the edge

The same thing every other page here does: it builds a larger grid, drops your photograph into it at one pixel to one pixel, and paints the space around it. What differs is who decides what goes in that space. On the rest of this site it is arithmetic — the edge reflected, smeared, blurred, averaged or swept. Here it is a model that has seen a great many photographs and is being asked what usually carries on past an edge like yours.

The model is MI-GAN, it is MIT-licensed, and the file is 26,639,447 bytes — 25.4 MB. It is fetched from cdn.pictureeditor.com the first time you ask for it and never before, and the 13.3 MB runtime that executes it comes from this domain. Both are then kept in your browser’s cache. Decline the download and every control on this page still works; the fill falls back to the edge stretch, which is what the preview shows until the model has run.

Your photograph is not an input to the result in the way you might expect. The network is handed the canvas with the new border already cut away — multiplied out to nothing before the tensor is built — so it can see the outside of your picture and cannot see the space it is filling. When it answers, only the border is taken from its answer. Your photograph is drawn back over the top afterwards at full size, so not one row of it is resampled, softened or shifted.

A word on this site has just changed meaning and it is worth separating the two senses. The readout above the canvas has always said what percentage of the finished picture was “built here”, and on every other page that is arithmetic: a reflection, a smear, a blur, a colour. Nothing is invented in the ordinary sense — every value comes from a value you already had. On this page it does not. The border is generated. It is plausible rather than true, nobody photographed it, and if the picture is evidence of something then this is the page not to use.

The limit is mechanical and the rail states it with your numbers in it. The canvas may be at most 2 times the area of your photograph, and the deepest new edge must be small enough that there is roughly one and a half times its depth in real picture beside it. Inside that, a wall carries on being a wall, a sky stays a sky, gravel and grass and water and a thrown-out background all continue without a join anybody would find. Outside it the model does not fail loudly — it produces a confident, smooth, wrong picture — which is exactly why the button goes away instead of letting you find out.

Subject matter matters as much as size. A straight architectural line will bend. A tiled floor will lose its grid. Lettering comes back as letter-shaped marks that spell nothing. A person standing against the edge of the frame grows something that is not their arm. Those are the same subjects the deterministic fills fail on, for a related reason: there is genuine information outside the frame and nothing inside it can recover it.

The border is generated at up to 1,536 pixels down its long edge and then enlarged onto your export canvas, which is the one resolution compromise in this page and it is worth naming. The model works at 512 square and nothing changes that; running a single 512 square across a 4,000-pixel canvas would give every generated pixel eight canvas pixels to cover and the new edge would arrive as a visible smear beside a sharp photograph. Instead the canvas is covered by overlapping squares at a working size, each one crossfaded into its neighbour. Your photograph is never part of that working copy; it is drawn on afterwards at its own size.

Everything else behaves as it does elsewhere on the site. JPG, PNG and WebP all write; the quality slider and the file-size estimate work the same; the camera information is dropped unless you keep it. Change the shape or move the anchor after a run and the generated border no longer fits the hole it was painted for, so it stops being drawn and the rail says so rather than leaving a misaligned band in your file.

What does the model actually do?

It fills a hole. MI-GAN is an inpainting network: it is shown a picture with a region cut out of it and asked what was behind that region. Extending a canvas is the same operation with the hole outside the original frame — your photograph is placed in a larger grid, the empty border is declared to be the hole, and the network continues whatever was running off the edge inward. It is never shown a single pixel from inside the hole, which is why this is a continuation of your picture rather than a blur of something already painted there.

Why will it not turn a phone photo into a widescreen frame?

Because that is not a continuation, it is a new scene. A 9:16 still taken to 16:9 needs more than a thousand new columns on each side of a photograph about a thousand wide, so most of the result would be generated and the model has almost nothing to carry outward. This page stops offering the model past 2 times the area of your photograph and tells you the numbers when it does. For that conversion the blurred fill is the right answer and always was.

What gets downloaded, and when?

Nothing at all until you press the button that names the figures. Then two things: the model itself, 26,639,447 bytes exactly — that is 25.4 MB — from cdn.pictureeditor.com, and 13.3 MB of WebAssembly runtime from this domain, which is the program that executes the model. Both are cached in your browser, so a second run downloads nothing and works with the network switched off, and the rail carries a control that deletes them again.

Does my photograph go anywhere?

No. The direction of travel is the whole point: the model comes to your device, your picture does not go to the model. One file is requested from another domain and it is the model's weights; that request carries nothing of yours beyond what any request for a file carries. The inference runs on your own processor, the result is composited on a canvas in this tab, and the file is written by your own browser.

How long does one run take?

About 2.4 seconds per 512-pixel square on single-threaded WebAssembly, which is the path every visitor is on — this site cannot turn on the cross-origin isolation that multi-threaded WebAssembly needs without breaking the advertising it is funded by. A canvas is covered by between one and nine squares depending on how deep the new edge is, and the button tells you how many before you start. The page stops responding while each square runs.