Why AI roto always looks cut out

AI roto is a small miracle if you remember paying by the hour for hand roto. Magic Mask in Resolve, Roto Brush in After Effects, and a dozen cousins will track a subject through a shot and hand you a silhouette in seconds. And then you composite it, and something is off. The subject sits on the background instead of in it. Clean edges, good tracking, and it still reads as a sticker.

The instinct is to blame edge quality and start feathering. The actual problem is one level deeper, and feathering makes it worse.

A mask answers the wrong question

Segmentation models answer, for each pixel: subject or background. One bit. The answer is usually right, and the question is wrong, because real edge pixels are not either-or. A strand of hair, a blurred hand, the rim of a shoulder where the lens rolled off focus: those pixels are 20, 50, 80 percent subject, and the remainder is whatever was behind them. Photography mixes; masks decide.

So every semi-transparent pixel gets rounded. Hair wisps become either solid tentacles or nothing. Motion blur becomes a hard paddle. And there is a subtler wound: the color inside those edge pixels still contains the old background. Even with a perfect-looking silhouette, the edge carries a rim of the original scene into the new one. Your eye may not name it, but it sees it.

Feathering blurs the decision boundary. It replaces wrong-hard with wrong-soft, a uniform haze where the real edge varied. That is the sticker look’s second form.

When a mask is exactly right

None of this makes AI roto a bad tool. Grading is the obvious case: isolating a face for a secondary correction does not need fractional alpha at the hairline. Same for tracked blurs, vignettes, sky replacements behind hard architecture, and quick social cutouts where the sticker look is the aesthetic. If the mask feeds a grade rather than a composite, use the mask and move on.

When you shot green, ask a different question

If the footage was shot against green or blue, you can do categorically better, because the screen makes the mixing recoverable. Key Studio Pro runs the CorridorKey neural keyer, which answers the question a mask cannot: for every pixel, how much is subject, and what color is the subject’s share with the screen removed. The output is a linear alpha with true intermediate values plus an unmixed foreground, which is the difference between a cutout and a photograph.

1. Run the green or blue screen shot through the app.
2. Compare its Comp preview against your mask tool’s output at the hairline and on the fastest-moving frame.
3. Composite with the passes, or ProRes 4444 with alpha if you are staying in an NLE.

The boundary of that promise

This works because the screen constrains what was behind the subject. On footage shot against a normal scene, that constraint does not exist, and unmixing against an arbitrary background is a far harder problem. There, AI roto plus comp craft is still the honest state of the art: light wraps, edge blends, and grain matching to talk the eye out of the sticker. The keyer path is for the material you shot to be keyed.

Common questions

Why does my roto edge chatter frame to frame?

The model re-decides the boundary every frame, and noise flips marginal pixels. Temporal smoothing helps and trades away accuracy on fast motion, which is where blur handling matters most.

Can I combine a mask with a chroma key?

Yes, and it is a classic move: the mask as a garbage matte to isolate the subject, the keyer for the edge. It inherits the keyer’s edge limits, so on hair-heavy shots see the hair guide before committing an evening to it.

Is a neural keyer just a fancier mask?

No, and the output proves it: a mask gives you a shape, a keyer gives you a linear alpha and a reconstructed foreground color. The second pair is what a composite is actually made of.