Keep object IDs stable through long video occlusions

Long-form video tracking fails in a familiar way: an object disappears behind an obstruction, then returns under a new track ID. The reverse error also happens. Two similar objects cross paths, and the annotation keeps one ID alive when it should have split. Both errors teach a model the wrong continuity.

The fix starts with annotation policy, not with filling every blank frame. Decide what counts as an occlusion, what evidence is enough to retain an ID, and how to represent uncertainty. Then make those decisions testable across clips and annotators.

Define the track before labeling

Start with the unit of identity. For a multi-object tracking annotation task, specify whether an ID belongs to a physical object across the entire clip, or only while it remains visible. For most long-form tracking datasets, persistent identity is the useful target: an object keeps its ID when temporarily hidden, but a new object receives a new one.

Write down what evidence can establish a match after an occlusion. Depending on the footage, that may include appearance, motion, position relative to the scene, or a distinctive attribute. Avoid rules such as “use context” without defining what context means. If an object cannot be matched confidently, provide an uncertainty or review path instead of forcing a guess.

Also define the box policy. State whether boxes should cover the visible portion of a partly hidden object or estimate its full extent. Choose one convention for the dataset and spell out how to treat truncation at the image edge. Mixing visible-only boxes with full-object estimates creates noisy training targets even when the track ID is correct.

Mark both sides of the occlusion

For each occlusion, label the last frame where the object is identifiable and the first frame where it becomes identifiable again. These are the boundary frames. Record the object’s track ID on both sides when the match is supported by the policy. If the object is partly visible during the gap, annotate it according to the chosen box convention rather than treating the whole interval as missing.

Boundary predictions can be useful for downstream checks, but keep predictions distinct from observations. A box interpolated or estimated for a hidden frame is not the same as an annotated visible box. Store an explicit status such as observed, partially occluded, or inferred if the schema supports it; otherwise, keep inferred frames out of the ground-truth labels and document the gap. This prevents evaluation code from treating a plausible estimate as direct evidence.

Do not interpolate by default across a long occlusion. Linear bounding box interpolation assumes steady movement and can produce boxes through walls, around corners, or across a scene change. If your workflow needs interpolated boxes, set a maximum gap and permitted motion conditions. Review longer or unusual gaps manually. The policy should say whether interpolation is for training labels, visualization, or only a QA aid.

Set an ID-retention rule

Track-ID retention needs a limit and a tie-breaker. Set a maximum number of hidden frames or elapsed time for automatic continuity, based on the video’s frame rate and the task. Beyond that threshold, require review rather than silently reusing the old ID. A fixed threshold will not resolve every case, but it makes disagreements visible and measurable.

When two candidates could match the returning object, require annotators to compare the same evidence in the same order. For example: check distinctive appearance, then plausible motion and scene position, then any available contextual cues. If two candidates remain plausible, flag the segment for adjudication. Do not resolve ambiguity by choosing whichever ID is nearest in the frame; proximity alone can cause ID swaps during crossings.

Keep split and merge rules explicit too. If an object leaves the frame and later returns, decide whether its identity persists and how long the policy allows that link. If one visible object is actually two objects that separate, define where the split occurs and how each identity is assigned. These cases often reveal that a guideline describes normal occlusion but says little about identity itself.

QA the failure points, not just the boxes

Review samples around every occlusion boundary, plus clips with long gaps, crossings, re-entry, and partial visibility. Check for IDs that change without a policy reason, IDs reused for another object, and inferred boxes accidentally exported as observed labels. Track these as separate error categories: a geometrically good box can still carry the wrong identity.

Measure the rate of ID changes near occlusions and the number of unresolved cases sent to review. Break results out by clip or batch so a guideline change does not conceal a difficult source segment. A structured export check can catch missing track IDs, invalid boxes, and disallowed frame gaps; a QA gate for video exports is a useful companion to human review, not a replacement for it.

Before scaling, test the rules on a small, mixed sample. Include simple disappearances as well as ambiguous crossings and long occlusions. Compare annotator decisions against your acceptance criteria, revise the guideline where disagreements cluster, then run another check before broad production. XYNTRIQ describes a sample-batch process in which it labels a small sample to a client’s guidelines and shares the output; that gives a team a way to assess the rules before committing to a larger scope.

The goal is not to invent certainty for every hidden frame. It is to preserve identity when the evidence supports it, expose uncertainty when it does not, and keep the same decision rule from clip to clip. That is how a tracking dataset stops teaching the model to swap identities at the exact moment the scene gets difficult.