Maintaining the 3D locations of active objects, even when they are out of sight, is critical for correctly locating them over short and long time scales in egocentric video.