Apple patents method to improve 3D depth maps using face indices

1. A method comprising: at a device including an image sensor, one or more processors, and non-transitory memory: capturing, using the image sensor, an image of a physical environment; obtaining a depth map for the image of the physical environment, wherein the depth map includes a plurality of depth map elements respectively associated with a plurality of areas of the image and each of the plurality of depth map elements includes a depth value representing a distance between the image sensor and a point in the physical environment represented by the respective area of the image; obtaining a face-index map for the image of the physical environment, wherein the face-index map includes a plurality of face-index map elements respectively associated with the plurality of the areas of the image and each of the plurality of face-index map elements includes a list of faces of a projection of a 3D polygon mesh of the physical environment in the respective area of the image; determining a number of distinct face indices of the face-index map elements corresponding to a subset of the plurality of the areas of the image nearest a pixel of the image; in response to determining that the number of distinct face indices is one, determining a depth value for the pixel via interpolation of depth values of depth map elements corresponding to the subset of the plurality of areas of the image nearest the pixel; and in response to determining that the number of distinct face indices is two or more, determining the depth value for the pixel as a distance between the image sensor and a point of the 3D polygon mesh corresponding to the pixel.