SDF Flashcards
Incorporate SDF related papers knowledge
DeepSDF - what is the input and the output of the network
Input - latent vector related to a certain shape and a query 3D point in space
Output - the SDF value of the 3D point in space for that shape
DeepSDF - why are they not using an encoder-decoder framework
They say that they’re not sure of the usefulness of the encoder and want to use computational resources better.
DeepSDF - What is being done with the posterior of the shape code zi, given the shape samples Xi?
pθ (zi |Xi ) = p(zi ) (xj ,sj )∈Xi pθ (sj |zi ; xj )
DeepSDF - what is the prior distribution of the latent codes
Gaussian distribution of means = 0 and the same variance in each dimension (while each covariance is 0)
DeepSDF - what is the paper expression of the SDF likelihood
exp(-loss(NN(latent_code, 3D point), SDF))
The exponent of minus loss.
The loss is between the NN to the SDF in a 3D point.
The input to the NN is the latent code and that 3D point.
What is TSDF
Truncated SDF
DeepSDF - How to get the TSDF
Convert the mesh into voxels and then calculate the SDF?
ASDF - what is ASDF
Articulated SDF
ASDF - What is their contribution?
1 Separate the code area for shape and for articulation
2 No prior assumption on the geometry or joint type, location and range.
3 Adjust the model in inference using Test-time adaptation.
ASDF - What are their findings?
That their model can generalise well to out-of-distribution and unseen data (partial point cloud + depth images).
Re-param pix int - What is re-parametrisation of the pixel integral?
In rendering optimisation we integrate the rays that go through a pixel. Reparametrisation is to reformulate the integral so it would be done more robustly.
Re-param pix int - Where is the problem they are trying to fix?
The neural SDF method for geometry rendering can’t use edge sampling for rendering optimisation.
Re-param pix int - What are their findings?
That the reparametrisation of the pixel integral gives correct neural SDF differentials.
Re-param pix int - What does “differentiable rendering” means? “Differentiable” to what?
It means that the output of the rendering of a 3D scene is a differentiable function in respect to its variables. The variables are: geometries, materials, light or camera.
Re-param pix int - Why is it good to differentiate the rendering? What are possible purposes?
In order to optimise the rendering process for 3D reconstruction, Inverse rendering or Neural Scene representation.
Re-param pix int - What is the old way to deal with edges in mesh-based rendering differentiation?
To do edge based sampling meaning to explicitly consider the silhouette in the sampling to build the differentials around the edge.
Re-param pix int - Why SDFs do not have a simple parametric form amenable to sampling?
Because they are implicitly define the edge as the area in which SDF(x)=0.
Re-param pix int - What is their contribution?
They develop a continuous warping function so neural SDFs rendering can be differentiable.
Re-param pix int - How they developed a continuous warping function?
They use the distance to the surface and quadrature on sphere tracer points.
Open3D - why should one divide the world to blocks and only then to voxels
Because blocks are a more coarse unit that helps optimise the rendering because of the sparse nature of objects in a scence. Voxels help to add more fine details.
Open3D - why we do not maintain a voxel hash map?
we can preserve the data locality instead of scattering adjacent data uniformly into the memory.
DeepSDF - Formulate getting the SDF of a point from the model
f_theta(zi, x) sim= SDFi(x)
Where
f is the model
theta are learned parameters
zi is a latent vector of indexed shape I
x is a point in 3D
DeepSDF - what are the model’s 4 abilities?
1) To represent the training data.
2) To use the learned feature representation to reconstruct unseen shapes.
3) To apply shape priors to complete partial shapes.
4) To learn smooth and complete shape embedding space.
IM-NET - what are the differences to DeepSDF?
IM-NET utilizes 3D CNN encoder and an MLP decoder approach to build latent space and an occupation function to present the 3D shape implicitly [12].
IM-NET - Explain what happens to a 3D mesh from input to output.
3D mesh is being converted to 3D point cloud.
Features are generated with 3D PointNET, or 3D CNN.
Features have 128 dimension.
Features are being decoded to an occupation function in space (3D coordinates with 1 if the place is occupied in space and 0 otherwise).
IM-NET - how did they do 3D shape generation?
The authors employed latent-GANs [13] on feature vectors learned by a 3D autoencoder as the second stage of the training.
The Wasserstein GAN loss with gradient penalty was applied to stabilize the GAN training. [12]
DeepSDF - What causes similar latent vectors to be close to each other in the higher dimensional space?
Regularisation of the code.
Backpropagation of the code and NN to produce similar SDF for similar shapes.
The decoder is an NN architecture which is continuous and differentiable which produces smooth SDFs for input perturbations.
DeepSDF - how do they do shape completion?
They initialise a random latent vector.
Optimise this latent vector with Backpropagation while the NN weights are frozen with the given data.
Use the optimised latent vector and the NN to complete the shape (querying SDF in unseen areas).
AutoSDF - Descrive the high level architecture steps for the VQ-VAE learning stage.
They patch the input 3D TSDF
They encode each separately
They restrict to learnt codebook words.
They gather into a grid (the same patches order as 3D)
They decode jointly
AutoSDF - what formulation the paper is using to use the transformer architecture?
The probability of the features can be decomposed to a multiplication of the probability of each feature patch given the previous seen ones.
For transformers - the product of the log likelihood of observing a query token given all previous tokens.
AutoSDF - How does the paper create order invariance in inference?
When training the transformer on the completion of the next latent code given the previous lantent codes -> they are shuffling what the transformer ‘sees’ as the previous latent codes in random order and random choice.
AutoSDF - How does the paper deals with conditioned 3D shape generation?
It break down the likelihood to 2 terms
1 The likelihood of the following latent vector given the past.
2 The distribution of the latent vectors given the condition.
1 is learned using transformers while 2 is done using domain encoders and a decoder to the latent vector space.
