SDF

Question 1

DeepSDF - what is the input and the output of the network

Answer 1

Input - latent vector related to a certain shape and a query 3D point in space
Output - the SDF value of the 3D point in space for that shape

Question 2

DeepSDF - why are they not using an encoder-decoder framework

Answer 2

They say that they’re not sure of the usefulness of the encoder and want to use computational resources better.

Question 3

DeepSDF - What is being done with the posterior of the shape code zi, given the shape samples Xi?

Answer 3

pθ (zi |Xi ) = p(zi ) (xj ,sj )∈Xi pθ (sj |zi ; xj )

Question 4

DeepSDF - what is the prior distribution of the latent codes

Answer 4

Gaussian distribution of means = 0 and the same variance in each dimension (while each covariance is 0)

Question 5

DeepSDF - what is the paper expression of the SDF likelihood

Answer 5

exp(-loss(NN(latent_code, 3D point), SDF))

The exponent of minus loss.
The loss is between the NN to the SDF in a 3D point.
The input to the NN is the latent code and that 3D point.

Question 6

What is TSDF

Answer 6

Truncated SDF

Question 7

DeepSDF - How to get the TSDF

Answer 7

Convert the mesh into voxels and then calculate the SDF?

Question 8

ASDF - what is ASDF

Answer 8

Articulated SDF

Question 9

ASDF - What is their contribution?

Answer 9

1 Separate the code area for shape and for articulation
2 No prior assumption on the geometry or joint type, location and range.
3 Adjust the model in inference using Test-time adaptation.

Question 10

ASDF - What are their findings?

Answer 10

That their model can generalise well to out-of-distribution and unseen data (partial point cloud + depth images).

Question 11

Re-param pix int - What is re-parametrisation of the pixel integral?

Answer 11

In rendering optimisation we integrate the rays that go through a pixel. Reparametrisation is to reformulate the integral so it would be done more robustly.

Question 12

Re-param pix int - Where is the problem they are trying to fix?

Answer 12

The neural SDF method for geometry rendering can’t use edge sampling for rendering optimisation.

Question 13

Re-param pix int - What are their findings?

Answer 13

That the reparametrisation of the pixel integral gives correct neural SDF differentials.

Question 14

Re-param pix int - What does “differentiable rendering” means? “Differentiable” to what?

Answer 14

It means that the output of the rendering of a 3D scene is a differentiable function in respect to its variables. The variables are: geometries, materials, light or camera.

Question 15

Re-param pix int - Why is it good to differentiate the rendering? What are possible purposes?

Answer 15

In order to optimise the rendering process for 3D reconstruction, Inverse rendering or Neural Scene representation.

Question 16

Re-param pix int - What is the old way to deal with edges in mesh-based rendering differentiation?

Answer 16

To do edge based sampling meaning to explicitly consider the silhouette in the sampling to build the differentials around the edge.

Question 17

Re-param pix int - Why SDFs do not have a simple parametric form amenable to sampling?

Answer 17

Because they are implicitly define the edge as the area in which SDF(x)=0.

Question 18

Re-param pix int - What is their contribution?

Answer 18

They develop a continuous warping function so neural SDFs rendering can be differentiable.

Question 19

Re-param pix int - How they developed a continuous warping function?

Answer 19

They use the distance to the surface and quadrature on sphere tracer points.

Question 20

Open3D - why should one divide the world to blocks and only then to voxels

Answer 20

Because blocks are a more coarse unit that helps optimise the rendering because of the sparse nature of objects in a scence. Voxels help to add more fine details.

Question 21

Open3D - why we do not maintain a voxel hash map?

Answer 21

we can preserve the data locality instead of scattering adjacent data uniformly into the memory.

Question 22

DeepSDF - Formulate getting the SDF of a point from the model

Answer 22

f_theta(zi, x) sim= SDFi(x)
Where
f is the model
theta are learned parameters
zi is a latent vector of indexed shape I
x is a point in 3D

Question 23

DeepSDF - what are the model’s 4 abilities?

Answer 23

1) To represent the training data.
2) To use the learned feature representation to reconstruct unseen shapes.
3) To apply shape priors to complete partial shapes.
4) To learn smooth and complete shape embedding space.

Question 24

IM-NET - what are the differences to DeepSDF?

Answer 24

IM-NET utilizes 3D CNN encoder and an MLP decoder approach to build latent space and an occupation function to present the 3D shape implicitly [12].

Question 25

IM-NET - Explain what happens to a 3D mesh from input to output.

Answer 25

3D mesh is being converted to 3D point cloud. Features are generated with 3D PointNET, or 3D CNN. Features have 128 dimension. Features are being decoded to an occupation function in space (3D coordinates with 1 if the place is occupied in space and 0 otherwise).

Question 26

IM-NET - how did they do 3D shape generation?

Answer 26

The authors employed latent-GANs [13] on feature vectors learned by a 3D autoencoder as the second stage of the training. The Wasserstein GAN loss with gradient penalty was applied to stabilize the GAN training. [12]

Question 27

DeepSDF - What causes similar latent vectors to be close to each other in the higher dimensional space?

Answer 27

Regularisation of the code. Backpropagation of the code and NN to produce similar SDF for similar shapes. The decoder is an NN architecture which is continuous and differentiable which produces smooth SDFs for input perturbations.

Question 28

DeepSDF - how do they do shape completion?

Answer 28

They initialise a random latent vector. Optimise this latent vector with Backpropagation while the NN weights are frozen with the given data. Use the optimised latent vector and the NN to complete the shape (querying SDF in unseen areas).

Question 29

AutoSDF - Descrive the high level architecture steps for the VQ-VAE learning stage.

Answer 29

They patch the input 3D TSDF They encode each separately They restrict to learnt codebook words. They gather into a grid (the same patches order as 3D) They decode jointly

Question 30

AutoSDF - what formulation the paper is using to use the transformer architecture?

Answer 30

The probability of the features can be decomposed to a multiplication of the probability of each feature patch given the previous seen ones. For transformers - the product of the log likelihood of observing a query token given all previous tokens.

Question 31

AutoSDF - How does the paper create order invariance in inference?

Answer 31

When training the transformer on the completion of the next latent code given the previous lantent codes -> they are shuffling what the transformer 'sees' as the previous latent codes in random order and random choice.

Question 32

AutoSDF - How does the paper deals with conditioned 3D shape generation?

Answer 32

It break down the likelihood to 2 terms 1 The likelihood of the following latent vector given the past. 2 The distribution of the latent vectors given the condition. 1 is learned using transformers while 2 is done using domain encoders and a decoder to the latent vector space.