— very beautiful
Deep autoregressive models are sequence models,
yet feed-forward (i.e. not recurrent);
generative models, yet supervised.
They are a compelling alternative to RNNs for sequential data, and GANs for generation tasks.
Difference bw RNN and Autoregressive model -
previous states are not provided as hidden in case of autoregressive models, they are provided as just another input
Autoregressive model -
simply
feed forward model which predicts future values from past valuestradeoffs -
stable, parallel, easy to optimize training, faster inference computations,
+ no backprop needed
Generative model does P(X,Y) -GAN and VAE
Discriminative model does P(Y|X)
autoregressive models have worked for both continuous and discrete data -
audio (WaveNet), images( PixelCNN+), text( Transformer)
But GANs, can’t model discrete data
But problem is, unlike GAN, we cannot train on noise,
( we can only train on input label )
→ so Gan can also convert noise into something semantic
::: How to solve this problem
—
— latent vector
— convolutional encoder
—deconvolutional decoder
doing things with latent vector seems to have a big effect
smth to do with discret and continuoss and output lenght, like text is discrete, and its output length is variable, so even this is not a problem, as we can use a stop token.
but problem with images and audio
backpropagation cant learn longer before time, or have long term memory