Irregularly-sampled time series occur in many domains including healthcare. They can be challenging to model because they do not naturally yield a fixed-dimensional representation as required by many standard machine learning models. In this paper, we consider irregular sampling from the perspective of missing data. We model observed irregularly sampled time series data as a sequence of index-value pairs sampled from a continuous but unobserved function. We introduce an encoder-decoder framework for learning from such generic indexed sequences. We propose learning methods for this framework based on variational autoencoders and generative adversarial networks. We focus on the continuous-time case and introduce continuous convolutional layers that can interface with existing neural network architectures. We investigate two applications of this framework: interpolation and time series classification. Experiments show that our models are able to achieve competitive or better classification results on irregularly sampled multivariate time series classification tasks compared to recent RNN models while offering significantly faster training times.