In this work, we pose a question whether it is possible to design and train an autoencoder model in an end-to-end fashion to learn latent representations in multivariate Bernoulli space, and achieve performance comparable with the current state-of-the-art variational methods. Moreover, we investigate how to generate novel samples and perform smooth interpolation in the binary latent space. To meet our objective, we propose a simplified deterministic model with a straight-through estimator to learn the binary latents and show its competitiveness with the latest VAE methods. Furthermore, we propose a novel method based on a random hyperplane rounding for sampling and smooth interpolation in the multivariate Bernoulli latent space. Although not a main objective, we demonstrate that our methods perform on par or better than the current state-of-the-art methods on common CelebA, CIFAR-10 and MNIST datasets. PyTorch code and trained models to reproduce published results will be released with the camera ready version.