In the context of structure to structure transformation tasks, sequences of discrete symbolic operations (e.g., op codes or programs) are an important tool but are difficult to learn due to their non-differentiability. To support learning sequences of symbolic operations, we propose a differentiable tree interpreter which compiles high-level symbolic tree operations into subsymbolic matrix operations on tensors. We introduce a novel Differentiable Tree Machine (DTM) architecture which combines our interpreter with an external memory and an agent which learns to sequentially select tree operations to perform a desired transformation. On synthetic supervised tree-to-tree transduction tasks, we demonstrate that our approach achieves drastically better compositional generalization and interpretability than baselines such as the Transformer and LSTM.