Diagram interpretation, an essential element in question answering, is the problem of identifying visual entities, properties, relations and correspondences to specialized knowledge repositories. Previous work uses hand-engineered rules for understanding specific diagrams. In this project, we introduce a unified, end-to-end framework to diagram interpretation and reasoning that is applicable to a wide range of diagrams.