Energy usage and thermal comfort are the pillars of smart buildings. Many research works have been proposed to save energy while maintaining a comfortable thermal condition. However, most of them either make the over-simplified assumption on thermal comfort with unsatisfied comfort performance or deal with the single-zone thermal control only with limited practical impact. A few preliminary pieces of research on multi-zone control are available, but they fail to keep pace with the latest advancements in the deep learning-based control techniques. In this paper, we investigate the multi-zone thermal control with optimized energy usage and canonical thermal comfort modeling. We adopt the emerging multi-agent deep reinforcement learning techniques and propose to model each zone as an agent. A multi-agent framework is established to support the information exchange among the agents and enable the intelligent thermal control in the heterogeneous zones. Accordingly, we mathematically formulate a problem to optimize both energy and comfort. A multi-zone thermal control algorithm (MOCA) is proposed to solve the problem by deriving optimal control policies. We validate the performance of MOCA through simulation in professional TRNSYS, configured based on our real-world laboratory. The results are promising with up to 15.4% energy-saving as well as satisfied thermal comfort in different zones.