Urban and building morphologies, usually determined at the design stage, have been proven as important determinants of energy usage at later operational stages. However, questions remain regarding the identification of the key determinants that influence urban building energy usage. To address this, in this study, an urban building dataset of 539 residential buildings and 153 public buildings was used to extract the building morphology factors as the determinants. A principal component analysis was performed to identify the key determinants for three buildings groups—residential buildings, residential blocks, and public buildings. The results show that the key determinants for residential buildings are their orientation, ratio of obstruction height to the canyon width from the south and west directions, shape coefficient, perimeter-to-area ratio, and building aspect ratio. The key determinants for public buildings are similar to those for residential buildings with the exception of the ratio of the obstruction height to the canyon width from the south direction. The key determinants for residential blocks are the ratio of the obstruction height to the canyon width from the south and west directions, mass space proportion, building aspect ratio, and floor area ratio. The findings of this study provide insights into the key drivers of urban building energy usage and the strategies that could be used to improve urban energy planning.