Energy use in buildings has great variability. In order to design and operate low energy buildings as well as to establish building energy codes and standards and effective energy policy, it is crucial to understand and quantify key factors influencing building energy performance. This study investigates air-conditioning (AC) energy use of four office buildings in four locations: Beijing, Taiwan, Hong Kong, and Berkeley. Building simulation was employed to quantify the influences of key factors, including climate, building envelope and occupant behavior. Through simulation of various combinations of the three influencing elements, it is found that climate can lead to AC cooling consumption differences by almost two times, while occupant behavior resulted in the greatest differences (of up to three times) in AC cooling consumption. The influence of occupant behavior on AC energy consumption is not homogeneous. Under similar climates, when the occupant behavior in the building differed, the optimized building envelope design also differed. Overall, the optimal building envelope should be determined according to the climate as well as the occupants who use the building.