Physics-based simulation of energy use in buildings is widely used in building design and performance rating, controls design and operations. However, various challenges exist in the modeling process. Model parameters such as people count and air infiltration rate are usually highly uncertain, yet they have significant impacts on the simulation accuracy. With the increasing availability and affordability of sensors and meters in buildings, a large amount of measured data has been collected including indoor environmental parameters, such as room air dry-bulb temperature, humidity ratio, and CO2 concentration levels. Fusing these sensor data with traditional energy modeling poses new opportunities to improve simulation accuracy. This study develops a set of physics-based inverse algorithms which can solve the highly uncertain and hard-to-measure building parameters such as zone-level people count and air infiltration rate. A simulation-based case study is conducted to verify the inverse algorithms implemented in EnergyPlus covering various sensor measurement scenarios and different modeling use cases. The developed inverse models can solve the zone people count and air infiltration at sub-hourly resolution using the measured zone air temperature, humidity and/or CO2 concentration given other easy-to-measure model parameters are known.