During recent years, urban building energy modeling has become known as a novel approach for identification, support and improvement of sustainable urban development initiatives and energy efficiency measures in cities. Urban building energy models draw the required information from the energy analysis of buildings in the urban context and suggest options for effective implementation of interventions. The growing interest in urban building energy models among researchers, urban designers and authorities has led to the development of a diversity of models and tools, evolving from physical to more advanced hybrid models. By critically analyzing the published research, this paper incorporates an updated overview of the field of urban building energy modeling and investigates possibilities, challenges and shortcomings, as well as an outlook for future improvements. The survey of previous studies identifies technical bottlenecks and legal barriers in access to data, systematic and inherent uncertainties as well as insufficient resources as the main obstacles. Furthermore, this study suggests that the main route to further improvements in urban building energy modeling is its integration with other urban models, such as climate and outdoor comfort models, energy system models and, in particular, mobility models.