The energy performance of a building is strongly influenced by its level of solar exposure, in turn affected by the climate, built context, and building morphological characteristics. Since these are typically fixed at the early-design phase, performance assessment methods based on solar considerations at the urban scale are essential to support early decision-making. As the adaptation of the well-developed building performance simulation methods to the urban scale lead to complexity issues, it is of interest to verify whether simpler metrics can act as performance indicators, as is often done at the building level with quantities such as form factor. This paper first presents a review of existing methods that aim at evaluating aspects such as passive heating and photovoltaic potential. We classify each evaluation metric based on its level of complexity, before choosing a representative subset to be further examined. The goal is to test whether simple metrics fulfill their role as performance indicators when applied to neighborhoods. To do so, the selected metrics, including geometrical parameters (e.g. compactness) and solar exposure levels (e.g. annual irradiation), are compared with simulation results (e.g. heating need), taken as reference values. This is done for two sets of designs: six distinct typologies and a large number of variants from a base case typology. Results, which show various levels of correlation between simple metrics and the reference values, highlight the limitations of using any one metric and indicate a need to revise the definition and evaluation metrics of various performance criteria for neighborhood designs.