You may already have worked out that this suburb was named due to its close proximity to an actual observatory - the Royal Observatory, Cape of Good Hope, founded in 1820.
Being so close to Cape Town itself makes this suburb one of the oldest in the province, and the reason for it looking old and worn out.
Similarly to Woodstock, Observatory (affectionately known as Obz) escaped segregation laws of the apartheid era, developing a interesting mixed culture of its own, and when I think of Observatory, for some reason I think of artists, musicians, hippies, crystals, dream catchers, incense, and hipsters. :P