Make sure you're in front of the building door before clicking the button.
Open Building Door