This 'issue' will always occur when you get back to the initial room (A) without talking to the correct 'brother'.
The way to deal with it is to go through all rooms, B, C, D, E and everytime, check at the NPC if you can turn in the quest or not. If you forgot to check at the NPC for quest or skip a room by accident, there is a good chance that you cannot complete the quest at the end.