The macro, is incorrectly named. The usage of the macro MAILBOX_MAX_MSGS_IN_FIFO is as mentioned below.

This is only used as a mask for the bits 2:0 for the MAILBOX_MSG_STATUS register and not really the actual number of messages. As the mask needs to be 0x7 for the last three bits, the value of (MAILBOX_MAX_MSGS_IN_FIFO - 1) i.e. 0x7 fits fine. But this is an issue with the naming of the macro.