Rx: Only backoff the peer timeout once