This reverts commit 90819abb56.
This logic was wrong, the original code is correct. The direct
impact is that we allocate up to approximately a squared amount
of memory compared to what we should allocate.
Acked-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>