It has a lot to do with crystal formation. (I’ll explain more, be patient)
Lipstick has to be a matrix - a mesh network of linked wax crystals woven through a semi-solid gel. Too much wax, and you have a waxy lump with no payoff - too little, and you have a gooey mess with no structure.
To pull this off, not only do you have to have just the right wax mixture, and just the right amount of wax to gel ratio, but your wax crystals also need to be just the right size. Too small, and your wax matrix won’t form, too large, and your stick will feel gritty on the lips.
So, how do you get your wax matrix to be near perfect? Since wax crystals grow larger slowly, only start to grow as your lipstick mass cools down below it’s melting point, and (if you’ve formulated your lipstick properly), will not grow at all at room temperature, you can precisely control the size of your wax crystals by limiting the time they have to grow. You can do that by chilling your lipstick mold as soon as the lipstick is poured. Some folks even pre-chill their molds.