In a medium bowl, whisk together flour, cocoa mix, baking soda and salt.
In a large mixing bowl, beat together butter and sugar until creamy. Add the eggs, one at a time, mixing until combined after each. Mix in milk and vanilla. Add the dry ingredients and mix just until combined.
Using a teaspoon cookie scoop, drop the dough onto parchment lined baking sheets. Bake until the cookie are set, about 8 minutes. Remove and carefully place a marshmallow, cut side down, in the center of each cookie. Place back in the oven and continue baking until the marshmallow begins to look puffy, about 1 to 2 minutes. Remove and let cookies sit on the baking sheet for 1 minute before moving them to a wire rack to cool completely.
To make the glaze: Place confectioners' sugar in a medium bowl. In a small saucepan, add butter, milk and cocoa. Cook over medium heat until the mixture comes to a boil - let cook for 1 minute. Remove from the heat and let cool for 5 minutes. Pour mixture into the bowl of confectioners' sugar and beat until creamy.
Immediately spoon a little of the glaze over each of the cooled cookies.