Preheat oven to 350°F Butter and flour a 9X5 loaf pan.
In a small bowl using a fork, mash the bananas. You should have 11/3 cups. In a separate bowl, stir together the all purpose flour, whole-wheat flour, baking powder baking soda, salt, nutmeg, orange zest and nuts, set aside.
In a large bowl, whisk the eggs until blended. Stir the brown sugar, butter, and milk. Add the dry ingredients in 3 batches alternatively with the mashed bananas in 2 batches, beginning and ending with the dry ingredients, just until moistened. Be careful not to overmix!
Pour the batter into the prepared pan smoothing the top with a rubber spatula. Bake for 55-60 minutes or until toothpick inserted into center comes out clean. If the top begins to brown too much, cover with a sheet of aluminum foil the last 15 minutes of baking.
Transfer to a wire rack and let cool in the pan for 5 minutes. Turn out into wire rack and let cool completely, before slicing.