In a large mixing bowl, combine flour, sugar, baking soda, salt, nutmeg, and cinnamon. Blend together until well combined. Add chopped apple, and mix until apple is evenly distributed and covered with flour.
In a medium bowl, combine buttermilk, eggs, banana, vanilla, and oil. Mix well, and slowly incorporate into dry ingredients. Gently fold together just until combined.
In a small bowl, (like how I'm downsizing each time? ;) haha) combine 1/2 cup granulated sugar and 1 tablespoon cinnamon. Mix well.
In a greased loaf pan (9x3x5), pour 1/4 of the batter, repeat with a second loaf pan. Sprinkle half of the cinnamon swirl mixture over batter; repeat. Cover both pans with remaining batter and swirl a butter knife through to make the marbled appearance.
Bake for an hour to an hour and 5 minutes, or until golden brown and a toothpick inserted near the center comes out nearly clean. (A little gooey is okay, but if loaves start getting too dark before they're fully baked, cover with aluminum foil).
For the glaze, mix together milk and powdered sugar and drizzle over cooled bread.