For brown butter:
Simmer 2 cups plus 2 tablespoons butter in a large nonstick skillet over medium heat until browned bits form, 6-8 minutes. Pour into a 2-cup measuring cup. If needed, add more butter to measure 1 3/4 cups plus 2 tablespoons (added butter will melt).
Whisk flour, sugar, and salt in a medium bowl. Stir in 3/4 cup plus 2 tablespoons brown butter (reserve remaining butter for cake); stir until moist clumps form. Stir in ginger. Set aside.
Preheat oven to 325°. Butter pan generously. Whisk all-purpose flour and next 7 ingredients in a medium bowl. Using an electric mixer, beat remaining 1 cup browned butter and sugar in a large bowl until light and thick, 2-3 minutes. Add eggs one at a time, beating to blend between additions. Beat in sour cream, milk, and vanilla. Stir in flour mixture just to blend.
Spoon half of cake batter into prepared pan; smooth top. Scatter 1 cup of topping over. Spoon remaining batter in dollops over; smooth. Add almonds to remaining topping; squeeze to form 1/2" clumps and scatter evenly over batter in pan.
Bake until a tester inserted into center of cake comes out clean, about 1 hour 20 minutes. Let cool in pan on a wire rack for 10 minutes. Run a knife around edge of pan to release cake. Remove pan sides; let cool completely.