Cook jam and brandy in 10-inch nonstick skillet over medium heat until reduced to thick syrup, 2 to 3 minutes. Remove skillet from heat and place plums cut-side down in syrup. Return skillet to medium heat and cook until plums shed their juices and thick syrup is again formed, about 5 minutes, shaking pan to prevent plums from sticking. Cool plums in pan, about 20 minutes.
Adjust oven rack to middle position and heat oven to 350°F Grease and flour 9-inch springform pan. Process sugar and almonds in food processor until nuts are finely ground, about 1 minute. Add flour, baking powder, and salt; pulse to combine. Add butter and pulse until mixture resembles coarse sand, about ten 1-second pulses. Add eggs, vanilla, and almond extract (if using) and process until smooth, about 5 seconds, scraping bowl once if needed (batter will be very thick and heavy).
Transfer batter to prepared pan; using spatula, spread batter evenly to pan edges and smooth surface. Stir plums to coat with syrup. Arrange plum halves, skin-side down, evenly over surface of batter. Bake until cake is golden brown and wooden skewer inserted into center comes out with few crumbs attached, 40 to 50 minutes. Run paring knife around sides of cake to loosen. Cool in pan on wire rack until just warm or to room temperature, at least 30 minutes. Remove cake from pan and dust with confectioners’ sugar. Cut into wedges and serve.
Note:This recipe works best with Italian plums, which are also called prune plums. If substituting regular red or black plums, use an equal weight of plums, cut them into eighths, and stir them a few times while cooking. Arrange slices, slightly overlapped, in two rings over surface of cake. Do not use canned Italian plums. Blanched whole almonds can be used but must be processed 30 seconds longer until finely ground. The brandy can be omitted, but then you will need to melt the jam with 1 tablespoon water before adding the plums. Don’t add the leftover plum cooking liquid to the cake before baking; reserve it and serve with the finished cake or over ice cream. The cake can be served with lightly sweetened whipped cream.