Preheat oven to 300°F. Place the challah cubes on a large rimmed baking sheet and toast until golden and crisp, about 30 minutes stirring halfway through. Set aside. Increase oven temperature to 325°F.
Grease a 2 quart baking dish (or two smaller casserole dishes) and set aside.
While the bread is toasting prepare the apple mixture. Heat a medium saucepan over medium heat and add 3 tablespoons of butter. Once melted, add the apples, granulated sugar and 2 tablespoons water. Cook, stirring frequently, until the apples are softened, about 15 minutes. Add the apple brandy and stir until the sauce is syrupy, about 1 minute. Remove from the heat.
Layer half the toasted bread cubes in the prepared pan. Using a slotted spoon, scoop out half the apples and spread evenly over the bread. Add the remaining bread to the pan, then top with the remaining drained apples. Reserve the apple juice and set aside.
In a large mixing bowl with a spout, whisk together the egg yolks, brown sugar, half & half, milk, cinnamon, nutmeg, vanilla and salt. Add the reserved apple juice and whisk again to combine.
Slowly pour the egg mixture over the bread and apples. Gently press down on the bread using the back of a spatula to ensure all pieces are moistened. Allow the mixture to soak for 15-20 minutes. Cover the pan with foil and bake for 45 minutes.
Remove the foil from the pudding, drizzle with 2 tablespoons melted butter, then sprinkle with 2 tablespoons coarse sugar. Return to the oven and bake, uncovered, for 20 to 25 minutes.
Remove from the oven and cool on a rack for 30 minutes (or more) before serving. Store leftovers in the refrigerator. Serve with Apple Brandy Sauce or ice cream if desired.