Preheat the oven to 325°F. Grease a 9x5-inch loaf pan and line with parchment paper, then set aside.
In a medium bowl add the flours, salt, baking soda, and baking powder, and mix until just combined with a fork.
In a large bowl or the bowl of a stand mixer, cream together the sugar, butter, and ¼ cup peanut butter until light and fluffy. Add the eggs, vanilla, and milk, and mix until smooth.
Slowly add in the flour mixture while the mixer is running.
Fold in the mashed bananas and pour into the prepared loaf pan.
Scoop the remaining ¼ cup peanut butter into a ziplock bag, and cut the corner off the bag. Pipe three lines of peanut butter across the top of the batter, then drag a toothpick or skewer through the batter to swirl the peanut butter.
Bake for 60-70 minutes, or until a toothpick inserted into the center of the loaf comes out clean.