In a large saucepan, combine the flour and brown sugar. Stir in milk until smooth. Cook and stir over medium-high heat until thickened and bubbly. Reduce heat; cook and stir 2 minutes longer. Remove from the heat. Stir a small amount of hot filling into egg yolks; return all to pan, stirring constantly. Bring to a gentle boil; cook and stir 2 minutes longer. Remove from the heat; stir in butter and vanilla. Cool to room temperature without stirring. Fold in the whipped cream. Layer a third of the pudding in a 2-quart glass bowl; top with half of the bananas. Repeat layers. Top with remaining pudding. Sprinkle with nuts if desired. Cover and refrigerate for at least 1 hour before serving.