Heat 2 tablespoons oil in Dutch oven over medium-high heat until shimmering. Add onion, garlic, red pepper flakes (if using), and bay leaf. Cook, stirring frequently, until onion is translucent, 3 to 5 minutes. Stir in tomatoes and their juice. Using potato masher, mash until no pieces bigger than 2 inches remain. Stir in sugar and bread (Crustless, torn into one inch pieces); bring soup to boil. Reduce heat to medium and cook, stirring occasionally, until bread is completely saturated and starts to break down, about 5 minutes. Remove and discard bay leaf.
Transfer half of soup to blender. Add 1 tablespoon oil and process until soup is smooth and creamy, 2 to 3 minutes. Transfer to large bowl and repeat with remaining soup and oil. Rinse out Dutch oven and return soup to pot. Stir in chicken broth and brandy (if using). Return soup to boil and season to taste with salt and pepper. Serve soup in individual bowls. Sprinkle each portion with pepper and chives and drizzle with olive oil.