Yeah it's not like we're wasting powder really. But we need consistent pressure, and it helps to have the powder burning completely inside the barrel (ideally, most in the chamber). This takes bottling the charge in the chamber, so that it's confinement is constant through the burn. Any unburned powder slug traveling with the bullet will burn differently and inconsistently.
You're exactly correct, bottling the charge in the chamber is very important because pressure affects the burn of discrete kernels of powder - it's physics, Gay-Lussac's Law states that the pressure of a gas at a constant volume is directly proportional to the temperature. Meaning higher pressure = hotter temps. Hotter peak temps will burn through kernels faster, so hitting the correct, highest pressure in the chamber does more for having a consistent burn than any amount of barrel length. That's why a pistol length barrel will hit 80% or better of maximum possible velocities for a chambering from the longest possible barrel; the most meaningful mass of powder is burned very early on because of 60k+ chamber pressures in any decent modern round.
QL shows the point in the burn where peak pressure is reached. Running one of my N565 300 RUM loads pressure peaks at 65000psi when the bullet is at 3" of travel, and goes below 30k PSI by 14"of travel.
Velocity pickup from a longer barrel is "free" in the sense all that velocity was already paid for in the chamber and the longer barrel is keeping the bullet on the wave longer. There's a 14% velocity pick up by doubling barrel length with that 300 RUM load meaning the last few inches barrel, while still helpful, are returning a significantly smaller benefit that the first 14 inches.
@Ernie made this very good point in another thread that if a shooter needs to hit an arbitrary velocity for a bullet then short barrels can get you there - because 80-90% of velocity comes from the chamber burn. A 300 RUM in a 14" will still outrun a 30-06 in any length because the first few inches of the barrel are all it takes to burn the most significant amount of powder.
I still run long tubes for the free velocity on most of my rifles because I don't care about how they'll pack down, but I have multiple 8-16" barreled SBRs and just got my first bolt action pistol. They're a fun distraction from my mega-thumpers projects.
The incandescent flash at the end of our barrels often isn't "unburned powder". It is completed burn gasses escaping the barrel that are still hot enough to incandesce.
That's right, primary flash is the wavefront of the hot ejecta gasses hitting cooler air, the gas is hot and visual light is a form of radiation. There's no secondary flash from re-combustion in the ejecta after oxygen exposure from a rifle because there's more than sufficient oxygen in the case and volume in the barrel to consume all the propellant.
Very short revolvers and SBRs will occasionally shoot sparks - those are partially consumed kernels that can't burn fast enough in low pressure to be completely consumed. They don't show up as a fireball of consumed kernels but unburned gasses because the surface area of the kernel never burned through.