Think about what occurs if a bullet comes apart unevenly. It has fluids and tissues acting against it as it penetrates and moves forward through the animal. If the bullet comes apart unevenly, there will be an imbalance induced by that right away, and if there's an uneven amount of material hanging off on one side of the bullet, it will alter the path of the bullet as it gets pulled in a different direction from the opposing force.
Spinning faster creates more centrifugal force, and helps the petals open more evenly and shed more reliably. Once that occurs, it allows the remaining shank to also continue in a straight line of penetration.
If the bullet wasn't fully stable in flight, it'll impact at an angle and cause the bullet to not open evenly in most cases as well. This can occurred even if it was stable, but the nose hits bone at an angle and deforms the tip unevenly. Either occurrence typically results in the bullet not performing as designed at that point and it typically will just tumble at that point the rest of the way through the animal until it stops or exits.
Tumbling is definitely instability. What caused the tumbling isn't always from instability though, if that makes sense. Sometimes it is, like if it was unstable in flight, like I said.