In Missouri back in the 80's and 90's roughly 98.7% of bucks killed were 1.5 year olds. Spikes, forkies, sixes, small 8's and 10's. The was data from the MDC. They would cut back the skin and identify the tri molar which is an almost 100% guarantee its a 1.5 year old. My wife checked in a 115" 2 year old and the guys at the check station KNEW there was no way a woman killed a buck that big. This was about 1990.
Antler restrictions hit about 2000. I was so thankful they did. It took about 5 years for them to EDUCATE people on deer age. Back in the 80's a gray colored forky was an old old deer. Nobody had a clue what age should look like. By 2010 they had learned.
In 2010 I knew of over a dozen deer that went from 184 up to 218 in my county. Then in 2011 and especially the drought of 2012 EHD dropped our deer population back to 1960 levels. That was devastating. By about 2019 the herd was getting back to a good age structure. All six of my kids have grown up with 4 on one side as a rule. Yes it may have cost some of them a buck as they grew up but they still shot does. Ironically my 4 daughters self imposed their own limits on themselves that they then passed many bucks that most would call shooters.
I know a lot of guys that pass a lot of 140" deer and go buckless many years. The 4 points on a side rule don't matter to them at all anymore. They refuse to shoot a buck before his time.
One reason I pushed my game warden to look into an antler restriction was inbreeding. We were witnessing most of the 1.5 year old bucks doing the breeding along with button bucks even. We had some deer showing up with massive overbites(inbreeding trait). Why?????? The bucks don't start dispersing and traveling great distances to breed until 2-4. So all the baby bucks were breeding their moms and sisters etc. do that for 30 years and it takes a toll.
If you can pass a buck one to two years they will at least be a decent buck with more meat.
So I LOVE what Ant. Res. have done to educate people. If all you have ever seen is a 1 or 2 year old buck you have no clue what 3 - 6 yr olds look like. Bill Jordan of Monster bucks said his first video they made was full of high 120's and people all over the USA were eating up those "Monster bucks" on film.
As far as negatively affecting genetics I don't think it is hardly even possible in most populations of deer. Bucks are traveling 10 miles to breed does, most will end up with 4 on one side by 4 or 5 years of age anyway. Kids can now shoot anything they want so that opens up letting a kid tag a 5 year old six point(dang rare anyway).
A buck in Iowa was a giant 8 point his 3-5 years. His 6th year he added on about 15 extra stickers and small typical tines. Roger Sapper say that is why he doesn't believe in culling 8 pointers until they hit 6 years of age.