What happens if you clean in the barrel in the middle of the shooting? Are the next 2 shots touching again?
From what I've noticed a clean barrel will affect point of impact more than a cold barrel will. Although if your factory barrel gets heated it could start stringing shots all over the place. (*could)
I would shoot 10 shots that are loaded identical into a group. You may find that just as many end up being over by bullet hole #3. If that happens it will tell you that is just the accuracy of your rifle/ammo.
You may also want to experiment with how you are holding that rifle. The majority of my centerfire rifles I do not have to hold onto overly tight. I've got used to more of a relaxed hold, but not quiet "free recoil". I have grown to enjoy shooting this way more because it's easier to keep the crosshairs steady. I did have this one rifle though that sometimes acted erratic with the groups. For the longest time I really thought that the scope base/rings or something was loose and shifting on me. Just out of curiosity I tried holding onto the rifle super tight and shot a group of 5. I didn't have much faith in the test. The crosshairs were moving a bit because I was holding onto the rifle so tight. (Which by the way, was the way my dad always taught me to hold a rifle starting 25 years ago). I assumed this would translate to a larger group. However it did not. When I compared targets, the "tightly held" target was obviously the smaller group. And it looked like a group, not a couple good shots along with a couple flyers an inch away. So now I know with that specific rifle, I need to hold onto tighter than my other rifles.
Hope this helps.