While I hate the "only one gun" debates, I will throw my experiences in -
Through the decades and having used, owned and still own numerous 6mm's and varied 224's cartridges and rifles, I have to say my fast twist 22/250AI's (almost identical to a 22CM) and 22/243's with the VLD heavies "ballistically" outperform my varied 6mm's at longer ranges. This is no small thing for me to admit, for a long time, I was, and still am, a dedicated 243, 243AI and 6mmAI fan! In the faster twists with VLD's, they work well at the longer ranges, but when true 22 cal VLD's came along, I discovered I could make those long shots with less drift and drop than my old 6's.
As for game and where legally allowed, I have killed numerous deer sized and smaller animals with varied 22 cartridges and loads with varied bullet choices. Whether it is the 6's or 22's, bullet selection is always critical, and of course, the old bullet placement discussion.
As for the op's request of strictly "only one gun" of the 2 calibers, I would choose my 6mmAI's for the versatility of using 55gr for smaller varmints with the heavier varmint bullets for less drift, and then switch to the medium game bullets for those. Thus, "only" giving the nod to the slightly heavier, not necessarily better BC, projectiles for longer range kills on deer sized and similar game. Even that decision is somewhat based on my bias for my old friends.
(edited to add) - Part of "my" decision also rests in the fact several states do not allow anything smaller than 6mm for medium game, and since this was an "only one gun" op, I kept with the dual purpose of the two choices.