I’ve been doing that too. Rarely had any difficulty, but I suppose if you don’t have strong fingers, I could see it being an issue. I had an XT mounted inside a box, up a tree, which I was swapping out for an XT2 yesterday, so I can take advantage of recordings on demand. That gave me a devil of a job, not least because I was also up a ladder, and over a river, but I managed in the end, and heard that satisfying click. Wasn’t keen on taking both hands off the top of the ladder, but in the end decided I had no choice, and managed to ‘wrestle’ it in, as you described. I couldn’t even get the old one out without getting two hands to it, so that decision had already been made, in retrospect.
Not funny if it’s your issue, but slightly comical in that we read many community reports of, ‘barely fastens on’ or ‘would drop off in a stiff breeze’ etc. Your experience is more like mine, in that it takes a fair old shove to get them in.