I agree with you that trust is the critical factor. I am not anti-gun, or anti gun-ownership--I would be quite hypocritical if I was. But I also decide my level of trust based upon the context. I have been to gatherings such as you describe, and have no problem with them at all. I have quite enjoyed the ones I have been to, minus the ribbing I generally take for being one of the worst shots.
But when I see photos of a gun-toting group of New Black Panthers outside a voting location in Philadelphia years back--CLEARLY intimidating voters by their presence, and sending a message--I am concerned about the impact on voters. Now I see essentially the same thing in a gun-toting group of MAGA wearing guys outside a state capitol--CLEARLY intimidating legislators by their presence, and sending a message--and I am concerned about the impact on legislation. I also see armed groups violating rules on the size of parties, and threatening a shootout if police show up.
The same people who supported the NBP gun-toting group condemn the current ones. And those who condemned the NBPs praise the current ones. I find both very wrong, and very much a step towards mob rule. I also find it deplorable that for many of the people I hear talk about it their position has no consistence about gun rights, but rather they support gun rights only for those who agree with them.
I am strongly against ANY group that uses guns for intimidation in these ways. I do not see either of these is self defense. Both are a clear "we disagree with the way things are and see no problem in subverting the legal process by implied force". But at least I can appreciate the few people I have heard support BOTH as OK means of expression, or who condemn BOTH.
However when I ask either side to explain the contradiction in their support of one and condemnation of the other it almost always (I only use "almost" because I may not remember one instance--I really do think it has been "always") winds up that they support the people who politically think/look like them, and condemn the group that politically thinks/looks different than them. People essentially say "I am scared of a MAGA guy with a gun, but not scared of a Brother with a gun", or the opposite.