From the owner's manual;
"Insert a 1/4 inch nut driver (Guild NKWREN neck wrench or equivalent). The guitar should remain tuned to pitch during this procedure."
I always go counterclockwise first and make sure nut is free before going in clockwise direction, a drop of machine oil on old dry threads doesn't hurt either. The bigger the strings the harder you have to turn the nut, and there is a snapping point as many have found out.
If as tight as you can get it by hand isn't enough (with nut driver, roughly 1/2 turn from seated is about maxed out), I usually go for a smaller string size.
If you own a good heavy 24" rule, great for checking neck angles, check your neck for bow/relief, you might find on older guitars and basses that the neck is permanently bowed some, and will have too much dip without any strings on the neck, and with truss rod maxed out.