I had a similar experience. I was stuck to a little over 5 minutes for the swings with the 32kg. I stopped S&S as it was really hard on my hands. Grip strength was probably too weak for the 32kg bell and I had to grip it really hard, which resulted in torn calluses every few weeks. Not fun. After stopping S&S, I got stronger in the barbell deadlift and did minimal conditioning. Definitively less than while I was doing S&S. This was confirmed by a resting heart rate that was about 5BPM more after a few months of stopping S&S than it was right after. I did swings sparingly, probably about one 100 swings session every 7-14 days.
Surprise: when I retested the swings after a while (I am too lazy to go through my training logs to know how many months that was), I nailed the swings in less than 5 minutes. I wasn't expecting that. The name says it all: Strongfirst. Why didn't I achieve the goal with the swings only? I may have had technical deficiencies, or tried to go heavy too quickly, or simply lacked patience. However, my guess is that grip strength was lacking, which put the brake on my force production during the swings, which stopped progress. After deadlifting for a while, not only did my grip go up, but my body was overall stronger, and the swings got easier.