I have both the f/4 non-IS and f/2.8 IS lenses. I seldom use either. I got the f/4 when I got a 30D and added the f/2.8 when I got a 1Ds. A Sigma 18-200 lives on the 30D and a Sigma 18-250 lives on my 550D. Frequently a Canon 28-300 is on my 5D and the 1Ds has a 100-400. For low light when I don’t need focal length, I use faster prime lenses.
But, last weekend I was shooting in low light with the 70-200 f/2.8 in the 5D at ISO 10000 to get some shutter speed.
Of those two lenses, I think your choice of the f/2.8 with IS, while it is heavier and more expensive, was the right choice. The extra stop gives less depth of field while IS lets you shoot at slower shutter speeds, so it is more versatile.