Rock Band is vastly superior. Guitar Hero and GH2 can give it a run for it's money, but GH3's revamped look, style, and feel without Harmonix costs it some major points. Flailing hammer-ons can only be fun for so long, and the last two sets are ridiculously hard. The singer and drummers are incredibly ugly and blockish, and the display is annoying and hard(er) to read.
I think
Zero Punctuation covered my feelings on the game pretty well.
Rock Band delivers a much more complete experience. Even leaving out the added drums and karaoke features, the guitar experience is much greater. Ignoring the controllers for now (another rant for another time, although ultimately I prefer the Rock Band Stratocaster over the GH3 Les Paul), the songs have a much wider range in skill level and play-styles. The HuD is much better organized and easier to read, and the black backing provides much better contrast than the multi-colored fretboards in GH. You're required much more accuracy to hit the notes, especially with hammer-ons, so even when some of the licks are a bit less flashy, they are still very skilled. The option to use the solo frets is nice, although personally I find them a bit small for my liking (although I have a feeling I'll need to master them if I ever want to beat Green Grass or Ride the Lightning). Character customization (and generally nicer looking characters) is a huge plus, same with the much more detailed tour mode. The down-loadable content is updated every week, with at least three new songs (much better than the mere handful that GH3 has put out). Given the drums and singing on top of that, Rock Band is a much more complete, much more polished, much more well-rounded game that I enjoy vastly more than GH3. I still play GH3 from time-to-time as a change of pace, but there's no way that it can take Rock Band's place.