Something that would probably help is using javascript to track how many people have actually had your comment viewable in their browser. Then take the ratio of views to votes.
Eventually you might also find that you can determine the average # of views of your comment purely based on the location of it on the page, in which case the javascript tracking would no longer be necessary. Although it's also possible that the characteristics of comment viewing are too complex to be able to simplify it to that.
Also, maybe instead of a fixed score required to be orange, take a distribution of all users and only make the top X percent qualify.