This has not happened on me on Windows Phone yet but it has happened to me on other devices. In the past it's always been something along the lines of the carrier "turn off the indicator" signal was lost (or never sent), and I fixed it by leaving myself another voicemail, then listening to and deleting that one. That sent the "all clear" down to the phone and the indicator would go away. (This was particularly bad when I was on Sprint, where this happened about 50% of the time.)
I'm assuming you are not using Visual Voicemail because that would be different and you wouldn't have this problem I would expect - or if you did just bringing up the Voicemail page would be enough to clear the flag because the phone is refreshing the list of messages at that point.