Try to get a reliably-failing test case, but at the end of the day it's still a bug. Make sure to state that e.g. "hey, this only fails ~1/2 the time".
(Note that one way to dance around the issue if it's impolitic is to write a new test case that runs the test e.g. 100 times.)
I once reported a HW bug that occurred ~1-in-100k times on ~1/10 devices... (And I found it on iteration 9986/10k, and even that was only because I forgot to stop it before I left for the weekend...) Root cause was one of those "if the stars line up" bugs - a write in a 1 clock cycle window would tristate an output for a bit, and everything would be OK as long as said output retained its value for long enough.