This is the simplest pattern I thought of that actually works. But there are probably other things you could do.

Someone mentioned related projects:

https://github.com/xjdr-alt/entropix

https://github.com/IINemo/lm-polygraph

I did a few explorations of looking at parts of generation (e.g. first n-tokens, singular tokens) - it has roots for sure.