A little tip for LLM-as-classifier that worked for me.Use binary questions rather than multi-class. Then you can use the consistency as another signal of your pipeline working or not, on top of accuracy.
Otterly99 · · focus · HN ↗
Use binary questions rather than multi-class. Then you can use the consistency as another signal of your pipeline working or not, on top of accuracy.