Evaluating whether to add AI detection as a required pre-publish gate in our CMS workflow. concern is that false positives will create publishing delays and frustrate writers. anyone running this in production. specifically: what threshold did you set, how do you handle disputes, and has it actually caught real issues or mostly created friction
Running CMS pre-publish detection on 4 editorial sites for the last 11 months. data: 1) Threshold at 70% AI consensus across 3 detectors (we use 2-of-3 majority above 70% = flag). Below that auto-publishes. 2) Flag dispute workflow: writer can override with editor approval and a documented reason. Override is logged and reviewed monthly. 3) Real issues caught: 14 in 11 months, mostly junior writers using AI to draft sections without disclosure. About half were caught by the gate, half by editor review. The gate is not a unique catcher but it’s an additional layer. 4) False positive friction: ~3% of submissions trigger the gate falsely. Average dispute resolution is 20 minutes. Manageable at our volume. 5) Cultural impact: the existence of the gate changed writer behavior. Self-reported AI assistance disclosure went from ~30% to ~80% within 3 months of rollout. Writers preferred to disclose upfront than navigate the dispute process. Worth it for us. But the ‘reasonable threshold + clear dispute path’ design is what makes it workable. Strict gates without dispute paths create real problems.
the disclosure rate change is the underrated finding. its not just about catching - its about shifting the culture toward upfront disclosure. that benefit outlasts any specific detector accuracy.
we run a lighter version of this (advisory only, no gate) and we saw similar disclosure rate change. just the existence of the check shifted behavior. agencies considering this should weigh the cultural benefit alongside the detection benefit.
@CyberVortex the 70% threshold and override workflow is what i was missing. drafting policy this week with that framework. expecting same culture shift on disclosure rates.