Language plays a key part in social justice efforts, but language alone is not the solution to addressing social injustice. In this keynote I explore the long, colonial histories of language stigma and shaming, showing how working-class and racialised communities have long been framed as linguistically deficient and in need of remediation if they are to experience justice. I show how these reductive narratives underpin education policy in England, materialising in different forms and labels such as the word gap, oracy, and tiered vocabulary. I show how deficit discourses and anti-Black linguistic racism permeate Ofsted inspection procedures. I end with some visions for linguistic justice, showing how the problem is not how marginalised people talk, but how powerful people listen.