Anti-woke arguments ‘destructive’ to people who have worked hard, lawyer and refugee says