Ipswich captain Jason De Vos flicked on a throw-in with his head to find Richards who slammed the ball home.
Former Tottenham and Derby man Mounir El Hamdaoui put Willem ahead in style with a long-range strike past Shane Supple 15 minutes into the second half.
Town made five second-half substitutes, including Indridi Sigurdsson, who was forced off after a clash of heads.