Apart from the filtration, it is possible that you are getting some fogging exposure from the light bounced from the white walls. That would give unwanted tone to the highlight areas in your prints and thereby reduce the overall contrast. A couple of large pieces of black card or thin foamboard, behind and against the wall to the side of the enlarger, would reduce that problem considerably.
Check that the projected image actually changes from "white" (used for focussing) to "filtered" (yellow-ish or magenta-ish depending on what you dialled in) when you operate the colour-filter lever.
Most colour-head enlargers do not have a filter-drawer, because the condenser head is usually completely replaced. Without a drawer, how were you using the multigrade filters? The under-lens filter set from Ilford has a very good holder and robust filters, but the usual above-lens filter squares are an imperfect solution for regular use under the lens. That said, it should still let you get some sort of result.