When chain drilling for a slot which is what you are talking about, drill every other hole 1.5 diameters away from each other then come back and drill the in between holes. This will allow you to take a MUCH larger percentage of waste out with the drill bit which means a lot less filing by hand. Make sure the part is clamped to the drill press table. Put a handle on your mill bastard file. Hold the workpiece between hardwood strips in a vise with the finish line about .020 inch above the vise jaw. While filing, keep the file parallel with the floor and rock back and forth on the balls of your feet using your body to move the file, do not use your arms or you will tire quickly. This also works when using a hack saw. You can saw or file all day long using this technique.
Do be careful when using a drill press for milling. Drill press chucks are pressed on and as a result a side load can cause them to fly off the arbor.