Yes, it is a possible problem if the oil should migrate to the lens surface.
If it were mine, I would remove the lens elements, and with the aperture blades closed completely as possible, soak both sides with lighter fluid or even acetone.
Work the aperture back and forth as the liquid evaporates. Then repeat the process. Leave the lenses out until the following day to eliminate the possibility of fumes affecting them.
This may not be the absolutely best method, but it has worked for me more than once.