That's not going to work because the focussing cam of the lens has to be matched to the goggles.
Any non-goggled 35 M lens will focus fine on an M3 but the framing in the viewfinder will be off: to compensate for the framing, there has to be a wider field of view (Leitz made a few accessory viewfinders that went over the outside of the viewing field only: there was a hole in the middle to match the split image. It didn't work well. In order to match the change in viewing field magnification the rangefinder window had to be accommodated, and then the cam changed. Doing only two out of three is useless. Most people would use an accessory viewfinder or buy an M2/4/6/7 or Voigtlander