This is all about splitting hairs. Often good enough is good enough. But like any 'craft', best practices don't hurt and might help doing a better job. They're your skis and your ass and I would expect low probability of a major problem unless you end up with spinners.
Most skis should indicate what diameter is recommended for that ski. I would think that driving a 4.1mm shanked screw into a 3.5mm (or 3.6mm) hole in metal/titanal/carbon and other less 'bendable/compressible material' could contribute to 'volcanoing'. In more 'bendable/compressible material', the screw shank in the ski core of a 3.5mm hole, will compress the material slightly and might even provide more retention.
In woodworking, pre-drilling in hardwood, the bit diameter matches the screw shank. In soft woods, the pre-drill diameter is slightly smaller than the screw shank. Same basic reasoning, I think.