The time you have noticed is due to compilation time of the shader. As soon as you add a texture the shader must be recompiled to take this information in account. The best idea here should be to have a white texture always attached to your object even when in wireframe mode so that the switch to real texture won't trigger a compilation