Mastering is a dark art. Many home producers, and even many intermediate studio professionals don't know what to give a mastering engineer. Where should the peaks be? Should I be compressing the main output? What file type do they want? It is easier to get ready for the final stage of the production process than you might think. And while many mastering engineers have preferences, modern technologies have greatly increased what is acceptable to send.
I am going to break this post down into two key parts: file type, and audio process. So far as file type is concerned, we are thinking more about what a computer thinks about a file from a storage perspective. From an audio process perspective, we are looking more about how a listener would perceive playback of that file.
Regarding file type, the short answer is 'the highest resolution wave file possible, without upsampling'. What does this mean? When you exporting the track from your daw, hitting 'export' or 'bounce' depending on the DAW, there are a bunch of options. '24 bit? 16 bit?', '44.1khz? 192khz?' – in general, these are options that would indicate the 'resolution' of your audio. Even phrasing it that way is an oversimplification, and perhaps slightly misleading, but for the purpose of this post lets roll with it. These numbers are concerned with the amount of information being outputted to your output file. So higher is better? I wish it were so simple. This is really going to be based on the audio interface you used to record, and the settings that were chosen with your interface. If an audio file is recorded at 16 bits, exporting at 24 bits will not enhance anything. You cannot add information that has not been recorded. So ideally, select the settings that match the highest fidelity of any recorded tracks. If you are unsure, and don't feel like searching these numbers out, just select 24 bit and 96khz. Nothing bad will happen, and your mastering engineer will not get mad at you.
What about the actual audio processes? Here are some non-negotiables:
No limiters on the output. If you run a limiter on your snare, that is fine. But as soon as you've added a limiter to your output, the mastering engineer is no longer able to control the dynamics of the track.
Peaks should be no higher than -1db. Many meters that you use will to determine peaks might actually not catch true peaks. leaving at least 1db of headroom lets your mastering engineer do their job without having to worry about true peaks, which limits their ability to do limiting in a controlled way.
Do not hit the 'normalize' function on your export. The 'normalize' function makes all peaks hit exactly 0. Basically negates any attempt you made at achieving the prior point.
Be very cautious with compression on your output. If you have a compressor on your output, just be mindful of what your 'dynamic range' is. If you have a meter (which you should) on your output, make sure that the 'LUFS' when compared to your peaks are differentiated by at the very least 12db. This is the softest rule, but the closer your peaks are to your 'LUFS', the less your mastering engineer will be able to control the output. I would recommend that your LUFS be somewhere in the -20 range, with your peaks around -3db. This is slightly conservative, and there is some flexibility here, but it is a decent starting point.
So now that I have rambled about this all, the big 'why prep at all?' is this:
If you are having an external mastering engineer work on your mix– which I recommend from as an objective standpoint I can as a mastering engineer myself– you should trust them to make some calls. A mastering engineer is very capable of controlling dynamics, tones, peaks, etc. Overworking a stereo output to send to a mastering engineer is similar to 'mixing' a vocal onto an instrumental track, and having all of the instrumentals on that single track. Sure, you can do it, but the end result won't be nearly where it could be if you had all the instruments separated.
TL;DR: 24bit, 96khz, peaks: -3db, LUFS: -20, no normalization (