Threads inside a machine share memory, so communication is free and correctness is hard. Ranks across machines share nothing, so correctness is easier and communication is the cost you spend the rest of your life optimising. Most real scientific codes do both: MPI between nodes, OpenMP inside each one, and increasingly a GPU underneath that.

If five per cent of the runtime is inherently serial, twenty times is the best you can ever do — with a thousand cores or a million. In practice communication overhead makes it worse than that, and past some point adding ranks makes a job slower. Measure the scaling curve for your actual system before requesting a large allocation; the answer is frequently that you should be running more small jobs rather than one large one.

A shared cluster is the right answer for a job that genuinely needs hundreds of nodes and tight interconnect, and it is free at the point of use in most institutions — you pay in queue time. A rented machine is the right answer when the job fits on one node, when you need it now, or when you would otherwise spend a week waiting to find out that the input file was wrong.