"totally agree that large formats are completely unsuitable for macro work"
Sorry..... but just have to disagree with that statement. I spent almost 30 years making innumeral photo-macrographs onto 4x5" film on close to a daily basis in a Govt research insitution. Should the need arise to photograph something the size of a postage stamp ter ARE better ways. A Scanning Electron Microscope used to allows you to expose at MANY hundreds of times magnfication onto 4x5 film. An SEM is however a rather expensive piece of equipment... and you cannot carry it around 'out into the field'.... But I don't think you will be able to get that 'wow' exposure in colour
Ken
To fill the frame with, say, a postage stamp on 5"x4", you'd need an RR (reproduction ratio) of around 5:1. Even using a 50mm lens you'd have a bellows extension of 300mm and a subject distance of less than 60mm. Plus need a very sturdy tripod to keep the whole rig steady.
In short, any subject that requires a high magnification - macro or telephoto - is better done on a small format.
As for 10x8 vs 5x4 in general; there's a law of diminishing returns. Theoretically 10x8 could give twice the image quality over 5x4, but since it's 4 times the area, then its costs are also nearly 4 times as great for film and processing. Outsourced scanning is also going to be a lot more costly for 10x8 as well. With no guarantee that the end print quality will be noticeably better.