I didn't say we did, rather I think we had to agree to disagree.
One could put it like that. We certainly had no choice (since it was apparent!) that we "agreed that we were disagreeing"
The only way your theory would work, would be if the house were an entirely sealed tunnel, only openable at the ends, with the fan the diameter of the tunnel. Then has much fresh cold air would be drawn in at the far end, as is extracted warm air. Other than that very limited scenario, dilution of cold air with warm air becomes a major factor, so not as great a heat loss as you suggest.
As I've just written to JohnD, what I've said is not "theory" but, rather, fact. As I've said, one simply cannot get away from the fact that if one pumps X litres of ('heated') air out of the house, it must be replaced by X litres of (potentially very cold) outside air entering the house somewhere 9through 'ventilation', even if 'accidental') - and that remains the case regardless of the layout of the house and the location of the ventilation relative to the fan.
As I also recently wrote, the best one can do is to restrict the replacement of heated air with cold outside air to one part of the house (e.g. just the bathroom), by having local ventilation (not common in bathrooms) and sealing the room from the rest of the house (obviously not usually done). In that situation, the temp of the bathroom air would fairly rapidly fall until it was at the same temp of the outside air, but (since the room was 'sealed') the rest of the house would not be affected.
There is one way that one could largely avoid the heat loss, but it would be silly, because it would render the fan essentially useless. As above, "X litres out, X litres in" is inevitable, and I have been assuming that the "X litres out" are X litres of heated (and 'water-laden) air (which is what one
wants to 'extract'). If one located a substantial ventilator very close to the extraction fan, then most of the air being pumped out of the room/house would be the ('cold') air drawn in from the outside world, leaving the heated (and 'water-laden') air within the room largely unaffected. In other words, the extractor would merely be 'recirculating' cold outside air, from the outside and then immediately back to the outside.
However, since the whole point of the fan is to remove the 'water-laden' (heated) air, such a situation would mean that the extractor would not be achieving anything! Put another way, if one wants to extract heated water-laden air from the room, then like it or not, one has to extract heated water-laden air from the room - and the only thing it can be replaced by is cold air which has entered the house from outside.
To what extent that heat loss affects different parts of the house will obviously depend upon the relative locations of the fan and the ventilation - as I've said, it could theoretically be restricted to just one room or, at the other extreme (if the ventilation was very distant from the extractor) might affect most of the house. Is that perhaps what you mean by 'dilution'?
Kind Regards, John