Very good point.
Suppose we wanted to test the impact of carbon dioxide enrichment on 100 terrestrial pine trees. We would have one field planted with trees maintained in enriched conditions, and we would have another with ambient carbon dioxide levels. In an ideal world, we would have numerous fields replicating the study, but we have to rely upon other researchers to replicate the experiment. Sure if we had enough fields that could account for experimental failures (like the enrichment system failing to provide enough nutrients and for variation between fields), but is that really necessary. Is it reasonable and is it possible. Experimental research always suffers from this kind of criticism. If there is belief afterwards that the experiment did not account for an unknown variable then there is the option of Bayesian statistics afterwards, and if something was missed that it can be investigated in the future. Few experiments are designed perfectly. But it is often still possible to add a drop to the knowledge pool.
I would have scolded your assessors if they did not have legitimate belief for an unknown variable affecting your Master's project. That is why it is better to discuss experiments in advance and agree the terms of reference. If they told me that an experiment was not possible on just two fields, then I would turn around and ask them to prove it. Let them try refuting a conjecture with absence of evidence. If they could show me that the carbon dioxide enrichment system has a 50% chance of breaking, then I would agree that this risk was not factored. But if they had no data, then my evidence is valid and their conjecture is an imaginary bias.
So taking the example of an aquarium, if the two tanks had statistically significant results in plant growth after the experiment was repeated... and then it was found that the temperature on one was slightly higher. That would mean that the next experiment would have to show that that temperature difference was a valid variable. Both tanks would be planted and have identical conditions except for temperature... so on and so forth. To me, it doesn't matter whether you have one tank of a hundred, if you are testing two populations of cloned planted with defined variables and controlled conditions then you will be able to draw enough evidence for me to believe it.
It is necessary, reasonable, and possible. It is standard practice for agricultural experiments! First, you can usually divide fields into individual plots that receive the treatments (randomly assigned to control for within-field variation) so the experimental unit is not the entire field and it is quite common to have the whole experiment run in several different fields and always, always over multiple years. If you're lucky you may have other research partners to help, but that's not always the case. It is an enormous pain to be sure, but it's done that way for a mathematical reason, not conjecture.
I don't know anything about how other types of research is done, but the design for these kinds of plant/field experiments is pretty well established and
not routinely criticized. Probably every student has a moment where they are like, "Ughhhh, why do I have to do this much work anyway????" but by the end of my program I had learned how the statistics work, and in many cases I was required to read the foundational papers on which the practices were based. The concept of pseudoreplication, which is what we were talking about with the tank being the experimental unit,
goes back to this paper from 1986, but a lot of the ins and outs of experimental design were pioneered in the 1950s and 60s. Nobody is making extra work for no reason, and in fact a lot of effort has gone into figuring out the minimum amount of work needed to move the science forward.
But to address the last bit, it is assumed that there is going to be variation within the experiment that you can't control for no matter what, be it temperature or what else. If you conduct the same experiment twice, even if you try to get it perfect, you are going to end up with results that are different. You hope that they are not that different, but you don't know how it's going to turn out until you do it, in part because you don't know the answer to the question you are research ahead of time. This is why replication and randomization is so important - they work to distribute all the little differences (aka experimental error) more or less evenly between treatment groups and without it you can end up with very, very wrong conclusions.
That one for me. The question then becomes
"how small in volume can the tanks be?" Before that invalidates the DOE.
That <"
is also true">. I think the only way you could really do this is by <"
metadata analysis">.
It's a good question that I don't really know the answer to. Well, the answer is that it can be as small as you want so long as the size of the tank is not itself influencing the growth of the plant so much that you wouldn't be able to generalize the results of the experiment, but I don't know what that is. You might be able to go pretty small, but obviously it'd have to be uniform and the tanks would have to be randomized in space (can't have all the tanks from one treatment grouped together). I definitely haven't thought through how best to control for error in this sort of experiment.
As for metadata analysis, that's a weaksauce desperation move for when you can't do proper randomized, controlled experiments. That's sort of a joke, but one of the joys of a plant experiment is that we can work with literal clones that we have total control over. This is why we can get away with just 6 or so replications per treatment instead of the sometimes thousands of subjects needed for human research. It's quite a bargain if you look at it that way!