It was a dust storm across Mars in 1971 that made biologists wonder if nuclear war might endanger all life on planet Earth: marine as well as human. The question of whether there can be any winner in a nuclear war is arising more emphatically now in laboratories in the Soviet Union and the United States. A bold new theory states that with the explosion of even a fraction of the stored up arsenal of nuclear power presently on Earth, there will be a so-called ''nuclear winter,'' caused by a devastating dust storm of soot and smoke, like that observed temporarily on Mars.
The development of the dust storm, which will cut off sunlight, had not previously been visualized. Rarely has a revolutionary concept appeared so suddenly at what could be a critical historical juncture.
It was publicly aired at a conference of 100 natural scientists here Oct. 31 -Nov. 1, and repeated last week at a meeting sponsored by Sens. Edward M. Kennedy (D) of Massachusetts and Mark O. Hatfield (R) of Oregon, with four leading US and four front-rank Soviet scientists participating.
Putting ideology aside, they argued that under the new postulates life might become impossible on Earth in a nuclear war. Not even cockroaches and other insects could adjust.
''The new results of findings on climate catastrophe seem to raise the stakes of nuclear war enormously,'' testified Carl Sagan of Cornell University. Arguing that nuclear weapons may precipitate calamity on all countries, he said, ''The possibility has arisen that, past a certain point, more nuclear weapons do not increase national security.''
E. P. Velikhov, vice-president of the Soviet Union's Academy of Sciences, discussed new research findings in the USSR similar to those in America. Nuclear war, accompanied by the release of radioactivity, toxic smoke, and drought, would black out sunlight for many months, he said, causing temperatures to fall precipitously around the world, killing plants and animals.
Until now, said Sergei Kapitza of the Moscow Physico-Technical Institute, it was arguable that nuclear arsenals deterred nuclear war as a ''tacit mutual hostage arrangement between the opposing nuclear powers.'' No longer, he argued. He agreed with US counterparts that recent findings mean that use of nuclear weapons is ''suicidal.''
''Toxic smog,'' said Paul Ehrlich of Stanford University, ''would follow; soot and smoke would come from burned off areas. After a nuclear war in the spring, three feet or more of ice would form on all bodies of fresh water, at least in the north temperate zone.''
Dr. Ehrlich summarized views of the new thinking: ''A large-scale nuclear war would leave scattered survivors in the Northern Hemisphere facing extreme cold, hunger, water shortages, and heavy smog.'' He declared flatly at one point, ''Nuclear war would entrain the extinction of humanity.''
Hearings took place in the historic colonnaded Senate caucus room, where the McCarthy hearings were held and the investigation of Watergate was pursued. There were interruptions from the jammed-in audience, which included protesters against the appearance of Russian witnesses. As one protester was ejected, he shouted, ''Kennedy speaks treason!''
The eight witnesses at the long table were in agreement as to the global danger if not as to what should be done about it.
Senators Kennedy and Hatfield discussed the possibility of a nuclear freeze. ''Every president since 1945,'' Mr. Kennedy said at the start, ''has met at the summit during his first term, with the leader of the Soviet Union.'' He urged Mr. Reagan ''to take that fundamental step back from the brink of war.''
Later he added that he is ''disturbed'' that members of the administration still seem to think nuclear war is ''winnable and survivable.''
The new concept of a ''nuclear winter'' touched off by war has outdated much of recent scientific discussion. ''Almost all medical and environmental estimates of the consequences of nuclear war have proved to be underestimates,'' said H. Jack Geiger of City College of New York. He ridiculed the earlier premise of ''buying a little fallout shelter for the back yard.''
Witnesses noted the slow development of the new theory on ''nuclear winter.''
''Climate catastrophe,'' Dr. Sagan called it. Others said not even cockroaches would survive.