MERCURY, Nevada, Aug. 17— Amid smiles and applause from Soviet and American scientists, the first phase of a joint experiment to verify the size of underground nuclear tests was carried out today on a remote plateau in the Nevada desert.

It was the first time Soviet scientists had ever taken part in an experiment involving the detonation of a nuclear device at the American test site in the Nevada desert.

''We have every indication that today's test has been a success,'' said Joseph F. Salgado, deputy secretary at the United States Department of Energy, the agency that oversees this nation's nuclear testing program.

''We are going forward,'' said Igor M. Palenykh, head of the Soviet delegation to the nuclear testing talks in Geneva. ''We are satisfied.'' #2 Monitoring Methods Compared Today's test is part of recent Soviet-American efforts to make progress on arms control, particularly around issues of nuclear testing.

Before the experiment today, 30 demonstrators from American Peace Test, a protest group, tried to block the main road leading into the Nevada test site. Twelve people were arrested and cited for attempting to block traffic. They were later released.

The test gave scientists an opportunity to compare two methods of monitoring the size of nuclear tests, and whether they had been conducted. An on-site method favored by the United States was used under an official United States-Soviet agreement. At the same time, under the auspices of a private American group and the Soviet Academy of Sciences, experts from both countries monitored the blast using an off-site method favored by the Soviet Union.

Officials said the test was successful in the sense that it had allowed them to gather the data they need. But they added that they could not reach conclusions about the two methods until they study the data. A second phase of the joint verification experiment is scheduled to take place Sept. 14 at the Soviet nuclear test site at Semipalatinsk. Again, American and Soviet scientists will test on-site methods for estimating the yield of nuclear explosions. Treaties of 1974 and 1976

Treaties limiting the size of nuclear tests were signed in 1974 and 1976. Neither side has ratified the treaties, however, because American officials have not agreed with the Soviet position that the yield of weapons tests can be accurately calculated using distant instruments. Instead, the American side has pressed for on-site verification methods that place instruments down a hole drilled parallel to the shaft in which nuclear weapons are exploded.

In today's tests, Soviet Government scientists supervised the taking of measurements at the American nuclear test site while a joint team of scientists from the Natural Resources Defense Council, a private environmental organization based in New York, and the Soviet Academy of Sciences took measurements at three distant sites, using seismic instruments.

Under the treaties, nuclear explosions may not exceed 150 kilotons, the equivalent of 150,000 tons of TNT. The bomb that destroyed Hiroshima had the force of about 13,000 tons of TNT.

According to C. Paul Robinson, chief American negotiator at the Geneva talks on nuclear testing, a breakthrough was made late last year when the Soviets agreed to conduct the joint verification experiment. Both sides are closely monitoring nuclear explosions at each other's nuclear test sites.

''We will compare for the first time the yardsticks we use to verify yield,'' Mr. Robinson said.

The first part of the experiment took place this morning. At 10 o'clock (P.D.T.) a 150 kiloton nuclear bomb was detonated underground on a mile high plateau in the northwest corner of the Nevada Test Site. A circular patch of ground about 100 feet in diameter lifted about 15 feet off the ground, according to an estimate made by Ted Valk, an electronics engineer from Lawrence Livermore Laboratory.

Thirty miles away, at a command center in the heart of the Nevada test site, 20 American and Soviet officials sat watching a television monitor showing an aerial view of the bomb site. After the explosion, the group sat quietly, smiling at one another.

But when Dr. Viktor N. Mikhailov, leader of the Soviet team at the Nevada Test Site, finally received a phone call saying that the Soviet instruments had worked and that data had been recorded, the room broke into applause.

Dr. Mikhailov later said, ''I always dreamed of seeing how U.S. experts work at the Nevada Test Site. I can tell you they are excellent experts, very friendly, very warm people. We have cut a window into their hearts as well.'' Data Available in 20 Minutes

Nuclear tests are customarily given code names honoring places, people or things. Today's test was named Kearsarge, after a mountain pass between California and Nevada.

Although so-called raw data was available within 20 minutes of today's explosion, American and Soviet scientists said it would take several weeks to fully analyze information from both tests. Results of the experiment will be carefully compared in Geneva later in the fall, Mr. Robinson said.

Mr. Palenykh said at a news conference that ''the Soviet Union was perfectly willing at any time to cease nuclear testing along with the U.S.'' After the United States failed to reciprocate a recent Soviet moratorium on testing, he said, ''we agreed to enter full scale stage-by-stage negotiations with the U.S. to resolve questions of verification of the two threshold treaties.''

Diagram