In an earlier post I mentioned how the Cavendish experiment allowed us to weigh the Earth - to determine the mass of the Earth mE. Newton knew the acceleration due to gravity on the surface of the Earth and was able to use that to find the product G mE; Cavendish determined G directly, and was thus able to solve for mE. He would also have been able to find the mass of the sun as follows:
mE aE = G mE mS / rE2
G, rE, and aE = vE2 / rE are known, so we can solve for mS.
But calculating the mass of the moon is trickier.
Once we were able to put a satellite around the moon we could measure its orbital radius and speed, deduce the acceleration, and use that plus the known G to calculate the mass of the moon. But prior to that we were limited to techniques like:
The moon does not exactly orbit the Earth, but instead orbits the center of mass of the moon/Earth system. By careful observation we can determine where this center of mass is. We can then measure the distance between the center of mass and the Earth's center. This plus the known mass of the Earth and the distance of the Earth from the Moon allows us to determine the mass of the Moon.
If we're lucky enough to see a foreign object come close to the moon, we can determine how much it is accelerated by the Moon. This will allow us to determine the mass of the Moon using the technique above. (We won't be able to determine the mass of the foreign object, but we don't need it.)
When the USSR launched Sputnik, American scientists really wanted to know what its mass was. But because none of the techniques above were useful, they were unable to determine it.