Einstein was right and Silberstein was wrong, but it’s beautifully subtle why.
Einstein’s point was that in a region free of singularities, the circumference of an infinitesimal circle should be 2π times its radius. He then showed that an infinitesimal circle perpendicular to, and centered around, the line connecting the two mass points in Silberstein’s solution has the wrong radius, hence the line connecting the two masses must be singular.
Silberstein countered by pointing out an error in Einstein’s derivation and then showing that a particular quantity rigorously vanishes along this line, implying that yes, the circumference of the infinitesimal circles in question do, in fact, have the right radii. This solution appeared in his paper in Physical Review in 1936.
Einstein then, in his published Letter that he wrote in reply to Silberstein’s paper, provided a technical argument discussing square roots and derivatives that, while correct, is not very enlightening. (It’s one of those comments that make perfect sense once you know what the devil he is talking about, but there’s no possible way you can actually understand what he’s talking about from the comment alone. Not that I haven’t been guilty of committing the same sin, even writing things that I myself wasn’t able to understand a few months later without going back to my notes or calculations. But I digress.)
By solving for the metric and obtaining an explicit formula, I was able to better understand this mystery.
First, it should be noted that the axis connecting the two mass points is also the origin of the radial coordinate, so the coordinate system itself is singular here even if there is no physical singularity. To verify that there is no physical singularity at this location, we can draw tiny, infinitesimal circles centered around this axis and check if they obey the rules of Euclidean geometry (one fundamental rule in curved spacetime is that in a small enough region, things always look Euclidean.) This was the basis of Einstein’s argument.
We can express the ratio of the angular and radial components of the metric in the form of an exponential expression, say exp(N) where N is some expression. Clearly, we must have N(r → 0) = 0 in order for the circumference of an infinitesimal circle centered around the axis to be 2π its radius.
As it turns out, N appears in the form  + C where  is just some complicated expression. It turns out that this part  is constant along the axis, but its constant value is different between the two mass points vs. outside the mass points. I can draw the axis and the two mass points like this:
The bracketed expression is constant along this line (not constant outside the line, such as above or below it but that’s not relevant now) but has one constant value between the two points (the two X’s) and another constant value on the two sides. The two constant values are not the same.
We can choose the constant C to make N vanish either between the two mass points or outside the two mass points but not both at the same time. When N vanishes outside, but not between, the two mass points, the singularity that remains serves as a “strut” holding the two mass points apart:
Or, when N vanishes outside, it’s as if the two mass points were attached by “ropes” to the “sphere at infinity”, hanging from there in static equilibrium as gravity pulls them together:
So how come Silberstein was able to dismiss Einstein’s criticism? He was able to do so by making a particular choice of the sign of a square root that made N vanish both between and outside the two mass points. But this is where it helps to write N in the form that I obtained, symbolically as  + C, with the bracketed part having one constant value along the axis between the two masses and another value outside. Sure we can make N vanish everywhere along the axis… by allowing C to have different values between vs. outside the masses.
This is fine along the axis… the masses themselves are singularities, so they represent a discontinuity anyway, so we are free to choose different integration constants on two sides of a discontinuity. This is indeed what Silberstein has done by choosing a particular sign of a square root in his formulation of the metric.
But, as Einstein pointed out, such a choice of sign leads to a discontinuity in derivatives. More explicitly, what happens is that the function N is defined not just along the axis but everywhere in spacetime… to get from a point on the axis between the two masses to a point outside, we need not go along the axis, we can leave the axis, go around the mass, and then return to the axis. As we do this, we encounter no singularity but the value of C must jump from one constant to another at some point. In other words, by removing the “strut” or “rope” singularity from the axis, we introduced a much worse singular “membrane” that separates regions of space.
One morale of this story is that when we use a coordinate system like polar coordinates that is not well defined at the origin, we must be ultra careful about that spot… since the coordinate system is singular here, this is where things can go wrong even if they appear perfect everywhere else.