When you click on links to various merchants on this site and make a purchase, this can result in this site earning a commission. Affiliate programs and affiliations include, but are not limited to, the eBay Partner Network.
I had the same thing with my drivers side mirror. I ended up breaking the mirror glass out and shimming the spring steel retainer that rides over the ball socket. I bought a replacement mirror glass from CC for $12 and some change. It works perfect now, a $12 fix instead of a $60 replacemt mirror.
The mirror is held in place by 4 dots of silicone. There is not mich you can do but break the glass to remove it.
You don't need to damage the mirror glass. Just put some paint thinner (not mineral spirits) in a tub/pan and stick the mirror on its face so that thinner covers the head. Then use aluminum foil to seal it off and prevent fumes/vapors leaking out. In the morning, the RTV holding the mirror in place will be disolved or softened enough to get the glass out. Once out, you will see a metal plate with two rivets--one on either side of the swivel. You need to make a little sheet metal shim to fit under one [or both] rivet heads so that the swivel will tighten up. The shim can look like a circle with a slot in one spot going to the center, or it can be a little square with the slot. Just get the shim between the metal plate and the rivet head to take up the "slop" in the system. Then glue the mirror back in place with RTV or silicone sealant on the 4 locating trunions. DO NOT PUT ANY WEIGHT ON THE MIRROR AS THE RTV IS CURING; it might leave a permanent bow in the glass.
I just soaked my mirror in gas for a day & out it came!
To fix the sloppyness I ground off one "rivet" & drilled a hole in it's place, now screw in a sheet metal screw & you can adjust the tightness of the ball socket