It would be much less distorting, and easier politically I think, to means test SSI benefits. e.g. if you have $100k in income in retirement (from RMDs, passive income, whatever) or net assets over say, $5 million, then you don't get SSI. Just phase it out over $X, like lots of other benefits.
SSI was only meant to help the destitute elderly, widows, and oprhans. It was never meant to be a public pension. And when the retirement age was set at 65, that was the avg life expectancy - half the people were never going to collect. So I think means-testing it is a good thing to do, even though it means I will never get a dime.