The Second Estate: A Q&A with Ray Madoff

One of the most frustrating aspects of economic inequality in this country is how easily the rich avoid the meager systems we do have in place to check runaway wealth. Tax evasion, shell companies, and tricks with charitable giving abound.

But, it turns out, the rich aren’t just dodging their obligations under our tax code. They’re actually using it to amass even more wealth.

Ray Madoff, now a professor at Boston College Law School, and I have worked together for 20 years on defending the estate tax, charity reform, and tax policy. We don’t always agree.

For example, I’m a supporter of wealth taxes and Ray is more skeptical. But we love working together, and this past week we sat down to discuss her excellent new book The Second Estate: How the Tax Code Made an American Aristocracy.

This interview has been edited for length and clarity.

Chuck Collins: When you were growing up, did you say, “I want to grow up and be a tax nerd?”

Ray Madoff: I thought I was going to run an orphanage — even though I think the orphanage system was already long gone by then!

CC: So, like a Jewish Mother Teresa?

RF: Hah! Later, when I went to law school, I thought I was going to be like a litigator that focused on battered women or saving the whales. I was kind of shocked to find that what I really liked was taxes. I studied philosophy in college, and a friend of mine has a theory that all philosophy majors that go to law school become tax lawyers, because it’s a certain type of analytic thinking.

CC: What do you think about the statistic that says, “the richest 1 percent pay 40 percent of the taxes,” and “46 percent of Americans then don’t pay taxes?”

RM: The reason those figures are misleading is that when they refer to the top 1 percent, they’re referring to those with the most taxable income. People with the most taxable income do pay a lot of taxes, but the problem is the wealthiest Americans — those with huge assets — are able to avoid taxable income. They are just as likely to be in the 46 percent of non-income taxpayers as they are in the top 1 percent of payers. That statistic makes it sound like a whole bunch of the country is getting off scot-free. It’s failing to account for payroll taxes, which are quite burdensome, and they have gone up as other tax rates have come down over time.

CC: In your book you talk about how the tax code constantly needs to be enforced and updated to address avoidance and loopholes.

RM: Yes, any tax system requires enforcement.  It’s been well proven that a small investment in the IRS enforcement produces tremendous revenue for the country. There is absolutely no justification for the current state a defunded IRS. It only provides an additional benefit to the wealthiest who would otherwise be required to pay taxes. It’s troubling for the country that we would allow flagrant tax avoidance to occur due to a lack of enforcement.

In addition to enforcement, it also needs constant legislative and regulatory oversight. I would say that the income tax has been somewhat regularly revised but where we really see the problem is the estate tax. So, if we start with the fact that wealthy Americans can avoid taxable income because they don’t take salaries and they don’t sell their assets. They instead borrow, and then on top of it, they inherit great wealth, and under our just basic tax rules, any money received by inheritance or gift is free of income taxes, also life insurance, entirely free of income taxes. You don’t even have to report it. Somebody who is given $10 million or $100 million doesn’t have to tell anyone that they’ve received that money. What we count on instead is the estate tax system, and that’s where we’ve seen a huge problem in terms of lack of upkeep.

We should get rid of the estate tax and instead expand the base of the income tax to include investment gains and inheritances as taxable income. It’s critically important that we have systems that feel fair to people.

CC: I’m curious, what do you think of an annual wealth tax?

RM: I can totally understand why the wealth tax is so appealing and attractive, and why it seems like the obvious solution to the problem. However, I worry about the practical limitations of such a system insofar as it counts on annual valuation. The problem is particularly acute with respect to the enormous amounts of wealth owned through complex partnerships and other assets that are difficult to value. What you’re going to have is the whole estate planning world hiding value.

In addition, wealthy people would be incentivized to move their money out of the easy to value stock market and instead invest in privately held interests that are more difficult to value, and this would impose significant hardships on the tens or hundreds of millions of Americans who count on the stock market for their retirement and other savings. While it would work great in theory, it is ultimately both unworkable and unwise.

CC: In terms of building support for taxing the very wealthy, how about this scenario. We say to save the earth, we need to tax the climate destructive behaviors and concentrations of wealth that are causing the most harm”?

RM: My problem with this approach is that it seems to describe the status quo as one where the rich are already paying taxes and we need for them to pay more. But a far bigger problem is how the wealthy have been able to write themselves out of the tax system altogether.

I think that our first step in tax policy is bringing the wealthiest Americans into our tax system. Our total government revenue in 2024 was $5 trillion and we had a deficit of about 2 trillion because we spent almost $7 trillion. And then you realize that the wealthiest 1 percent owned $50 trillion it matters whether this group pays taxes. It matters because our growing debt is a problem.

CC: Well, if only the message could get out further, then people could shift that narrative.

RM: I feel so urgent about my project, which is really a project about public awareness. How do we grow public awareness? And my concern is that politicians across the political spectrum have effectively spread a message that you don’t need to understand taxes. And that is baloney.

The Second Estate is available now from University of Chicago Press.