You mentioned the fm towns marty 3D sphere which gave me an idea but then I looked it up on youtube and it was nothing like my idea.
The idea is to use a normal VDP2 layer with small premade tiles and a CPU can decide how to composite the tiles to recreate what looks like any shadows.
The VDP1 draws arbitrary lines from the left line to the right using a greedy algorithm which causes pixels to be overdrawn. This avoids PS1-like gaps but causes the moire pattern with transparency. The other thing that causes the moire pattern was demonstrated well in one of Jon Burton's Sonic R videos. So unless each line is perfectly vertical or horizontal it'll overdraw causing transparency to be bollocks. This applies to skewed and rotated quads, not just perspective corrected quads.
One of the things you might have heard is that CRTs and composite video blurs the dithering enough to be a convincing shadow just drawing a dithered black oval over the ground. I didn't understand just how effective this was until I saw this composite capture of Z-treme on youtube:
https://youtu.be/x7cW9wgIW00?t=50Anyways you can tell how, back in the day, a team on a deadline scraping for ounces of power on a not quite adequate for 3D system would just draw an oval and be done with it.
I think one of the major issues with performance is that we're using an incredibly old jack-of-all-trades graphics library instead of tightly focused custom routines. Unlike a more straightforward graphics system, the saturn's really hard to write efficient graphic routines or even wrap your head around especially the memory scheduling. If two chips try to access the same memory chip in the same cycle, one of them will be halted until it gets access. This can basically throw away many cycles of work the saturn would otherwise be able to accomplish.
Anyways we got some interesting ideas floating around.