Ax5 shares core with G series, and ax15 shares core with R series. Best combo is taking an ax15 and using a toyota turbo bellhousing and input shaft, as it was the only application (in us) that used the R series trans behind a 4 cyl engine, which allows for the acme adapter, but the turbos are rare.
I still like the idea of a rotary... but only if you feel like reinventing the wheel even more so than you would already be doing with a diesel. I am also aware of the inherent downsides of a rotary such as reliability.
TDI is roughly the same size as the rotary... I was thinking of using that space for storage box, however I ended up moving my whole driveline forward to use that length to squeeze in a dual case and a little bit longer rear driveshaft.
Keep in mind you will likely kill some of that space with turbo hoses and intercooler.
You are definitely in the realm of things that will just require that you research everywhere you can. TDICLUB.com is a great VW TDI resource... 4btswaps.com has some good info on several diesels, but primarily the cummins. Search every forum out there, from jeep forums, to toyota forums, to zuk forums, etc.
A lot of the info I gained that helped was from people swapping my engine into other random vehicles, as it hasn't been swapped into very many jeeps.
No one can answer all questions when it comes to diesel swaps, especially people who haven't done them. Search for the threads where people have actually done something similar, if not what you actually have in mind... just keep researching!
Eventually you just have to pull the trigger, commit to one route... and then work through the little challenges as you come across them.