I'm going to start doing some mini blog posts from now on to keep everyone updated on our research, because it seems people don't notice some of the stuff we do otherwise.
Zedd's been doing a lot of work on cross-map BSP injection lately, and he's been having some trouble with the terrain shaders. Even though Assembly has support for shader extraction and injection, the map was still crashing. One shader that was causing problems for us in m70_bonus.map was levels\solo\m70_bonus\shaders\ground\m70_bonus_cinematic_ground.rmtr. Its actual pixl tag is shaders\terrain_templates\_0_0_0_0_0_0_0.pixl.
If you go to that tag in a newer version of Assembly (one that supports shaders), you might notice that all of the shader pointers in it are null, even though the shader works fine and certainly isn't just a null shader. How can this be? Well, we noticed that just above the shader pointer in one of the blocks, there was an int32 with a value of 321. This is usually set to -1 for shaders with valid code pointers. Obviously, there must be something to this.
We first speculated that this integer was an index into the resource table in zone. (That's a pretty safe guess when you're dealing with stuff like this.) At index 321 in zone, the parent resource was...a bitmap of a spartan helmet. Zedd did some zedding around with Assembly's injection code anyway, but couldn't get anything to work reliably. OK, that's probably not any good. Time to move on.
Stumped, I decided that we should set a memory read breakpoint on that integer and see what the game does with it. The breakpoint hit at address 0x82182E2C in the TU1 default.xex. Opening that up in IDA, you see this (not including irrelevant code):
lwz %r8, 0x50(%r3)...mulli %r10, %r8, 0x58
So this means that the value is an index into an array with an element size of 0x58. And it just so happens that shader tag blocks are that large, but there weren't 300-something entries in the pixl tag.
Turns out that, starting in Reach, there's a global_cache_file_pixel_shaders ('gpix') tag which contains a bunch of commonly-used shaders, and this is an index into a block in that tag. The actual shader code can be found there. This means that the corresponding block entry needs to be extracted and injected into the target map in order to make the shader work.
Well, looks like we have some work ahead of us.