Doom 3 engine release and game code

Ok that explains a few things

Btw Sikkpin will join Inside3d and i given him the temporary registration details as registering is down ATM I hope one of the admins can register him for real.
And some cool news from him as it seems hees trying to cut a deal with justin for finishing raynors backend he would agree to make all of justins shaders
so we might see finshed glsl backend soon.

That's good news.

I've personally been playing with the code again recently, and while I don't have intentions to embark on a serious project, I'm feeling keen enough on using the info in the ARB/NV10/NV20 and R200 backends to create a GPL-friendly GLSL interaction shader.

Cool looking forward to what you might come up with.
When Sikk joins here you could have a chat with him about shaders this guy is a wizard with ARB2 so you might end up with something very cool.

Cleaned up my modded version with sikks 1.2 version integrated.

Most work in materials and textures.

Found out a lot of the textures where non power of two and while Doom3 can load these i found out it sometimes bugs view so i ran all textures through a tool i have for that and made them power of two.
The HD menu had a few fixes but needs a little tweaking in some areas (the sikkmod menu will overlap other opened menus it should close them, and you cannot overwrite savegames).

Engine optimized for multicore processors with SSE and i changed a few of the autoset options so i matches the highres replacment textures better (turning compression totally off in high and ultra setting).
Ill updated the torrent for the new version soon.

To begin with, here are basic conversions of the main interaction shaders to GLSL. These work perfectly, but THEY ARE NOT GPL-SAFE. I initially did them so that I'd have a reference for sanity-checking, etc, and you may be interested in seeing what they look like. I haven't bothered with any performance or quality modifications to these; they're just basic line-by-line conversions, taking each chunk of asm code and translating it to it's GLSL equivalent. If you see something in there that you think looks stupid, it probably is.

Vertex shader:

  Code:

in vec4 attrTexCoords;
in vec4 attrTangents0;
in vec4 attrTangents1;
in vec4 attrNormal;

uniform vec4 PP_LIGHT_ORIGIN;
uniform vec4 PP_VIEW_ORIGIN;
uniform vec4 PP_LIGHT_PROJECT_S;
uniform vec4 PP_LIGHT_PROJECT_T;
uniform vec4 PP_LIGHT_PROJECT_Q;
uniform vec4 PP_LIGHT_FALLOFF_S;
uniform vec4 PP_BUMP_MATRIX_S;
uniform vec4 PP_BUMP_MATRIX_T;
uniform vec4 PP_DIFFUSE_MATRIX_S;
uniform vec4 PP_DIFFUSE_MATRIX_T;
uniform vec4 PP_SPECULAR_MATRIX_S;
uniform vec4 PP_SPECULAR_MATRIX_T;

uniform vec4 PP_COLOR_MODULATE;
uniform vec4 PP_COLOR_ADD;

uniform vec4 PP_CONSTANT_DIFFUSE;
uniform vec4 PP_CONSTANT_SPECULAR;

void main ()
{
   vec4 r0 = PP_LIGHT_ORIGIN - gl_Vertex;

   gl_TexCoord[0] = vec4
   (
      dot (attrTangents0.xyz, r0.xyz),
      dot (attrTangents1.xyz, r0.xyz),
      dot (attrNormal.xyz, r0.xyz),
      1.0
   );

   gl_TexCoord[1] = vec4
   (
      dot (attrTexCoords, PP_BUMP_MATRIX_S),
      dot (attrTexCoords, PP_BUMP_MATRIX_T),
      0.0,
      1.0
   );

   gl_TexCoord[2] = vec4
   (
      dot (gl_Vertex, PP_LIGHT_FALLOFF_S),
      0.5,
      0.0,
      1.0
   );

   gl_TexCoord[3] = vec4
   (
      dot (gl_Vertex, PP_LIGHT_PROJECT_S),
      dot (gl_Vertex, PP_LIGHT_PROJECT_T),
      0.0,
      dot (gl_Vertex, PP_LIGHT_PROJECT_Q)
   );

   gl_TexCoord[4] = vec4
   (
      dot (attrTexCoords, PP_DIFFUSE_MATRIX_S),
      dot (attrTexCoords, PP_DIFFUSE_MATRIX_T),
      0.0,
      1.0
   );

   gl_TexCoord[5] = vec4
   (
      dot (attrTexCoords, PP_SPECULAR_MATRIX_S),
      dot (attrTexCoords, PP_SPECULAR_MATRIX_T),
      0.0,
      1.0
   );

   vec4 r1 = PP_VIEW_ORIGIN - gl_Vertex;
   vec4 r2 = PP_LIGHT_ORIGIN - gl_Vertex;
   
   r1 = vec4 (normalize (r1.xyz), 1.0);
   r2 = vec4 (normalize (r2.xyz), 1.0);

   vec3 halfangle = r1.xyz + r2.xyz;

   gl_TexCoord[6] = vec4
   (
      dot (attrTangents0.xyz, halfangle),
      dot (attrTangents1.xyz, halfangle),
      dot (attrNormal.xyz, halfangle),
      1.0
   );

   gl_FrontColor = ((gl_Color * PP_COLOR_MODULATE) + PP_COLOR_ADD) * PP_CONSTANT_DIFFUSE;

   // we must use ftransform as Doom 3 needs invariant position
   gl_Position = ftransform ();
}

Fragment Shader:

  Code:

uniform samplerCube normalCubeMap;
uniform sampler2D bumpImage;
uniform sampler2D lightFalloffImage;
uniform sampler2D lightImage;
uniform sampler2D diffuseImage;
uniform sampler2D specularImage;
uniform sampler2D specularLookup;

uniform vec4 PP_LIGHT_ORIGIN;
uniform vec4 PP_VIEW_ORIGIN;
uniform vec4 PP_LIGHT_PROJECT_S;
uniform vec4 PP_LIGHT_PROJECT_T;
uniform vec4 PP_LIGHT_PROJECT_Q;
uniform vec4 PP_LIGHT_FALLOFF_S;
uniform vec4 PP_BUMP_MATRIX_S;
uniform vec4 PP_BUMP_MATRIX_T;
uniform vec4 PP_DIFFUSE_MATRIX_S;
uniform vec4 PP_DIFFUSE_MATRIX_T;
uniform vec4 PP_SPECULAR_MATRIX_S;
uniform vec4 PP_SPECULAR_MATRIX_T;

uniform vec4 PP_COLOR_MODULATE;
uniform vec4 PP_COLOR_ADD;

uniform vec4 PP_CONSTANT_DIFFUSE;
uniform vec4 PP_CONSTANT_SPECULAR;


void main ()
{
   vec4 specular = vec4 (normalize (gl_TexCoord[6].xyz), 1.0);

   vec4 light = (textureCube (normalCubeMap, gl_TexCoord[0].xyz) * 2.0) - 1.0;

   vec4 localNormal = ((texture2D (bumpImage, gl_TexCoord[1].xy).agbr)) * 2.0 - 1.0;

   light = vec4
   (
      dot (light.xyz, localNormal.rgb),
      dot (light.xyz, localNormal.rgb),
      dot (light.xyz, localNormal.rgb),
      dot (light.xyz, localNormal.rgb)
   );

   light *= texture2DProj (lightImage, gl_TexCoord[3]);
   light *= texture2DProj (lightFalloffImage, gl_TexCoord[2]);

   vec4 color = texture2D (diffuseImage, gl_TexCoord[4].st) * PP_CONSTANT_DIFFUSE;

   specular = vec4
   (
      dot (specular.xyz, localNormal.rgb),
      dot (specular.xyz, localNormal.rgb),
      dot (specular.xyz, localNormal.rgb),
      dot (specular.xyz, localNormal.rgb)
   );

   vec4 specularResult = texture2D (specularLookup, specular.st);

   specularResult *= PP_CONSTANT_SPECULAR;

   vec4 specularMap = texture2D (specularImage, gl_TexCoord[5].st) * 2.0;

   color = specularResult * specularMap + color;

   color *= light;

   gl_FragColor = color * gl_Color;
}

You'll need a working GLSL backend for these (I have one now ) but otherwise they take the same inputs as the ASM programs.

Here's the current GLSL backend and shaders: http://pastebin.com/8FDsxdPF

Advance warning that the shaders are most likely still not 100% GPL-safe, but they're getting there. The next step will be to go over the backends we have that are GPL-safe (ARB/NV10/NV20/R200), determine what can be reproduced from them, and come up with something that is safe.

"Wot? No qgl?" Yes - life's too short to load extensions manually. I just used GLEW. So you can't just copy-n-paste this into a project and expect it to work; you need to do some extra stuff yourself. But it should be obvious. It replaces draw_arb2, by the way.

twiches... ok gotta say thats extremely cool gonna try this out at once.

Done and works like a champ i gotta say im impressed. Looks even better than the old ARB2 backend. One question though if one where to write a shader for glsl specific stuff that ARB2 cannot handle
how would one be able to ? unless hardcoded in the engine. Or are there ways to get around that ?.

Now i just have to remove raynors backend though it doesnt seem to interfere with anything its not really needed anymore.
Btw i removed all the other backends as they where newer used on modern cards so after this only the ARB2 backend remains.
Seems i will have to toothcomb the source and remove the backend switches completely xD.

It'll load "interaction_vs.glsl" and "interaction_fs.glsl" from glprogs in your gamedir if you have them - that's what I used for development (so you can be certain that it works), together with the "reloadARBprograms" command, otherwise turnaround times are just so slow that it's hardly worth bothering. The hard-coded in-engine versions are just there as fallbacks, and to get the basic shaders under the GPL.

Remember though that the shaders in that code are NOT GPL-friendly - they're relatively straight ports from the original ARB code.

What IS GPL-friendly in them is the matrix multiplications in the vertex shader (they can be reconstructed from TexGen calls elsewhere) and the specular power calculation in the fragment shader (which was a straight port from the in-engine R_SpecularTableImage function, hence the weird calculation instead of a proper power function).

Earlier on today I reconstructed the full fragment shader from the R200 backend; I haven't yet plugged it in and tested it, but when I do I'll post the code and that WILL be GPL-friendly too.

The only remaining item would be the half-angle calculation in the vertex shader. I haven't found any comparable formula in any of the classic literature or tutorials I've trawled through so far, so - assuming that things remain the same - the final GPL-friendly vertex shader may have a slightly different calculation.

Remember though that the shaders in that code are NOT GPL-friendly - they're relatively straight ports from the original ARB code.

I've been told many, many times that one can't license over ARB shaders. Mostly because there isn't much leeway in doing it differently. So that's pretty much a gray area.

License schmeisence, it's game content, it wasn't part of the GPL source distribution, so it still falls under the same terms as the rest of the game content. We've honestly been over this discussion so many times both here and elsewhere that it's ancient news by now and all the old-timers are worn out by it. We were talking about it back in 2002 and it'll still be coming up in 2022. The general consensus is that unless it's explicitly GPL then it's not, and that's not going to change. Nobody wants to be a legal test case.

So - here's the reconstructed fragment shader. The only thing that needs to change is the main function.

  Code:

void main ()
{
   vec4 lightProject = texture2DProj (lightImage, projCoords);
   vec4 lightFalloff = texture2D (lightFalloffImage, fallCoords);
   vec3 bumpMap = texture2D (bumpImage, bumpCoords).agb * 2.0 - 1.0;
   vec4 incomingLight = lightProject * lightFalloff;

   incomingLight *= max (dot (normalize (lightDir), bumpMap), 0.0);

   vec4 diffMap = texture2D (diffuseImage, diffCoords);
   vec4 specMap = texture2D (specularImage, specCoords);
   float specularPower = clamp ((max (dot (normalize (halfAngle), bumpMap), 0.0) - 0.75) * 4.0, 0.0, 1.0);

   specularPower *= specularPower;

   vec4 finalSpec = specularPower * specMap * 2.0 * u_constant_specular;
   vec4 finalDiff = diffMap * u_constant_diffuse;

   gl_FragColor = (finalDiff + finalSpec) * incomingLight * Color;
}

This is 95% taken from the R200 backend, which - as the shader is entirely specified in engine code - falls under the GPL. Therefore so does this.

The only differences of note (the other 5%) are:

- normalization with math rather than the cubemap in two places.
- max (dot (x, y), 0) instead of just dot (x, y) in two places - which is more correct by the classic lighting equations (but has no visible difference in-game so far as I can see, it just seemed right to do).

What's interesting about this one is that it does a standard texture2D on the falloff texture, whereas the ARB-derived version uses texture2DProj (the original ARB shader used TXP, the original R200 backend uses a standard unprojected texture lookup via glSampleMapATI) - no visible difference.

While I could get fancy and add oodles of extra quality here (correcting that specular power would be an obvious place to start), that's not the objective. The objective is to provide GPL shaders that are as equivalent as possible to the ARB shaders.

With the GLSL backend I gave above you can use r_useGLSL 0 or r_useGLSL 1 to switch between GLSL and ARB programs at runtime, which is handy for verifying that the results you're getting are consistent.

@mh: So your GLSL backend is complete and GPL compatible? Where is the catch?

I don't have a GPL-compatible vertex shader yet, that's the catch.

Is it in works?

Yup.

The necessary calculations are all in idSIMD_Generic::CreateTextureSpaceLightVectors and idSIMD_Generic::CreateSpecularTextureCoords in fact, so the shader code can be derived from those.

I've found one interesting place where the normal mapping turns "inside-out" with this code, so consider anything I've posted so far as a work-in-progress snapshot rather than something you should grab and use straight away.