
If you are using d3dx9_25.dll then that's the April 2005 release of the DirectX SDK. What I can't remember is when the LAA flag was added to Effects for Direct3D 9. Instead you have to use GetParameterByName to get the handle and use that instead. You must also not use the alias trick where you can use a "string name" instead of a "handle" on all the effect methods. You then must use the D3DXFX_LARGEADDRESSAWARE flag when creating all effects. To enable this, you define D3DXFX_LARGEADDRESS_HANDLE before including d3dx9.h headers. This assumption is not true for LargeAddressAware. It really only helps if your application needs a little more room close to 2GB of VA, not if if needs a lot more.Ī key problem with the legacy DirectX SDK Direct3D 9 era effects system is that it assumed the high-bit of the effect "handle" was free so it could use it, and without the bit the handle was an address to a string. LargeAddressAware is a bit of a hack, so it may or may not help your case. Is there anywhere else I should be injecting the Large Address Aware flag before this? I understand that a call to D3DXGetShaderConstantTable will need to be fixed to use D3DXGetShaderConstantTableEx, but its not even reached yet. Is D3DXCreateEffect supposed to accept the Large Address Aware flag? I found a wine test using it, but digging into DX9 assembly, the error it throws is caused by an internal function returning HResult Invalid Call when any bit out of FFFFF800 in flags is set, which leads me to believe CreateEffect is not supposed to accept this flag. Injecting the flag (1 << 17) = 0x20000 to D3DXCreateEffect makes the shader fail compilation in another way.

The only API call the game uses that I can find relies on it is D3DXGetShaderConstantTable, but the issues happen before it is ever called. Unfortunately our knowledge of DX9 is very limited, I've seen that DX9 has a flag D3DXCONSTTABLE_LARGEADDRESSAWARE but I'm not sure where exactly Its supposed to go. Forcing the engine to malloc 2GB before compilation makes the shaders fail to compile every time. The resulting pointer pointed to unallocated memory. It took a valid pointer that looked like 0x8xxxxxx3, checked that bits 0x80000003 are lit and if yes, it bit inverts the pointer and derefs it.

I believe the issue is lack of Large Address Awareness: I noticed one of the d3dx9_25.dll crashes doing something that would hint as such. Usually it returns HResult "out of memory", sometimes d3dx9_25.dll prints random errors in a popup, or it just outright segfault. The issue is usually with D3DXCreateEffect. We are running into issues with an old closed-source game engine failing to compile shaders when memory nears 2GB.
