Racing Ghosts (As this unfolds)

draw the model with glColorMask(0,0,0,0)

I had no idea you could do that (just write to the depth buffer by turning all color off). That is a very nice trick and concept there, thanks Spike.

Joining the train late:

So, can the "ghost" player actually interact with monsters, doors and other elements ? Is this an actual fake player guided/positioned by the demo information ?

Also, can you playback a ghostdemo and record a regular demo at the same time, rinse, repeat so I could simulate several players together in the last demo recorded ? I can see this being EXTREMELY useful for machinima purposes.

Last, please, make all those details (ghost transparency/ghost waits or not byt the player/whatever) configurable by cvars.

Regardless the answers: great idea and good job, Baker.

/ Note: Really I wouldn't have done this qbism didn't start talking about it more. But I was getting bored with whatever code I was working on.

To a master of getting ideas rolling, a taste of your own medicine.
The video looks great, pausing the ghost and the potentially contextual info above his head. I agree w/ frag about the machinima possiblities, especially in combination with the new demo tools.

There's a question as to the extent of the ghosting, and maybe it can vary by application. Could rockets, etc. fired by the ghost be shown? (Transparent, pale, or otherwise distinct from actual rockets) Monsters as the next step, maybe, but that seems confusing even if ghosted. Too much undesirable "double-image" potential.

RE: SDA, standard quake engine with no sv_ changes according to the rules. They are just that good (or we stink that much...)

/ Note: Really I wouldn't have done this qbism didn't start talking about it more. But I was getting bored with whatever code I was working on.

To a master of getting ideas rolling, a taste of your own medicine.

After a few years of learning and listening, I thankfully understand the engine deep enough now to be able to implement unusual and strange ideas instead of just think about them. Took long enough ... but there is a hell of a lot to know.

Monsters, rockets, etc.

Demos are client-side from a player-point-of-view. The only entity guaranteed to be known in any given frame is the player. If a monster or rocket isn't in what the server thinks the player-point-of-view can see, that entity isn't sent to the client so won't be in a demo that frame. Well, and static entities like torches, func_illusionary, etc and temporary entities.

[A demo recorded with sv_novis 1 --- which few engines have --- wouldn't have that issue.]

This stuff is all technically possible managing circumstances carefully, but in practice with any given existing demo wouldn't be consistent [none of the existing demos would have what is needed].

Regarding rockets and weapon effects, those emitted by the ghost should be in the demo most of the time. Although the demo doesn't "know" who they're from. So all of those items in view of the original player would be seen. Again, the chance of "double-image" confusion, if the ghost overlay contains a grenade launched by an ogre from the same position as the real ogre in-game as an example.

There's going to be one more engine with sv_novis soon. Although it sounds like it could be dangerous without enough support from increased limits, protocol, etc.

sv_novis 1 really just started life as a hack to allow entities to be seen through translucent water. It doesn't pretend to be robust, and really does need protocol and other limit extensions. If SV_WriteEntitiesToClient and client-side visedict addition are coded to fail gracefully (rather than crash or overflow buffers) - which is the case with ID Quake - then the worst that can happen is some entities don't get added or drawn.

A few things can firm it up. In SV_WriteEntitiesToClient you could do a first pass through the entities writing those that are visible, then do a second pass writing those that are not. That would mean that if an entity fails to send it's probably not that big a deal as it likely would not have been visible anyway. Sorting entities by distance to the client and sending from nearest to farthest would also help some, at a (small) performance cost (this only need be done for the entities that are not visible). There are also some entities that you probably must send (monsters, doors, plats, other players) and some you could get away with not bothering to send if need be (gibs, heads, dead bodies, bubble sprites) so there's another sort key. Basically, it amounts to being a bit more clever about how you manage sending entities to the client. I haven't bothered doing any of this in my own code, as I've never extended use of sv_novis 1 beyond it's initial purpose as a hack (plus my limits are already stupidly high anyway so I don't really need to).

Client-side it's quite trivial to just extend the visedicts array. Extending cl_entities is a bit more work (if you want to avoid reserving too much memory for this array), but since this is defined at size MAX_EDICTS which is the same size used for server-side, it's probably not necessary (unless you've been mucking around with the size of the server-side array, in which case you're in "might crash" country even without sv_novis 1 support - another of the nice little traps Quake lays for you).

There's one more place server-side that PVS is used and that should probably be addressed, and that's in PF_checkclient where it's used to check if enemies can see you for accelerating aim tests. Sending all entities to the client is also something that should probably be done if the viewpoint is inside a solid leaf (i.e. we're noclipping).

Extending cl_entities is a bit more work (if you want to avoid reserving too much memory for this array), but since this is defined at size MAX_EDICTS which is the same size used for server-side

FitzQuake 0.85 always had a max_edicts cvar dilemma, 666 protocol supports 32000 edicts/entities but that uses a ton of memory, so FitzQuake had the max_edicts cvar which defaults to 1024 (not big enough for some of the king-sized big maps). FitzQuake Mark V ... max_edicts is no longer necessary. It also doesn't use too much memory either A smallish hunk allocation of a mere 64 MB can run ARWOP, and uses only about 50% of the 64 MB. In fact this single change not only essentially kills the need for max_edicts cvar, but combined with raising the default memory from 32 MB to 64 MB, effectively kills the need for a -mem or -heapsize command line parameter.

It isn't much code or a lot of changes either. The mechanics are simple: when loading the map, see how many entities are in the entities section of the BSP. Allocate 50% more than that (with a minimum). Some can still use max_edicts cvar (but why?), it defaults to 0 meaning "check the entities count while loading a map".

I guess my approach went like this: The number of entities needed to allocate memory for isn't some mystery, it can be counted and estimated at level load time.

  Code:

SV_SpawnServer in sv_main.c

/////////////////////////////////////////////////////////////////////////////////

#ifdef SUPPORTS_SMART_EDICTS // Baker change (moved down and reworked)
// Earliest point we can read the entity string, since we need the model loaded.
// 1. Read entity_string from memory
#ifdef SUPPORTS_WRONG_CONTENT_PROTECTION // Baker change
   entity_string = (char *)COM_LoadHunkFile_Limited (va ("maps/%s.ent", sv.name), sv.worldmodel->loadinfo.searchpath );
#else // Baker change +
   entity_string = (char *)COM_LoadHunkFile (va ("maps/%s.ent", sv.name) );
#endif // Baker change -
   if (entity_string)
      Con_Printf ("External .ent file: Using entfile maps/%s.ent\n", sv.name);
   else
      entity_string = sv.worldmodel->entities; // Point it to the standard entities string

// 2. If maxedicts.value == 0 (automatic), look at the entity string
   if (max_edicts.value)
   {
      // Manual setting of maxedicts
      sv.max_edicts = max_edicts.value;
   }
   else
   {
      // Automatic estimation of a good maxedicts #
      int classname_count;

      classname_count = COM_StringCount (entity_string, "classname");

      // Automatic estimation of a good maxedicts #

      sv.max_edicts = classname_count * 1.5;
      Con_DPrintf ("Setting server edicts maximum to %i edicts on classname count of %i\n", sv.max_edicts, classname_count);
   }

// 3. Allocate the memory
   sv.max_edicts = CLAMP(512, sv.max_edicts, 8192);
   sv.edicts = Hunk_AllocName (sv.max_edicts * pr_edict_size, "edicts");

// 4. Now do this.

// leave slots at start for clients only
   sv.num_edicts = svs.maxclients + 1;
   for (i=0 ; i<svs.maxclients ; i++)
   {
      ent = EDICT_NUM(i+1);
      svs.clients[i].edict = ent;
   }

#endif // Baker change +

////////////////////////////////////////////////////////////////
CL_ClearState in cl_main.c

#ifdef SUPPORTS_SMART_EDICTS // Baker change

   if (sv.active)
      cl_max_edicts = sv.max_edicts; // Server is active
   else if (max_edicts.value)
      cl_max_edicts = max_edicts.value; // Client-only (demo or connected to a server) and max_edicts has non-zero value (should be rare)
   else
      cl_max_edicts = 8192; // Client-only and max_edicts is 0.  Set to 8192.  cl edicts are tiny.

   cl_max_edicts = CLAMP (MIN_EDICTS, cl_max_edicts, MAX_EDICTS);
#else // Baker change +

Baker: nice. I'd recently though of doing something like this, but not gotten around to it. 1.5x is probably plenty thanks to all the lights and static entities that wind up being freed immediately.

I changed sv.edicts from an array of edict_t to an array of edict_t *, which needs a little more work and some seriously ugly code (particularly in PR_ExecuteProgram) but accomplishes the same goal. With hindsight my memory management system didn't need it; I could just MEM_RESERVE enough space for 32k edicts in a new CQuakeHeap and pull from that, but I balk at the reworking required elsewhere. Maybe someday.

There's just something seems deeply wrong about crashing the host, potentially at runtime (edicts can be allocated at runtime), and expecting the player to increase a cvar to some arbitrary higher value that they don't know and may potentially may be exceeded again.

guessing the number of entities required is a horrible thought indeed, especially if your guess can be wildly affected by the number of lights on a map which are _generally_ removed immediately after spawning and thus do not contribute to the required limit.
Your 8k limit will cause maps like that 10k knights map to fail.
Your 512 minimum limit may result in crashes if you have 16 players all on some moderatly large/open map spamming nail-spewing grenades, depending on the mod. Especially if its also running a mod like frikbots that auto-adds waypoints to the map.
I do not like mere guesses.

mem_reserve memory is really only one step up from bss. the only difference is that you're likely to have functions to re-reserve it again. which means that you're now allocating and releasing lots of pages instead of simply reusing. in a worse-case scenareo you're still bumping in to the hunk limit (compounded by the size of the map, which is likely to be huge if you have 6k+ ents specified on it).
I'd recommend just using malloc if doing so didn't also break assumptions about qc's ev_pointer types.
FTE has a separate heap for such vm-accessible memory, giving mem_reserve type behaviour for ents/strings, solving qccx pointer hack exploits, and fixing all 64-bit vm issues. Your milage may vary.

MEM_RESERVE needn't be like that. You know you've a max of 32k edicts, you know how big an edict is, so your first VirtualAlloc call reserves space for this total size. As needed you commit in reasonably sized blocks - memory usage only ever goes up as required, so there is no runtime release/realloc happening, and if you commit enough for - say - 128 new edicts each time you need more space, you've got very little runtime allocs happening. Most of the time you're going to be referencing an edict slot that has already been allocated. When starting a new map you wipe and recreate. It's basically a primitive garbage collector. Very fast, no fragmentation, never runs out of space, immune to leaks, everything in linear contiguous memory so it's nicely cache-friendly, very little extra bookkeeping or boilerplate needed, but also very non-portable.

What situation "needs" 32000 entities? Besides experimental maps designed for the purpose, I haven't been able to find a bigmap that threatens the Msg_WriteShort limit of 8192 edicts. My highly scientific process includes running around Unforgiven maps, stirring up as many monsters as possible while occasionally checking "entities" in the console.

No situation, but in theory Fitz 666 supports it, and that means it can happen (in practice you'll crash elsewhere long before you reach that figure).

guessing the number of entities required is a horrible thought indeed, especially if your guess can be wildly affected by the number of lights on a map which are _generally_ removed immediately after spawning and thus do not contribute to the required limit.

I didn't remove the ability to set a host_maxedicts --- which can still be done like typing host_maxedicts 4096 before loading something -- I expanded upon it. So a user can override the estimation via cvar. So I didn't subtract from a capability or take anything away from the user.

Your 512 minimum limit may result in crashes if you have 16 players all on some moderatly large/open map spamming nail-spewing grenades, depending on the mod.

What I should have done in the above is add to the allocation 128 entities per svs.maxclients or cl.maxclients in the above code to address this scenario. My goal was avoiding allocating a colossal 8192 or 32000 server edicts on level load and eliminating true need to mess with the cvar but I did understand that some weird mod might require manually setting the maxedicts so I left that as an option.

The 10000 knights scenario --- unless I'm using presumably the unreleased DirectQ which may have a way that optimizes that to actually make it playable --- am I really going to be playing that?

This isn't perfect, but I think it fully adequate and better than Quake hardcoded 600 and FitzQuake 0.85 cvar default 1024 --- and I didn't remove the ability to set it via cvar. I should have added a per player allocation of X (128?) per svs.maxclients/cl.maxclients to the "automatic" setting of 0 to avoid the 16-players firing grenades that shoot nails.

/Full dynamic in blocks of 128 is better of course. But this gets probably 98% of the benefit with 2% of the work as it doesn't require much effort.

DirectQ's renderer can probably handle that number of knights but it's network code can't, and I expect it's progs interpreter will have some trouble too. I'll probably add an MDL benchmarking test that just spawns these models client-side; it's fun to test these things out.

The big problem with cvar-izing max edicts is that it places responsibility onto the player for something that should be the engine coder's job - it's a cop-out, it's saying "I can't be bothered fixing this right, you deal with it". It's true that making an estimation the way you've done so is a good first step, and a hell of a lot better than what was there before, but it's still got a case where this estimation can be exceeded and you don't seem to have a graceful failure case for that (I'll admit that I haven't looked at your code in a while though). Adding another 128 (and probably lower-bounding at 1024) will help for sure, and will make the failure case quite unlikely, but it's still there.