for those working on dos quake ports

http://code.google.com/p/realm/download ... z&can=2&q=

my own build. optimized a bit better and uses ncurses for the terminal emulation. has a built in menu for ease of use.
compiled with mingw64 + dragonegg (llvm backend).

try it out and let me know how it works for you.

Ahh, it's not DOS Quake but a DOSBox! Works with Engoo DOS Quake but not Super8. Engoo does not work with -mem 46, although it does work on DosBOX 0.74 with that setting. Super8 won't work on any setting. Possibly not able to use memory above 16Mb? Wild guess. Yes, I did set DosBOX to 64Mb.

Engoo has detailed error reporting and reports a segfault with -mem 46. (also with -mem 31). Super8 just kicks out back to the prompt.

Anyway the menu is great.

I stole some of FTE's memory functions if that helps.

The earliest part of engoo development was actually trying to get Quake to run in 4-6mb - not much luck.

It's possible that Super8 is bloated above 16Mb minimum memory requirement, just by allocating more memory than standard DOS Quake.

The skybox code allocates enough RAM for 6 512x512 textures, even if no skybox is ever loaded. This alone consumes 1,572,864 bytes.

I've rewritten the code to only allocate RAM for skyboxes when loading them, but the code for freeing that memory (when the skybox is turned off, or when a different one gets loaded, or when (re)loading maps) doesn't seem to work, as the memory usage reported by Windows only goes higher. Maybe it's OS-dependent.

The earliest part of engoo development was actually trying to get Quake to run in 4-6mb - not much luck.

What is special about 4-6 MB?

And why are you thinking it should be able to run in such a small amount of memory?

The reason I am curious is that I recently did some memory experiments continuing a thought-experiment and I've developed a rather bizarre memory management system in just the last 48 hours that is laughably efficient and hilarious and simple. And niche in the sense that modern programmers don't care about such things (I sort of don't either, but I want minimum footprint and tracking of all allocations to check for memory leaks or accidental code oversights).

It is so efficient, if I had the time I'd consider replacing 100% of fixed memory allocations (like static whatever whatever) if I had the time.

It is untested in a performance based environment where speed matters (but I don't see problems), and although I don't know of the consequences of unaligned memory blocks and such and don't have the kind of experience MH and Spike do, it might solve your issue. (And there is a macro in there to make it aligned --- well --- it isn't a macro but a function but it can be turned into a macro easy enough, but not having MH/Spike experience I don't know other than maybe aligning it to 16 byte blocks how to maximize the performance for speed.)

Except I don't know globals would fit into it and software is all about globals.

Either way ... here is code ... I thought I'd have more free time in coming days, but my free time has been zero'ed out for at least a month ...

  Code:

/// Header file
#ifndef _MEMORY_H
#define _MEMORY_H

///////////////////////////////////////////////////////////////////////////////
// Bare minimum to use this file:
#include <stdio.h>      // size_t

typedef struct memlink_s
{
   void*               addy;
   size_t               allocSize;
   
   struct memlink_s*      allocPrev;
   struct memlink_s*      allocNext;
} memlink_t;


///////////////////////////////////////////////////////////////////////////////
//
//   Memory Functions
//
///////////////////////////////////////////////////////////////////////////////

size_t Boundary_Add_And_Pad_To_Next (const size_t originalSize, const size_t addBlockSize);

void  Memory_free            (const void *ptr);
void* Memory_malloc            (const size_t size, const char *Reason);               // memory allocation space is "undefined"
void* Memory_calloc            (const size_t n, const size_t size, const char *Reason);   // zero-filled memory allocation
void* Memory_realloc         (      void *in_ptr, const size_t size, const char *Reason);
char* Memory_strdup            (const char* text);


///////////////////////////////////////////////////////////////////////////////
//
// Memory stratification support: Create memory chains
//
///////////////////////////////////////////////////////////////////////////////


size_t MemoryChain_MemoryUsed (memlink_t* headnode);

// Allocations
void* MemoryChain_malloc (const size_t allocSize, const char* Reason, memlink_t* headnode);
void* MemoryChain_calloc (const size_t allocCount, const size_t allocSize, const char* Reason, memlink_t* headnode);
void* MemoryChain_strdup (const char* myString, memlink_t* headnode);

// Reallocation: Does realloc and refreshes node
void* MemoryChain_realloc (void *in_ptr, const size_t size, const char* Reason);

// Freeing them
void MemoryChain_freeAlloc (void* pointer, memlink_t* headnode);
void MemoryChain_freeAllocChain (memlink_t* headnode);



#endif // !_MEMORY_H

// HEADER ^^^^ CODE BELOW
/***********************************************************************************************************\
*****
**
**   memory.c:   Memory functions
**
*****
\***********************************************************************************************************/


#include "memory.h"




///////////////////////////////////////////////////////////////////////////////
// Bare minimum to use this file:
#include <string.h>         // String funcs
#include <stdlib.h>         // Malloc and friends
#include "eX_environment.h"   // Host_FatalError


size_t Boundary_Add_And_Pad_To_Next (const size_t originalSize, const size_t addBlockSize)
{
   const size_t   boundaryCrossAmount   = originalSize % addBlockSize; // 0 = even, anything else isn't
   const size_t   padToBoundaryAmount   = addBlockSize - boundaryCrossAmount;
   const size_t   newSize            = originalSize + addBlockSize + padToBoundaryAmount;
   
   return newSize;
}

///////////////////////////////////////////////////////////////////////////////
//   Simple memory
///////////////////////////////////////////////////////////////////////////////

void Memory_free (const void *ptr)
{
   if (ptr == NULL)      Host_FatalError ("Memory_free failed: \"%s\"", "Attempted to free NULL pointer");
   
   free ((void *)ptr);
   
   ptr = NULL;
}


void* Memory_malloc (const size_t size, const char *Reason)
{
   const void   *ptr         = malloc (size);
   
   if (ptr == NULL)      Host_FatalError ("Memory_malloc failed: \"%s\" on %i bytes", Reason, size);
   return (void *)ptr;
}

void* Memory_calloc (const size_t n, const size_t size, const char *Reason)
{
   const void   *ptr      = calloc (n, size);
   const int allocsize      = (int)(n * size);
   
   if (ptr == NULL)      Host_FatalError ("Memory_calloc: \"%s\" on %i bytes", Reason, allocsize);
   return (void *)ptr;
}

char* Memory_strdup (const char* text)
{
   return strdup (text);
}

void* Memory_realloc (void *in_ptr, const size_t size, const char *Reason)
{
   const void   *ptr      = realloc((void *)in_ptr, size);
   
   if (ptr == NULL)      Host_FatalError ("Memory_realloc: \"%s\" on %i bytes", Reason, size);
   return (void *)ptr;
}


typedef enum // Tis private
{
   MEMORYOP_MALLOC,
   MEMORYOP_CALLOC,
   MEMORYOP_REALLOC,
   MEMORYOP_STRDUP,
   
} memoryop_t;

///////////////////////////////////////////////////////////////////////////////
//   Internal function prototypes for links
///////////////////////////////////////////////////////////////////////////////

void* sMemoryChain_Register_OP (const memoryop_t allocType, const size_t allocCount, const size_t allocSize, const char* Reason, memlink_t* headnode);
void sMemoryChain_RemoveLink (memlink_t* myNode, memlink_t* headnode);
memlink_t* sMemoryChain_FindAllocation (const void* pointer, memlink_t* headnode);

///////////////////////////////////////////////////////////////////////////////
//   Public memory link prototypes
///////////////////////////////////////////////////////////////////////////////

// Removes link
void MemoryChain_freeAlloc (void* pointer, memlink_t* headnode)
{
   // Run through list.  Dealloc pointer.  Reassign previous and next links.  Free link too.
   memlink_t* currentNode = sMemoryChain_FindAllocation (pointer, headnode);
   
   // Remove it
   sMemoryChain_RemoveLink(currentNode, headnode);
   
   Memory_free (pointer);      // Goodbye pointer
   Memory_free (currentNode);   // Goodbye node.
}

// Removes all links
void MemoryChain_freeAllocChain (memlink_t* headnode)
{
   while (headnode)
      MemoryChain_freeAlloc (headnode->addy, headnode);
}

size_t MemoryChain_MemoryUsed (memlink_t* headnode)
{
   size_t totalBytesAllocated = 0;
   
   for (memlink_t* currentNode = headnode; currentNode; currentNode = currentNode->allocNext)
   {
      totalBytesAllocated += currentNode->allocSize;
      totalBytesAllocated += sizeof(memlink_t);
   }
   return totalBytesAllocated;   
}

void* MemoryChain_malloc (const size_t allocSize, const char* Reason, memlink_t* headnode)
{
   return sMemoryChain_Register_OP (MEMORYOP_MALLOC, 1, allocSize, Reason, headnode);
}

void* MemoryChain_calloc (const size_t allocCount, const size_t allocSize, const char* Reason, memlink_t* headnode)
{
   return sMemoryChain_Register_OP (MEMORYOP_CALLOC, allocCount, allocSize, Reason, headnode);
}

void* MemoryChain_strdup (const char* myString, memlink_t* headnode)
{
   const size_t   allocSize   = strlen (myString) + 1;
   void*         pointer      = sMemoryChain_Register_OP (MEMORYOP_STRDUP, 1, allocSize, "String dup", headnode);
   
   // Must copy the string into there ...
   strcpy (pointer, myString);
   return pointer;
}

///////////////////////////////////////////////////////////////////////////////
//   Internal memory link functions
///////////////////////////////////////////////////////////////////////////////

// The big boss ...
void* sMemoryChain_Register_OP (const memoryop_t allocType, const size_t allocCount, const size_t allocSize, const char* Reason, memlink_t* headnode)
{
   memlink_t*      newNode         = Memory_calloc (1, sizeof(memlink_t), "Child node");
   
   if (headnode == NULL)
   {
      headnode = newNode;  // No need to specify next/previous since chain was empty
   }
   else   
   {
      memlink_t* previousNode = headnode;
      
      // Run through list until hits end which is NULL
      for (memlink_t* currentNode   = headnode->allocNext; currentNode != NULL; previousNode = currentNode, currentNode = currentNode->allocNext);
      
      // Previous is final node
      previousNode->allocNext = newNode;
      newNode->allocPrev      = previousNode;
   }
   
   // Determine alloc size
   
   newNode->addy      = allocType == MEMORYOP_CALLOC ? Memory_calloc (allocCount, allocSize, Reason): Memory_malloc (allocSize, Reason);
   newNode->allocSize   = allocType == MEMORYOP_CALLOC ? allocCount * allocSize : allocSize; // If strdup, this must be provided to us!   
   
   return newNode->addy;
}





memlink_t* sMemoryChain_FindAllocation (const void* pointer, memlink_t* headnode)
{
   memlink_t* currentNode = headnode;
   for (; &currentNode->addy != pointer; currentNode = currentNode->allocNext);
   return currentNode;
}

void sMemoryChain_RemoveLink (memlink_t* myNode, memlink_t* headnode)
{
   do
   {
      // What if headnode (which has no previous, but requires us to rewrite top)
      if (myNode == headnode)  { headnode = myNode->allocNext; break; } // Next could be NULL which is ok
      
      // We are not headnode, therefore we do have a previous
      myNode->allocPrev->allocNext      = myNode->allocNext; 
      
      if (myNode->allocNext)      
         myNode->allocNext->allocPrev   = myNode->allocPrev;      
      
   } while (0);
}

You could just make a global memlink_t* mainMemoryChain and use that in the args. You call MemoryChain_malloc instead of malloc, MemoryChain_calloc instead of calloc, MemoryChain_free instead of free, MemoryChain_strdup instead of strdup and would kill everything Hunk_Alloc with MemoryChain_calloc.

You could create independent temporarily memory chains and use MemoryChain_freeAllocChain to deallocate every single allocation in that temp chain (think Hunk_FreeToLowMark)--- like model loading or whatever needs a ton of temp allocations that normally don't clear until the next level.

Not tested as much as I'd like and for the next month --- unfortunately it appears I'm going to be far too busy to do Quakey stuffs.

(No doubt this isn't a new idea in the universe of C. But I'm rather picky about memory allocations and wanted a system to measure, track with minimal overhead. The function MemoryChain_MemoryUsed can be used to get an aggregate byte count of all memory used by a specific chain. Like if you make a memory chain called memlink_t* modelLoadingTemps, calling that function and feeding it that pointer would say total amount of memory used and calling MemoryChain_freeAllocChain would free it all with a single line.)

NOTE: Didn't have time to write MemoryChain_realloc (or did I? Well it might be in there. Just checked ... nope. I don't believe that Quake uses that anywhere anyway).

a modern malloc implementation will generally have an alignment of 16 bytes as this is the alignment required for the largest register size (sse registers). aligning to 4 bytes would be slow for such registers (although if you use the faster instructions it'll outright crash).
misaligned access is slower, as for any writes, the cpu must read the two blocks to each side, replace the central bytes, and write the whole thing back. on arm, you'll get an exception and the kernel will emulate this really really slowly (or just outright crash).
if you're using random access, like linked lists, you would generally want to ensure that all your data fits within a single cache line.

large global objects are bad in terms of code design. they wreck modularity and if you use them for passing data around then you're really evil. but in terms of memory use, they're not that bad.
Remember that in any system with virtual memory (ie: anything running windows or non-crippled linux) the system will only allocate memory when those pages are actually poked. if you do not poke them, they consume no physical memory even if they consume 100mb of your address space.
the actual problems only come when you no longer need that data (switching from some really huge map to a really tiny one), but for games the working set's maximum capacity is the only real concern.
I repeat though... AVOID USING GLOBALS! THEY'RE EVIL.

progressively reallocing larger and larger blocks of memory has its own issues. malloc+free are slow, as is the memcpy required to move all the data into it.
if you have such allocations coupled with smaller allocations, you can find that you run out of chunks large enough, as each free block has some small chunk of data sitting in the middle which cannot be moved (java has the upper hand here!).
Thus for both reasons, you should always over-allocate and consume the extra as needed, to avoid excessive reallocs.

if you have such allocations coupled with smaller allocations, you can find that you run out of chunks large enough, as each free block has some small chunk of data sitting in the middle which cannot be moved (java has the upper hand here!).
Thus for both reasons, you should always over-allocate and consume the extra as needed, to avoid excessive reallocs.

I take it then if I allocate say a float --- which is 4 bytes --- I should really allocate 16 bytes, use 4 for the float, and then if some other allocation comes rolling in like a size 4 float use part of that remaining 12 bytes.

theoretically yes, but tracking such allocations is a nightmare unless you have some sort of quake-hunk design where the order of frees is directly related to the order of mallocs (ie: all at once). otherwise its a bit of a pain to manage.
implementing malloc on top of malloc is also a bit futile... but if you can avoid calling malloc by allocating 50 objects in the same block (by being really lazy with your frees), then you'll use less memory in tracking each individual allocation, and less cpu in allocating memory (and freeing).

from a code design perspective, having to explicitly free each chunk of memory allows the code to be more dynamic and reusable, etc.

back in the dos days, there was no virtual memory. any pages which were allocated stayed allocated. any paging was purely to work around dos weirdness, otherwise it wouldn't have used any paging at all.
large reallocs would have required large blocks of memory. a big gap could not be released back into any unused pools, thus fragmentation within your small 8mb of ram was a very real concern.
A hunk is thus really useful to avoid such issues. Doesn't help when you have external objects stored in there though (like file handles).

blood also bitches about mem even if i allocate 64 mb and allow it to use dos high mem (it does run though). maybe the daum gui code does not pass the setting correctly since if using my build with another gui frontend it works with no problems ?.
btw it also happens with the standard daum build so seems to be a bug.

theoretically yes, but tracking such allocations is a nightmare unless you have some sort of quake-hunk design where the order of frees is directly related to the order of mallocs (ie: all at once). otherwise its a bit of a pain to manage.
.
.
.
A hunk is thus really useful to avoid such issues. Doesn't help when you have external objects stored in there though (like file handles).

Thanks for detail info, yeah I'm kind of working from the ground up. Trying to learn to innate details like memory alignment, paging, thinking about the slowness of memmove, allocating in blocks exceeding needed size as to avoid constant memmove-ing. But also keeping an eye on very small footprint models. Took a look at VirtualAlloc function in WINE source code (I figured that would be somewhat insightful ...)

Back on topic: Still not sure the magic of 4-6MB. DOSQuake needed with 8 MB or 16MB, didn't it.

if you trap interrupt 14, you could implement your own disk-backed virtual memory system. this would allow you to run using less memory at least, but you'll need to decide which pages to purge from memory.
you will need to know how to edit the page tables that your dos extender sets up...

Generally you'll reserve 64k pages at a time and commit from that. When each 64k page is full you get another. Reserved address space is not the same as actually using physical memory - it's just preventing other allocations from coming from it so that the allocator that reserved it can use it. You can reserve gigabytes of address space if you want and see no impact at all on physical memory usage, and the physical memory that you don't commit will still be available for use by other apps. Memory = reserve + commit. The Quake hunk is actually a rather crude homegrown version of this (as a challenge: mod Quake to have multiple hunks); a modern OS will (should!) have a grown-up memory API that lets you do it a lot more flexibly (and also get extra safety by marking pages as read-only, write-only, no-execute, no-access, etc).

I seem to remember that some old DOS apps had a nasty habit of just malloc'ing all the memory in your system on startup (the joy of no multitasking!) which would of course cause colossal problems when you try to run them on Windows. Might be one reason why DOS Box limits the memory available.

On Windows malloc and free are just wrappers around the native API (HeapAlloc, which in turn uses VirtualAlloc, AKA the big daddy of Windows memory management). You may as well use the native API and get the extra flexibility (multiple independent heaps! free all memory with one HeapDestroy and no allocation tracking needed! automatic thread-safety! defragmentation!) unless you've got a real reason to recreate all the code needed yourself (learning and having fun is a real reason). Dunno about Linux.

With this implementation the biggest bottleneck is going to be linked-list walking during free-all operations. Like any other memory allocator, it's going to be prone to "lots of small allocations" syndrome but that's a common failing, not a specific flaw; it's best if you know the size of your objects in advance and allocate them all in one go, but if not you can allocate a 64k page and reuse that for subsequent allocations until it fills up.

Your Memory_Free doesn't do what you think it does, by the way, and you're modifying some const params in other functions. Not sure if that's a big deal here (const rules make my brain melt - a lot of modern code seems to be just a wall of "const" noise that it's difficult to pick much signal out of, plus they're only compile-time safety that can be circumvented if you get tricksy so I'm naturally suspicious of a false sense of security here - C++ 11 should have really defaulted to const for everything with needing to explicitly specify non-const, but I suppose the damage was done there a long time ago).

Reserved address space is not the same as actually using physical memory

it is the same in dos, sadly. There's no such thing as reserved unless you create your own page/memory manager as otherwise all page faults are fatal. Its not impossible to do, at least, but debugging it is a pain.

Reserved address space is not the same as actually using physical memory

it is the same in dos, sadly. There's no such thing as reserved unless you create your own page/memory manager as otherwise all page faults are fatal. Its not impossible to do, at least, but debugging it is a pain.

It was pointless for a single task, single user OS.