EuroAssembler

Posted: **08 Sep 2023 20:03**

On 2023/09/08 ecm says at Stack Overflow:

 In your tsrup example, it is said that a UMB must be allocated OFFSET# TsrTop - Shift bytes. However, the AllocUMB function is called with MOV CX,OFFSET# TsrTop / 16 so it appears to allocate too much. Is this an oversight in your example, or am I mistaken? –

When Action I: Install TSR to upper memory was selected. in the code Main.I: indeed says

      MOV CX,OFFSET# TsrTop / 16     ; Number of paragraphs for the resident code.
      CALL AllocUMB

AllocUMB is called here with CX=0x13, i.e. 304 bytes which covers the entire PSP + the size of resident code.
I am not sure now why I chose to allocate this much upper memory, perhaps my DOS 5 refused to terminate and left resident the incomplete PSP in those times (30 years ago).
Nevertheless, tsrup.com /I works in DosBox and only shortened PSP + resident is finally allocated in the upper memory (9 paragraphs = 144 bytes).

>mem /c
Modules using memory below 1 MB:
 Name          Total            Conventional     Uppper Memory
 --------     ----------------    ----------------    ----------------
 SYSTEM      28,720   (28K)   28,720  (28K)               0    (0K)
 COMMAND    3,216    (3K)     3.216    (3K)               0    (0K)
 tsrup              144    (0K)            0    (0K)           144     (0K)
 Free         703,952 (687K)  622,176 (608K)      81,776 (80K)

If you are developping TSR for the real DOS, you can try calling AllocUMB with

CX=(OFFSET# TsrTop - Shift) / 16     ; Number of paragraphs for the resident code.

I might try it myself in future release, but I don't promise for sure. Memory allocation is not easy to debug in DosBox and TSR programs are not in my concern nowadays, sorry.
Anyway, thank you for checking.

Posted: **09 Sep 2023 08:22**

I am not sure now why I chose to allocate this much upper memory, perhaps my DOS 5 refused to terminate and left resident the incomplete PSP in those times (30 years ago).

I don't think so. You are relocating only the PSP contents up to 53h with your relocated resident code after that point already.

Nevertheless, tsrup.com /I works in DosBox and only shortened PSP + resident is finally allocated in the upper memory (9 paragraphs = 144 bytes).

Yes, after your reply I inspected the code more and you do actually terminate with function 31h in either case, so you will eventually shrink the block to 144 Bytes when running InstallUpper as well. However, that means if only a very small UMB is available, you may fail to allocate it when it is between 144 and 288 Bytes in size, even though that would suffice to install the resident portion.

I was confused by your initial over-allocation because I assumed that you'd use function 4Ch to terminate when installing into a UMB. This is what Ralf Brown's TSRs do (when installing into a UMB) and what my TSRs do (always). In this technique, the MCB owner of the allocated (U)MCB is changed to avoid the block being freed by function 4Ch.

Terminating with function 4Ch also happens to close all file handles, which is useful if a user redirected like in > NUL. If you use function 31h and do not close all file handles, then you will leak a Process Handle and also possibly a System File Table entry. The Process Handle Table is not that important, but the SFT impacts all processes on the system.

Another thing is you do not clear the environment block reference in the PSP to zero. I do that in TSRs that I terminate with function 31h.

And there are some different opinions on how much of the PSP is re-usable. DOS keeps at least 60h Bytes of the PSP block when function 31h is passed a zero length. Ralf Brown's TSRs keep only the first 40h Bytes. I'd go with 60h to be safe.

If you are developping TSR for the real DOS, you can try calling AllocUMB with

CX=(OFFSET# TsrTop - Shift) / 16 ; Number of paragraphs for the resident code.

I might try it myself in future release, but I don't promise for sure. Memory allocation is not easy to debug in DosBox and TSR programs are not in my concern nowadays, sorry.
Anyway, thank you for checking.

I know that this is the calculation that you pass to function 31h. However, it is arguably not entirely right.

The TsrTop label is aligned with an ALIGN 16 directive in front of it. But this alignment may be no longer valid when you subtract the Shift value, as that value is possibly not aligned to a paragraph. If the shift equate modulo 16 happens to be 15 (as happens if JMP Main: is a short jump), then the alignment may be wrong. I created a small NASM example program to demonstrate how this can go wrong:

PSP.Reserved53: equ 53h

	org 256
start:
jmp strict short $

TsrBottom:
	times LEN int3
	align 16
TsrTop:

Shift EQU TsrBottom - start + 256 - PSP.Reserved53
paras equ (TsrTop - start + 256 - Shift)/16

%assign shiftnumber Shift
%assign wantalloc LEN + PSP.Reserved53
%assign havealloc paras*16
%warning shift is shiftnumber, want wantalloc bytes, have havealloc bytes

The - start + 256 is equivalent to an OFFSET# keyword for €ASM I believe. It is a standard idiom that I often use to calculate a scalar number from a label in NASM, which allows to use the scalar in division etc.

If you run the program with the LEN smacro set to 14, you get an error:

nasm test.asm -DLEN=14
test.asm:19: warning: shift is 175, want 97 bytes, have 96 bytes [-w+user]

175 is equal to 0AFh, showing that the shift equate is not paragraph aligned. The TSR top label is paragraph aligned, but with subtracting the shift equate, this alignment is no longer valid. When the align directive inserts exactly zero bytes, we want to allocate 53h + 14 = 97 Bytes, but we actually allocate 96 Bytes.

What you want to do is not align TsrTop and instead do the alignment, once, when calculating the paragraphs value. That is you want (OFFSET# TsrTop - Shift + 15) / 16. (This calculation still works with the align directive left where it is, but that's double-booking the alignment and may cause too much space to be allocated for alignment, >= 16 Bytes.)

I found later that your jump at the program entrypoint is actually a near jump because it jumps across both the resident part and the help message, so it cannot be a short jump. That means your alignment calculation is actually correct, as the shift equate modulo 16 is equal to zero. But I do think that it would be better to do it my way, because that hardens it against failure if one changes the offset into the PSP to something other than 53h, or changes the initial entrypoint jump to be shorter or longer than 3 Bytes, such as by moving the help message.

Finally, in InstallUpper you have this comment:

      ; The following instructions actually run in deallocated memory but they are not overwritten yet by DOS.

The relocated PSP actually will also still refer to the Process Handle Table of the unrelocated PSP, so that's another reference to free memory. Obviously, I don't recommend doing that. You are in good company though, I believe that some DOS command.com also runs from freed memory at some point. And the original AMIS uninstall function design hints that it expected the TSR to free itself before returning, which in a naive implementation also runs the last return instruction from free memory. (To avoid this and still implement the entire uninstaller in the resident program, you need some tricks like returning via a stack trampoline that frees the memory block.)

Posted: **09 Sep 2023 16:24**

ecm wrote: ↑09 Sep 2023 08:22 However, that means if only a very small UMB is available, you may fail to allocate it when it is between 144 and 288 Bytes in size, even though that would suffice to install the resident portion.

I thought it necessary to tell DOS that PSP of the resident has changed (was moved to upper memory). This is done in InstallUpper with

  DosAPI AH=0x50 ; Set current PSP to BX=segment of upper memory.

Perhaps that's why AllocUMB was called with that many paragraphs - to keep PSP valid.

ecm wrote: ↑09 Sep 2023 08:22 Another thing is you do not clear the environment block reference in the PSP to zero.

The block is freed at the begining of Main and nobody is going to free it again, so why bother?

ecm wrote: ↑09 Sep 2023 08:22 What you want to do is not align TsrTop and instead do the alignment, once, when calculating the paragraphs value.

Yes, that seems legit, when we change

MOV CX,OFFSET# TsrTop / 16     ; Number of paragraphs for the resident code.

into

MOV CX=(OFFSET# TsrTop - Shift + 15) / 16     ; Number of paragraphs for the resident code.

before CALL AllocUMB.
Your link to Retrocomputing is very inspirative, thank you.

Posted: **09 Sep 2023 16:55**

vitsoft wrote: ↑09 Sep 2023 16:24
ecm wrote: ↑09 Sep 2023 08:22 However, that means if only a very small UMB is available, you may fail to allocate it when it is between 144 and 288 Bytes in size, even though that would suffice to install the resident portion.
I thought it necessary to tell DOS that PSP of the resident has changed (was moved to upper memory). This is done in InstallUpper with
  DosAPI AH=0x50 ; Set current PSP to BX=segment of upper memory.
Perhaps that's why AllocUMB was called with that many paragraphs - to keep PSP valid.

I don't think DOS behaves any differently either way.

And yes, if you want to stay resident with the TSR function 31h but not leave a trace of your original process block then you do have to relocate your process and take care to free the original memory allocation. Meanwhile, I always use function 4Ch to terminate my process and (as mentioned) let my resident block stay allocated through MCB trickery. (This has the advantage of not needing any part of a PSP to remain resident.)

I do also relocate my process, before allocating the resident block, but this is to avoid memory fragmentation if the process was originally loaded to the same memory area in which we want to install. (I use a modified function 4Ch termination with a Parent PSP and Parent Return Address of my choosing to actually relocate the process.)

ecm wrote: ↑09 Sep 2023 08:22 Another thing is you do not clear the environment block reference in the PSP to zero.

The block is freed at the begining of Main and nobody is going to free it again, so why bother?

Some software wants to inspect the environment blocks of processes it can find. Putting a zero in there allows such software to reliably detect that we no longer have an environment allocated to our process. This can avoid wrong results, mostly informational but it is conceivable that someone would modify something it believes to be an environment block (but could have been allocated to something else meanwhile). I think of caution like this as hardening my applications.

For an example of someone else expecting that a resident TSR process's environment may still be used, refer to this note I made in my AMIS review:

leaves resident a modified/reallocated shorter environment block to support memory mappers reading the command name from the string behind the environment variables

ecm wrote: ↑09 Sep 2023 08:22 What you want to do is not align TsrTop and instead do the alignment, once, when calculating the paragraphs value.
Yes, that seems legit, when we change
MOV CX,OFFSET# TsrTop / 16     ; Number of paragraphs for the resident code.
into
MOV CX=(OFFSET# TsrTop - Shift + 15) / 16     ; Number of paragraphs for the resident code.
before CALL AllocUMB.
Your link to Retrocomputing is very inspirative, thank you.

I'm glad to help!

Posted: **09 Sep 2023 17:03**

ecm wrote: ↑09 Sep 2023 16:55
vitsoft wrote: ↑09 Sep 2023 16:24 Your link to Retrocomputing is very inspirative, thank you.
I'm glad to help!

Re-reviewing my answer there, it turns out Ralf Brown also chose to zero the environment block field of the resident PSP:

        xor     ax,ax
        xchg    ax,es:[002Ch]           ; get and zero environment segment
        mov     es,ax
        mov     ah,49h                  ; since we will be going resident,
        int     21h                     ;   discard our environment

EuroAssembler

DOS TSR program in upper memory

DOS TSR program in upper memory

Re: DOS TSR program in upper memory

Re: DOS TSR program in upper memory

Re: DOS TSR program in upper memory

Re: DOS TSR program in upper memory