Windows stack limit checking retrospective: amd64, also known as x86-64

Our survey of stack limit checking reaches the modern day with amd64, also known as x86-64. This time, there are two versions of the function, one for user mode and one for kernel mode. We’ll look at the user mode version.

Actually, there are two user mode versions. One is in msvcrt, the legacy runtime.

; on entry, rax is the number of bytes to allocate
; on exit, stack has been validated (but not adjusted)

chkstk:
    sub     rsp, 16
    mov     [rsp], r10          ; save temporary register
    mov     [rsp][8], r11       ; save temporary register

    xor     r11, r11            ; r11 = 0
    lea     r10, [rsp][16][8]   ; r10 = caller's rsp
    sub     r10, rax            ; r10 = desired new stack pointer
    cmovb   r10, r11            ; clamp underflow to zero

    mov     r11, gs:[StackLimit]; user mode stack limit

    cmp     r10, r11            ; are we inside the limit?
    jae     done                ; Y: nothing to do

    and     r10w, #-PAGE_SIZE   ; round down to page start

probe:
    lea     r11, [r11][-PAGE_SIZE]  ; move to previous page
    test    [r11], r11b         ; probe it
    cmp     r10, r11            ; finished probing?
    jb      probe               ; N: keep going

done:
    mov     r10, [rsp]          ; restore temporary register
    mov     r11, [rsp][8]       ; restore temporary register
    add     rsp, 16             ; clean up stack
    ret

Bonus reading: Windows is not a Microsoft Visual C/C++ Run-Time delivery channel.

The other is in ucrtbase, the so-called universal runtime. That one is identical except that the probing is done by writing rather than reading.

    mov     byte ptr [r11], 0   ; probe it

In both cases, the function ensures that the stack has expanded the necessary amount but leaves it the caller’s responsibility to adjust the stack after the call returns. This design preserves compliance with shadow stacks (which Intel calls Control-Flow Enforcement Technology, or CET).

A typical usage might go like this:

    mov     eax, #17328         ; desired stack frame size (zero-extended)
    call    chkstk              ; validate that there is enough stack
    sub     rsp, rax            ; allocate it

Next time, we’ll wrap up the series with a look at AArch64, also known as arm64.

The post Windows stack limit checking retrospective: amd64, also known as x86-64 appeared first on The Old New Thing.