原创 关于NT内核的apc机制

Asynchronous Procedure Calls (APCs) are a fundamental building block in NT's
asynchronous processing architecture. An understanding of this mechanism is
essential to better understand how NT works and performs several core system
operations.

Basically, APCs allow user programs and system components to execute code
in the context of a particular thread and, therefore, within the address space
of a particular process. The I/O Manager executive subsystem, for example, uses
APCs to complete I/O operations initiated asynchronously. When a device driver
calls IoCompleteRequest to notify the I/O Manager that it is through processing
an asynchronous I/O request, the I/O subsystem queues an APC to the thread that
initiated the I/O request. The next time the thread runs at low IRQL, the
pending APC is delivered. This APC's role is copying the results of the I/O
operation and status information from system memory into a buffer in the
thread's virtual address space. Other interesting ways that APCs are used in NT
are to get or set a thread's context and to suspend a thread's execution. The
POSIX environment subsystem also uses APCs to simulate the delivery of POSIX
signals to POSIX processes.

Although APCs are extensively used throughout NT, documentation on how to
use them is completely lacking. In this article, I'll detail how NT processes
APCs and document the exported NT functions available to device-driver writers
to use APCs in their programs. I'll also show a most-likely implementation of
the NT's APC dispatcher subroutine KiDeliverApc to allow you to better grasp the
inners of APC delivery.

APC Objects

In NT there are two possible kinds of APCs: user mode and kernel mode. User
APCs execute in user mode in the target thread's current process context and
require "permission" from that thread to run. Specifically, user-mode APCs
require the target thread to be in an alertable wait state for being
successfully delivered. A thread enters such a state by calling one of the
system functions KeWaitForSingleObject, KeWaitForMultipleObjects,
KeWaitForMutexObject, or KeDelayExecutionThread and specifying the wait as
"alertable." Alternatively, a user thread can cause user-mode APCs to be
delivered to it by calling the undocumented alert-test service
KeTestAlertThread.

When a user-mode APC is delivered to a thread waiting alertably, the wait
state is satisfied with the completion status STATUS_USER_APC. On return to user
mode, the kernel transfers control to the APC routine and resumes the thread's
execution when the APC routine completes.

In contrast to user APCs, kernel APCs execute in kernel mode and can be
classified as regular or special. As will become evident later in this article,
when APCs are delivered to a particular thread, special kernel-mode APCs don't
require permission from that thread to run, whereas regular kernel-mode APCs
require that certain conditions hold before they are successfully executed.
Besides, special kernel-mode APCs can only be blocked when running at raised
IRQL, and they can preempt the execution of a regular kernel-mode APC.

Every APC waiting to execute resides in a thread-specific, kernel-managed
queue and every thread in the system contains two APC queues, one for user-mode
APCs and another for kernel-mode APCs.

NT represents an APC by a kernel control object named "KAPC." Although
the DDK doesn't explicitly document APCs, the KAPC object is clearly declared in
NTDDK.H as shown in Listing 1. From this declaration of the KAPC object, the
self-explanatory fields are Type and Size. The Type field identifies this kernel
object as an APC. In NT, every kernel or executive object is tagged with a type
so that functions can ensure they are handed the appropriate object type. Size
contains a word-aligned value that indicates the length in bytes that this
structure takes up in memory. Spare0 may seem a little obscure, but it's not
used for anything meaningful except for memory-alignment reasons. The other
field's descriptions are less obvious and I'll explain them throughout the
following sections.

APC Environments

At any time during its execution, provided that the current IRQL is at
Passive Level, a thread may need to temporarily execute code in another process
context. To perform this operation, a thread typically calls the system function
KeAttachProcess. On return from this call, the thread is executing within the
address space of another process. Any APCs previously waiting to execute in the
thread's owning-process context cannot be delivered at this time because the
owning-process address space is not currently available. However, new APCs can
be perfectly targeted at this thread to execute in the new process address
space. Even new APCs can be targeted at this thread to execute in the thread's
owning-process context when it finally detaches itself from the new process.

To achieve this degree of control over APC delivery, NT maintains two APC
environments or states per thread. Each APC environment contains the APC queues
for user-mode and kernel-mode APCs, a pointer to the current process object, and
three control variables that indicate: whether there is any pending kernel-mode
APCs (KernelApcPending), whether there are any regular kernel-mode APC in
progress (KernelApcInProgress), and whether there is a user-mode APC pending
(UserApcPending). The locations of these APC environments are kept in an array
of two pointers at the ApcStatePointer field in a thread object.

The main APC environment is the one located at the ApcState field in a
thread object. APCs waiting to execute in the thread's current process context
(whatever it is) reside in the ApcState's queues. Whenever the NT's APC
dispatcher and other system components query a thread for pending APCs, they
check the main APC environment and, in case there are any, they are delivered at
this moment or at some later time, modifying its control variables accordingly.
The second APC environment is located at the SavedApcState field in a thread
object and is used as a backup place for the main APC environment while a thread
is temporarily attached to another process.

The first element in ApcStatePointer always points to the APC environment
used for the thread's owning-process context, while the second element points to
the APC environment used for the thread's new process context, in case the
thread is attached to another process. For example, if a thread is running in
its owning-process address space, the first element of the ApcStatePointer array
contains the address of ApcState and the second element the address of
SavedApcState (in this case SavedApcState is empty). The ApcStateIndex field of
the thread object contains the value OriginalApcEnvironment, indicating the
thread is running in its owning-process context.

When a thread calls KeAttachProcess to execute subsequent code in another
process context, the content of ApcState is copied to SavedApcState. Next,
ApcState is cleaned up, its APC queues are reinitialized, its control variables
set to zero, and the current process field is set to the new process. These
steps successfully guarantee that any APCs previously waiting to execute within
the thread's owning-process address space are not delivered while the thread is
running in a different process context. Next, the thread's ApcStateIndex field
is set to AttachedApcEnvironment, indicating the thread is executing in a
different process context. Later, the ApcStatePointer array content is updated
to reflect the new state, having its first element point to SavedApcState and
its second element to ApcState, indicating that the APC environment for the
thread's owning-process context is now located at SavedApcState, and the one for
the thread's new process context at ApcState. Finally, the current process
context is switched for that of the new process.

What determines the target APC environment for an APC object is the
ApcStateIndex field. The value of ApcStateIndex is taken as an index into
ApcStatePointer to obtain a pointer to the target APC environment. Later, this
pointer is used to place the APC object in its corresponding queue within the
APC environment.

When the thread is detaching from the new process (KeDetachProcess), any
pending kernel APCs waiting to execute in the new process address space are
delivered. Next, the content of SavedApcState is copied back to ApcState,
SavedApcState is cleaned up, the thread's ApcStateIndex field is set to
OriginalApcEnvironment, and ApcStatePointer is also updated; then the current
process context is switched for that of the thread's owning process.

Using APCs

Device drivers use two major functions to utilize APCs. The first,
KeInitializeApc (see Listing 2), is used to initialize an APC object. This
function takes a driver-allocated APC object, a pointer to the target thread
object, the APC environment index (which APC environment to place the APC object
in), the APC's kernel, rundown and normal routines pointers, the kind of APC
(user mode or kernel mode), and a context parameter.

KeInitializeApc first sets the Type and Size fields to the appropriate
values for an APC object. Then it checks the value of the Environment argument.
If it is CurrentApcEnvironment, the ApcStateIndex field is set to the target
thread's ApcStateIndex field; otherwise, the ApcStateIndex field is set to the
value of Environment. Next, this function sets the Thread, RundownRoutine, and
KernelRoutine fields directly from the function arguments. To accurately
determine the kind of APC, KeInitializeApc checks the value of the NormalRoutine
parameter. If it is NULL, the ApcMode field is set to KernelMode and
NormalContext is set to NULL. If NormalRoutine is nonNULL, in which case it must
point to a valid routine, the ApcMode and NormalContext fields are obtained from
the function arguments. Finally, KeInitializeApc sets the Inserted field to
FALSE, indicating the APC object has not been placed in its corresponding APC
queue yet.

From this explanation, you can easily realize that APC objects missing a
valid NormalRoutine are considered as kernel APCs. Specifically, they are
considered as special kernel-mode APCs. Indeed, this is the kind of APC the I/O
Manager uses to perform asynchronous I/O completion. Conversely, APC objects
that define a valid NormalRoutine are considered regular kernel-mode APCs, given
that ApcMode is KernelMode; otherwise, they are considered user-mode APCs. The
prototypes for KernelRoutine, RundownRoutine, and NormalRoutine are defined in
NTDDK.H as shown in Listing 3.

In general, every APC object must contain a valid KernelRoutine function
pointer, whatever its kind. This driver-defined routine will be the first one to
run when the APC is successfully delivered and executed by the NT's APC
dispatcher. User-mode APCs must also contain a valid NormalRoutine function
pointer, which must reside in user memory. Likewise, regular kernel-mode APCs
contain a valid NormalRoutine, which runs in kernel mode just like
KernelRoutine. Optionally, either kind of APC may define a valid RundownRoutine.
This routine must reside in kernel memory and is only called when the system
needs to discard the contents of the APC queues, such as when the thread exits.
In this case, neither KernelRoutine nor NormalRoutine are executed, just the
RundownRoutine. An APC without such a routine will be deleted.

Keep in mind that the action of delivering APCs to a thread only involves
calling the APC dispatcher subroutine KiDeliverApc at operating system
well-defined points, while executing an APC involves actually calling the APC
routines.

Once the APC object is completely initialized, device drivers typically
call KeInsertQueueApc (see Listing 4) to place the APC object in the target
thread's corresponding APC queue. This function takes a pointer to the APC
object initialized with KeInitializeApc, two system arguments, and a priority
increment. As well as the context parameter passed to KeInitializeApc, these
system arguments are simply passed to the APC's routines when they are executed
(see Listing 3).

Before KeInsertQueueApc places the APC object in the target thread's
corresponding APC queue, it first checks whether the target thread is APC
queueable. If it isn't, the function returns immediately with a FALSE result. If
APCs can be queued to the thread, the function sets the SystemArgument1 and
SystemArgument2 fields directly from the function arguments. Next, the function
calls KiInsertQueueApc to actually place the APC object in its corresponding APC
queue.

KiInsertQueueApc only takes an APC object and a priority increment, which
is applied in case the APC causes the target thread's wait state to be
satisfied. This function first obtains the thread's APC queue spinlock and holds
it while it is running to prevent other threads from simultaneously modifying
the thread's APC structures (note that on uniprocessor systems,
KiAcquireSpinLock just returns to the caller). Next, it checks the Inserted
field. If it is TRUE, this indicates the APC object has already been placed in
one APC queue and the function returns immediately with a FALSE result. If it is
FALSE, the function determines the target APC environment from the ApcStateIndex
field as explained earlier and proceeds to place the APC object in its
corresponding queue by linking it through the ApcListEntry field. The position
in which an APC object is placed is determined by the kind of APC. Specifically,
regular kernel-mode and user-mode APCs are placed at the end of their
corresponding APC queues. Conversely, a special kernel-mode APC is placed at the
front of its particular queue before the first regular kernel-mode APC, if there
is any already in the queue. If the APC is a kernel-defined user APC used when a
thread exits, it's also placed at the front of its corresponding queue and the
thread's main APC environment's UserApcPending control variable is set to TRUE.
Now, KiInsertQueueApc sets the Inserted field to TRUE, indicating the APC object
is already placed in its corresponding queue. Next, a check is performed to see
whether the APC was queued to the APC environment for the thread's current
process context. If not, the function immediately returns with a TRUE result. If
this is a kernel APC (whatever its kind), the KernelApcPending control variable
in the thread's main APC environment is set to TRUE.

In the couple-of-sentences allusion to APCs in the Win32 SDK
documentation, it's stated that after an APC has been successfully placed in its
queue, a software interrupt is issued and the APC is executed the next time the
thread is scheduled to run. However, this is not entirely true. Such a software
interrupt is only issued if the APC is directed at the calling thread and it's a
kernel-mode APC, whether regular or special. Later the function returns with a
TRUE result. If the APC is not directed at the calling thread, the target thread
is in a wait state at Passive Level; this is a regular kernel-mode APC; the
thread is not inside a critical region; and no other regular kernel-mode APC is
still in progress, then the thread is awakened with the completion status
STATUS_KERNEL_APC, but the wait state is not aborted. If this is a user-mode
APC, KiInsertQueueApc checks to see whether the target thread is in an alertable
wait state with WaitMode equal to UserMode. If it is, the main APC state's
UserApcPending control variable is set to TRUE and the wait state is satisfied
with the completion status STATUS_USER_APC. Finally, the function releases the
spinlock and returns a TRUE result to indicate the APC object has been
successfully queued.

As a supplement to the APC management functions described earlier, device
drivers can use the undocumented system service NtQueueApcThread (see Listing 5)
to directly queue a user-mode APC to a particular thread. Internally, this
function calls KeInitializeApc and KeInsertQueueApc to accomplish this task.

NT's APC Dispatcher

At well-defined points, NT checks whether a thread has pending APCs. Then,
the APC dispatcher subroutine KiDeliverApc is executed in the thread's context
to initiate APC delivery to the thread. Note that this behavior interrupts the
thread's normal execution flow, giving control to the APC dispatcher first and
later resuming the thread execution when KiDeliverApc completes.

For example, whenever a thread is scheduled to run, the last step of the
context swapping function SwapContext is to inspect whether the new thread has
"pending kernel APCs." If so, SwapContext either (1) requests an APC Level
software interrupt to initiate APC delivery as soon as the new thread runs at
low IRQL (Passive Level) or (2) returns with a TRUE result indicating that the
new thread has "pending kernel APCs." This decision is based upon the IRQL at
which the new thread will ultimately run when control passes to its restored
program counter. If it is higher than Passive Level, SwapContext makes decision
(1), and if it is Passive Level, the function makes decision (2).

The return value of SwapContext is only usable by certain system
functions that explicitly call SwapContext to force a context switch to another
thread. Then, when these system functions are resumed at some later time (when
they are rescheduled again), they usually check the return value of SwapContext
and if it is TRUE, they call the APC dispatcher to deliver kernel APCs to the
current thread. For example, the system function KiSwapThread is used by wait
services to relinquish the processor until the wait is satisfied. This function
calls SwapContext internally and, when its execution is resumed at the point
after the call to SwapContext (when the wait is satisfied), a check is performed
on SwapContext's return value. If it is TRUE, KiSwapThread lowers the IRQL to
APC Level and calls KiDeliverApc to deliver kernel APCs to the current
thread.

For user APCs, the kernel invokes the APC dispatcher only "whenever" a
thread is returning to user mode and the thread's main APC environment's
UserApcPending control variable is TRUE. For example, when the system service
dispatcher KiSystemService is about to return to user mode after completing a
system service request, it checks whether there are pending user APCs. If so, it
raises the IRQL to APC Level and invokes the APC dispatcher, indicating it
should deliver any pending user APCs. Upon execution, KiDeliverApc calls this
user APC's KernelRoutine. Later, the helper function KiInitializeUserApc is
called to set up the thread's trap frame so that on exit from kernel mode, the
thread starts executing in the user-mode APC dispatcher subroutine
KiUserApcDispatcher in Ntdll.dll. The helper function's job is to copy the
thread's previous execution state (which is stored in the trap frame created in
the thread's kernel stack when it entered kernel mode) to the thread's user-mode
stack, as well as the APC's normal routine address, normal context, and both
system arguments, modifying the trap frame's ESP register accordingly. Finally,
it loads the trap frame's EIP register with the address of KiUserApcDispatcher
in Ntdll.dll. When the trap frame is eventually dismissed and the kernel
transfers control to KiUserApcDispatcher, this function invokes the APC's
NormalRoutine, which address and arguments reside in the stack and, when the
routine completes, it calls NtContinue to resume the thread execution as if
nothing had happened using the previous execution context also in the stack.

Listing 6 shows the pseudocode for the NT's APC dispatcher subroutine
KiDeliverApc. As you can observe, its implementation is quite trivial. When the
kernel invokes KiDeliverApc to deliver a user-mode APC, PreviousMode is passed
as UserMode. TrapFrame points to the thread's trap frame and when it's invoked
for kernel APCs, PreviousMode is passed as KernelMode and TrapFrame as NULL.

Note that whenever KernelRoutine is called (whatever the kind of APC),
the pointers passed to it are from local copies of the APC's attributes and,
because the APC object has already been pulled off its queue, it's safe to free
the memory allocated for the APC in KernelRoutine. Besides, this routine has a
last chance to modify its arguments before they are passed on to other
routines.

Conclusion

APCs provide a very useful mechanism to execute code asynchronously in a
particular thread context. As a device driver writer, you can rely on APCs to
execute a routine in a particular thread context without that thread's
intervention or consent whenever no guarantee of its address space's
availability can be made. In this case, special kernel-mode APCs should be used
given the restrictions imposed on regular kernel-mode APC delivery. For user
application programmers, user-mode APCs can effectively be used to implement
some sort of callback notification mechanism.

//-------------------------------------------------------------------------------------------------------

几个函数声明和结构定义：

typedef struct _KAPC {
    CSHORT Type;
    CSHORT Size;
    ULONG
Spare0;
    struct _KTHREAD *Thread;
    LIST_ENTRY ApcListEntry;
   
PKKERNEL_ROUTINE KernelRoutine;
    PKRUNDOWN_ROUTINE RundownRoutine;
   
PKNORMAL_ROUTINE NormalRoutine;
    PVOID NormalContext;
    //
    //
N.B. The following two members MUST be together.
    //
    PVOID
SystemArgument1;
    PVOID SystemArgument2;
    CCHAR
ApcStateIndex;
    KPROCESSOR_MODE ApcMode;
    BOOLEAN Inserted;
}
KAPC, *PKAPC, *RESTRICTED_POINTER PRKAPC;
//------

NTKERNELAPI
VOID
KeInitializeApc (
    IN PRKAPC Apc,
    IN
PKTHREAD Thread,
    IN KAPC_ENVIRONMENT Environment,
    IN
PKKERNEL_ROUTINE KernelRoutine,
    IN PKRUNDOWN_ROUTINE RundownRoutine
OPTIONAL,
    IN PKNORMAL_ROUTINE NormalRoutine OPTIONAL,
    IN
KPROCESSOR_MODE ApcMode,
    IN PVOID NormalContext
    );

typedef enum _KAPC_ENVIRONMENT {
    OriginalApcEnvironment,
   
AttachedApcEnvironment,
    CurrentApcEnvironment
}
KAPC_ENVIRONMENT;
//------------

typedef
VOID
(*PKKERNEL_ROUTINE) (
    IN struct _KAPC *Apc,
   
IN OUT PKNORMAL_ROUTINE *NormalRoutine,
    IN OUT PVOID
*NormalContext,
    IN OUT PVOID *SystemArgument1,
    IN OUT PVOID
*SystemArgument2
    );

typedef
VOID
(*PKRUNDOWN_ROUTINE) (
    IN struct _KAPC *Apc
   
);

typedef
VOID
(*PKNORMAL_ROUTINE) (
    IN PVOID
NormalContext,
    IN PVOID SystemArgument1,
    IN PVOID
SystemArgument2
    );
//------------------

NTKERNELAPI
BOOLEAN
KeInsertQueueApc (
    IN PRKAPC Apc,
    IN
PVOID SystemArgument1,
    IN PVOID SystemArgument2,
    IN KPRIORITY
Increment
    );
//-----------------

NTSYSAPI
NTSTATUS
NTAPI
NtQueueApcThread (
 IN HANDLE
Thread,
 IN PKNORMAL_ROUTINE NormalRoutine,
 IN PVOID
NormalContext,
 IN PVOID SystemArgument1,
 IN PVOID
SystemArgument2
 );