Modern Microprocessors

Characteristics of Modern Microprocessors

  • Multicore processors: Modern processors can have between 2 (LCC- Low core count) to 24 (HCC – high core count) number of cores for parallelism at hardware level.
  • out-of-order execution:
    • In order to hide the ever widing gap between the CPU and main memory, all modern CPUs are pipelined and have execution reordering.
    • While CPU is waiting for some data or instructions from memory, it might work on something else. There is a complex logic designed in hardware to check the input dependencies between instructions.
  • Multi-level caches: There are atleast 3 levels of caches built in the processor which acts like local memory to the CPU.
  • Speculative execution: Modern CPUs predict the memory location of data to next instruction that may be executed.
  • Microops: The assembly instructions are divided into micro operations which makes it easy for execution and the result is then combined into one.
  • Register renaming: The compilers can compile the code with logical registers like EAX, EBX, EDX etc. The processors have a lot of temporary registers. So it renames one of its temporary registers with logical names.

Architecture diagram for Intel Nehalem Processor

Intel_Nehalem_arch

Image credit: By Appaloosa (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/)%5D, via Wikimedia Commons

CPU Package Overview

Screen Shot 2018-03-15 at 12.00.58 AM

Image credit: http://pages.rubrik.com/rs/794-OHF-673/images/vSphere_6.5_Host_Resources_Deep_Dive.pdf

Intel Xeon Uncore Elements

UNCORE ELEMENT

DESCRIPTION

RESPONSIBLE FOR

QPI Agent

Quick Path Interconnect

QPI caching agent, manages R3QPI and QPI Link Interface.

PCU

Power Controller

Core/Uncore power unit and thermal manager, governs P-State of the CPU, C-State of the Core and package. It enables Turbo Mode and can throttle cores when a thermal violation occurs.

Ubox

System Config Controller

Intermediary for interrupt traffic between system and core.

IIO

Integrated IO

Provides the interface to PCIe Devices.

R2PCI

Ring to PCI Interface

Provides interface to the ring for PCIe access.

IMC

Integrated Memory Controller

Provides the interface to RAM and communicates with Uncore through Home Agent.

HA

Home Agent

Responsible for ordering read/writes coming from Ring to IMC. Provides directory cache coherency.

SMI

Scalable Memory Interface

Provides IMC access to DIMMs.

High Core Count Architecture

Screen Shot 2018-03-15 at 12.04.59 AM.png

Image Credit: http://pages.rubrik.com/rs/794-OHF-673/images/vSphere_6.5_Host_Resources_Deep_Dive.pdf

Intel Xeon Processor Overview

GENERATION

BRANDING

YEAR

PROCESS

CADENCE

MAX CORES

Nehalem

X5500

2008

45nm

4

Westmere

X5600

2010

32nm

Tick

6

Sandy Bridge

E5-2600-v1

2012

32nm

Tock

8

Ivy Bridge

E5-2600-v2

2013

22nm

Tick

12

Haswell

E5-2600-v3

2014

22nm

Tock

18

Broadwell

E5-2600-v4

2016

14nm

Tick-Progress

22

Skylake

2P

2017

14nm

Architecture

28

Image Credit: http://pages.rubrik.com/rs/794-OHF-673/images/vSphere_6.5_Host_Resources_Deep_Dive.pdf

 

Interrupts, Signals and Exceptions

What is an Interrupt

Interrupt is an event that changes the program flow i.e. the instruction stream being executed by the CPU. Interrupts are also generated by various devices connected to the CPU or they caused by bugs within the software. Interrupts are way for hardware to signal to the processor.

Interrupts and Exceptions

  1. Exceptions Synchronous Interrupts:
    Caused by software and produced by control unit of CPU.
    Example: A bug in software or a page fault. Kernel handles these exceptions by following the steps defined (in kernel code) to recover from such a condition.
  2.  Interrupts Asynchronous Interrupts:
    Caused by hardware devices.
    Example: Keypress/mouse movement by user.

Interesting Points about Interrupts

  • Interrupts are asynchronous and they are nested.
  • An interrupt can occur while kernel is handling another interrupt. When kernel is executing some critical region, interrupts are disabled and kept the critical region as small as possible.
  • By disabling interrupts, Kernel guarantees that an interrupt handler will not preempt the critical code.
  • The interrupts and exceptions are identified by a number between 0 to 255.
  • The code executed by interrupt handler is not a process switch, rather its ran at an expense of the process that was running when interrupt was received.
  • Interrupt handling is critical for Kernel but since handling can take long time in the case of slow I/O devices. Hence the interrupt handling is divided into two parts
    • Urgent: Kernel executes this right away.
    • Bottom Halves: Deferred to execute later. (using various techniques like soft irqs, Tasklets, Task/ Work Queues)

Classification of Interrupts

  • Maskable
    • These are interrupt requests issued by I/O devices.
    • There are two states for a maskable interrupt.
      • Masked
      • Unmasked
    • The vectors of maskable interrupts are altered by programming the interrupt control.
  • Nonmaskable
    • They are always recognized by CPU.
    • The vectors of non-maskable interrupts are fixed.

Classification of Exceptions

  • Processor Detected
    • CPU detects an anomalous condition while executing an instruction
  • Faults:
    • Example: Page faults
    • Faults can be corrected and once corrected, program can be resumed.
  • Traps:
    • They can be reported immediately at the next instruction.
    • Used mainly for debugging.
  • Aborts:
    • These are severe errors like hardware failure
    • The process is terminated on receiving this signal.
  • Programmable Exceptions:
    • Often called as software interrupts.
    • Used to implement system calls and debugging.
Exception Number Explanation Type
0 divide by zero error Fault
1 Debug Trap or Fault
2 Not Used
3 Breakpoint Trap
4 Overflow Trap
5 Bounds check Fault
6 Invalid opcode Fault
7 Device not available Fault
8 Double Fault Fault
9 Coprocessor segment overrun Abort
10 Invalid TSS Fault
11 Segment not present Fault
12 Stack segment fault Fault
13 General protection Fault
14 Page Fault Fault
15 Reserved by Intel
16 Floating-point error Fault
17 Alignment check Fault
18 Machine check abort
19 SIMD floating point exception Fault

_Page Fault occurs when the process try to address a page in its address space but is not currently in RAM. When Kernel is handling this exception, it may suspend current process and switch to another process until the page is available in the RAM. The process switch is done because of high latency of RAM (200 ns or serveral hundred CPU cycles). _

Hardware for Interrupt Handling

  • Each hardware device connected to a computer has a single output line named as Interrupt Request (IRQ) line.
  • There is a hardware circuit called Programmable Interrupt Controller (PIC) to which all the IRQ lines are connected.
  • Interrupt Controller (PIC) monitors IRQ lines for raised signals.
  • In case of multiple signals raised simultenously, the signal with lower pine number is selected.
  • When there is a signal raised, its stored in signal vector, then the vector is sent to CPU and signal is raised to CPUs INTR pin to wait until the CPU acknowledges the signal.

 

PIC_Hardware_interrupt_path

Image credit: By Jfmantis – Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=18168230

APIC – Advanced Programmable Interrupt Controller

In modern multiprocessor systems, there is a local APIC chip per CPU. The APIC has following components:

  1. 32 bit registers
  2. Internal clocks
  3. Local timer device
  4. Two additional lines LINT 0 and LINT 1

local_apics_intel_xeon

Image credit: [Intel Software Developer manual vol 3.](https://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-vol-3a-part-1-manual.pdf)

Categories of Interrupts

  • I/O Interrupts
  • Timer Interrupts
  • Interprocessor interrupts

Types of Actions taken by Linux on interrupts

  • Critical
    • Critical actions are executed within the interrupt handler immediately.
  • Noncritical
    • These are quick to finish and hence executed by interrupt handler immediately.
  • Noncritical deferrable
    • These may be delayed for a long time interval without affecting the kernel operations

IRQ Distribution in Multiprocessor System

The kernel tries to distribute the IRQ signals coming from the hardware devices in a round-robin fashion among all the CPUs.

The interrupts coming from external hardware can be distributed within CPUs in following ways

  • Static Distribution
  • Dynamic Distribution

IRQ affinity

The kernel provides a functionality to redirect all the interrupts to a particular CPU. This is achieved by modifying Interrupt Redirection Table entries of the I/O APIC. IRQ affinity of particular interrupts can also be changed by writing a new CPU bitmap mask into the /proc/irq/n/smp_affinity file.

Interrupt Handling

When an interrupt is received, kernel runs inyerrupt handler or interrupt service routine code. These are ‘C’ functions. A data structure named Interrupt Descriptor Table (IDT) stores each interrupt or exception vector with the address of the corresponding interrupt or exception handler. That Table must be properly initialized before the kernel enables interrupts.

Top Halves vs Bottom Halves

  • Interrupt handling should be fast but there may be large amount of work involved, hence the handling is divided into two parts:
  • Top Half: Executed immediately and perform time critical work like acknoledging the interrupt.
  • Bottom Half: This part can be deferred like communicating with I/O.

Deferred Part of Interrupt Handling

  • As stated above the interrupt handling has two parts: critial and non critical (deferred handling).
  • SoftIRQs, Tasklets, WorkQueues etc are ways to process deferred part of interrupt handling which is also called as bottom halves.
SoftIRQs Tasklets
They are statically allocated. can also be allocated and initialized at runtime
softirqs are reentrant functions and must explicitly protect their data structures with spin lock Do not need synchronization because Kernel handles that for them.
provide the least serialization Tasklets of the same type are always serialized: in other words, the same type of tasklet cannot be executed by two CPUs at the same time
Easy to code

Work Queues

  • They defer work into Kernel Queue.
  • Functions in work queues run in process context and hence the function can be blocking functions or can sleep.
  • Function in a work queue is executed by a kernel thread, so there is no User Mode address space to access.

Exception Handling

  • The exceptions raised by CPU are handled by linux as error conditions.
  • Kernel sends a signal to the process about the erroneous condition.
  • Steps taken to handle exception:
    • Save registers to Kernel Stack
    • Invoke C-level function to handle exception.
    • call ret_from_exception() function and exit!

Signals

Signals are software generated interrupts. A signal is generated for a process (or sent to a process) when the event that causes the signal occurs. When the signal is generated, the kernel usually sets a flag of some form in the process table. A signal is delivered to a process when the action for a signal is taken. Between the time of generation and delivery, the signal is pending.

When a process receives a signal, it can do either of following.

  1. Ignore: except for SIGKILL and SIGSTOP all signals can ignored.
  2. catch the signal: call some callback on receiving this signal. Again,  SIGKILL and SIGSTOP can not be caught or blocked.
  3. Apply some default action.

On Termination of the process the memory image of the file is stored in the pwd of the process.

Reentrant functions: functions that are guaranteed to be safe to call from within a signal handler. These functions are async-safe functions meaning they block the signals before entering into a critical region