Add SMT/HT awareness to DragonFlyBSD scheduler

Name: Mihai Carabas

Email: mihai.carabas@gmail.com

Physical address: 10A-12 Merisor Street, Constanta, Romania

Phone number (include country and area code): +40730021550

 

I. Description:

In order to add support for the SMT to the scheduler, I have to establish answers for the following basic questions:

1. which CPUs are siblings (share the same core)
2. which is the status of each core
3. where to schedule

Before I begin, I must get a strong understanding of the current scheduler implementation [1]. Also I have to take a good look in the initialization of the CPU information that the kernel is holding. For this I will start and dig from the entry point of the kernel [2][6]. As for documenting, a good start is the article [3] mentioned in your project proposal and the current linux implementation of the scheduler [4]. Another place where I can look is at the ULE scheduler from FreeBSD [8][9].

I have found a technical report from 2005, written by James R. Bulpin, which describes the entire hyper-threading technology (from hardware to software) [7]. Though is an old paper and there are current advanced SMT-aware schedulers, the theoretical base is the same and is a good place to understand well how I can reach my final goal. In this article, there is a section provinding different tests from where I can start developing the testing suite. 

After a brief documentation I can start working on the following goals:

A. Detect how many threads have each core

• adding a generic way of detecting how many threads each core has (create a map of siblings for example). First, I will be looking into a simple extension where i would only want to differentiate between physical and logical cores. This can then be expanded so that I can take into account physical packages/CPUs in the system. I will also look in the FreeBSD ULE scheduler[8] because it has knowledge about the nodes in the system.
• guard all new added information regarding HyperThreading (physical/logical CPU discovery). For this, I will add an option (Enable HT) in the config file [5].
• create a mechanism that allows to modify the behaviour of the scheduler, from runtime, using sysctl (enable the new HT added opperations versus the original scheduler).

B. Effective SMT aware scheduling task

In order to achieve the final goal, I have to make a design of how threads will get scheduled. I will have to treat different cases:

• "passive" scheduling - select a free physical CPU (if available) for the next thread to schedule
• "active" scheduling" - if a physical CPU becomes idle and on another CPU are running two threads, migrate one of them on the idle physical cpu
• what task to get from a list - get the one that was scheduled on that domain (physical CPU), no matter on what logical CPU has run before
• tasks should stick to the same domain (physical CPU)

In order to achieve all cases presented above, a shared runqueue per physical CPU is required to fulfill all the requirements, as stated in the paper presenting the linux implementation [3]. But, in the first part of the project, I don't want to make changes to be too invasive. I will instead work with what it is now in usched_bsd4.c. After getting the basic idea working, I will try to made more radical changes, as stated above.

C. Testing and testing again

• benchmarks with different types of loads.
• intensive CPU long jobs
• much more CPU short jobs
• each benchmark will be run on the scheduler with and without HT support (using the sysctl option).
• generate some test-cases from the test scenarios presented in James Bulpin's article [7]
• another test scenario can be found here [10], where are compared sched_4bsd.c (HT-unaware) and sched_ule.c (HT-aware). Here they compile Apache2 with -j[2 | 3 |4] and they measure the overall time. Also they used synthetic tests using two utilities:  stream and ubench. We can
• at this step, not only test the performance of the SMT aware scheduler, but discover any bugs introduced by this feature. Bugs will be easily noticeable, as they tend to cause a massive drop in performance under certain workloads or simply panic the system.

II. Project timeline