Port of Xv6 for 25∞2
Go to file
rsc c8919e6537 kernel SMP interruptibility fixes.
Last year, right before I sent xv6 to the printer, I changed the
SETGATE calls so that interrupts would be disabled on entry to
interrupt handlers, and I added the nlock++ / nlock-- in trap()
so that interrupts would stay disabled while the hw handlers
(but not the syscall handler) did their work.  I did this because
the kernel was otherwise causing Bochs to triple-fault in SMP
mode, and time was short.

Robert observed yesterday that something was keeping the SMP
preemption user test from working.  It turned out that when I
simplified the lapic code I swapped the order of two register
writes that I didn't realize were order dependent.  I fixed that
and then since I had everything paged in kept going and tried
to figure out why you can't leave interrupts on during interrupt
handlers.  There are a few issues.

First, there must be some way to keep interrupts from "stacking
up" and overflowing the stack.  Keeping interrupts off the whole
time solves this problem -- even if the clock tick handler runs
long enough that the next clock tick is waiting when it finishes,
keeping interrupts off means that the handler runs all the way
through the "iret" before the next handler begins.  This is not
really a problem unless you are putting too many prints in trap
-- if the OS is doing its job right, the handlers should run
quickly and not stack up.

Second, if xv6 had page faults, then it would be important to
keep interrupts disabled between the start of the interrupt and
the time that cr2 was read, to avoid a scenario like:

   p1 page faults [cr2 set to faulting address]
   p1 starts executing trapasm.S
   clock interrupt, p1 preempted, p2 starts executing
   p2 page faults [cr2 set to another faulting address]
   p2 starts, finishes fault handler
   p1 rescheduled, reads cr2, sees wrong fault address

Alternately p1 could be rescheduled on the other cpu, in which
case it would still see the wrong cr2.  That said, I think cr2
is the only interrupt state that isn't pushed onto the interrupt
stack atomically at fault time, and xv6 doesn't care.  (This isn't
entirely hypothetical -- I debugged this problem on Plan 9.)

Third, and this is the big one, it is not safe to call cpu()
unless interrupts are disabled.  If interrupts are enabled then
there is no guarantee that, between the time cpu() looks up the
cpu id and the time that it the result gets used, the process
has not been rescheduled to the other cpu.  For example, the
very commonly-used expression curproc[cpu()] (aka the macro cp)
can end up referring to the wrong proc: the code stores the
result of cpu() in %eax, gets rescheduled to the other cpu at
just the wrong instant, and then reads curproc[%eax].

We use curproc[cpu()] to get the current process a LOT.  In that
particular case, if we arranged for the current curproc entry
to be addressed by %fs:0 and just use a different %fs on each
CPU, then we could safely get at curproc even with interrupts
disabled, since the read of %fs would be atomic with the read
of %fs:0.  Alternately, we could have a curproc() function that
disables interrupts while computing curproc[cpu()].  I've done
that last one.

Even in the current kernel, with interrupts off on entry to trap,
interrupts are enabled inside release if there are no locks held.
Also, the scheduler's idle loop must be interruptible at times
so that the clock and disk interrupts (which might make processes
runnable) can be handled.

In addition to the rampant use of curproc[cpu()], this little
snippet from acquire is wrong on smp:

  if(cpus[cpu()].nlock == 0)
    cli();
  cpus[cpu()].nlock++;

because if interrupts are off then we might call cpu(), get
rescheduled to a different cpu, look at cpus[oldcpu].nlock, and
wrongly decide not to disable interrupts on the new cpu.  The
fix is to always call cli().  But this is wrong too:

  if(holding(lock))
    panic("acquire");
  cli();
  cpus[cpu()].nlock++;

because holding looks at cpu().  The fix is:

  cli();
  if(holding(lock))
    panic("acquire");
  cpus[cpu()].nlock++;

I've done that, and I changed cpu() to complain the first time
it gets called with interrupts disabled.  (It gets called too
much to complain every time.)

I added new functions splhi and spllo that are like acquire and
release but without the locking:

  void
  splhi(void)
  {
    cli();
    cpus[cpu()].nsplhi++;
  }

  void
  spllo(void)
  {
    if(--cpus[cpu()].nsplhi == 0)
      sti();
  }

and I've used those to protect other sections of code that refer
to cpu() when interrupts would otherwise be disabled (basically
just curproc and setupsegs).  I also use them in acquire/release
and got rid of nlock.

I'm not thrilled with the names, but I think the concept -- a
counted cli/sti -- is sound.  Having them also replaces the
nlock++/nlock-- in trap.c and main.c, which is nice.


Final note: it's still not safe to enable interrupts in
the middle of trap() between lapic_eoi and returning
to user space.  I don't understand why, but we get a
fault on pop %es because 0x10 is a bad segment
descriptor (!) and then the fault faults trying to go into
a new interrupt because 0x8 is a bad segment descriptor too!
Triple fault.  I haven't debugged this yet.
2007-09-27 12:58:42 +00:00
.cvsignore New formatting scripts. 2006-09-05 15:50:55 +00:00
asm.h continuous quality management 2007-08-31 19:55:27 +00:00
bio.c formatting 2007-08-28 05:00:53 +00:00
bootasm.S fix comments 2007-09-15 20:05:47 +00:00
bootmain.c fix comments 2007-09-15 20:05:47 +00:00
bootother.S bootothers now in main 2007-08-29 19:20:49 +00:00
buf.h Another attempt at the bio.c comment. 2007-08-27 14:09:30 +00:00
BUGS sh 2007-09-05 15:55:43 +00:00
cat.c more consistent spacing 2007-08-28 18:32:08 +00:00
console.c use console lock 2007-09-27 12:29:25 +00:00
cuth delete unnecessary #include lines 2007-08-27 23:26:33 +00:00
defs.h kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
dev.h Change dev read/write functions 2007-08-28 17:49:49 +00:00
dot-bochsrc kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
echo.c Remove puts in favor of printf. 2007-08-24 20:03:40 +00:00
elf.h clumsy blank pages to make some files align better 2007-08-28 05:00:39 +00:00
exec.c oops - broke arg counting 2007-08-30 18:19:52 +00:00
fcntl.h nit 2007-08-28 04:13:24 +00:00
file.c remove _ from pipe; be like file 2007-08-28 04:22:35 +00:00
file.h Remove struct uinode. 2007-08-24 20:54:23 +00:00
forktest.c test that fork fails gracefully 2007-08-24 20:20:23 +00:00
fs.c final nits 2007-08-28 19:39:49 +00:00
fs.h comment nits 2007-08-24 19:52:49 +00:00
fsvar.h nits 2007-08-28 18:23:48 +00:00
grep.c add grep; add lost echo 2007-08-28 04:26:34 +00:00
ide.c make new Homework 8 work 2007-08-30 18:21:35 +00:00
init.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
initcode.S Rename main0 to main. 2007-08-27 23:32:16 +00:00
ioapic.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
kalloc.c delete unnecessary #include lines 2007-08-27 23:26:33 +00:00
kbd.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
kbd.h formatting; split kbd constants into kbd.h 2007-08-14 19:31:16 +00:00
kill.c add kill 2007-08-08 08:50:23 +00:00
lapic.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
ln.c add ln 2007-08-22 05:54:55 +00:00
ls.c Remove puts in favor of printf. 2007-08-24 20:03:40 +00:00
main.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
Makefile This should fix building on FreeBSD 2007-09-19 23:49:52 +00:00
mkdir.c more consistent spacing 2007-08-28 18:32:08 +00:00
mkfs.c fix comments 2007-09-15 20:05:47 +00:00
mmu.h formatting 2007-08-14 19:41:38 +00:00
mp.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
mp.h Simplify MP hardware code. 2007-08-27 22:53:31 +00:00
Notes nit in console interrupts 2006-09-07 00:00:33 +00:00
param.h simplify ide queuing 2007-08-24 19:32:36 +00:00
picirq.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
pipe.c spaces around else for rtm 2007-08-28 18:37:41 +00:00
pr.pl runoff changes 2006-09-08 13:53:18 +00:00
printf.c spaces around else for rtm 2007-08-28 18:37:41 +00:00
proc.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
proc.h kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
README spelling 2007-08-29 18:18:57 +00:00
rm.c more consistent spacing 2007-08-28 18:32:08 +00:00
runoff tweak 2007-08-30 14:12:19 +00:00
runoff.list final nits 2007-08-28 19:39:49 +00:00
runoff.spec nits 2007-08-28 05:01:04 +00:00
runoff1 PDF at http://am.lcs.mit.edu/~rsc/xv6.pdf 2007-08-22 06:01:32 +00:00
sh.c clumsy cd 2007-08-30 18:30:26 +00:00
show1 PDF at http://am.lcs.mit.edu/~rsc/xv6.pdf 2007-08-22 06:01:32 +00:00
sign.pl spacing fixes: no tabs, 2-space indents (for rtm) 2006-09-06 17:04:06 +00:00
spinlock.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
spinlock.h clumsy blank pages to make some files align better 2007-08-28 05:00:39 +00:00
stat.h more comments 2006-09-07 14:12:30 +00:00
string.c spaces around else for rtm 2007-08-28 18:37:41 +00:00
swtch.S replace setjmp/longjmp with swtch 2007-08-28 12:48:33 +00:00
symlink.patch symlink implementation 2007-08-30 18:36:38 +00:00
syscall.c spaces around else for rtm 2007-08-28 18:37:41 +00:00
syscall.h Replace yield system call with sleep. 2007-08-27 13:34:35 +00:00
sysfile.c believe it or not, this was working 2007-09-27 05:13:10 +00:00
sysproc.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
timer.c comments; rename irq_ to pic_ 2007-08-28 19:04:36 +00:00
toc.ftr nits 2007-08-28 19:25:04 +00:00
toc.hdr nits 2007-08-28 18:23:48 +00:00
trap.c kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
trapasm.S nits 2007-08-28 18:23:48 +00:00
traps.h typo 2007-08-08 08:27:32 +00:00
types.h align 2007-08-24 20:27:47 +00:00
ulib.c Remove puts in favor of printf. 2007-08-24 20:03:40 +00:00
umalloc.c spaces around else for rtm 2007-08-28 18:37:41 +00:00
user.h add struct stat decl 2007-08-28 04:20:25 +00:00
usertests.c make slow bigdir last test 2007-09-27 12:29:06 +00:00
usys.S Replace yield system call with sleep. 2007-08-27 13:34:35 +00:00
vectors.pl PDF at http://am.lcs.mit.edu/~rsc/xv6.pdf 2007-08-22 06:01:32 +00:00
wc.c more consistent spacing 2007-08-28 18:32:08 +00:00
x86.h kernel SMP interruptibility fixes. 2007-09-27 12:58:42 +00:00
xv6-rev0.tar.gz dist tarball 2006-09-08 15:44:42 +00:00
xv6-rev1.tar.gz DO NOT MAIL: xv6-rev1 2007-08-30 14:11:21 +00:00
xv6.pdf final xv6 for 2007 2007-08-30 14:09:14 +00:00
xv6.ps final xv6 for 2007 2007-08-30 14:09:14 +00:00
zombie.c delete unnecessary #include lines 2007-08-27 23:26:33 +00:00

xv6 is a re-implementation of Dennis Ritchie's and Ken Thompson's Unix
Version 6 (v6).  xv6 loosely follows the structure and style of v6,
but is implemented for a modern x86-based multiprocessor using ANSI C.

ACKNOWLEDGMENTS

xv6 is inspired by John Lions's Commentary on UNIX 6th Edition (Peer
to Peer Communications; ISBN: 1-57398-013-7; 1st edition (June 14,
2000)). See also http://pdos.csail.mit.edu/6.828/2007/v6.html, which
provides pointers to on-line resources for v6.

xv6 borrows code from the following sources:
    JOS (asm.h, elf.h, mmu.h, bootasm.S, ide.c, console.c, and others)
    Plan 9 (bootother.S, mp.h, mp.c, lapic.c)
    FreeBSD (ioapic.c)
    NetBSD (console.c)

The following people made contributions:
    Russ Cox (context switching, locking)
    Cliff Frey (MP)
    Xiao Yu (MP)

The code in the files that constitute xv6 is
Copyright 2006-2007 Frans Kaashoek, Robert Morris, and Russ Cox.

ERROR REPORTS

If you spot errors or have suggestions for improvement, please send
email to Frans Kaashoek and Robert Morris (kaashoek,rtm@csail.mit.edu). 

BUILDING AND RUNNING XV6

To build xv6 on an x86 ELF machine (like Linux or FreeBSD), run "make".
On non-x86 or non-ELF machines (like OS X, even on x86), you will
need to install a cross-compiler gcc suite capable of producing x86 ELF
binaries.  See http://pdos.csail.mit.edu/6.828/2007/tools.html.
Then run "make TOOLPREFIX=i386-jos-elf-".

To run xv6, you can use Bochs or QEMU, both PC simulators.
Bochs makes debugging easier, but QEMU is much faster. 
To run in Bochs, run "make bochs" and then type "c" at the bochs prompt.
To run in QEMU, run "make qemu".  Both log the xv6 screen output to 
standard output.

To create a typeset version of the code, run "make xv6.pdf".
This requires the "mpage" text formatting utility.
See http://www.mesa.nl/pub/mpage/.