The force_mwait variable iss defined either in
arch/x86/kernel/cpu/amd.c or in arch/x86/kernel/setup_64.c, but it is
only initialized and used in arch/x86/kernel/process.c. This patch
moves the declaration to arch/x86/kernel/process.c.
Signed-off-by: Thomas Petazzoni <thomas.petazzoni@free-electrons.com>
Cc: michael@free-electrons.com
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Jack Ren and Eric Miao tracked down the following long standing
problem in the NOHZ code:
scheduler switch to idle task
enable interrupts
Window starts here
----> interrupt happens (does not set NEED_RESCHED)
irq_exit() stops the tick
----> interrupt happens (does set NEED_RESCHED)
return from schedule()
cpu_idle(): preempt_disable();
Window ends here
The interrupts can happen at any point inside the race window. The
first interrupt stops the tick, the second one causes the scheduler to
rerun and switch away from idle again and we end up with the tick
disabled.
The fact that it needs two interrupts where the first one does not set
NEED_RESCHED and the second one does made the bug obscure and extremly
hard to reproduce and analyse. Kudos to Jack and Eric.
Solution: Limit the NOHZ functionality to the idle loop to make sure
that we can not run into such a situation ever again.
cpu_idle()
{
preempt_disable();
while(1) {
tick_nohz_stop_sched_tick(1); <- tell NOHZ code that we
are in the idle loop
while (!need_resched())
halt();
tick_nohz_restart_sched_tick(); <- disables NOHZ mode
preempt_enable_no_resched();
schedule();
preempt_disable();
}
}
In hindsight we should have done this forever, but ...
/me grabs a large brown paperbag.
Debugged-by: Jack Ren <jack.ren@marvell.com>,
Debugged-by: eric miao <eric.y.miao@gmail.com>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
This is v2, it's a little deference from v1 that I
had send to lkml.
use ACCESS_ONCE
use rcu_batch_after/rcu_batch_before for batch # comparison.
rcutorture test result:
(hotplugs: do cpu-online/offline once per second)
No CONFIG_NO_HZ: OK, 12hours
No CONFIG_NO_HZ, hotplugs: OK, 12hours
CONFIG_NO_HZ=y: OK, 24hours
CONFIG_NO_HZ=y, hotplugs: Failed.
(Failed also without my patch applied, exactly the same bug occurred,
http://lkml.org/lkml/2008/7/3/24)
v1's email thread:
http://lkml.org/lkml/2008/6/2/539
v1's description:
The code/algorithm of the implement of current callbacks-processing
is very efficient and technical. But when I studied it and I found
a disadvantage:
In multi-CPU systems, when a new RCU callback is being
queued(call_rcu[_bh]), this callback will be invoked after the grace
period for the batch with batch number = rcp->cur+2 has completed
very very likely in current implement. Actually, this callback can be
invoked after the grace period for the batch with
batch number = rcp->cur+1 has completed. The delay of invocation means
that latency of synchronize_rcu() is extended. But more important thing
is that the callbacks usually free memory, and these works are delayed
too! it's necessary for reclaimer to free memory as soon as
possible when left memory is few.
A very simple way can solve this problem:
a field(struct rcu_head::batch) is added to record the batch number for
the RCU callback. And when a new RCU callback is being queued, we
determine the batch number for this callback(head->batch = rcp->cur+1)
and we move this callback to rdp->donelist if we find
that head->batch <= rcp->completed when we process callbacks.
This simple way reduces the wait time for invocation a lot. (about
2.5Grace Period -> 1.5Grace Period in average in multi-CPU systems)
This is my algorithm. But I do not add any field for struct rcu_head
in my implement. We just need to memorize the last 2 batches and
their batch number, because these 2 batches include all entries that
for whom the grace period hasn't completed. So we use a special
linked-list rather than add a field.
Please see the comment of struct rcu_data.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Dipankar Sarma <dipankar@in.ibm.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Cc: Dhaval Giani <dhaval@linux.vnet.ibm.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
use a batch number(rcp->pending) instead of a flag(rcp->next_pending)
rcu_start_batch() need to change this flag, so mb()s is needed
for memory-access safe.
but(after this patch applied) rcu_start_batch() do not change
this batch number(rcp->pending), rcp->pending is managed by
__rcu_process_callbacks only, and troublesome mb()s are eliminated.
And codes look simpler and clearer.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Cc: "Paul E. McKenney" <paulmck@linux.vnet.ibm.com>
Cc: Dipankar Sarma <dipankar@in.ibm.com>
Cc: Gautham Shenoy <ego@in.ibm.com>
Cc: Dhaval Giani <dhaval@linux.vnet.ibm.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Real-time code needs to know the number of cycles per second
on SGI UV. The information is provided via a run time BIOS
call. This patch provides the linux side of that interface.
This is the first of several run time BIOS calls to be defined
in uv/bios.h and bios_uv.c.
Note that BIOS_CALL() is just a stub for now. The bios
side is being worked on.
Signed-off-by: Russ Anderson <rja@sgi.com>
Cc: Jack Steiner <steiner@sgi.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Ricardo M. Correia spotted that the use of __fls() in fls64() did
not seem to make sense. In fact fls64()'s implementation is fine,
but the description of __fls() was wrong. Fix that.
Reported-by: "Ricardo M. Correia" <Ricardo.M.Correia@Sun.COM>
Signed-off-by: Alexander van Heukelum <heukelum@fastmail.fm>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andi Kleen <andi@firstfloor.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
[ mingo@elte.hu: picked up this patch from Maciej, lets make apic=debug
print out more info - we had a lot of APIC changes ]
Signed-off-by: Ingo Molnar <mingo@elte.hu>
As a microoptimisation, make apic_verbosity unsigned. This will make
apic_printk(APIC_QUIET, ...) expand into just printk(...) with the
surrounding condition and a reference to apic_verbosity removed.
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
it's separate functionality that deserves its own file.
This also prepares 32-bit memtest support.
Signed-off-by: Yinghai Lu <yhlu.kernel@gmail.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
provide an empty partition_sched_domains() definition for the UP case:
include/linux/cpuset.h: In function ‘rebuild_sched_domains':
include/linux/cpuset.h:163: error: implicit declaration of function ‘partition_sched_domains'
Signed-off-by: Ingo Molnar <mingo@elte.hu>
This is based on Linus' idea of creating cpu_active_map that prevents
scheduler load balancer from migrating tasks to the cpu that is going
down.
It allows us to simplify domain management code and avoid unecessary
domain rebuilds during cpu hotplug event handling.
Please ignore the cpusets part for now. It needs some more work in order
to avoid crazy lock nesting. Although I did simplfy and unify domain
reinitialization logic. We now simply call partition_sched_domains() in
all the cases. This means that we're using exact same code paths as in
cpusets case and hence the test below cover cpusets too.
Cpuset changes to make rebuild_sched_domains() callable from various
contexts are in the separate patch (right next after this one).
This not only boots but also easily handles
while true; do make clean; make -j 8; done
and
while true; do on-off-cpu 1; done
at the same time.
(on-off-cpu 1 simple does echo 0/1 > /sys/.../cpu1/online thing).
Suprisingly the box (dual-core Core2) is quite usable. In fact I'm typing
this on right now in gnome-terminal and things are moving just fine.
Also this is running with most of the debug features enabled (lockdep,
mutex, etc) no BUG_ONs or lockdep complaints so far.
I believe I addressed all of the Dmitry's comments for original Linus'
version. I changed both fair and rt balancer to mask out non-active cpus.
And replaced cpu_is_offline() with !cpu_active() in the main scheduler
code where it made sense (to me).
Signed-off-by: Max Krasnyanskiy <maxk@qualcomm.com>
Acked-by: Linus Torvalds <torvalds@linux-foundation.org>
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Acked-by: Gregory Haskins <ghaskins@novell.com>
Cc: dmitry.adamushko@gmail.com
Cc: pj@sgi.com
Signed-off-by: Ingo Molnar <mingo@elte.hu>
There are already 7 of them - time to kill some duplicate code.
Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
Proc temporary uses stats from init_net.
BTW, TCP_XXX_STATS are beautiful (w/o do { } while (0) facing) again :)
Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
The only structure declared within is the netns_mib, which will
carry all our mibs within. I didn't put the mibs in the existing
netns_xxx structures to make it possible to mark this one as
properly aligned and get in a separate "read-mostly" cache-line.
Signed-off-by: Pavel Emelyanov <xemul@openvz.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
Use alternatives to select the workaround for the 11AP Pentium erratum
for the affected steppings on the fly rather than build time. Remove the
X86_GOOD_APIC configuration option and replace all the calls to
apic_write_around() with plain apic_write(), protecting accesses to the
ESR as appropriate due to the 3AP Pentium erratum. Remove
apic_read_around() and all its invocations altogether as not needed.
Remove apic_write_atomic() and all its implementing backends. The use of
ASM_OUTPUT2() is not strictly needed for input constraints, but I have
used it for readability's sake.
I had the feeling no one else was brave enough to do it, so I went ahead
and here it is. Verified by checking the generated assembly and tested
with both a 32-bit and a 64-bit configuration, also with the 11AP
"feature" forced on and verified with gdb on /proc/kcore to work as
expected (as an 11AP machines are quite hard to get hands on these days).
Some script complained about the use of "volatile", but apic_write() needs
it for the same reason and is effectively a replacement for writel(), so I
have disregarded it.
I am not sure what the policy wrt defconfig files is, they are generated
and there is risk of a conflict resulting from an unrelated change, so I
have left changes to them out. The option will get removed from them at
the next run.
Some testing with machines other than mine will be needed to avoid some
stupid mistake, but despite its volume, the change is not really that
intrusive, so I am fairly confident that because it works for me, it will
everywhere.
Signed-off-by: Maciej W. Rozycki <macro@linux-mips.org>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Adrian Bunk reported that enabling 4MB page size breaks the build.
The problem is that MAX_ORDER combined with the page shift exceeds the
SECTION_SIZE_BITS we use in asm-sparc64/sparsemem.h
There are several ways I suppose we could work around this. For one
we could define a CONFIG_FORCE_MAX_ZONEORDER to decrease MAX_ORDER in
these higher page size cases.
But I also know that these page size cases are broken wrt. TLB miss
handling especially on pre-hypervisor systems, and there isn't an easy
way to fix that.
These options were meant to be fun experimental hacks anyways, and
only 8K and 64K make any sense to support.
So remove 512K and 4M base page size support. Of course, we still
support these page sizes for huge pages.
Signed-off-by: David S. Miller <davem@davemloft.net>
With this commit all sparc64 header files are moved to asm-sparc.
The remaining files (71 files) were too different to be trivially
merged so divide them up in a _32.h and a _64.h file which
are both included from the file with no bit size.
The following script were used:
cd include
FILES=`wc -l asm-sparc64/*h | grep -v '^ 1' | cut -b 20-`
for FILE in ${FILES}; do
echo $FILE:
BASE=`echo $FILE | cut -d '.' -f 1`
FN32=${BASE}_32.h
FN64=${BASE}_64.h
GUARD=___ASM_SPARC_`echo $BASE | tr '-' '_' | tr [:lower:] [:upper:]`_H
git mv asm-sparc/$FILE asm-sparc/$FN32
git mv asm-sparc64/$FILE asm-sparc/$FN64
echo git mv done
printf "#ifndef %s\n" $GUARD > asm-sparc/$FILE
printf "#define %s\n" $GUARD >> asm-sparc/$FILE
printf "#if defined(__sparc__) && defined(__arch64__)\n" >> asm-sparc/$FILE
printf "#include <asm-sparc/%s>\n" $FN64 >> asm-sparc/$FILE
printf "#else\n" >> asm-sparc/$FILE
printf "#include <asm-sparc/%s>\n" $FN32 >> asm-sparc/$FILE
printf "#endif\n" >> asm-sparc/$FILE
printf "#endif\n" >> asm-sparc/$FILE
git add asm-sparc/$FILE
echo new file done
printf "#include <asm-sparc/%s>\n" $FILE > asm-sparc64/$FILE
git add asm-sparc64/$FILE
echo sparc64 file done
done
The guard contains three '_' to avoid conflict with existing guards.
In additing the two Kbuild files are emptied to avoid breaking
headers_* targets.
We will reintroduce the exported header files when the necessary
kbuild changes are merged.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
A manual inspection revealed that the following headerfiles
contained only trivial differences:
hw_irq.h idprom.h kmap_types.h kvm.h spinlock_types.h sunbpp.h unaligned.h
The only noteworthy change are that sparc64 had a volatile
qualifer that sparc missed in spinlock_types.h.
In addition a few comments were updated.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Used the following script to find equal header files:
SPARC64=`ls asm-sparc64`
for FILE in ${SPARC64}; do
cmp -s asm-sparc/$FILE asm-sparc64/$FILE;
if [ $? = 0 ]; then
printf "#include <asm-sparc/%s>\n" $FILE > asm-sparc64/$FILE
fi
done
A few of the equal files are a simple include from
asm-generic, but by including the file from asm-sparc
we know they are equal for sparc and sparc64.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Used the following script to copy the files:
cd include
set -e
SPARC64=`ls asm-sparc64`
for FILE in ${SPARC64}; do
if [ -f asm-sparc/$FILE ]; then
echo $FILE exist in asm-sparc
else
git mv asm-sparc64/$FILE asm-sparc/$FILE
printf "#include <asm-sparc/$FILE>\n" > asm-sparc64/$FILE
git add asm-sparc64/$FILE
fi
done
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Joined the two files as they contain distinct definitions.
Inspired by patch from: Adrian Bunk <bunk@kernel.org>
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Cc: Adrian Bunk <bunk@kernel.org>
sparc64 exports openprom.h to userspace so let sparc follow
the example.
As openprom.h pulled in another not-for-export vaddrs.h header
file it required a few changes to fix the build.
The definition af VMALLOC_* were moved to pgtable as this is
where sparc64 has them.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Copy content of sparc64 file to sparc file.
There is only minimal possibilities for further unification.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Bring the commit e55c57e0b5
("[SPARC64]: Report any user access faults in termios accessors")
over to sparc when unifying the two files.
The diff was manually inspected to contain no
other relevant changes.
This unification therefore changes functionality of sparc.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
The type of tcflag_t differs from 32 and 64 bit.
For 32 bit it is long
For 64 bit it is int
Altough these have same size then I was not sure that
it was OK to change the 64 bit version to long as this
is part of the ABI so it was made conditional.
:$ diff -u include/asm-sparc/termbits.h include/asm-sparc64/termbits.h
:-- include/asm-sparc/termbits.h 2008-06-13 06:42:07.000000000 +0200
:++ include/asm-sparc64/termbits.h 2008-06-13 06:42:07.000000000 +0200
:@@ -1,11 +1,11 @@
:-#ifndef _SPARC_TERMBITS_H
:-#define _SPARC_TERMBITS_H
:+#ifndef _SPARC64_TERMBITS_H
:+#define _SPARC64_TERMBITS_H
:
: #include <linux/posix_types.h>
:
: typedef unsigned char cc_t;
: typedef unsigned int speed_t;
:-typedef unsigned long tcflag_t;
:+typedef unsigned int tcflag_t;
:
: #define NCC 8
: struct termio {
:@@ -102,7 +102,7 @@
: #define IXANY 0x00000800
: #define IXOFF 0x00001000
: #define IMAXBEL 0x00002000
:-#define IUTF8 0x00004000
:+#define IUTF8 0x00004000
:
: /* c_oflag bits */
: #define OPOST 0x00000001
:@@ -171,7 +171,6 @@
: #define HUPCL 0x00000400
: #define CLOCAL 0x00000800
: #define CBAUDEX 0x00001000
:-/* We'll never see these speeds with the Zilogs, but for completeness... */
: #define BOTHER 0x00001000
: #define B57600 0x00001001
: #define B115200 0x00001002
:@@ -199,7 +198,7 @@
: #define B3500000 0x00001012
: #define B4000000 0x00001013 */
: #define CIBAUD 0x100f0000 /* input baud rate (not used) */
:-#define CMSPAR 0x40000000 /* mark or space (stick) parity */
:+#define CMSPAR 0x40000000 /* mark or space (stick) parity */
: #define CRTSCTS 0x80000000 /* flow control */
:
: #define IBSHIFT 16 /* Shift from CBAUD to CIBAUD */
:@@ -258,4 +257,4 @@
: #define TCSADRAIN 1
: #define TCSAFLUSH 2
:
:-#endif /* !(_SPARC_TERMBITS_H) */
:+#endif /* !(_SPARC64_TERMBITS_H) */
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
RLIM_INFINITY differ from 32 and 64 bit.
The rest is equal.
:$ diff -u include/asm-sparc/resource.h include/asm-sparc64/resource.h
:-- include/asm-sparc/resource.h 2008-06-13 06:46:39.000000000 +0200
:++ include/asm-sparc64/resource.h 2008-06-13 06:46:39.000000000 +0200
:@@ -1,11 +1,11 @@
: /*
: * resource.h: Resource definitions.
: *
:- * Copyright (C) 1995 David S. Miller (davem@caip.rutgers.edu)
:+ * Copyright (C) 1996 David S. Miller (davem@caip.rutgers.edu)
: */
:
:-#ifndef _SPARC_RESOURCE_H
:-#define _SPARC_RESOURCE_H
:+#ifndef _SPARC64_RESOURCE_H
:+#define _SPARC64_RESOURCE_H
:
: /*
: * These two resource limit IDs have a Sparc/Linux-specific ordering,
:@@ -14,13 +14,6 @@
: #define RLIMIT_NOFILE 6 /* max number of open files */
: #define RLIMIT_NPROC 7 /* max number of processes */
:
:-/*
:- * SuS says limits have to be unsigned.
:- * We make this unsigned, but keep the
:- * old value for compatibility:
:- */
:-#define RLIM_INFINITY 0x7fffffff
:-
: #include <asm-generic/resource.h>
:
:-#endif /* !(_SPARC_RESOURCE_H) */
:+#endif /* !(_SPARC64_RESOURCE_H) */
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
There were only a few trivial changes and a few additions
in the sparc64 variant of this file.
This patch copies the sparc64 specific bits to the sparc version
of fbio.h so they are equal. A later patch will merge the two.
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Padding in the sembuf structure made conditional
as only 32 bit sparc did so.
:$ diff -u include/asm-sparc/sembuf.h include/asm-sparc64/sembuf.h
:-- include/asm-sparc/sembuf.h 2008-06-13 06:42:07.000000000 +0200
:++ include/asm-sparc64/sembuf.h 2008-06-13 06:42:07.000000000 +0200
:@@ -1,21 +1,18 @@
:-#ifndef _SPARC_SEMBUF_H
:-#define _SPARC_SEMBUF_H
:+#ifndef _SPARC64_SEMBUF_H
:+#define _SPARC64_SEMBUF_H
:
: /*
:- * The semid64_ds structure for sparc architecture.
:+ * The semid64_ds structure for sparc64 architecture.
: * Note extra padding because this structure is passed back and forth
: * between kernel and user space.
: *
: * Pad space is left for:
:- * - 64-bit time_t to solve y2038 problem
:- * - 2 miscellaneous 32-bit values
:+ * - 2 miscellaneous 64-bit values
: */
:
: struct semid64_ds {
: struct ipc64_perm sem_perm; /* permissions .. see ipc.h */
:- unsigned int __pad1;
: __kernel_time_t sem_otime; /* last semop time */
:- unsigned int __pad2;
: __kernel_time_t sem_ctime; /* last change time */
: unsigned long sem_nsems; /* no. of semaphores in array */
: unsigned long __unused1;
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
Padding from 32 bit sparc kept using preprocessor magic
:$ diff -u include/asm-sparc/msgbuf.h include/asm-sparc64/msgbuf.h
:-- include/asm-sparc/msgbuf.h 2008-06-13 06:42:07.000000000 +0200
:++ include/asm-sparc64/msgbuf.h 2008-06-13 06:42:07.000000000 +0200
:@@ -7,17 +7,13 @@
: * between kernel and user space.
: *
: * Pad space is left for:
:- * - 64-bit time_t to solve y2038 problem
:- * - 2 miscellaneous 32-bit values
:+ * - 2 miscellaneous 64-bit values
: */
:
: struct msqid64_ds {
: struct ipc64_perm msg_perm;
:- unsigned int __pad1;
: __kernel_time_t msg_stime; /* last msgsnd time */
:- unsigned int __pad2;
: __kernel_time_t msg_rtime; /* last msgrcv time */
:- unsigned int __pad3;
: __kernel_time_t msg_ctime; /* last change time */
: unsigned long msg_cbytes; /* current number of bytes on queue */
Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
