David Howells
201a15428b
FS-Cache: Handle pages pending storage that get evicted under OOM conditions
...
Handle netfs pages that the vmscan algorithm wants to evict from the pagecache
under OOM conditions, but that are waiting for write to the cache. Under these
conditions, vmscan calls the releasepage() function of the netfs, asking if a
page can be discarded.
The problem is typified by the following trace of a stuck process:
kslowd005 D 0000000000000000 0 4253 2 0x00000080
ffff88001b14f370 0000000000000046 ffff880020d0d000 0000000000000007
0000000000000006 0000000000000001 ffff88001b14ffd8 ffff880020d0d2a8
000000000000ddf0 00000000000118c0 00000000000118c0 ffff880020d0d2a8
Call Trace:
[<ffffffffa00782d8>] __fscache_wait_on_page_write+0x8b/0xa7 [fscache]
[<ffffffff8104c0f1>] ? autoremove_wake_function+0x0/0x34
[<ffffffffa0078240>] ? __fscache_check_page_write+0x63/0x70 [fscache]
[<ffffffffa00b671d>] nfs_fscache_release_page+0x4e/0xc4 [nfs]
[<ffffffffa00927f0>] nfs_release_page+0x3c/0x41 [nfs]
[<ffffffff810885d3>] try_to_release_page+0x32/0x3b
[<ffffffff81093203>] shrink_page_list+0x316/0x4ac
[<ffffffff8109372b>] shrink_inactive_list+0x392/0x67c
[<ffffffff813532fa>] ? __mutex_unlock_slowpath+0x100/0x10b
[<ffffffff81058df0>] ? trace_hardirqs_on_caller+0x10c/0x130
[<ffffffff8135330e>] ? mutex_unlock+0x9/0xb
[<ffffffff81093aa2>] shrink_list+0x8d/0x8f
[<ffffffff81093d1c>] shrink_zone+0x278/0x33c
[<ffffffff81052d6c>] ? ktime_get_ts+0xad/0xba
[<ffffffff81094b13>] try_to_free_pages+0x22e/0x392
[<ffffffff81091e24>] ? isolate_pages_global+0x0/0x212
[<ffffffff8108e743>] __alloc_pages_nodemask+0x3dc/0x5cf
[<ffffffff81089529>] grab_cache_page_write_begin+0x65/0xaa
[<ffffffff8110f8c0>] ext3_write_begin+0x78/0x1eb
[<ffffffff81089ec5>] generic_file_buffered_write+0x109/0x28c
[<ffffffff8103cb69>] ? current_fs_time+0x22/0x29
[<ffffffff8108a509>] __generic_file_aio_write+0x350/0x385
[<ffffffff8108a588>] ? generic_file_aio_write+0x4a/0xae
[<ffffffff8108a59e>] generic_file_aio_write+0x60/0xae
[<ffffffff810b2e82>] do_sync_write+0xe3/0x120
[<ffffffff8104c0f1>] ? autoremove_wake_function+0x0/0x34
[<ffffffff810b18e1>] ? __dentry_open+0x1a5/0x2b8
[<ffffffff810b1a76>] ? dentry_open+0x82/0x89
[<ffffffffa00e693c>] cachefiles_write_page+0x298/0x335 [cachefiles]
[<ffffffffa0077147>] fscache_write_op+0x178/0x2c2 [fscache]
[<ffffffffa0075656>] fscache_op_execute+0x7a/0xd1 [fscache]
[<ffffffff81082093>] slow_work_execute+0x18f/0x2d1
[<ffffffff8108239a>] slow_work_thread+0x1c5/0x308
[<ffffffff8104c0f1>] ? autoremove_wake_function+0x0/0x34
[<ffffffff810821d5>] ? slow_work_thread+0x0/0x308
[<ffffffff8104be91>] kthread+0x7a/0x82
[<ffffffff8100beda>] child_rip+0xa/0x20
[<ffffffff8100b87c>] ? restore_args+0x0/0x30
[<ffffffff8102ef83>] ? tg_shares_up+0x171/0x227
[<ffffffff8104be17>] ? kthread+0x0/0x82
[<ffffffff8100bed0>] ? child_rip+0x0/0x20
In the above backtrace, the following is happening:
(1) A page storage operation is being executed by a slow-work thread
(fscache_write_op()).
(2) FS-Cache farms the operation out to the cache to perform
(cachefiles_write_page()).
(3) CacheFiles is then calling Ext3 to perform the actual write, using Ext3's
standard write (do_sync_write()) under KERNEL_DS directly from the netfs
page.
(4) However, for Ext3 to perform the write, it must allocate some memory, in
particular, it must allocate at least one page cache page into which it
can copy the data from the netfs page.
(5) Under OOM conditions, the memory allocator can't immediately come up with
a page, so it uses vmscan to find something to discard
(try_to_free_pages()).
(6) vmscan finds a clean netfs page it might be able to discard (possibly the
one it's trying to write out).
(7) The netfs is called to throw the page away (nfs_release_page()) - but it's
called with __GFP_WAIT, so the netfs decides to wait for the store to
complete (__fscache_wait_on_page_write()).
(8) This blocks a slow-work processing thread - possibly against itself.
The system ends up stuck because it can't write out any netfs pages to the
cache without allocating more memory.
To avoid this, we make FS-Cache cancel some writes that aren't in the middle of
actually being performed. This means that some data won't make it into the
cache this time. To support this, a new FS-Cache function is added
fscache_maybe_release_page() that replaces what the netfs releasepage()
functions used to do with respect to the cache.
The decisions fscache_maybe_release_page() makes are counted and displayed
through /proc/fs/fscache/stats on a line labelled "VmScan". There are four
counters provided: "nos=N" - pages that weren't pending storage; "gon=N" -
pages that were pending storage when we first looked, but weren't by the time
we got the object lock; "bsy=N" - pages that we ignored as they were actively
being written when we looked; and "can=N" - pages that we cancelled the storage
of.
What I'd really like to do is alter the behaviour of the cancellation
heuristics, depending on how necessary it is to expel pages. If there are
plenty of other pages that aren't waiting to be written to the cache that
could be ejected first, then it would be nice to hold up on immediate
cancellation of cache writes - but I don't see a way of doing that.
Signed-off-by: David Howells <dhowells@redhat.com >
2009-11-19 18:11:35 +00:00
..
2009-09-22 20:49:04 +01:00
2009-01-06 18:10:26 -08:00
2009-09-04 02:16:14 -07:00
2009-08-07 10:39:56 -07:00
2009-09-19 00:51:34 -03:00
2006-09-26 17:40:24 -04:00
2009-10-09 13:32:24 -07:00
2009-08-27 22:05:39 -07:00
2009-07-25 20:16:01 +02:00
2009-09-22 07:54:33 -07:00
2009-09-26 10:50:47 -07:00
2009-06-05 10:36:24 -07:00
2009-10-11 11:20:58 -07:00
2009-09-20 05:55:36 -07:00
2009-08-25 16:07:58 +02:00
2009-08-24 14:56:30 +02:00
2009-08-24 14:56:30 +02:00
2009-08-24 14:56:30 +02:00
2009-08-24 14:56:30 +02:00
2009-09-23 07:39:29 -07:00
2009-06-18 08:46:47 +10:00
2009-09-22 13:32:45 +01:00
2008-09-03 15:41:57 -07:00
2009-09-26 10:49:42 -07:00
2009-09-09 11:19:00 -04:00
2009-09-23 14:36:38 -04:00
2009-01-30 23:34:34 +05:30
2009-01-30 23:36:36 +05:30
2009-09-24 07:21:05 -07:00
2009-10-09 13:52:05 -07:00
2008-12-22 18:30:29 +00:00
2009-06-11 03:30:20 -07:00
2009-01-02 10:19:37 -08:00
2008-06-16 10:20:58 -07:00
2008-02-06 10:41:02 -08:00
2008-07-25 10:53:47 -07:00
2008-01-30 13:30:18 +01:00
2009-09-19 02:14:45 -04:00
2008-07-01 11:28:06 +10:00
2009-06-17 00:36:36 -04:00
2008-10-20 11:01:52 -07:00
2006-05-04 15:49:24 +01:00
2009-09-27 11:39:25 -07:00
2009-03-26 18:14:14 +01:00
2009-01-30 23:38:03 +05:30
2009-09-20 16:09:20 +05:30
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2009-09-23 07:39:29 -07:00
2007-12-17 19:28:16 -08:00
2007-02-09 17:08:57 +00:00
2009-09-01 01:14:07 -07:00
2005-06-21 19:07:41 -07:00
2009-09-08 17:42:50 -07:00
2009-02-08 09:56:11 -08:00
2008-03-27 14:51:39 -04:00
2009-10-06 00:26:26 -04:00
2009-01-30 23:38:37 +05:30
2009-01-29 16:00:33 -05:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2009-01-15 16:39:41 -08:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2008-05-14 23:24:09 -07:00
2005-04-16 15:20:36 -07:00
2008-12-03 22:12:38 -08:00
2005-04-16 15:20:36 -07:00
2007-02-08 16:01:09 -08:00
2009-01-30 23:39:08 +05:30
2005-04-16 15:20:36 -07:00
2009-09-30 16:12:20 -07:00
2007-02-08 15:13:47 +00:00
2008-02-08 09:22:38 -08:00
2008-02-04 13:16:39 +00:00
2008-03-04 13:41:23 +01:00
2009-04-08 20:47:48 +02:00
2008-07-24 10:47:41 -07:00
2007-10-17 08:42:47 -07:00
2005-04-16 15:20:36 -07:00
2009-03-26 18:14:15 +01:00
2009-03-26 18:14:15 +01:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2009-09-15 09:50:47 -07:00
2009-06-24 00:00:52 -04:00
2009-04-01 08:59:23 -07:00
2009-01-30 23:39:42 +05:30
2009-06-01 06:21:13 +00:00
2009-01-08 08:31:12 -08:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2009-09-26 00:10:40 +02:00
2009-09-21 21:03:58 +01:00
2005-04-16 15:20:36 -07:00
2008-10-20 08:52:42 -07:00
2009-01-30 23:40:06 +05:30
2009-09-24 07:21:01 -07:00
2009-09-11 14:33:31 +02:00
2007-10-18 14:37:29 -07:00
2009-08-21 09:26:15 -07:00
2009-04-23 10:06:35 +01:00
2008-06-06 11:29:10 -07:00
2009-09-11 14:33:32 +02:00
2009-10-05 11:03:58 +02:00
2006-04-27 16:46:56 +01:00
2009-10-01 21:19:34 +02:00
2009-01-04 13:33:20 -08:00
2009-09-22 07:17:38 -07:00
2008-11-28 12:38:38 +01:00
2005-04-16 15:20:36 -07:00
2008-05-29 01:38:46 -07:00
2009-03-28 13:30:43 -07:00
2009-04-16 07:47:49 -07:00
2009-06-16 19:47:48 -07:00
2009-06-15 15:49:34 +02:00
2008-04-28 08:58:19 -07:00
2008-01-28 14:54:10 -08:00
2009-09-21 15:14:51 +02:00
2009-01-15 16:39:41 -08:00
2009-07-31 12:28:46 +02:00
2006-12-07 08:39:30 -08:00
2005-04-16 15:20:36 -07:00
2009-06-11 21:36:09 -04:00
2005-04-16 15:20:36 -07:00
2009-01-30 23:44:08 +05:30
2008-07-04 10:40:05 -07:00
2008-11-07 22:56:00 -08:00
2009-10-01 16:11:11 -07:00
2009-01-30 23:44:41 +05:30
2008-02-03 17:45:46 +02:00
2005-04-16 15:20:36 -07:00
2009-06-04 17:45:43 +01:00
2009-07-10 17:32:55 +02:00
2009-08-28 20:25:24 +02:00
2009-03-26 18:14:15 +01:00
2009-09-23 07:39:29 -07:00
2008-11-09 11:17:33 -08:00
2005-04-16 15:20:36 -07:00
2008-04-30 08:29:54 -07:00
2008-07-26 20:53:14 -04:00
2009-02-03 19:32:24 +05:30
2008-07-25 10:53:33 -07:00
2005-04-16 15:20:36 -07:00
2009-01-23 22:28:48 -08:00
2009-04-30 19:24:24 +02:00
2009-06-18 13:03:57 -07:00
2009-03-26 18:14:23 +01:00
2009-03-12 21:33:03 +00:00
2007-10-17 08:42:47 -07:00
2009-06-12 21:46:57 +09:30
2008-08-26 10:26:54 +02:00
2005-04-16 15:20:36 -07:00
2008-04-30 08:29:54 -07:00
2009-09-24 07:20:57 -07:00
2009-10-02 10:54:05 -07:00
2009-07-16 09:19:16 -07:00
2009-03-24 16:38:26 -07:00
2008-06-04 14:56:12 +01:00
2008-01-30 13:32:42 +01:00
2009-08-15 19:02:07 +02:00
2009-10-29 07:39:24 -07:00
2008-06-11 19:13:45 -04:00
2009-09-24 09:34:53 +09:30
2009-06-16 19:47:31 -07:00
2005-04-16 15:20:36 -07:00
2009-06-11 21:36:10 -04:00
2009-07-12 12:22:34 -07:00
2007-07-17 10:23:04 -07:00
2005-09-12 08:48:08 -07:00
2006-12-08 08:28:39 -08:00
2008-12-25 11:01:43 +11:00
2008-04-30 08:29:54 -07:00
2007-05-10 18:24:13 +02:00
2008-07-12 08:22:32 -05:00
2009-09-23 11:01:25 -07:00
2009-07-14 20:29:57 +08:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2009-09-19 13:13:31 -07:00
2006-04-26 12:56:16 +01:00
2005-04-16 15:20:36 -07:00
2005-09-14 08:36:54 -04:00
2008-02-03 15:42:53 +02:00
2009-09-10 10:00:05 -07:00
2009-06-11 21:36:06 -04:00
2009-09-01 01:24:30 -07:00
2009-01-21 14:34:05 -08:00
2008-02-14 21:17:08 -08:00
2009-04-14 11:11:52 +02:00
2009-09-24 07:20:57 -07:00
2008-04-30 08:29:53 -07:00
2008-06-24 13:53:46 +02:00
2009-09-18 09:48:52 -07:00
2008-04-29 08:06:09 -07:00
2009-09-04 20:40:25 +01:00
2009-10-30 14:59:51 -07:00
2008-10-13 09:51:43 -07:00
2009-01-12 20:56:41 +01:00
2008-07-25 10:53:34 -07:00
2007-05-08 11:15:26 -07:00
2008-08-28 11:49:15 -05:00
2009-03-26 18:14:15 +01:00
2009-01-30 23:45:32 +05:30
2009-05-07 10:14:26 -05:00
2008-04-21 11:22:29 -05:00
2008-08-07 02:22:54 -04:00
2009-04-02 19:55:30 +01:00
2008-04-25 13:26:55 +01:00
2009-06-22 10:12:30 +01:00
2008-04-25 13:26:55 +01:00
2009-09-04 20:40:34 +01:00
2008-10-21 17:45:06 +01:00
2009-05-18 14:46:26 +01:00
2008-07-22 10:39:36 +10:00
2009-04-15 12:22:37 +02:00
2009-08-28 14:24:10 +10:00
2009-09-08 17:55:21 -07:00
2007-02-09 17:39:36 -05:00
2009-06-23 22:09:17 +01:00
2009-09-08 21:17:48 -04:00
2009-01-30 23:46:03 +05:30
2009-06-11 14:57:53 -04:00
2009-01-05 08:40:21 -08:00
2009-01-05 08:40:25 -08:00
2009-01-05 08:40:25 -08:00
2006-06-30 18:25:18 +02:00
2008-10-15 12:46:51 +01:00
2007-07-16 09:05:42 -07:00
2009-04-03 14:53:32 -07:00
2009-09-14 17:41:42 -07:00
2009-04-01 15:42:34 -07:00
2009-04-16 16:17:11 -07:00
2008-04-29 08:06:26 -07:00
2009-01-30 23:46:40 +05:30
2007-10-10 16:53:56 -07:00
2009-06-17 09:33:49 -07:00
2009-01-30 23:47:07 +05:30
2005-04-16 15:20:36 -07:00
2009-06-15 21:30:26 -07:00
2009-05-11 09:52:18 +02:00
2008-02-08 09:22:30 -08:00
2009-01-30 23:47:59 +05:30
2009-10-06 10:35:10 +02:00
2008-02-29 08:06:37 -08:00
2009-07-08 09:18:05 -07:00
2008-01-28 23:21:18 +01:00
2009-08-22 17:52:13 -05:00
2008-02-07 08:42:25 -08:00
2007-06-01 08:18:29 -07:00
2009-02-15 22:43:33 -08:00
2009-05-05 12:26:24 -07:00
2009-09-02 23:07:39 -07:00
2009-09-23 07:39:29 -07:00
2009-03-16 08:32:27 -06:00
2008-08-20 13:19:51 -07:00
2009-01-08 08:31:00 -08:00
2009-01-08 08:31:00 -08:00
2009-11-11 15:22:49 +01:00
2009-01-08 17:14:59 -08:00
2009-07-15 21:30:46 +02:00
2008-10-16 11:21:30 -07:00
2007-11-08 08:42:46 -05:00
2006-03-31 12:18:54 -08:00
2009-06-24 08:15:27 -04:00
2008-12-29 11:27:46 +02:00
2009-11-12 07:25:56 -08:00
2005-04-16 15:20:36 -07:00
2007-10-17 08:43:01 -07:00
2008-10-09 08:56:19 +02:00
2008-11-20 20:29:48 -08:00
2005-04-16 15:20:36 -07:00
2009-04-03 12:13:03 +02:00
2007-04-25 22:28:35 -07:00
2009-04-13 15:04:30 -07:00
2008-10-21 07:47:06 -04:00
2009-06-18 19:50:21 +08:00
2009-03-24 20:56:50 +01:00
2007-05-10 18:24:13 +02:00
2009-09-12 14:48:40 +02:00
2009-06-16 19:47:28 -07:00
2009-06-15 21:30:25 -07:00
2005-04-16 15:20:36 -07:00
2009-09-22 07:17:47 -07:00
2007-05-08 11:15:31 -07:00
2008-10-30 11:38:45 -07:00
2009-04-27 02:53:51 -07:00
2006-12-08 08:28:40 -08:00
2009-03-31 23:00:26 -04:00
2008-08-23 12:14:12 -07:00
2009-10-04 12:39:14 -07:00
2009-11-19 18:11:35 +00:00
2009-11-19 18:11:35 +00:00
2009-06-18 13:03:42 -07:00
2009-07-21 15:26:27 -04:00
2009-06-11 14:57:54 -04:00
2009-09-18 07:25:44 +02:00
2009-02-07 20:03:33 -05:00
2009-10-07 15:52:09 -04:00
2009-07-08 18:17:58 +02:00
2009-10-08 12:16:35 -07:00
2009-05-09 16:10:18 -07:00
2009-06-18 13:04:05 -07:00
2009-08-17 21:33:49 -07:00
2006-10-02 07:57:12 -07:00
2007-05-09 08:58:16 +02:00
2009-01-02 10:19:38 -08:00
2009-01-30 23:49:39 +05:30
2009-10-06 20:16:55 +02:00
2006-09-30 01:47:55 +02:00
2009-09-23 07:39:29 -07:00
2009-08-25 13:44:04 +01:00
2007-02-12 09:48:30 -08:00
2008-10-27 22:33:04 -04:00
2007-07-10 00:35:17 -04:00
2009-09-23 07:39:46 -07:00
2009-09-11 13:23:18 -07:00
2008-02-06 10:41:00 -08:00
2009-09-19 13:13:31 -07:00
2009-09-01 01:13:31 -07:00
2009-02-17 17:37:40 -08:00
2005-04-16 15:20:36 -07:00
2009-04-01 21:42:26 +02:00
2009-06-26 10:48:34 +02:00
2009-09-17 15:15:11 +02:00
2009-01-30 23:51:17 +05:30
2009-01-04 01:00:51 +01:00
2009-06-16 19:47:41 -07:00
2006-04-26 12:56:16 +01:00
2008-04-18 22:22:54 -04:00
2005-10-21 22:58:51 -04:00
2008-11-20 20:32:15 -08:00
2007-03-16 00:59:29 -04:00
2008-07-31 18:45:41 +02:00
2009-08-14 15:59:30 +02:00
2006-11-08 18:29:25 -08:00
2009-09-27 11:39:25 -07:00
2008-04-30 08:29:54 -07:00
2006-03-23 14:21:50 -08:00
2006-01-05 22:16:26 -08:00
2007-10-09 22:56:30 -04:00
2005-04-16 15:20:36 -07:00
2007-05-09 08:58:21 +02:00
2009-03-28 21:34:44 +01:00
2008-10-22 20:21:30 +02:00
2009-02-24 19:19:49 +01:00
2007-05-01 23:26:34 +02:00
2009-09-19 00:16:14 -03:00
2009-06-13 10:39:28 +01:00
2009-03-28 21:34:45 +01:00
2008-08-08 15:13:27 +01:00
2006-11-30 12:24:45 +00:00
2009-11-07 13:10:46 +01:00
2007-01-30 08:26:45 -08:00
2008-10-16 11:21:38 -07:00
2005-04-16 15:20:36 -07:00
2009-05-28 20:52:40 -04:00
2009-09-17 23:23:45 -07:00
2009-01-21 14:02:26 -08:00
2007-04-25 22:25:31 -07:00
2009-08-12 22:13:15 -07:00
2009-09-01 17:52:57 -07:00
2009-04-02 19:04:53 -07:00
2009-08-28 14:40:24 -04:00
2009-09-11 12:54:58 -07:00
2009-01-30 23:52:41 +05:30
2009-03-26 18:14:15 +01:00
2009-08-19 23:08:24 +04:00
2007-12-07 15:00:32 -05:00
2008-06-11 21:00:38 -07:00
2005-04-16 15:20:36 -07:00
2005-04-16 15:20:36 -07:00
2007-10-10 16:51:16 -07:00
2009-07-26 19:25:44 -07:00
2009-01-30 23:53:12 +05:30
2008-10-13 19:01:08 -07:00
2009-09-07 01:56:33 -07:00
2009-01-30 23:53:40 +05:30
2005-04-16 15:20:36 -07:00
2009-01-30 23:54:53 +05:30
2005-05-05 14:25:59 -07:00
2008-04-30 08:29:54 -07:00
2009-05-21 15:13:39 -07:00
2008-10-01 01:30:19 -07:00
2005-04-16 15:20:36 -07:00
2009-01-30 23:55:45 +05:30
2009-03-26 18:14:14 +01:00
2009-02-14 22:58:35 -08:00
2005-04-16 15:20:36 -07:00
2009-01-30 23:56:13 +05:30
2009-01-30 23:56:48 +05:30
2009-07-17 10:11:00 -07:00
2009-09-26 20:28:07 -07:00
2009-04-16 02:02:07 -07:00
2009-05-18 22:19:19 -07:00
2009-09-15 02:37:40 -07:00
2008-08-02 18:36:10 +01:00
2009-06-29 08:59:10 +10:00
2009-02-02 21:40:58 +05:30
2008-06-03 16:36:01 -07:00
2009-06-02 00:45:24 -07:00
2009-01-30 23:57:43 +05:30
2008-06-27 20:09:00 -07:00
2008-07-09 12:09:28 -04:00
2009-07-27 18:39:45 -07:00
2008-01-30 13:34:11 +01:00
2009-09-21 14:28:04 +02:00
2009-10-02 15:42:19 +02:00
2005-04-16 15:20:36 -07:00
2008-11-15 12:26:44 -08:00
2007-04-29 23:42:45 -04:00
2009-11-10 21:13:32 -08:00
2009-09-11 16:40:10 +01:00
2009-10-11 11:20:58 -07:00
2009-08-26 15:41:16 -07:00
2008-05-05 16:47:14 +10:00
2006-10-05 15:10:12 +01:00
2006-10-10 15:37:22 -07:00
2009-07-31 08:55:48 +02:00
2005-04-16 15:20:36 -07:00
2008-10-16 11:21:33 -07:00
2009-03-24 09:42:54 +00:00
2009-09-23 07:39:41 -07:00
2009-01-05 22:46:26 -05:00
2009-07-15 08:17:02 +01:00
2009-01-30 23:58:12 +05:30
2009-03-26 18:14:15 +01:00
2007-04-25 22:25:31 -07:00
2009-06-18 13:03:56 -07:00
2009-09-26 10:17:19 -07:00
2009-04-21 13:41:48 -07:00
2009-03-31 01:14:44 +04:00
2009-04-21 13:41:48 -07:00
2006-10-04 00:31:09 -07:00
2009-01-30 23:59:12 +05:30
2009-06-01 03:07:33 -07:00
2009-01-30 23:59:44 +05:30
2009-01-31 00:00:06 +05:30
2006-04-26 12:56:16 +01:00
2009-09-11 13:21:31 -07:00
2009-03-13 01:32:36 +01:00
2009-08-29 15:53:00 +02:00
2009-03-24 12:15:23 +01:00
2007-09-16 21:13:58 -07:00
2008-04-29 03:22:15 -04:00
2008-04-19 19:10:28 -07:00
2007-05-08 11:15:16 -07:00
2009-11-15 22:23:47 -08:00
2008-04-15 00:30:16 -07:00
2006-04-26 12:56:16 +01:00
2008-04-30 08:29:54 -07:00
2006-09-24 11:13:19 -04:00
2009-01-02 10:19:40 -08:00
2009-03-30 12:43:22 -03:00
2009-03-26 18:14:15 +01:00
2006-11-30 05:24:39 +01:00
2009-09-30 00:32:06 -04:00
2009-09-23 07:39:58 -07:00
2009-03-26 18:14:17 +01:00
2007-04-25 22:25:52 -07:00
2008-12-23 15:21:56 -05:00
2009-01-05 08:40:30 -08:00
2009-01-15 16:39:41 -08:00
2009-03-31 13:05:32 +10:30
2007-10-17 08:42:52 -07:00
2007-08-22 19:52:45 -07:00
2009-10-22 16:39:28 +10:30
2008-04-29 08:06:29 -07:00
2007-11-29 09:24:52 -08:00
2007-07-31 15:39:40 -07:00
2007-04-04 21:12:47 -07:00
2009-06-18 13:03:40 -07:00
2009-10-11 23:21:25 +02:00
2008-04-18 22:16:54 -04:00
2008-10-31 10:01:56 +01:00
2009-09-02 21:29:22 +10:00
2009-06-14 13:50:36 -07:00
2009-09-02 21:29:22 +10:00
2009-09-19 13:13:17 -07:00
2008-05-05 07:13:21 -05:00
2009-01-20 20:52:10 -08:00
2007-05-17 05:23:04 -07:00
2009-09-23 07:39:29 -07:00
2009-08-27 14:29:16 +01:00
2009-04-14 22:05:43 -04:00
2009-04-09 07:49:44 -07:00
2008-04-30 08:29:54 -07:00
2009-03-24 16:38:26 -07:00
2009-09-15 23:51:31 -04:00
2009-09-26 10:17:19 -07:00
2008-07-04 10:40:05 -07:00
2009-09-22 07:17:33 -07:00
2008-07-24 19:11:15 -07:00
2008-04-19 19:44:57 +02:00
2009-09-18 21:22:08 +02:00
2009-09-10 08:33:20 +03:00
2009-06-10 11:48:27 +03:00
2009-09-10 08:33:16 +03:00
2005-04-16 15:20:36 -07:00
2009-02-11 10:18:04 +01:00
2008-09-23 22:01:33 +01:00
2009-04-06 16:06:26 +01:00
2009-04-06 16:06:26 +01:00
2009-06-23 20:21:38 +01:00
2009-01-08 12:38:58 +00:00
2009-06-23 20:21:39 +01:00
2009-07-30 16:03:45 +09:30
2009-07-30 16:03:46 +09:30
2009-10-06 00:26:27 -04:00
2009-09-17 23:23:45 -07:00
2006-06-09 21:53:55 +02:00
2007-07-17 10:23:03 -07:00
2009-09-21 06:27:08 +02:00
2009-06-16 19:47:57 -07:00
2009-09-22 07:17:48 -07:00
2009-06-13 12:28:57 +02:00
2008-09-01 09:47:16 +02:00
2008-02-03 17:45:46 +02:00
2009-08-31 17:30:14 -10:00
2009-08-02 15:41:37 +02:00
2008-02-06 10:41:03 -08:00
2009-04-28 07:37:28 +02:00
2008-02-06 10:41:01 -08:00
2009-08-17 08:37:18 +10:00
2007-07-16 09:05:34 -07:00
2006-06-05 12:29:17 -07:00
2009-09-23 07:39:42 -07:00
2009-06-16 19:47:48 -07:00
2008-12-29 04:59:31 -08:00
2009-02-27 16:07:32 +09:00
2008-06-12 10:47:58 +02:00
2009-03-26 18:14:15 +01:00
2009-06-09 00:53:21 +04:00
2008-04-15 19:35:41 -07:00
2008-03-27 14:51:39 -04:00
2005-04-16 15:20:36 -07:00
2007-11-14 18:45:41 -08:00
2008-04-29 08:06:01 -07:00
2007-05-09 12:30:49 -07:00
2007-10-10 16:54:03 -07:00
2008-11-16 18:59:45 -08:00
2009-08-18 20:13:03 -07:00
2009-09-24 07:20:59 -07:00
2009-09-23 07:39:41 -07:00
2009-04-05 11:04:19 -07:00
2008-07-24 10:47:21 -07:00
2009-09-22 07:17:35 -07:00
2009-01-09 16:54:41 -08:00
2008-04-24 14:07:43 -03:00
2009-06-16 08:40:20 +02:00
2009-01-06 15:59:00 -08:00
2009-04-29 17:32:35 -07:00
2009-01-31 00:00:35 +05:30
2009-09-19 12:50:38 -07:00
2009-05-25 00:55:30 -07:00
2009-07-25 20:18:16 +02:00
2009-09-21 15:14:56 +02:00
2009-09-22 07:17:37 -07:00
2009-09-27 11:39:25 -07:00
2009-09-24 17:20:20 -07:00
2009-05-02 15:36:10 -07:00
2008-06-19 13:33:11 +02:00
2009-04-29 11:33:34 +02:00
2005-04-16 15:20:36 -07:00
2009-09-22 07:17:42 -07:00
2009-09-22 07:17:31 -07:00
2009-09-24 07:21:04 -07:00
2009-07-08 09:31:56 -07:00
2009-09-23 07:39:43 -07:00
2009-09-25 00:32:58 +09:30
2009-01-05 08:40:13 +10:30
2009-10-29 08:56:17 +10:30
2009-06-11 21:36:02 -04:00
2009-04-01 07:38:54 -04:00
2006-11-30 04:40:22 +01:00
2009-09-30 16:12:20 -07:00
2009-09-30 16:12:20 -07:00
2009-01-31 00:01:11 +05:30
2008-06-06 11:29:12 -07:00
2009-03-20 10:48:14 -07:00
2008-04-07 12:15:39 -05:00
2008-02-13 16:21:18 -08:00
2009-04-30 09:01:34 +02:00
2008-09-05 06:33:59 +02:00
2008-01-26 15:04:01 +00:00
2008-01-26 15:03:59 +00:00
2006-12-13 09:05:53 -08:00
2009-09-21 15:14:51 +02:00
2008-04-29 08:06:24 -07:00
2008-04-30 08:29:54 -07:00
2007-03-06 13:26:27 +01:00
2009-03-27 14:44:00 -04:00
2006-12-13 09:05:53 -08:00
2009-02-14 22:58:35 -08:00
2006-01-08 20:14:07 -08:00
2009-03-04 00:03:08 -08:00
2009-07-12 20:11:17 -07:00
2009-02-15 22:43:33 -08:00
2009-09-30 16:12:20 -07:00
2009-10-07 13:53:11 -07:00
2009-02-14 22:58:35 -08:00
2009-09-30 16:12:20 -07:00
2009-09-24 15:44:05 -07:00
2009-03-28 23:55:58 -07:00
2005-04-16 15:20:36 -07:00
2009-06-17 18:02:11 -07:00
2009-06-17 18:02:11 -07:00
2007-05-09 12:30:54 -07:00
2007-07-10 23:40:48 -04:00
2009-09-22 07:54:33 -07:00
2006-03-20 13:44:40 -05:00
2009-08-09 15:06:19 -04:00
2009-09-26 10:17:19 -07:00
2009-01-31 00:02:27 +05:30
2009-04-03 16:42:43 +01:00
2008-12-23 15:21:37 -05:00
2008-07-09 12:09:24 -04:00
2009-06-17 12:24:34 -07:00
2009-06-17 18:02:11 -07:00
2009-03-10 20:33:18 -04:00
2008-02-01 16:42:05 -05:00
2009-11-15 17:17:38 +09:00
2009-08-14 09:13:43 -04:00
2009-08-19 23:08:22 +04:00
2009-06-15 21:44:43 -07:00
2009-08-03 09:56:52 +02:00
2009-01-06 15:59:00 -08:00
2009-08-07 10:39:55 -07:00
2009-09-15 02:37:40 -07:00
2006-07-14 21:53:54 -07:00
2009-03-31 23:00:27 -04:00
2009-02-03 19:33:51 +05:30
2006-04-26 12:56:16 +01:00
2005-06-25 16:25:03 -07:00
2009-01-08 16:25:18 +11:00
2008-05-16 23:22:28 +10:00
2008-12-21 14:21:14 +11:00
2009-01-09 15:49:06 -07:00
2009-07-22 09:27:18 -07:00
2009-05-02 15:36:10 -07:00
2008-07-25 22:34:40 -04:00
2009-04-27 02:53:45 -07:00
2009-09-22 07:17:38 -07:00
2009-07-20 16:33:53 +02:00
2009-01-07 09:59:50 -08:00
2009-09-24 07:20:58 -07:00
2009-04-01 08:59:13 -07:00
2009-09-24 07:53:22 -07:00
2007-10-16 09:43:02 -07:00
2008-05-25 10:55:11 +02:00
2009-08-22 13:07:09 +02:00
2009-04-01 08:59:13 -07:00
2005-04-16 15:20:36 -07:00
2009-04-07 08:44:06 -07:00
2009-06-02 09:53:22 -07:00
2008-10-13 10:10:37 -07:00
2005-05-05 16:36:31 -07:00
2008-02-14 21:13:33 -08:00
2009-09-14 17:39:12 -07:00
2009-11-12 07:25:57 -08:00
2009-09-15 09:50:48 -07:00
2009-06-16 14:29:30 -07:00
2008-07-28 14:56:09 -07:00
2009-09-09 14:19:11 -07:00
2009-03-20 10:47:49 -07:00
2009-02-02 20:15:47 +03:00
2009-01-06 17:10:04 -08:00
2009-09-15 09:39:44 -07:00
2009-08-14 15:00:53 +09:00
2009-09-22 15:42:01 +02:00
2009-10-23 09:35:02 +02:00
2009-07-13 08:01:47 +10:00
2009-03-26 18:14:15 +01:00
2008-09-14 17:24:26 +02:00
2005-04-16 15:20:36 -07:00
2009-01-15 16:39:41 -08:00
2007-02-12 09:48:46 -08:00
2009-09-24 15:38:57 -07:00
2008-01-23 19:33:58 -06:00
2009-05-26 21:16:25 -07:00
2009-01-08 08:31:12 -08:00
2009-01-08 08:31:12 -08:00
2009-02-14 22:58:35 -08:00
2009-05-11 14:13:10 +02:00
2009-01-31 00:03:40 +05:30
2009-02-10 17:18:17 -08:00
2009-04-24 08:54:21 +02:00
2009-07-22 00:28:39 +02:00
2008-12-29 09:39:53 -05:00
2008-08-05 14:33:50 -07:00
2009-08-23 00:04:44 +02:00
2008-07-14 14:30:21 -07:00
2009-09-14 20:27:00 +02:00
2008-03-13 10:09:27 +11:00
2009-09-22 07:17:49 -07:00
2009-09-22 07:17:47 -07:00
2009-10-04 15:05:10 -07:00
2005-06-23 09:45:33 -07:00
2009-06-24 16:58:48 -04:00
2008-12-12 17:00:07 +01:00
2009-07-30 17:49:15 +04:00
2006-10-03 23:01:26 +02:00
2009-03-13 16:09:12 -07:00
2009-03-26 18:14:14 +01:00
2008-06-11 21:00:38 -07:00
2009-06-18 13:04:04 -07:00
2009-07-29 19:10:36 -07:00
2009-10-04 03:23:17 +02:00
2008-05-23 20:32:54 +02:00
2007-10-19 20:35:04 +02:00
2007-10-19 11:53:41 -07:00
2005-04-16 15:20:36 -07:00
2009-09-23 07:39:41 -07:00
2008-10-23 08:55:01 -07:00
2008-08-15 17:15:23 +02:00
2009-06-18 13:03:51 -07:00
2008-07-03 13:25:05 +01:00
2009-03-30 15:22:01 +02:00
2009-06-11 21:36:12 -04:00
2008-09-02 19:21:38 -07:00
2009-03-26 02:18:35 +01:00
2009-09-22 07:17:24 -07:00
2009-01-15 16:39:41 -08:00
2009-06-16 19:47:30 -07:00
2008-04-19 19:10:33 -07:00
2009-09-27 11:39:25 -07:00
2009-01-31 00:05:59 +05:30
2008-11-12 17:17:17 -08:00
2009-06-11 08:51:08 -07:00
2005-04-16 15:20:36 -07:00
2009-01-10 06:04:33 -08:00
2009-09-19 08:53:22 +02:00
2009-04-14 18:41:15 +02:00
2009-10-05 21:02:02 +02:00
2009-10-15 11:17:17 +02:00
2009-08-23 19:13:02 -07:00
2008-08-15 08:35:42 -07:00
2006-12-13 09:05:49 -08:00
2008-01-30 13:31:47 +01:00
2009-06-24 08:17:06 -04:00
2009-06-24 08:17:06 -04:00
2009-06-11 21:35:58 -04:00
2009-06-18 13:03:46 -07:00
2009-05-17 11:45:45 -07:00
2009-09-24 07:20:57 -07:00
2009-10-01 16:11:13 -07:00
2008-10-30 11:38:45 -07:00
2008-05-25 07:09:47 +02:00
2009-09-01 12:48:21 -04:00
2009-09-04 19:42:22 -04:00
2009-01-06 15:59:28 -08:00
2005-11-07 07:53:46 -08:00
2005-11-07 07:53:46 -08:00
2008-05-03 10:01:33 -07:00
2009-09-24 07:53:22 -07:00
2009-05-09 10:49:41 -04:00
2006-07-13 13:21:35 -07:00
2005-04-16 15:20:36 -07:00
2009-04-17 20:37:21 -07:00
2005-04-16 15:20:36 -07:00
2007-05-02 11:56:33 +01:00
2009-04-02 19:04:51 -07:00
2009-04-01 08:59:24 -07:00
2008-10-16 11:21:30 -07:00
2009-09-09 03:42:23 -07:00
2008-02-13 16:21:18 -08:00
2008-04-30 08:29:54 -07:00
2009-09-15 02:44:17 -07:00
2005-04-16 15:20:36 -07:00
2009-07-31 12:28:45 +02:00
2006-04-26 12:56:16 +01:00
2009-09-24 07:53:22 -07:00
2007-11-29 09:24:54 -08:00
2009-06-03 09:14:45 -04:00
2007-10-14 12:41:51 -07:00
2006-10-04 07:55:14 -07:00
2006-01-08 20:14:07 -08:00
2009-04-19 10:47:45 -07:00
2008-11-14 10:39:18 +11:00
2009-09-24 07:21:04 -07:00
2007-10-16 09:43:17 -07:00
2009-03-26 18:14:15 +01:00
2009-09-15 11:37:33 +10:00
2008-07-25 10:53:42 -07:00
2008-07-24 08:31:21 -04:00
2008-07-18 04:07:44 -07:00
2009-09-24 07:47:35 -04:00
2008-04-25 00:25:08 +02:00
2008-02-07 08:42:34 -08:00
2009-09-19 13:13:19 -07:00
2009-09-30 21:46:59 +02:00
2009-04-08 14:33:38 -07:00
2007-02-14 08:09:52 -08:00
2009-06-22 11:32:24 -07:00
2009-05-08 23:22:26 +09:00
2009-09-19 13:13:30 -07:00
2006-04-26 12:56:16 +01:00
2009-07-26 11:17:01 -07:00
2009-08-28 19:57:33 -04:00
2009-08-28 19:57:30 -04:00
2009-08-24 19:52:38 +09:00
2009-05-03 17:57:17 +09:00
2008-02-07 08:42:30 -08:00
2009-09-15 09:50:49 -07:00
2009-04-13 15:04:29 -07:00
2009-09-24 07:21:01 -07:00
2009-01-31 00:06:33 +05:30
2009-10-30 12:25:12 -07:00
2009-06-13 08:58:43 +02:00
2009-06-15 15:50:49 +02:00
2009-08-06 11:36:25 +03:00
2009-11-19 18:10:57 +00:00
2009-09-14 20:19:06 +03:00
2008-03-04 16:35:13 -08:00
2008-07-25 10:53:30 -07:00
2008-04-30 08:29:54 -07:00
2008-04-30 08:29:54 -07:00
2008-07-25 10:53:26 -07:00
2005-04-16 15:20:36 -07:00
2008-09-23 08:09:13 -07:00
2005-04-16 15:20:36 -07:00
2008-09-09 17:41:42 +02:00
2008-10-22 07:00:38 -04:00
2008-06-20 14:05:53 -06:00
2009-09-24 09:34:40 +09:30
2009-04-28 01:53:11 -07:00
2009-04-27 02:45:02 -07:00
2009-10-05 00:24:36 -07:00
2009-02-15 22:43:33 -08:00
2005-04-16 15:20:36 -07:00
2008-06-17 16:19:24 -07:00
2007-04-28 22:06:01 -04:00
2009-03-27 12:18:56 -04:00
2005-04-16 15:20:36 -07:00
2009-01-31 00:07:00 +05:30
2009-02-10 00:02:54 +01:00
2009-08-31 18:08:51 +02:00
2008-02-03 17:45:46 +02:00
2009-06-10 11:48:14 +02:00
2009-08-31 18:08:51 +02:00
2009-05-12 11:11:48 +02:00
2006-10-04 07:55:30 -07:00
2008-05-26 16:15:32 +02:00
2009-06-12 23:04:12 +02:00
2008-07-20 17:12:37 -07:00
2006-12-07 02:14:08 +01:00
2007-05-08 11:15:18 -07:00
2006-03-26 08:57:00 -08:00
2006-10-01 00:39:18 -07:00
2009-01-05 08:40:14 +10:30
2008-10-03 11:46:14 -05:00
2009-10-29 07:39:25 -07:00
2009-04-10 15:48:52 +02:00
2005-11-07 07:53:28 -08:00
2009-03-26 18:14:14 +01:00
2009-11-17 17:40:33 -08:00
2008-04-30 08:29:54 -07:00
2009-01-14 19:56:50 -08:00
2009-09-24 07:53:22 -07:00
2009-09-16 11:50:05 +02:00
2009-07-28 14:19:20 +09:00
2009-04-02 19:05:01 -07:00
2005-04-16 15:20:36 -07:00
2009-09-23 09:21:05 -07:00
2009-09-24 07:21:04 -07:00
2008-07-21 21:55:02 -07:00
2008-10-20 08:52:42 -07:00
2009-07-06 13:57:03 -07:00
2008-04-29 08:06:27 -07:00
2008-07-27 16:12:28 -07:00
2008-10-16 11:21:32 -07:00
2009-09-18 09:48:52 -07:00
2009-01-31 00:08:18 +05:30
2009-09-01 18:25:07 -07:00
2007-02-09 16:23:15 +00:00
2009-09-02 01:03:43 -07:00
2008-11-11 09:30:23 -08:00
2008-10-13 09:51:40 -07:00
2005-06-23 20:59:16 -07:00
2008-07-08 02:38:40 -07:00
2006-12-11 14:34:42 -08:00
2009-03-27 16:58:22 -04:00
2009-04-06 11:14:03 +02:00
2008-12-30 09:05:12 +10:30
2009-06-13 12:00:26 +02:00
2008-03-10 18:01:18 -07:00
2009-09-24 07:20:56 -07:00
2009-02-15 22:43:32 -08:00
2009-08-30 22:26:34 +02:00
2009-02-18 15:37:53 -08:00
2009-03-27 12:59:54 +08:00
2009-03-26 18:14:14 +01:00
2009-06-16 19:47:27 -07:00
2006-03-31 12:18:56 -08:00
2008-05-21 14:52:30 -07:00
2009-07-05 19:16:11 -07:00
2009-10-14 15:02:34 +02:00
2007-02-12 09:48:42 -08:00
2009-02-03 11:23:12 +11:00
2009-02-26 18:44:06 +01:00
2009-10-24 11:07:51 +02:00
2009-09-24 07:21:00 -07:00
2009-09-24 07:20:57 -07:00
2009-09-15 09:50:47 -07:00
2006-09-28 17:53:59 -07:00
2006-10-01 00:39:29 -07:00
2009-09-27 13:35:16 -07:00
2006-06-28 14:59:05 -07:00
2009-08-04 13:46:30 -07:00
2009-09-19 13:13:35 -07:00
2009-06-19 08:08:50 +02:00
2009-03-02 11:06:49 +01:00
2009-09-23 07:39:48 -07:00
2008-04-17 14:22:23 +02:00
2008-06-16 17:14:11 -07:00
2007-10-19 11:53:42 -07:00
2009-03-24 16:38:24 -07:00
2009-07-29 19:10:39 -07:00
2009-06-12 18:01:45 +02:00
2005-04-16 15:20:36 -07:00
2006-10-02 07:57:23 -07:00
2009-03-24 16:20:34 -07:00
2009-09-23 06:46:23 -07:00
2009-09-23 06:46:39 -07:00
2009-02-27 16:26:21 -08:00
2005-04-16 15:20:36 -07:00
2009-03-26 18:14:14 +01:00
2005-04-16 15:20:36 -07:00
2009-09-23 18:13:10 -07:00
2009-08-26 12:39:29 +01:00
2008-05-09 07:45:18 -07:00
2007-12-26 19:36:35 -08:00
2005-04-16 15:20:36 -07:00
2009-09-24 09:57:08 -07:00
2005-04-16 15:20:36 -07:00
2007-10-12 14:51:04 -07:00
2009-09-19 00:18:36 -03:00
2009-03-30 12:43:15 -03:00
2008-07-26 13:25:25 -03:00
2009-10-22 16:39:28 +10:30
2009-10-22 16:39:28 +10:30
2009-10-22 16:39:30 +10:30
2009-09-23 07:39:29 -07:00
2009-10-22 16:39:28 +10:30
2009-09-23 22:26:32 +09:30
2009-10-22 16:39:28 +10:30
2009-06-12 22:16:37 +09:30
2009-07-30 16:03:46 +09:30
2009-10-22 16:39:28 +10:30
2009-09-23 22:26:31 +09:30
2009-06-16 19:47:52 -07:00
2009-08-14 15:00:52 +09:00
2009-09-22 07:17:30 -07:00
2006-04-26 12:56:16 +01:00
2009-09-19 13:13:25 -07:00
2009-09-19 13:13:26 -07:00
2009-06-18 13:03:58 -07:00
2009-09-15 16:51:30 +02:00
2008-05-29 01:43:48 -07:00
2006-06-20 19:00:30 +02:00
2009-05-28 18:02:32 -07:00
2009-07-15 08:53:39 -07:00
2008-11-19 14:47:04 +00:00
2009-09-04 20:20:58 +04:00
2009-09-23 03:49:27 +04:00
2009-10-14 09:16:42 -07:00
2009-09-24 07:21:04 -07:00
2006-03-22 00:01:31 -08:00
2009-09-10 10:11:22 +10:00
2009-03-26 18:14:14 +01:00
2007-10-12 14:05:17 +10:00
2005-04-16 15:20:36 -07:00
2006-06-22 15:05:58 -07:00
2007-10-10 16:53:57 -07:00
2005-04-16 15:20:36 -07:00
2006-06-23 07:43:00 -07:00
2006-06-22 15:05:58 -07:00