Debian Bug report logs - #605009
serious performance regression with ext4

Package: dpkg; Maintainer for dpkg is Dpkg Developers <debian-dpkg@lists.debian.org>; Source for dpkg is src:dpkg (PTS, buildd, popcon).

Reported by: Michael Biebl <biebl@debian.org>

Date: Fri, 26 Nov 2010 09:36:01 UTC

Severity: important

Found in version dpkg/1.15.8.6

Fixed in version dpkg/1.15.8.7

Done: Guillem Jover <guillem@debian.org>

Bug is archived. No further changes may be made.

Toggle useless messages

View this report as an mbox folder, status mbox, maintainer mbox

Report forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 09:36:04 GMT) (full text, mbox, link).

Acknowledgement sent to Michael Biebl <biebl@debian.org>:
New Bug report received and forwarded. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 09:36:04 GMT) (full text, mbox, link).

Message #5 received at submit@bugs.debian.org (full text, mbox, reply):

Package: dpkg
Version: 1.15.8.6
Severity: important

Hi!

I'm using ext4 with the default mount option and the fs was created with
the default settings from /etc/mke2fs.conf.
Since the latest upgrade, performance suffered badly.

E.g. installing vim-runtime took ~8 secs with 1.15.8.5, now it takes
~44secs.

It was suggested that I use the nodelalloc mount option for ext4.
But that not only takes away some of the benefits of ext4, it also
requires explicit configuration.

dpkg should work properly(with good performance) out-of-the-box on ext4.

Cheers,
Michael

-- System Information:
Debian Release: squeeze/sid
  APT prefers unstable
  APT policy: (500, 'unstable'), (200, 'experimental')
Architecture: i386 (i686)

Kernel: Linux 2.6.32-5-686 (SMP w/1 CPU core)
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages dpkg depends on:
ii  coreutils               8.5-1            GNU core utilities
ii  libbz2-1.0              1.0.5-6          high-quality block-sorting file co
ii  libc6                   2.11.2-7         Embedded GNU C Library: Shared lib
ii  libselinux1             2.0.96-1         SELinux runtime shared libraries
ii  xz-utils                5.0.0-2          XZ-format compression utilities
ii  zlib1g                  1:1.2.3.4.dfsg-3 compression library - runtime

dpkg recommends no packages.

Versions of packages dpkg suggests:
ii  apt                           0.8.9      Advanced front-end for dpkg

-- no debconf information

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 14:57:06 GMT) (full text, mbox, link).

Acknowledgement sent to 605009@bugs.debian.org, debian-devel@lists.debian.org:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 14:57:06 GMT) (full text, mbox, link).

Message #10 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi,

adding debian-devel, debian-boot, debian-kernel and Theodore Y. Ts'o in CC
because I'm fed up with this problem. Sorry for the massive crosspost, you
might want to follow up only on -devel and on the bug report.

Some clones/reassign should probably result from this discussion anyway.

On Fri, 26 Nov 2010, Michael Biebl wrote:
> I'm using ext4 with the default mount option and the fs was created with
> the default settings from /etc/mke2fs.conf.
> Since the latest upgrade, performance suffered badly.
> 
> E.g. installing vim-runtime took ~8 secs with 1.15.8.5, now it takes
> ~44secs.
>
> It was suggested that I use the nodelalloc mount option for ext4.
> But that not only takes away some of the benefits of ext4, it also
> requires explicit configuration.
> 
> dpkg should work properly(with good performance) out-of-the-box on ext4.

There's currently no way we're going to fix this.

It all started with report of corrupted (zero-length) files on ext4/ubifs
(see http://bugs.debian.org/430958). We did the right thing to fix this
which is to call fsync() on the fly on each file that dpkg unpacks. That
was introduced in dpkg 1.15.6. (Ubuntu had more than 80 bug reports
related to this and Debian very few because we don't use ext4 by default)

That was disastrous in terms of performance, so we then grouped the
fsync()+rename() calls at the end of the unpack phase before updating the
dpkg database. That can be witnessed in dpkg 1.15.7.

That was ok everywhere except on ext4. For some unknown reasons, with the
nodelalloc option the performance are again reasonable (but we discovered
that only recently). Ubuntu is using ext4 by default and this performance
was not acceptable and we tried to work-around the problem by replacing
all the fsync() by a single sync() on Linux only (because Linux's sync()
is synchronous while it's not necessarily the case on other systems). This
was enabled in dpkg 1.15.7.2 in response to http://bugs.debian.org/578635.

Now using sync() appears to be a bad choice since it resulted in RC bugs
like http://bugs.debian.org/595927 and http://bugs.debian.org/600075 and
was an annoyance for people using dpkg on tmpfs to get super-high speed
(http://bugs.debian.org/588339).

So we reverted that hackish work-around of using sync() and we're back
to the situation where dpkg is very slow on ext4. But we have added a new
option --force-unsafe-io that can be used when data safety is not
important (in a temporary chroot, during initial installation, etc.) as
requested in http://bugs.debian.org/584254.

Where does that leaves us? Here are various options to consider (they are
not necessarily mutually exclusive):

----
1/ This problem and the known work-arounds should be documented in the
release notes.

2/ d-i should be modified to use --force-unsafe-io if it detects a dpkg
version >= 1.15.8.6.

3/ d-i might want to setup the nodelalloc option by default when a
partition is formatted with ext4 and mounted as / or /usr.

4/ Theodore might want to find out why ext4 is behaving so badly under
this usage pattern while ext3 doesn't... i.e. fix
https://bugzilla.kernel.org/show_bug.cgi?id=15910
Or at least suggest another usage pattern which result in the same
guaranty for dpkg but without the poor performance (and sync() is not the
right answer as experience showed us).

Note that doing N x fsync() followed by N x rename() is not giving us any
better result that doing N x (fsync()+rename()).

Note that calling posix_fallocate() before writing the files that are
fsynced() did not help either (Mike Hommey thought it could be a way to
emulate nodelalloc at the dpkg level only but apparently not).

What has not been tried: reordering the fsync() based on FIEMAP
information. Or using fdatasync() instead of fsync() and calling fsync()
on directories.

5/ maybe the debian-kernel team wants to discuss the issue on LKML. Both
for the bad ext4 performances (see above) and the (incorrect?) behaviour
of sync() which is never finishing under important I/O loads (cf
https://bugzilla.kernel.org/show_bug.cgi?id=18632).
----

But right now from the point of view of dpkg maintainers, this bug is a
"wontfix" at our level.

Just to sum up what dpkg --unpack does in 1.15.8.6:
1/ set the package status as half-installed/reinst-required
2/ extract all the new files as *.dpkg-new
3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
   rename(foo.dpkg-new, foo)
4/ set the package status as unpacked

Note that the directories are not fsynced() because we mainly don't care
if the rename is recorded or not, as long as the installed file always has
the content of either the old or the new file. This could be fixed but is
unlikely to help in getting better performances I guess.

The only thing we want to achieve is that:
- when the package is set to unpacked status, all changes have been
  committed
- when the process is abruptly interrupted, we don't leave corrupted files
  around (like unwanted empty files)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 15:24:03 GMT) (full text, mbox, link).

Acknowledgement sent to Mathias Behrle <mathiasb@mbsolutions.selfip.biz>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 15:24:03 GMT) (full text, mbox, link).

Message #15 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 15:53:27 +0100):

> That was ok everywhere except on ext4.

JFTR: I am experiencing those problems as well on XFS.

Cheers,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Acknowledgement sent to Aron Xu <happyaron.xu@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 15:30:04 GMT) (full text, mbox, link).

Message #20 received at 605009@bugs.debian.org (full text, mbox, reply):

Low performance with Btrfs as well, :(

(Even Btrfs is not supported in squeeze, I think this could help on
digging whether it is a more generic problem than EXT4 only.)

-- 
Regards,
Aron Xu

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 15:36:03 GMT) (full text, mbox, link).

Message #25 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi,

On Fri, 26 Nov 2010, Mathias Behrle wrote:
> * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
>   Nov 2010 15:53:27 +0100):
> 
> > That was ok everywhere except on ext4.
> 
> JFTR: I am experiencing those problems as well on XFS.

Can you give us figures to quantify the slowdown that you experience?
Please compare dpkg 1.15.8.5 and 1.15.8.6.

(I suppose that's the problem you're referring to)

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 16:39:05 GMT) (full text, mbox, link).

Message #30 received at 605009@bugs.debian.org (full text, mbox, reply):

I'm reinstalling vim-runtime using the following command to do this benchmark:
$ time sudo dpkg -i vim-runtime_7.2.330-1ubuntu3_all.deb

Results:
1.15.8.5: 3.02s user 1.96s system 16% cpu 30.364 total
1.15.8.6: 3.48s user 2.86s system 9% cpu 1:04.43 total

My system infomation:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 10.04.1 LTS
Release:        10.04
Codename:       lucid

$ uname -r
Linux aron-desktop 2.6.37-5-generic #14~lucid1-Ubuntu SMP Sat Nov 20
06:26:57 UTC 2010 i686 GNU/Linux

/ is btrfs mounted with (rw,compress)

The hard drive is an old PATA one so the performance is not comparable
to the mainstream SATA ones.


-- 
Regards,
Aron Xu

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 21:15:13 GMT) (full text, mbox, link).

Message #35 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 16:33:06 +0100):

> On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > 26 Nov 2010 15:53:27 +0100):
> > 
> > > That was ok everywhere except on ext4.
> > 
> > JFTR: I am experiencing those problems as well on XFS.
> 
> Can you give us figures to quantify the slowdown that you experience?
> Please compare dpkg 1.15.8.5 and 1.15.8.6.
> 
> (I suppose that's the problem you're referring to)

I have to correct my previous mail.

I am experiencing since some time a substantial slowdown on my XFS file
systems (could also be related to upgrading to kernel 2.6.32). This slowdown is
also quite perceivable on system upgrades with apt-get/aptitude. Now for the
first time Bug#605009 came to my knowledge and I guessed it could also be a
cause for the meanwhile rather painful upgrades on my desktop machine.

I just upgraded my testing machine to dpkg 1.15.8.6. and tested with
'aptitude -o DPkg::options::="--force-unsafe-io"' and see that my guess
was at least partly wrong. Unpacking of the debs still causes very heavy load
on the harddisk while final install seems to be faster.

So the problems I am facing with XFS are probably at least not closely related
to #605009.

Sorry for the noise,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Fri, 26 Nov 2010 22:36:06 GMT) (full text, mbox, link).

Acknowledgement sent to Ted Ts'o <tytso@mit.edu>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Fri, 26 Nov 2010 22:36:06 GMT) (full text, mbox, link).

Message #40 received at 605009@bugs.debian.org (full text, mbox, reply):

On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote:
> Just to sum up what dpkg --unpack does in 1.15.8.6:
> 1/ set the package status as half-installed/reinst-required
> 2/ extract all the new files as *.dpkg-new
> 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
>    rename(foo.dpkg-new, foo)

What are you doing?

1) Suppose package contains files "a", "b", and "c".  Which are you
doing?

a)  extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a);
    extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b);
    extract c.dpkg-new ; fsync(c.dpkg-new); rename(c.dpkg-new, c);

or

b)  extract a.dpkg-new ; fsync(a.dpkg-new);
    extract b.dpkg-new ; fsync(b.dpkg-new);
    extract c.dpkg-new ; fsync(c.dpkg-new);
    rename(a.dpkg-new, a);
    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);

or

c)  extract(a.dpkg-new);
    extract(b.dpkg-new);
    extract(c.dpkg-new);
    fsync(a.dpkg-new);
    fsync(b.dpkg-new);
    fsync(c.dpkg-new);
    rename(a.dpkg-new, a);
    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);


(c) will perform the best for most file systems, including ext4.  As a
further optimization, if "b" and "c" does not exist, of course it
would be better to extract into "b" and "c" directly, and skip the
rename, i.e.:

d)  extract(a.dpkg-new);
    extract(b);			# assuming the file "b" does not yet exist
    extract(c);			# assuming the file "c" does not yet exist
    fsync(a.dpkg-new);
    fsync(b);
    fsync(c);
    rename(a.dpkg-new, a);

... and then set the package status as unpacked.

I am guessing you are doing (a) today --- am I right?  (c) or (d)
would be best.

						- Ted

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Sat, 27 Nov 2010 08:03:03 GMT) (full text, mbox, link).

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 08:03:03 GMT) (full text, mbox, link).

Message #45 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi Ted,

Ted Ts'o wrote:

> 1) Suppose package contains files "a", "b", and "c".  Which are you
> doing?
> 
> a)  extract a.dpkg-new ; fsync(a.dpkg-new); rename(a.dpkg-new, a);
>     extract b.dpkg-new ; fsync(b.dpkg-new); rename(b.dpkg-new, b);
>     extract c.dpkg-new ; fsync(c.dpkg-new); rename(c.dpkg-new, c);
> 
> or
> 
> b)  extract a.dpkg-new ; fsync(a.dpkg-new);
>     extract b.dpkg-new ; fsync(b.dpkg-new);
>     extract c.dpkg-new ; fsync(c.dpkg-new);
>     rename(a.dpkg-new, a);
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);
> 
> or
> 
> c)  extract(a.dpkg-new);
>     extract(b.dpkg-new);
>     extract(c.dpkg-new);
>     fsync(a.dpkg-new);
>     fsync(b.dpkg-new);
>     fsync(c.dpkg-new);
>     rename(a.dpkg-new, a);
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);
> 
> 
> (c) will perform the best for most file systems, including ext4.
[...]
> I am guessing you are doing (a) today --- am I right?  (c) or (d)
> would be best.

We are doing (c) today.

Regards,
Jonathan

Acknowledgement sent to Mathias Behrle <mathiasb@mbsolutions.selfip.biz>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 08:42:03 GMT) (full text, mbox, link).

Message #50 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

* Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 16:33:06 +0100):

> On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > 26 Nov 2010 15:53:27 +0100):
> > 
> > > That was ok everywhere except on ext4.
> > 
> > JFTR: I am experiencing those problems as well on XFS.
> 
> Can you give us figures to quantify the slowdown that you experience?
> Please compare dpkg 1.15.8.5 and 1.15.8.6.
> 
> (I suppose that's the problem you're referring to)

I have to correct my previous mail.

I am experiencing since some time a substantial slowdown on my XFS file
systems (could also be related to upgrading to kernel 2.6.32). This slowdown is
also quite perceivable on system upgrades with apt-get/aptitude. Now for the
first time Bug#605009 came to my knowledge and I guessed it could also be a
cause for the meanwhile rather painful upgrades on my desktop machine.

I just upgraded my testing machine to dpkg 1.15.8.6. and tested with
'aptitude -o DPkg::options::="--force-unsafe-io"' and see that my guess
was at least partly wrong. Unpacking of the debs still causes very heavy load
on the harddisk while final install seems to be faster.

So the problems I am facing with XFS are probably at least not closely related
to #605009.

Sorry for the noise,

-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 08:57:03 GMT) (full text, mbox, link).

Message #55 received at 605009@bugs.debian.org (full text, mbox, reply):

On Sat, 27 Nov 2010, Jonathan Nieder wrote:
> > c)  extract(a.dpkg-new);
> >     extract(b.dpkg-new);
> >     extract(c.dpkg-new);
> >     fsync(a.dpkg-new);
> >     fsync(b.dpkg-new);
> >     fsync(c.dpkg-new);
> >     rename(a.dpkg-new, a);
> >     rename(b.dpkg-new, b);
> >     rename(c.dpkg-new, c);
> > 
> > 
> > (c) will perform the best for most file systems, including ext4.
> [...]
> > I am guessing you are doing (a) today --- am I right?  (c) or (d)
> > would be best.
> 
> We are doing (c) today.

Actually we are doing this:
    extract(a.dpkg-new);
    extract(b.dpkg-new);
    extract(c.dpkg-new);
    fsync(a.dpkg-new);
    rename(a.dpkg-new, a);
    fsync(b.dpkg-new);
    rename(b.dpkg-new, b);
    fsync(c.dpkg-new);
    rename(c.dpkg-new, c);

But as I said, I tried (c) and it's not performing noticably better than
the above.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Sat, 27 Nov 2010 09:45:06 GMT) (full text, mbox, link).

Acknowledgement sent to Guillem Jover <guillem@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 09:45:06 GMT) (full text, mbox, link).

Message #60 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi!

On Fri, 2010-11-26 at 16:52:54 -0500, Ted Ts'o wrote:
> On Fri, Nov 26, 2010 at 03:53:27PM +0100, Raphael Hertzog wrote:
> > Just to sum up what dpkg --unpack does in 1.15.8.6:
> > 1/ set the package status as half-installed/reinst-required
> > 2/ extract all the new files as *.dpkg-new
> > 3/ for all the unpacked files: fsync(foo.dpkg-new) followed by
> >    rename(foo.dpkg-new, foo)

> What are you doing?

We already had this conversation some time ago in
<https://bugzilla.kernel.org/show_bug.cgi?id=15910>.

> 1) Suppose package contains files "a", "b", and "c".  Which are you
> doing?

[...]

Anyway, dpkg is currently doing the variation on c that Raphaël
posted, including making backups so that it can rollback the entire
package if something goes wrong.

> (c) will perform the best for most file systems, including ext4.

Well it does not, and that's also what was mentioned in the bug
report. Something we've found out recently (as Raphaël mentioned too)
is that with nodelalloc the performance issues *and* the zero-length
issues disappear, which seems like a clear win to me, and so IMO
changing the default file system mount option to nodelalloc seems to
be the way to go.

> As a further optimization, if "b" and "c" does not exist, of course
> it would be better to extract into "b" and "c" directly, and skip the
> rename, i.e.:

> d)  extract(a.dpkg-new);
>     extract(b);			# assuming the file "b" does not yet exist
>     extract(c);			# assuming the file "c" does not yet exist
>     fsync(a.dpkg-new);
>     fsync(b);
>     fsync(c);
>     rename(a.dpkg-new, a);
> 
> ... and then set the package status as unpacked.

That would make possible for partial files to appear on their final path
and thus available for external use while they are being extracted. I
don't think that's a good idea.

regards,
guillem

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Sat, 27 Nov 2010 10:48:03 GMT) (full text, mbox, link).

Acknowledgement sent to Mathias Behrle <mathiasb@mbsolutions.selfip.biz>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 10:48:03 GMT) (full text, mbox, link).

Message #65 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

* Betr.: " Fw: Bug#605009: serious performance regression with ext4" (Fri, 26
  Nov 2010 22:15:06 +0100):

> * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri, 26
>   Nov 2010 16:33:06 +0100):
> 
> > On Fri, 26 Nov 2010, Mathias Behrle wrote:
> > > * Betr.: " Re: Bug#605009: serious performance regression with ext4" (Fri,
> > > 26 Nov 2010 15:53:27 +0100):
> > > 
> > > > That was ok everywhere except on ext4.
> > > 
> > > JFTR: I am experiencing those problems as well on XFS.
> > 
> > Can you give us figures to quantify the slowdown that you experience?
> > Please compare dpkg 1.15.8.5 and 1.15.8.6.

Finally some real numbers:

command:
time dpkg -i openjdk-6-doc_6b18-1.8.2-4_all.deb


1.15.8.6:
=========
dpkg 1.15.8.6 with force-unsafe-io (1. install):
real    8m12.093s
user    0m1.737s
sys     0m2.290s

dpkg 1.15.8.6 with force-unsafe-io (2. run = replace):
real    13m57.933s
user    0m1.657s
sys     0m3.596s

dpkg 1.15.8.6 without force-unsafe-io (3. run = replace):
real    19m50.008s
user    0m1.940s
sys     0m4.240s


1.15.8.5:
=========
dpkg 1.15.8.5 (4. run = replace):
real    15m13.063s
user    0m1.723s
sys     0m6.453s


Seems, that 1.15.8.5 performs much better than 1.15.8.6 (without
force-unsafe-io).

Filesystem:
/ xfs     defaults,noatime,nodiratime,logbufs=8
on an lvm volume


Cheers,
-- 

    Mathias Behrle
    MBSolutions
    Gilgenmatten 10 A
    D-79114 Freiburg

    Tel: +49(761)471023
    Fax: +49(761)4770816
    http://m9s.biz
    UStIdNr: DE 142009020
    PGP/GnuPG key availabable from any keyserver, ID: 0x8405BBF6

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Sat, 27 Nov 2010 14:57:05 GMT) (full text, mbox, link).

Acknowledgement sent to Olaf van der Spek <olafvdspek@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 27 Nov 2010 14:57:05 GMT) (full text, mbox, link).

Message #70 received at 605009@bugs.debian.org (full text, mbox, reply):

On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o <tytso@mit.edu> wrote:
> I am guessing you are doing (a) today --- am I right?  (c) or (d)
> would be best.

Are there any plans to provide an API for atomic (non-durable) file
updates, not involving fsync?
Would be simpler (for apps), faster and more general (because it makes
less assumptions).

Olaf

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 04:15:03 GMT) (full text, mbox, link).

Acknowledgement sent to Ted Ts'o <tytso@mit.edu>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 04:15:03 GMT) (full text, mbox, link).

Message #75 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

I did some experimenting, and I figured out what was going on.  You're
right, (c) doesn't quite work, because delayed allocation meant that
the writeout didn't take place until the fsync() for each file
happened.  I didn't see this at first; my apologies.

However, this *does* work:

    extract(a);
    sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WRITE); 
    extract(b.dpkg-new);
    sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WRITE); 
    extract(c.dpkg-new);
    sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WRITE); 

    sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
    sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
    sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 

    fdatasync(a);
    fdatasync(b.dpkg-new);
    fdatasync(c.dpkg-new);

    rename(b.dpkg-new, b);
    rename(c.dpkg-new, c);

This assumes that files b and c existed beforehand, but a is a new file.

What's going on here?  sync_file_range() is a Linux specific system
call that has been around for a while.  It allows program to control
when writeback happens in a very low-level fashion.  The first set of
sync_file_range() system calls causes the system to start writing back
each file once it has finished being extracted.  It doesn't actually
wait for the write to finish; it just starts the writeback.

The second series of sync_file_range() calls, with the operation
SYNC_FILE_RANGE_WAIT_BEFORE, will block until the previously initiated
writeback has completed.  This basically ensures that the delayed
allocation has been resolved; that is, the data blocks have been
allocated and written, and the inode updated (in memory), but not
necessarily pushed out to disk.

The fdatasync() call will actually force the inode to disk.  In the
case of the ext4 file system, the first fdatasync() will actually push
all of the inodes to disk, and all of the subsequent fdatasync() calls
are in fact no-ops (assuming that files 'a', 'b', and 'c' are all on
the same file system).  But what it means is that it minimizes the
number of (heavyweight) jbd2 commits to a minimum.

It uses a linux-specific system call --- sync_file_range --- but the
result should be faster performance across the board for all file
systems.  So I don't consider this an ext4-specific hack, although it
probably does makes things faster for ext4 more than any other file
system.

I've attached the program I used to test and prove this mechanism, as
well as the kernel tracepoint script I used to debug why (c) wasn't
working, which might be of interest to folks on debian-kernel.
Basically it's a demonstration of how cool ftrace is.  :-)

But using this program on a file system composed of a 5400rpm laptop
drive running LVM and LUKS, I get:

mass-sync-tester -d:	dpkg current: time:  0.83/ 0.01/ 0.00

versus

mass-sync-tester -n:	dpkg fixed: time:  0.07/ 0.00/ 0.01

	       	 	       	      	   - Ted

[mass-sync-tester.c (text/x-csrc, attachment)]

[do-sync-test (text/plain, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 06:21:03 GMT) (full text, mbox, link).

Acknowledgement sent to Guillem Jover <guillem@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 06:21:03 GMT) (full text, mbox, link).

Message #80 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

Hi Ted!

On Sun, 2010-11-28 at 23:11:52 -0500, Ted Ts'o wrote:
> I did some experimenting, and I figured out what was going on.  You're
> right, (c) doesn't quite work, because delayed allocation meant that
> the writeout didn't take place until the fsync() for each file
> happened.  I didn't see this at first; my apologies.

Thanks for the analysis.

> However, this *does* work:
> 
>     extract(a);
>     sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WRITE); 
>     extract(b.dpkg-new);
>     sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WRITE); 
>     extract(c.dpkg-new);
>     sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WRITE); 

The man page and the kernel sources seem to indicate this might block
depending on the size of the request queue?

>     sync_file_range(fd.a, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
>     sync_file_range(fd.b, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 
>     sync_file_range(fd.c, 0, 0, SYNC_FILE_RANGE_WAIT_BEFORE); 

>     fdatasync(a);
>     fdatasync(b.dpkg-new);
>     fdatasync(c.dpkg-new);
> 
>     rename(b.dpkg-new, b);
>     rename(c.dpkg-new, c);

> What's going on here?  sync_file_range() is a Linux specific system
> call that has been around for a while.  It allows program to control
> when writeback happens in a very low-level fashion.  The first set of
> sync_file_range() system calls causes the system to start writing back
> each file once it has finished being extracted.  It doesn't actually
> wait for the write to finish; it just starts the writeback.

Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
instead, skimming over the kernel source seems to indicate it might
end up doing more or less the same thing but in a portable way?

Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
against dpkg?

thanks,
guillem

[dpkg_fadvise.patch (text/x-diff, attachment)]

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 06:27:03 GMT) (full text, mbox, link).

Message #85 received at 605009@bugs.debian.org (full text, mbox, reply):

(pruned cc list)

Guillem Jover wrote:

> Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> instead, skimming over the kernel source seems to indicate it might
> end up doing more or less the same thing but in a portable way?

Probably a silly question, but what does "The specified data will not
be accessed in the near future" have to do with preventing delayed
allocation?

Put another way: if this works now, is it likely to continue to work?

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 07:03:05 GMT) (full text, mbox, link).

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 07:03:05 GMT) (full text, mbox, link).

Message #90 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi Guillem,

Guillem Jover wrote:

> Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> instead, skimming over the kernel source seems to indicate it might
> end up doing more or less the same thing but in a portable way?
> 
> Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
> against dpkg?

ext4: yes, this brings the running time down from 34sec to 10.5sec,
just like sync_file_range with SYNC_FILE_RANGE_WRITE does.

Regards,
Jonathan

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 13:18:10 GMT) (full text, mbox, link).

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 13:18:10 GMT) (full text, mbox, link).

Message #95 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, 29 Nov 2010, Jonathan Nieder wrote:
> Guillem Jover wrote:
> > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> > instead, skimming over the kernel source seems to indicate it might
> > end up doing more or less the same thing but in a portable way?
> 
> Probably a silly question, but what does "The specified data will not
> be accessed in the near future" have to do with preventing delayed
> allocation?

It means we don't need to keep it in RAM since we're not going to
read/modifiy it again in the near future. Thus the writeback can be
started right now since delaying it will not save us anything.

At least that's the way I understand the situation.

> Put another way: if this works now, is it likely to continue to work?

Well, it will always work (the code is unlikely to introduce failures),
but the resulting behaviour is entirely up to the kernel to decide. So
there's no guaranty that the optimization will last.

On the other hand, the whole point of posix_fadvise() is to give hints to
the kernel so that he can decide on the best course of action. So I hope
the interpretation above is one the main motivation behind that hint.

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Acknowledgement sent to Ian Jackson <ijackson@chiark.greenend.org.uk>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 13:24:06 GMT) (full text, mbox, link).

Message #100 received at 605009@bugs.debian.org (full text, mbox, reply):

Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
> On Fri, Nov 26, 2010 at 10:52 PM, Ted Ts'o <tytso@mit.edu> wrote:
> > I am guessing you are doing (a) today --- am I right? =C2=A0(c) or (d)
> > would be best.
> 
> Are there any plans to provide an API for atomic (non-durable) file
> updates, not involving fsync?

Yes.  Such an API has already been defined by POSIX, SuSv3, et al.
It's called "rename".

dpkg used it correctly before filesystem bugs (and associated
intransigence by fs maintainers[1]) forced the current dpkg
maintainers to add a whole pile of pointless fsyncs.

dpkg does _not_ need durable updates; it just needs atomicity and
correctness.  If after a crash the system is rewound to some earlier
state that's absolutely fine.

I'm told that the Linux fs maintainers have now accepted that 
   open("file.new",O_CREAT);
   write();
   close();
   rename("file.new","file");
should not result, even after a crash, in "file" containing garbage.

If this is the case then all the fsyncs can be taken out again.

Ian.

[1] The view that fsync is the new IAC DONT RANDOMLY-LOSE.

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 13:33:05 GMT) (full text, mbox, link).

Message #105 received at 605009@bugs.debian.org (full text, mbox, reply):

clone 605009 -1
reassign -1 base-installer
retitle -1 d-i should use dpkg --force-unsafe-io to optimize installation time
severity -1 wishlist
thanks

I put severity wishlist but it would still be nice if this could be
implemented for squeeze. It should not be a major change.

On Fri, 26 Nov 2010, Raphael Hertzog wrote:
> 2/ d-i should be modified to use --force-unsafe-io if it detects a dpkg
> version >= 1.15.8.6.

dpkg has become slower due to the usage of fsync() to ensure data
consistency in case of crashes/power failures.

In the context of d-i, a crash means restarting the installation 
so it's not a big deal to not use fsync() and on the opposite, it's rather
important that it goes fast...

So it would be nice if base-installer could auto-detect whether dpkg >=
1.15.8.6 is used and if yes, then it could add a temporary configuration
file in /etc/dpkg.cfg.d/ with "force-unsafe-io" inside.

That file should obviously be dropped at the end of the installation.

Thanks.
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Bug 605009 cloned as bug 605384. Request was from Raphael Hertzog <hertzog@debian.org> to control@bugs.debian.org. (Mon, 29 Nov 2010 13:33:05 GMT) (full text, mbox, link).

Acknowledgement sent to Michael Biebl <biebl@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 13:45:07 GMT) (full text, mbox, link).

Message #112 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

On 29.11.2010 07:18, Guillem Jover wrote:

> 
> Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
> against dpkg?

I'm using ext4 (as already mentioned), my small benchmark is (re)installing
vim-runtime using dpkg -i

1.15.8.5:
real    0m9.259s
user    0m4.212s
sys     0m0.752s

1.15.8.6:
real    0m41.766s
user    0m4.248s
sys     0m1.028s

1.15.8.6 (force-unsafe-io):
real    0m8.404s
user    0m4.244s
sys     0m0.752s

1.15.8.6 (+patch posix_fadvise):
real    0m7.635s
user    0m4.272s
sys     0m0.840s



HTH,
Michael


-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 14:00:06 GMT) (full text, mbox, link).

Acknowledgement sent to Bastian Blank <waldi@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 14:00:06 GMT) (full text, mbox, link).

Message #117 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 01:22:25PM +0000, Ian Jackson wrote:
> Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
> > Are there any plans to provide an API for atomic (non-durable) file
> > updates, not involving fsync?
> Yes.  Such an API has already been defined by POSIX, SuSv3, et al.
> It's called "rename".

Please quote the appropriate standards. "rename" is atomic, aka there is
no time both the old and the new name exists.

> I'm told that the Linux fs maintainers have now accepted that 
>    open("file.new",O_CREAT);
>    write();
>    close();
>    rename("file.new","file");
> should not result, even after a crash, in "file" containing garbage.

It does not contain gargabe. It is empty.

> If this is the case then all the fsyncs can be taken out again.

Again: Please quote the standard instead of crying. Your view of things
disallows many of the recent improvements in filesystems, so you have to
show evidence. All the databases and other reliable data handing tools
uses fsync since a long time, because the writes may or may not hit the
disk otherwise.

Bastian

-- 
Even historians fail to learn from history -- they repeat the same mistakes.
		-- John Gill, "Patterns of Force", stardate 2534.7

Message #122 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 02:16:02PM +0100, Raphael Hertzog wrote:
> 
> It means we don't need to keep it in RAM since we're not going to
> read/modifiy it again in the near future. Thus the writeback can be
> started right now since delaying it will not save us anything.
> 
> At least that's the way I understand the situation.

Yes, that's correct.  The fadvise() will do two things; it will start
the writeback, and also make these memory pages be the most likely to
be discarded.  This last might or might not be a good thing.  If you
are installing a large number of packages, discarding will avoid more
useful things from being discard, and might help the interactive
"feel" of the machine while the install is going on in the background.

OTOH, if you are only installing one package, it might cause some file
that will be needed by the postinstall script to be pushed out of the
page cache prematurely.

So the fadvise() does the same thing as SYNC_FILE_RANGE_WRITE, which
is to say, start an asynchronous writeback of the pages in the file.
It will not do a SYNC_FILE_RANGE_WAIT_BEFORE, which assures that the
writebacks are complete before attempting to start doing the
fdatasync().

> > Put another way: if this works now, is it likely to continue to work?
> 
> Well, it will always work (the code is unlikely to introduce failures),
> but the resulting behaviour is entirely up to the kernel to decide. So
> there's no guaranty that the optimization will last.

Exactly.  I think the real question is whether you want to also give
the hint that the pages for that particular file should be first in
line to be discarded from the page cache.  

> On the other hand, the whole point of posix_fadvise() is to give hints to
> the kernel so that he can decide on the best course of action. So I hope
> the interpretation above is one the main motivation behind that hint.

The main motivation is to make the pages easily discardable; the fact
that it happens to start writeback is really a side effect.  So for
backup programs, including rsync when it is being used for backups,
using POSIX_FADV_DONTNEED is definitely a good idea.  Whether or not
it is a good idea for dpkg really depends on whether you think the
files are going to be used soon after they are written --- either
because the user has just installed the new toy and wants to play with
it (i.e., "apt-get install tuxracer; tuxracer") or because of a
post-install script.

On the other hand, if the user was just updating a random set of
progams that aren't actually going to be used right away (i.e.,
"apt-get update; apt-get upgrade"), in that case the
POSIX_FADV_DONTNEED would probably be a good thing.

The reason why I suggested using sync_file_range() is because it is
very specifically directed at forcing the writeback to happen, which
is not quite the same thrust as posix_fadvise().

Regards,

						- Ted

Acknowledgement sent to Olaf van der Spek <olafvdspek@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 14:42:09 GMT) (full text, mbox, link).

Message #127 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 2:22 PM, Ian Jackson
<ijackson@chiark.greenend.org.uk> wrote:
> Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
>> Are there any plans to provide an API for atomic (non-durable) file
>> updates, not involving fsync?
>
> Yes.  Such an API has already been defined by POSIX, SuSv3, et al.
> It's called "rename".

Probably not an issue for dpkg, but in general:
Don't you reset meta-data that way?
Require a second file (name), permission to write to it and assume
it's on the same volume?

Olaf

Acknowledgement sent to Olaf van der Spek <olafvdspek@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 14:48:03 GMT) (full text, mbox, link).

Message #132 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 2:56 PM, Bastian Blank <waldi@debian.org> wrote:
> Again: Please quote the standard instead of crying. Your view of things
> disallows many of the recent improvements in filesystems, so you have to
> show evidence. All the databases and other reliable data handing tools
> uses fsync since a long time, because the writes may or may not hit the
> disk otherwise.

If the durable guarantee isn't necessary, fsync isn't the fastest way, right?

Olaf

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 15:00:03 GMT) (full text, mbox, link).

Message #137 received at 605009@bugs.debian.org (full text, mbox, reply):

Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
> Probably not an issue for dpkg, but in general:
> Don't you reset meta-data that way?

Yes.  If you want to keep the metadata you must copy it.

> Require a second file (name), permission to write to it and assume
> it's on the same volume?

It will be on the same volume because it's in the same directory.

This is the standard way that ordinary files for which reliability was
important have been updated on Unix for decades.  fsync is for files
which need synchronisation with things external to the computer (or at
least, external to the volume) - eg, email at final dot.

Ian.

Acknowledgement sent to Olaf van der Spek <olafvdspek@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 15:06:03 GMT) (full text, mbox, link).

Message #142 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 3:58 PM, Ian Jackson
<ijackson@chiark.greenend.org.uk> wrote:
> Olaf van der Spek writes ("Re: Bug#605009: serious performance regression with ext4"):
>> Probably not an issue for dpkg, but in general:
>> Don't you reset meta-data that way?
>
> Yes.  If you want to keep the metadata you must copy it.

I guess that's hard.

>> Require a second file (name), permission to write to it and assume
>> it's on the same volume?
>
> It will be on the same volume because it's in the same directory.

What about the other issues?

> This is the standard way that ordinary files for which reliability was
> important have been updated on Unix for decades.  fsync is for files
> which need synchronisation with things external to the computer (or at
> least, external to the volume) - eg, email at final dot.

I know. I agree the rename trick should work, but my point is that a
better way for atomic non-durable file updates would be good to have.

Olaf

Message #147 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 02:58:16PM +0000, Ian Jackson wrote:
> 
> This is the standard way that ordinary files for which reliability was
> important have been updated on Unix for decades.  fsync is for files
> which need synchronisation with things external to the computer (or at
> least, external to the volume) - eg, email at final dot.

This is simply not true.  And I'm speaking as someone who has been
doing Unix/Linux kernel development since the BSD 4.3 days.  (Well,
BSD 4.3+Tahoe days, to be precise.)

fsync() has always been the only guarantee that files would be on
disk.  In fact the way BSD worked, there was no guarantee that
rename() would provide any kind of file synchronization primitive;
that's actually something new.  No, in the old days, if you really
cared about a file, you would fsync() it.  Period.  End of paragraph.

It was just that in those days, the main things people cared about
where either source/text files (so the editors of the day would do the
right thing) or e-mail (and no, just for the final delivery; for all
MTA's).

The problem that caused people to get this wrong idea was because (a)
back then Unix machines tended to be more reliable, because they were
run by professionals in machine rooms, very often with UPS's.  Also,
(b) people weren't loading craptastic video drivers with buggy
proprietary kernel modules; they may have used proprietary drivers,
but kernels weren't changing all the time, and there was a lot more
careful testing of drivers before they were unloosed onto the world.

Finally (c), ext3 had as an accident of how it provided protection
against old file blocks showing up newly allocated files (something
which BSD 4.3 did __not__ protect against, by the way), had the
behaviour that renaming over a file __usually__ (but not always)
provided atomic guarantees.

(c) was especially unfortunate, because it never applied to all Linux
file systems, just to ext3, and because the fact that it did this was
also responsible for disastrous desktop performance when you had the
combination of large number of streaming writes (i.e., bittorrent,
video ripping/copying, etc.) going on in the background combined with
foreground GUI applications that were fsync-happy() --- i.e., firefox.

Lots of users have complained about the desktop performance problem,
but the reality is we can't really solve that without also taking away
the magic that made (c) happen.  Whether you solve it by using
data=writeback and stick with ext3, or switch to ext4, or switch to
XFS, or switch to btrfs --- all of these will solve the desktop
performance problem, but they also leave you vulnerable to file loss
in the case of system crashes and applications that don't use
fsync()/fdatasync().

Hence the fact that all file system developers, whether they were
btrfs developers or XFS developers or ext4 developers, made the joke
at the file system developers summit two years ago, that what the
application programmers really wanted was O_PONY, with the magic pixie
dust.   Unfortunately:

	http://www.linuxformat.com/files/nopony.jpg

							- Ted

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 15:24:06 GMT) (full text, mbox, link).

Message #152 received at 605009@bugs.debian.org (full text, mbox, reply):

Ted Ts'o wrote:
> On Mon, Nov 29, 2010 at 02:16:02PM +0100, Raphael Hertzog wrote:

>> It means we don't need to keep it in RAM since we're not going to
>> read/modifiy it again in the near future. Thus the writeback can be
>> started right now since delaying it will not save us anything.
>>
>> At least that's the way I understand the situation.
>
> Yes, that's correct.  The fadvise() will do two things; it will start
> the writeback, and also make these memory pages be the most likely to
> be discarded.

That explanation helps a lot.  Thanks, both.  (Guillem, I like your
patch very much then.  Most files being unpacked in a dpkg run aren't
going to be read back again soon.  Perhaps some other kernels will
also interpret it as a hint to start writeback.)

> The reason why I suggested using sync_file_range() is because it is
> very specifically directed at forcing the writeback to happen, which
> is not quite the same thrust as posix_fadvise().

I suppose a patch to adopt the optimization you suggested ought to
also do the sync_file_range() directly to be explicit.

Acknowledgement sent to Olaf van der Spek <olafvdspek@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 15:30:03 GMT) (full text, mbox, link).

Message #157 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 4:18 PM, Ted Ts'o <tytso@mit.edu> wrote:
> Lots of users have complained about the desktop performance problem,
> but the reality is we can't really solve that without also taking away
> the magic that made (c) happen.  Whether you solve it by using
> data=writeback and stick with ext3, or switch to ext4, or switch to
> XFS, or switch to btrfs --- all of these will solve the desktop
> performance problem, but they also leave you vulnerable to file loss
> in the case of system crashes and applications that don't use
> fsync()/fdatasync().

Are you saying you can't guarantee the atomic property without the
durable property? I don't think that's right.

I also don't understand why you can't solve the old contents issue
without negatively affecting performance.

> Hence the fact that all file system developers, whether they were
> btrfs developers or XFS developers or ext4 developers, made the joke
> at the file system developers summit two years ago, that what the
> application programmers really wanted was O_PONY, with the magic pixie
> dust.   Unfortunately:

I think a lot would be happy with O_ATOMIC.

Olaf

Message #162 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 09:21:44AM -0600, Jonathan Nieder wrote:
> 
> That explanation helps a lot.  Thanks, both.  (Guillem, I like your
> patch very much then.  Most files being unpacked in a dpkg run aren't
> going to be read back again soon.  Perhaps some other kernels will
> also interpret it as a hint to start writeback.)

Most files won't, but consider a postinstall script which needs to
scan/index a documentation file, or simply run one or more binaries
that was just installed.  I can definitely imagine situations where
using POSIX_FADV_DONTNEED could actually hurt performance.  Is it
enough to worry about?  Hard to say; for a very long dpkg run, the
files might end up getting pushed out of memory anyway.  But if you
are only installing one package, and you are doing this on a
particularly slow disk, using POSIX_FADV_DONTNEED could actually hurt
in a measurable way.

If you are only installing a one or a few packages, and/or you can
somehow divine the user's intention that they will very shortly use
the file --- for example, if dpkg is being launched via packagekit to
install some font or codec, then using POSIX_FADV_DONTNEED is probably
the wrong answer.  So at the very least I'd recommend having command
line options to enable/disable use of posix_fadvise().

Regards,

						- Ted

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Mon, 29 Nov 2010 16:06:02 GMT) (full text, mbox, link).

Message #167 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 10:32:44AM -0500, Ted Ts'o wrote:
> Most files won't, but consider a postinstall script which needs to
> scan/index a documentation file, or simply run one or more binaries
> that was just installed.  I can definitely imagine situations where
> using POSIX_FADV_DONTNEED could actually hurt performance.

Kernel installation. It will use depmod and read every file again.

Bastian

-- 
	"That unit is a woman."
	"A mass of conflicting impulses."
		-- Spock and Nomad, "The Changeling", stardate 3541.9

Acknowledgement sent to Jonathan Nieder <jrnieder@gmail.com>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Mon, 29 Nov 2010 16:09:07 GMT) (full text, mbox, link).

Message #172 received at 605009@bugs.debian.org (full text, mbox, reply):

Ted Ts'o wrote:

> Most files won't, but consider a postinstall script which needs to
> scan/index a documentation file, or simply run one or more binaries
> that was just installed.  I can definitely imagine situations where
> using POSIX_FADV_DONTNEED could actually hurt performance.

Hmm.  Maybe file triggers could suppress DONTNEED.  A nice side-effect
might be to encourage use of file triggers. :)

> If you are only installing a one or a few packages, and/or you can
> somehow divine the user's intention that they will very shortly use
> the file --- for example, if dpkg is being launched via packagekit to
> install some font or codec, then using POSIX_FADV_DONTNEED is probably
> the wrong answer.

Even then, documentation extracted to /usr/share/doc doesn't need to
be cached.  I suppose some timing and tweaking would be needed.

The cases to optimize most for imho are

 1) initial installation (would use --force-unsafe-io)
 2) apt-get dist-upgrade; xterm
 3) apt-get install openoffice.org; oowriter
 4) apt-get build-dep <anything>; dpkg-buildpackage

since these are the most painful when they are slow.

So I agree, this is not cut and dry.  But being able to use the same
function and have it (hopefully) do something roughly sane on a
variety of kernels is very appealing.

Jonathan

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Tue, 30 Nov 2010 00:12:05 GMT) (full text, mbox, link).

Acknowledgement sent to Michael Biebl <biebl@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 00:12:05 GMT) (full text, mbox, link).

Message #177 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

On 29.11.2010 14:41, Michael Biebl wrote:
> I'm using ext4 (as already mentioned), my small benchmark is (re)installing
> vim-runtime using dpkg -i
> 
> 1.15.8.5:
> real    0m9.259s
> user    0m4.212s
> sys     0m0.752s
> 
> 1.15.8.6:
> real    0m41.766s
> user    0m4.248s
> sys     0m1.028s
> 
> 1.15.8.6 (force-unsafe-io):
> real    0m8.404s
> user    0m4.244s
> sys     0m0.752s
> 
> 1.15.8.6 (+patch posix_fadvise):
> real    0m7.635s
> user    0m4.272s
> sys     0m0.840s

Something interesting I noticed:

I created the ext4 file system on a spare partition and installed a chroot.
After running the test, I exited the chroot, immediately unmounted the partition
and measured how long it took:

1.15.8.5: 0.4s
1.15.8.6: 7.2s
1.15.8.6+patch: 18.4s

If I wait a bit (+60s) after exiting the chroot and then do the unmount, it is
almost instant (< 0.2s) in all three cases.

Michael
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Acknowledgement sent to Michael Biebl <biebl@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 00:39:06 GMT) (full text, mbox, link).

Message #182 received at 605009@bugs.debian.org (full text, mbox, reply):

[Message part 1 (text/plain, inline)]

On 29.11.2010 14:41, Michael Biebl wrote:
> On 29.11.2010 07:18, Guillem Jover wrote:
> 
>>
>> Could someone with ext4/btrfs/xfs/etc test w/ and w/o the attached patch
>> against dpkg?
> 
> I'm using ext4 (as already mentioned), my small benchmark is (re)installing
> vim-runtime using dpkg -i


Same test on an xfs partition

1.15.8.5:
real    0m22.494s
user    0m3.384s
sys     0m1.312s

1.15.8.6:
real    0m48.842s
user    0m3.360s
sys     0m1.696s

1.15.8.6 (force-unsafe-io):
real    0m18.967s
user    0m3.412s
sys     0m1.144s

1.15.8.6 (w/ patch posix_fadvise):
real    0m48.524s
user    0m3.872s
sys     0m1.892s


Looks like POSIX_FADV_DONTNEED has no effect on xfs.
-- 
Why is it that all of the instruments seeking intelligent life in the
universe are pointed away from Earth?

[signature.asc (application/pgp-signature, attachment)]

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Tue, 30 Nov 2010 07:03:03 GMT) (full text, mbox, link).

Acknowledgement sent to Raphael Hertzog <hertzog@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 07:03:03 GMT) (full text, mbox, link).

Message #187 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi,

On Tue, 30 Nov 2010, Michael Biebl wrote:
> Something interesting I noticed:
> 
> I created the ext4 file system on a spare partition and installed a chroot.
> After running the test, I exited the chroot, immediately unmounted the partition
> and measured how long it took:
> 
> 1.15.8.5: 0.4s

Expected, this version calls sync() so there's nothing to write back when
you umount the filesystem.

> 1.15.8.6: 7.2s
> 1.15.8.6+patch: 18.4s

I can't explain the difference between both but it doesn't matter much IMO.

> If I wait a bit (+60s) after exiting the chroot and then do the unmount, it is
> almost instant (< 0.2s) in all three cases.

60s is more than the default writeback delay so again it's normal.

$ sudo cat /proc/sys/vm/dirty_writeback_centisecs
500

 -> writeback after 5 sec

Cheers,
-- 
Raphaël Hertzog ◈ Debian Developer

Follow my Debian News ▶ http://RaphaelHertzog.com (English)
                      ▶ http://RaphaelHertzog.fr (Français)

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Tue, 30 Nov 2010 09:09:03 GMT) (full text, mbox, link).

Acknowledgement sent to Mike Hommey <mh@glandium.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 09:09:03 GMT) (full text, mbox, link).

Message #192 received at 605009@bugs.debian.org (full text, mbox, reply):

On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote:
> > What's going on here?  sync_file_range() is a Linux specific system
> > call that has been around for a while.  It allows program to control
> > when writeback happens in a very low-level fashion.  The first set of
> > sync_file_range() system calls causes the system to start writing back
> > each file once it has finished being extracted.  It doesn't actually
> > wait for the write to finish; it just starts the writeback.
> 
> Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> instead, skimming over the kernel source seems to indicate it might
> end up doing more or less the same thing but in a portable way?

On the other hand, there is no guarantee that other kernels do the same,
nor that Linux will keep doing it in the future. Using sync_file_range
and possibly the corresponding BSD syscall seems a better solution.
(and apparently the assumption with fadvise doesn't work with xfs)

Mike

Acknowledgement sent to Samuel Thibault <sthibault@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 09:39:05 GMT) (full text, mbox, link).

Message #197 received at 605009@bugs.debian.org (full text, mbox, reply):

Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit :
> On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote:
> > > What's going on here?  sync_file_range() is a Linux specific system
> > > call that has been around for a while.  It allows program to control
> > > when writeback happens in a very low-level fashion.  The first set of
> > > sync_file_range() system calls causes the system to start writing back
> > > each file once it has finished being extracted.  It doesn't actually
> > > wait for the write to finish; it just starts the writeback.
> > 
> > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> > instead, skimming over the kernel source seems to indicate it might
> > end up doing more or less the same thing but in a portable way?
> 
> On the other hand, there is no guarantee that other kernels do the same,

Err, that's posix.

Samuel

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Tue, 30 Nov 2010 10:27:05 GMT) (full text, mbox, link).

Acknowledgement sent to Bastian Blank <waldi@debian.org>:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Tue, 30 Nov 2010 10:27:05 GMT) (full text, mbox, link).

Message #202 received at 605009@bugs.debian.org (full text, mbox, reply):

On Tue, Nov 30, 2010 at 10:35:11AM +0100, Samuel Thibault wrote:
> Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit :
> > On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote:
> > > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> > > instead, skimming over the kernel source seems to indicate it might
> > > end up doing more or less the same thing but in a portable way?
> > On the other hand, there is no guarantee that other kernels do the same,
> Err, that's posix.

What is POSIX? What exactly is written in the standard? Please quote.

Okay, here is the quote[1]:
| POSIX_FADV_DONTNEED
| Specifies that the application expects that it will not access the
| specified data in the near future.

sync_file_range is Linux-specific and documented to do exactly what we
want[2]. posix_fadvise with POSIX_FADV_DONTNEED is not documented to do
what we want but only does it as a side-effect (and may hurt others
because it evicts anything of the cache).

Bastian

[1]: http://www.opengroup.org/onlinepubs/9699919799/functions/posix_fadvise.html
[2]: fs/sync.c
-- 
	"That unit is a woman."
	"A mass of conflicting impulses."
		-- Spock and Nomad, "The Changeling", stardate 3541.9

Message #207 received at 605009@bugs.debian.org (full text, mbox, reply):

On Tue, Nov 30, 2010 at 10:35:11AM +0100, Samuel Thibault wrote:
> Mike Hommey, le Tue 30 Nov 2010 10:07:55 +0100, a écrit :
> > On Mon, Nov 29, 2010 at 07:18:17AM +0100, Guillem Jover wrote:
> > > > What's going on here?  sync_file_range() is a Linux specific system
> > > > call that has been around for a while.  It allows program to control
> > > > when writeback happens in a very low-level fashion.  The first set of
> > > > sync_file_range() system calls causes the system to start writing back
> > > > each file once it has finished being extracted.  It doesn't actually
> > > > wait for the write to finish; it just starts the writeback.
> > > 
> > > Hmm, ok so what about posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)
> > > instead, skimming over the kernel source seems to indicate it might
> > > end up doing more or less the same thing but in a portable way?
> > 
> > On the other hand, there is no guarantee that other kernels do the same,
> 
> Err, that's posix.

Being posix doesn't guarantee that posix_fadvise(fd, 0, 0,
POSIX_FADV_DONTNEED) is going to start write back, which is the desired
effect, but not what you may actually get, depending on the kernel.

Mike

Added tag(s) pending. Request was from Guillem Jover <guillem@debian.org> to control@bugs.debian.org. (Fri, 10 Dec 2010 12:24:16 GMT) (full text, mbox, link).

Message sent on to Michael Biebl <biebl@debian.org>:
Bug#605009. (Fri, 10 Dec 2010 12:24:18 GMT) (full text, mbox, link).

Message #212 received at 605009-submitter@bugs.debian.org (full text, mbox, reply):

tag 605009 pending
thanks

Hello,

Bug #605009 reported by you has been fixed in the Git repository. You can
see the changelog below, and you can check the diff of the fix at:

    http://git.debian.org/?p=dpkg/dpkg.git;a=commitdiff;h=203c99b

---
commit 203c99b0dda40159d824944e407a2f720f1088ed
Author: Guillem Jover <guillem@debian.org>
Date:   Wed Dec 8 06:29:07 2010 +0100

    dpkg: On Linux initiate writeback of unpacked files ASAP
    
    To avoid performance degradation on filesystems with "allocate on
    flush" semantics (like xfs, ubifs, hfs+, and ext4 without nodelalloc),
    start writing back each file once it has finished being extracted.
    This doesn't actually wait for the write to finish; it just starts
    the writeback.
    
    The sync_file_range() call has been available since Linux 2.6.17.
    On non-Linux systems we can skip it.
    
    Closes: #605009
    
    Suggested-by: Ted Ts'o <tytso@mit.edu>
    Based-on-patch-by: Jonathan Nieder <jrnieder@gmail.com>
    Signed-off-by: Guillem Jover <guillem@debian.org>

diff --git a/debian/changelog b/debian/changelog
index d5aee9b..d241d06 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -4,6 +4,9 @@ dpkg (1.15.8.7) UNRELEASED; urgency=low
   * Defer symlink renames so that there's never a point were a symlink
     is broken, this is particularly important for shared libraries.
     Closes: #605536
+  * On Linux use sync_file_range() to initiate asynchronous writeback
+    of just unpacked files. Suggested by Ted Ts'o <tytso@mit.edu>.
+    Thanks to Jonathan Nieder <jrnieder@gmail.com>. Closes: #605009
 
   [ Updated manpage translations ]
   * French (Christian Perrier).

Information forwarded to debian-bugs-dist@lists.debian.org, Dpkg Developers <debian-dpkg@lists.debian.org>:
Bug#605009; Package dpkg. (Sat, 11 Dec 2010 20:21:02 GMT) (full text, mbox, link).

Acknowledgement sent to sedat.dilek@gmail.com:
Extra info received and forwarded to list. Copy sent to Dpkg Developers <debian-dpkg@lists.debian.org>. (Sat, 11 Dec 2010 20:21:03 GMT) (full text, mbox, link).

Message #217 received at 605009@bugs.debian.org (full text, mbox, reply):

Hi,

finally, this bug is getting fixed.

I have tried to build new dpkg 1.15.8.7 from sid GIT branch [1].
Looks like autpoint is missing in the Build-Depends (see below P.S.).

# apt-get install autopoint

...helps to build the Debian package.


Regards,
- Sedat -

[1] http://git.debian.org/?p=dpkg/dpkg.git;a=shortlog;h=refs/heads/sid

P.S.:

sd@tbox:~/src/dpkg/dpkg$ LC_ALL=C debuild
 dpkg-buildpackage -rfakeroot -D -us -uc
dpkg-buildpackage: export CFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export CPPFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: export CXXFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export FFLAGS from dpkg-buildflags (origin: vendor): -g -O2
dpkg-buildpackage: export LDFLAGS from dpkg-buildflags (origin: vendor):
dpkg-buildpackage: source package dpkg
dpkg-buildpackage: source version 1.15.8.7
dpkg-buildpackage: source changed by Sven Joachim <svenjoac@gmx.de>
 dpkg-source --before-build dpkg
dpkg-buildpackage: host architecture i386
dpkg-source: info: using options from dpkg/debian/source/options:
--compression=bzip2
 fakeroot debian/rules clean
dh_testdir
[ ! -f Makefile ] || /usr/bin/make distclean
rm -rf build-tree
dh_clean
 dpkg-source -b dpkg
dpkg-source: info: using options from dpkg/debian/source/options:
--compression=bzip2
dpkg-source: info: using source format `3.0 (native)'
dpkg-source: info: building dpkg in dpkg_1.15.8.7.tar.bz2
dpkg-source: info: building dpkg in dpkg_1.15.8.7.dsc
 debian/rules build
dh_testdir
autoreconf -v -i
autoreconf: Entering directory `.'
autoreconf: running: autopoint
Can't exec "autopoint": No such file or directory at
/usr/share/autoconf/Autom4te/FileUtils.pm line 345.
autoreconf: failed to run autopoint: No such file or directory
autoreconf: autopoint is needed because this package uses Gettext
make: *** [configure] Error 1
dpkg-buildpackage: error: debian/rules build gave error exit status 2
debuild: fatal error at line 1325:
dpkg-buildpackage -rfakeroot -D -us -uc failed
sd@tbox:~/src/dpkg/dpkg

Reply sent to Guillem Jover <guillem@debian.org>:
You have taken responsibility. (Mon, 20 Dec 2010 02:33:06 GMT) (full text, mbox, link).

Notification sent to Michael Biebl <biebl@debian.org>:
Bug acknowledged by developer. (Mon, 20 Dec 2010 02:33:06 GMT) (full text, mbox, link).

Message #222 received at 605009-close@bugs.debian.org (full text, mbox, reply):

Source: dpkg
Source-Version: 1.15.8.7

We believe that the bug you reported is fixed in the latest version of
dpkg, which is due to be installed in the Debian FTP archive:

dpkg-dev_1.15.8.7_all.deb
  to main/d/dpkg/dpkg-dev_1.15.8.7_all.deb
dpkg_1.15.8.7.dsc
  to main/d/dpkg/dpkg_1.15.8.7.dsc
dpkg_1.15.8.7.tar.bz2
  to main/d/dpkg/dpkg_1.15.8.7.tar.bz2
dpkg_1.15.8.7_amd64.deb
  to main/d/dpkg/dpkg_1.15.8.7_amd64.deb
dselect_1.15.8.7_amd64.deb
  to main/d/dpkg/dselect_1.15.8.7_amd64.deb
libdpkg-dev_1.15.8.7_amd64.deb
  to main/d/dpkg/libdpkg-dev_1.15.8.7_amd64.deb
libdpkg-perl_1.15.8.7_all.deb
  to main/d/dpkg/libdpkg-perl_1.15.8.7_all.deb



A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 605009@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Guillem Jover <guillem@debian.org> (supplier of updated dpkg package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@debian.org)


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Format: 1.8
Date: Mon, 20 Dec 2010 02:26:26 +0100
Source: dpkg
Binary: libdpkg-dev dpkg dpkg-dev libdpkg-perl dselect
Architecture: source amd64 all
Version: 1.15.8.7
Distribution: unstable
Urgency: low
Maintainer: Dpkg Developers <debian-dpkg@lists.debian.org>
Changed-By: Guillem Jover <guillem@debian.org>
Description: 
 dpkg       - Debian package management system
 dpkg-dev   - Debian package development tools
 dselect    - Debian package management front-end
 libdpkg-dev - Debian package management static library
 libdpkg-perl - Dpkg perl modules
Closes: 605009 605099 605248 605506 605536 606315 607253 607259 607292 607302 607306 607437 607501
Changes: 
 dpkg (1.15.8.7) unstable; urgency=low
 .
   [ Guillem Jover ]
   * Defer symlink renames so that there's never a point were a symlink
     is broken, this is particularly important for shared libraries.
     Closes: #605536
   * On Linux use sync_file_range() to initiate asynchronous writeback
     of just unpacked files. Suggested by Ted Ts'o <tytso@mit.edu>.
     Thanks to Jonathan Nieder <jrnieder@gmail.com>. Closes: #605009
   * On non-Linux use posix_fadvise(POSIX_FADV_DONTNEED) to notify the kernel
     dpkg does not need the unpacked files any longer, and that it can start
     writeback to be able to evict them from the cache at a later point.
   * Fix stanza delimiting on -L, -s and -p output. This was making the output
     for multiple packages unrealiable to parse. Closes: #606315
 .
   [ Updated dpkg translations ]
   * Basque (Iñaki Larrañaga Murgoitio). Closes: #607253
   * Catalan (Guillem Jover).
   * Czech (Miroslav Kure). Closes: #605099
   * Esperanto (Felipe E. F. de Castro). Closes: #607437
   * French (Christian Perrier).
   * German (Sven Joachim).
   * Indonesian (Arief S Fitrianto). Closes: #605248
   * Italian (Milo Casagrande). Closes: #607306
   * Japanese (Kenshi Muto). Closes: #607259
   * Norwegian Bokmål (Hans Fredrik Nordhaug).
   * Portuguese (Miguel Figueiredo). Closes: #605506
   * Russian (Yuri Kozlov). Closes: #607292
   * Simplified Chinese (Aron Xu).
   * Slovak (Ivan Masár). Closes: #607302
   * Spanish (Javier Fernandez-Sanguino).
   * Thai (Theppitak Karoonboonyanan). Closes: #607501
 .
   [ Updated man page translations ]
   * French (Christian Perrier).
   * German (Helge Kreutzmann).
   * Spanish (Omar Campagne).
 .
   [ Updated dselect translations ]
   * Spanish (Javier Fernandez-Sanguino).
 .
   [ Updated scripts translations ]
   * German (fix by Sven Joachim).
   * Spanish (Javier Fernandez-Sanguino).
Checksums-Sha1: 
 d53e3f1156b59503e2f786a7b8214dcbd9b29c4f 1208 dpkg_1.15.8.7.dsc
 c11aefddda4ebc8a78171343a201495fe64e1837 5251036 dpkg_1.15.8.7.tar.bz2
 b7587044f347f88521cb5f2df026f722175d73b7 430924 libdpkg-dev_1.15.8.7_amd64.deb
 c381b706eff13f6a97ec161462d058c4142ba5f5 2391526 dpkg_1.15.8.7_amd64.deb
 e62bc3b332a292f8f5f1838ac4706c7e57ffffd5 898796 dselect_1.15.8.7_amd64.deb
 30bb5e7b4a49b830b161aa356bb116b1b1df5cb9 805800 dpkg-dev_1.15.8.7_all.deb
 64657a7f4939f5aaf34b1c0d2564aeacdaa3b398 687168 libdpkg-perl_1.15.8.7_all.deb
Checksums-Sha256: 
 1451e2109b445665340f57e4aad269f3a0838bd4415d016ab5b89f999550a69f 1208 dpkg_1.15.8.7.dsc
 1ec1376471b04717a4497e5d7a27cd545248c92116898ce0c53ced8ea94267b5 5251036 dpkg_1.15.8.7.tar.bz2
 62351dc3823dc0f8b00348beebc9ea1feaf82db53f194d6105c5bd4781a8f557 430924 libdpkg-dev_1.15.8.7_amd64.deb
 8c711eebc3decb5ffaa4e184baa985b635a86fc0399f72e424004ab84414c943 2391526 dpkg_1.15.8.7_amd64.deb
 f39b3324b5841cb2d20fb6ebd1ea9a1264823bfb345afacace8dc25bdb6a12e2 898796 dselect_1.15.8.7_amd64.deb
 b28fab3670bab1f3e0aedb01b50b2ed33a1380f5f5d9608657a96be1411f073d 805800 dpkg-dev_1.15.8.7_all.deb
 54e050449f21daf6842f08bbd3fdbf07b82d6328070aab87c775e6765123b8c3 687168 libdpkg-perl_1.15.8.7_all.deb
Files: 
 a1734a4024208e967710c5232b0307a7 1208 admin required dpkg_1.15.8.7.dsc
 d1731d4147c1ea3b537a4d094519a6dc 5251036 admin required dpkg_1.15.8.7.tar.bz2
 ca98309242298799c56f70c4216da84e 430924 libdevel optional libdpkg-dev_1.15.8.7_amd64.deb
 29aa8a1e4fe70cb4033529518c10f617 2391526 admin required dpkg_1.15.8.7_amd64.deb
 d2d0d56e446a6bc9e1c35ac3ba67a7e9 898796 admin optional dselect_1.15.8.7_amd64.deb
 89ad9bfd49f6e9633a00d9213efbd691 805800 utils optional dpkg-dev_1.15.8.7_all.deb
 67287ec82679e9d1ee1106a6d7229079 687168 perl optional libdpkg-perl_1.15.8.7_all.deb

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEARECAAYFAk0Ot60ACgkQuW9ciZ2SjJvH2gCggmtKzf7f4hpQKXU7ZTWoGLIL
3xwAnjkyz5teXZQsZRlzGgsYaDLHrZyZ
=GPh1
-----END PGP SIGNATURE-----

Bug archived. Request was from Debbugs Internal Request <owner@bugs.debian.org> to internal_control@bugs.debian.org. (Thu, 20 Jan 2011 07:28:34 GMT) (full text, mbox, link).

Send a report that this bug log contains spam.

Debian bug tracking system administrator <owner@bugs.debian.org>. Last modified: Fri Apr 19 17:59:49 2024; Machine Name: buxtehude

Debian Bug tracking system

Debbugs is free software and licensed under the terms of the GNU Public License version 2. The current version can be obtained from https://bugs.debian.org/debbugs-source/.

Debian Bug report logs - #605009 serious performance regression with ext4

Debian Bug report logs - #605009
serious performance regression with ext4