Bug#864597: upgrade-reports: jessie -> stretch: gnome fails to upgrade: cycle found while processing triggers

[Cyril Brulebois]

Package: upgrade-reports
Severity: critical
Justification: makes upgrade from stable abort

[ X-D-Cc:
debian-***@lists.debian.org
pkg-java-***@lists.alioth.debian.org
pkg-gnome-***@lists.alioth.debian.org ]

Hi,

Regression spotted by Pere in some debian-edu job, but also seen since
the 2nd of June in normal gnome chroot installation then upgrade from
jessie to stretch:
https://jenkins.debian.net/view/edu_devel/job/chroot-installation_jessie_install_education-desktop-gnome_upgrade_to_stretch/
https://jenkins.debian.net/job/chroot-installation_jessie_install_gnome_upgrade_to_stretch/

I've managed to reproduce it locally with basically a debootstrap of
jessie, installation of gnome, then switch sources.list from jessie to
stretch, then update & upgrade & dist-upgrade.

I've bisected the archive using snapshot.debian.org and found out:
- 20170601T212625Z = last timestamp found to be OK;
- 20170602T033358Z = first timestamp to be KO.

Since logs are a bit too heavy for a bug report, I've uploaded them
there:
https://people.debian.org/~kibi/jessie2stretch/gnome/

$timestamp.log is the output of the installation & dist-upgrade process,
while $timestamp.log.clean is a cleaned version (with Get: lines edited
to remove the package indice and the timestamp, then sort them by block,
so as to avoid a huge diff).

Then I've generated upgrade.diff by diffing both clean versions:
https://people.debian.org/~kibi/jessie2stretch/gnome/upgrade.diff

This file consists mainly of some differences which should help us
pinpoint the exact issue (first part of the diff), but also of a big
diff at the end, since the OK log goes on with the install while the KO
one is cut rather quickly. Actual error follows:
| Unpacking default-jre-headless (2:1.8-58) over (2:1.7-52) ...
| Processing triggers for libc-bin (2.24-10) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| Processing triggers for desktop-file-utils (0.23-1) ...
| Processing triggers for man-db (2.7.6.1-2) ...
| Processing triggers for libglib2.0-0:amd64 (2.50.3-2) ...
| (Reading database ... 129883 files and directories currently installed.)
| Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing tzdata-java (2017b-0+deb8u1) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| dpkg: cycle found while processing triggers:
| chain of packages whose triggers are or may be responsible:
| gnome-menus -> desktop-file-utils
| packages' pending triggers which are or may be unresolvable:
| gnome-menus: /usr/share/applications
| shared-mime-info: /usr/share/mime/packages
| desktop-file-utils: /usr/share/applications
| dpkg: error processing package gnome-menus (--remove):
| triggers looping, abandoned
| Processing triggers for desktop-file-utils (0.23-1) ...
| Errors were encountered while processing:
| gnome-menus
| E: Sub-process /usr/bin/dpkg returned an error code (1)

By looking at the diff before that, this might have been triggered (no
pun intended) by the ca-certificates-java update, which included changes
in the required java version, which might explain why this block was
present in the OK log but no longer in the KO one?
| -dpkg: openjdk-7-jre-headless:amd64: dependency problems, but removing anyway as you requested:
| - ca-certificates-java depends on openjdk-7-jre-headless | java7-runtime-headless; however:
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| - ca-certificates-java depends on openjdk-7-jre-headless | java7-runtime-headless; however:
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| -

I don't see any immediate solutions (mostly because it's the first dpkg
triggers cycle I encounter), but this looks like something that really
should be fixed before the stretch release, hence the severity and the
x-d-cc list.


KiBi.

[Niels Thykier]

Post by Cyril Brulebois
Package: upgrade-reports
Severity: critical
Justification: makes upgrade from stable abort
Hi,
Regression spotted by Pere in some debian-edu job, but also seen since
the 2nd of June in normal gnome chroot installation then upgrade from
https://jenkins.debian.net/view/edu_devel/job/chroot-installation_jessie_install_education-desktop-gnome_upgrade_to_stretch/
https://jenkins.debian.net/job/chroot-installation_jessie_install_gnome_upgrade_to_stretch/

Hi,

Thanks for reporting this.

Timing being what it is, we need to act fast if we want to fix this for
stretch. I have CC'ed maintainers of (presumably) relevant packages;
please be ready to upload a fix to your package if needed.

Long story short:
=================

* dpkg reports a trigger cycle between gnome-menus and
desktop-file-utils.

* desktop-file-utils has an "interest" (implicit "-await") trigger, so
it is plausible. I have not verified the dependency relationship.

* shared-mime-info also has an "interest" (implicit "-await") trigger.
It may (or may not) be an accomplice to this issue.

* gnome-menus only have "interest-noawait" and should therefore not be
able to start the trigger cycle (but it can be part of one).

Things to do / research:
=======================

* @desktop-file-utils: The trigger seems to be used for a cache. Is
there any reason for using the implicit "-await" trigger or can this
cache update be deferred like the "man-db" cache can?

* @shared-mime-info: The trigger seems to be used for a cache. Is
there any reason for using the implicit "-await" trigger or can this
cache update be deferred like the "man-db" cache can?

* @Andreas: Could you have a look at the upgrade ordering in this case.
You are usually pretty good at spotting if we need Breaks, so I would
be delighted if you could give it a go.
(E.g. if gnome-menus need to break desktop-file-utils in jessie even
if desktop-file-utils in stretch uses a "-noawait" trigger)


I have left the remaining parts of KiBi's mails below for the newly
CC'ed people.

Thanks,
~Niels

Post by Cyril Brulebois
I've managed to reproduce it locally with basically a debootstrap of
jessie, installation of gnome, then switch sources.list from jessie to
stretch, then update & upgrade & dist-upgrade.
- 20170601T212625Z = last timestamp found to be OK;
- 20170602T033358Z = first timestamp to be KO.
Since logs are a bit too heavy for a bug report, I've uploaded them
https://people.debian.org/~kibi/jessie2stretch/gnome/
$timestamp.log is the output of the installation & dist-upgrade process,
while $timestamp.log.clean is a cleaned version (with Get: lines edited
to remove the package indice and the timestamp, then sort them by block,
so as to avoid a huge diff).
https://people.debian.org/~kibi/jessie2stretch/gnome/upgrade.diff
This file consists mainly of some differences which should help us
pinpoint the exact issue (first part of the diff), but also of a big
diff at the end, since the OK log goes on with the install while the KO
| Unpacking default-jre-headless (2:1.8-58) over (2:1.7-52) ...
| Processing triggers for libc-bin (2.24-10) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| Processing triggers for desktop-file-utils (0.23-1) ...
| Processing triggers for man-db (2.7.6.1-2) ...
| Processing triggers for libglib2.0-0:amd64 (2.50.3-2) ...
| (Reading database ... 129883 files and directories currently installed.)
| Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing tzdata-java (2017b-0+deb8u1) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| gnome-menus -> desktop-file-utils
| gnome-menus: /usr/share/applications
| shared-mime-info: /usr/share/mime/packages
| desktop-file-utils: /usr/share/applications
| triggers looping, abandoned
| Processing triggers for desktop-file-utils (0.23-1) ...
| gnome-menus
| E: Sub-process /usr/bin/dpkg returned an error code (1)
By looking at the diff before that, this might have been triggered (no
pun intended) by the ca-certificates-java update, which included changes
in the required java version, which might explain why this block was
present in the OK log but no longer in the KO one?
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| -
I don't see any immediate solutions (mostly because it's the first dpkg
triggers cycle I encounter), but this looks like something that really
should be fixed before the stretch release, hence the severity and the
x-d-cc list.
KiBi.

[Andreas Beckmann]

[ added Guillem to the recipients, he is the expert for trigger cycles ]
[ added Doko to the recipients, since #863820 is a possible solution ]
[ therefore resending a full-quote ]

Post by Niels Thykier

Post by Cyril Brulebois
Package: upgrade-reports
Severity: critical
Justification: makes upgrade from stable abort
Hi,
Regression spotted by Pere in some debian-edu job, but also seen since
the 2nd of June in normal gnome chroot installation then upgrade from
https://jenkins.debian.net/view/edu_devel/job/chroot-installation_jessie_install_education-desktop-gnome_upgrade_to_stretch/
https://jenkins.debian.net/job/chroot-installation_jessie_install_gnome_upgrade_to_stretch/

Hi,
Thanks for reporting this.
Timing being what it is, we need to act fast if we want to fix this for
stretch. I have CC'ed maintainers of (presumably) relevant packages;
please be ready to upload a fix to your package if needed.
=================
* dpkg reports a trigger cycle between gnome-menus and
desktop-file-utils.
* desktop-file-utils has an "interest" (implicit "-await") trigger, so
it is plausible. I have not verified the dependency relationship.
* shared-mime-info also has an "interest" (implicit "-await") trigger.
It may (or may not) be an accomplice to this issue.
* gnome-menus only have "interest-noawait" and should therefore not be
able to start the trigger cycle (but it can be part of one).
=======================
there any reason for using the implicit "-await" trigger or can this
cache update be deferred like the "man-db" cache can?
there any reason for using the implicit "-await" trigger or can this
cache update be deferred like the "man-db" cache can?
You are usually pretty good at spotting if we need Breaks, so I would
be delighted if you could give it a go.
(E.g. if gnome-menus need to break desktop-file-utils in jessie even
if desktop-file-utils in stretch uses a "-noawait" trigger)

I could reproduce the failure in piuparts (after adding support for
two-stage (apt-get upgrade && apt-get dist-upgrade) upgrades.

I sent a full upgrade log to the bug (#864597) with trigger debugging
enabled (debug=70000), hopefully Guillem can get a clue from it.
If you need more debug output, just tell me.
I could also test with a proposed dpkg/stretch patch.

Breaks probably won't help much since gnome-menus, desktop-file-utils
and shared-mime-info are already upgraded during the apt-get upgrade phase.

Switching desktop-file-utils or/and shared-mime-info to -noawait
triggers does not solve the problem.

I can confirm that the ca-certificates-java change triggered the bug
(i.e. backing it out makes the problem go away, but this is not a solution).

I can also confirm that the changes I suggested in #863820 (adding
adding Breaks: tzdata-java to gcc-6-base) would fix this upgrade path.
It shakes the upgrade order quite well for this case ...
(I can post the log if someone is interested.)
According to piuparts, the upgrade paths of about 860 packages in
jessie2stretch-rcmd would be affected by this change (because
tzdata-java gets installed in jessie). These can be retested quickly.

I'm planning to do piuparts tests doing the two-stage jessie->stretch
upgrade on piuparts.d.o, too. (So far we only do one-stage
apt-get dist-upgrade)


Andreas

Post by Niels Thykier
I have left the remaining parts of KiBi's mails below for the newly
CC'ed people.
Thanks,
~Niels

Post by Cyril Brulebois
I've managed to reproduce it locally with basically a debootstrap of
jessie, installation of gnome, then switch sources.list from jessie to
stretch, then update & upgrade & dist-upgrade.
- 20170601T212625Z = last timestamp found to be OK;
- 20170602T033358Z = first timestamp to be KO.
Since logs are a bit too heavy for a bug report, I've uploaded them
https://people.debian.org/~kibi/jessie2stretch/gnome/
$timestamp.log is the output of the installation & dist-upgrade process,
while $timestamp.log.clean is a cleaned version (with Get: lines edited
to remove the package indice and the timestamp, then sort them by block,
so as to avoid a huge diff).
https://people.debian.org/~kibi/jessie2stretch/gnome/upgrade.diff
This file consists mainly of some differences which should help us
pinpoint the exact issue (first part of the diff), but also of a big
diff at the end, since the OK log goes on with the install while the KO
| Unpacking default-jre-headless (2:1.8-58) over (2:1.7-52) ...
| Processing triggers for libc-bin (2.24-10) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| Processing triggers for desktop-file-utils (0.23-1) ...
| Processing triggers for man-db (2.7.6.1-2) ...
| Processing triggers for libglib2.0-0:amd64 (2.50.3-2) ...
| (Reading database ... 129883 files and directories currently installed.)
| Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
| Removing tzdata-java (2017b-0+deb8u1) ...
| Processing triggers for hicolor-icon-theme (0.15-1) ...
| gnome-menus -> desktop-file-utils
| gnome-menus: /usr/share/applications
| shared-mime-info: /usr/share/mime/packages
| desktop-file-utils: /usr/share/applications
| triggers looping, abandoned
| Processing triggers for desktop-file-utils (0.23-1) ...
| gnome-menus
| E: Sub-process /usr/bin/dpkg returned an error code (1)
By looking at the diff before that, this might have been triggered (no
pun intended) by the ca-certificates-java update, which included changes
in the required java version, which might explain why this block was
present in the OK log but no longer in the KO one?
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| - Package openjdk-7-jre-headless:amd64 is to be removed.
| - Package java7-runtime-headless is not installed.
| - Package openjdk-8-jre-headless:amd64 which provides java7-runtime-headless is not configured yet.
| - Package default-jre-headless which provides java7-runtime-headless is not configured yet.
| - Package openjdk-7-jre-headless:amd64 which provides java7-runtime-headless is to be removed.
| -
I don't see any immediate solutions (mostly because it's the first dpkg
triggers cycle I encounter), but this looks like something that really
should be fixed before the stretch release, hence the severity and the
x-d-cc list.
KiBi.

[Michael Biebl]

Hi

Post by Andreas Beckmann
Switching desktop-file-utils or/and shared-mime-info to -noawait
triggers does not solve the problem.

So afaics there is nothing that can be done from the Debian GNOME team
side, right?

Post by Andreas Beckmann
I can confirm that the ca-certificates-java change triggered the bug
(i.e. backing it out makes the problem go away, but this is not a solution).
I can also confirm that the changes I suggested in #863820 (adding
adding Breaks: tzdata-java to gcc-6-base) would fix this upgrade path.
It shakes the upgrade order quite well for this case ...
(I can post the log if someone is interested.)
According to piuparts, the upgrade paths of about 860 packages in
jessie2stretch-rcmd would be affected by this change (because
tzdata-java gets installed in jessie). These can be retested quickly.

I see that #863820 is still open and apparently also affects other
upgrade paths. Shouldn't this be bumped to RC?
What are we going to do about this? A failure to dist-upgrade our
default desktop environment is pretty bad.
Is there some known workaround which we could document in the release
notes at least?
From what I can see, wouldn't it be better to drop the the Breaks:
tzdata-java from openjdk-8-jdk-headless again?

[Julien Cristau]

Post by Michael Biebl
Hi

Post by Andreas Beckmann
Switching desktop-file-utils or/and shared-mime-info to -noawait
triggers does not solve the problem.

So afaics there is nothing that can be done from the Debian GNOME team
side, right?

Post by Andreas Beckmann
I can confirm that the ca-certificates-java change triggered the bug
(i.e. backing it out makes the problem go away, but this is not a solution).
I can also confirm that the changes I suggested in #863820 (adding
adding Breaks: tzdata-java to gcc-6-base) would fix this upgrade path.
It shakes the upgrade order quite well for this case ...
(I can post the log if someone is interested.)
According to piuparts, the upgrade paths of about 860 packages in
jessie2stretch-rcmd would be affected by this change (because
tzdata-java gets installed in jessie). These can be retested quickly.

I see that #863820 is still open and apparently also affects other
upgrade paths. Shouldn't this be bumped to RC?
What are we going to do about this? A failure to dist-upgrade our
default desktop environment is pretty bad.
Is there some known workaround which we could document in the release
notes at least?
tzdata-java from openjdk-8-jdk-headless again?

It sounds like openjdk-8 added two Breaks recently, one or both of which
are causing trouble, and none of which fix anything as bad as this. So
I think we should remove the Breaks on tzdata-java from
openjdk-8-jdk-headless, and remove the Breaks on ca-certificates-java
from openjdk-8-jre-headless. Would that fix the current issue?

Post by Michael Biebl
From what I can see #863820 is at most a normal-severity issue. Not

upgrading some packages is way way better than a failed upgrade.

Cheers,
Julien

[Cyril Brulebois]

Post by Julien Cristau
It sounds like openjdk-8 added two Breaks recently, one or both of
which are causing trouble, and none of which fix anything as bad as
this. So I think we should remove the Breaks on tzdata-java from
openjdk-8-jdk-headless, and remove the Breaks on ca-certificates-java
from openjdk-8-jre-headless. Would that fix the current issue?
From what I can see #863820 is at most a normal-severity issue. Not
upgrading some packages is way way better than a failed upgrade.

I'm currently building an updated openjdk-8 with the attached changes
to see what the upgrade-desktop test cases would look like.

Note: I've chosen to remove the “Breaks: ${jrehl:Breaks}” line from
debian/control* so that I'm only touching these files, since d/rules
seems to only set it to tzdata-java anyway, depending on the target
dist.

I'll report the results afterwards, but as mentioned on IRC, even if
that looks good in the end, fewer than 2 days to become confident it
won't regress in other cases is really short. :(


KiBi.

[Andreas Beckmann]

Post by Julien Cristau
It sounds like openjdk-8 added two Breaks recently, one or both of
which are causing trouble, and none of which fix anything as bad as
this. So I think we should remove the Breaks on tzdata-java from
openjdk-8-jdk-headless, and remove the Breaks on ca-certificates-java
from openjdk-8-jre-headless. Would that fix the current issue?

What I had tested was reverting the recent ca-certificates-java change,
i.e. switching the dependency on
openjdk-8-jre-headless | java7-runtime-headless
back to
openjdk-7-jre-headless | java7-runtime-headless
(or similar)
That change was made later than the one in openjdk-8 and it proved to be
sufficient in my piuparts test to fix^Wnot activate that trigger cycle.


Andreas

[Cyril Brulebois]

Post by Andreas Beckmann

Post by Julien Cristau
It sounds like openjdk-8 added two Breaks recently, one or both of
which are causing trouble, and none of which fix anything as bad as
this. So I think we should remove the Breaks on tzdata-java from
openjdk-8-jdk-headless, and remove the Breaks on ca-certificates-java
from openjdk-8-jre-headless. Would that fix the current issue?

FWIW, there are absolutely no differences when removing those two breaks,
even with resolver debugging enabled.

Post by Andreas Beckmann
What I had tested was reverting the recent ca-certificates-java change,
i.e. switching the dependency on
openjdk-8-jre-headless | java7-runtime-headless
back to
openjdk-7-jre-headless | java7-runtime-headless
(or similar)
That change was made later than the one in openjdk-8 and it proved to be
sufficient in my piuparts test to fix^Wnot activate that trigger cycle.

I was just saying this on IRC:

“should I try to reinstate ca-certificates-java's old dependencies and
compare? from my initial bug report, that's the change in the archive
that led to the regression.”

and mentioned earlier that given the old dependencies have an OR anyway,
we shouldn't be introducing an uninstallable package. This would have
the nice effect of being a solution that can be implemented in hours
instead of days (given openjdk-8's build time). This doesn't mean we
should jump the gun anyway


I'll report back.


KiBi.

[Cyril Brulebois]

Post by Cyril Brulebois
“should I try to reinstate ca-certificates-java's old dependencies and
compare? from my initial bug report, that's the change in the archive
that led to the regression.”
and mentioned earlier that given the old dependencies have an OR anyway,
we shouldn't be introducing an uninstallable package. This would have
the nice effect of being a solution that can be implemented in hours
instead of days (given openjdk-8's build time). This doesn't mean we
should jump the gun anyway

I'll report back.

This is looking good so far: gnome upgrade path fixed, kde and xfce are
still looking good, and tests with other desktop tasks are still running.

If all succeed, I intend to NMU ca-certificates-java with the attached
changes. I could have reintroduced the old package, but I chose to retain
the svn to git changes, and to drop the version for the openjdk-7 case,
since even jessie had a higher version. Attached debdiffs against current
version, and previous one.


KiBi.

[Emmanuel Bourg]

Post by Cyril Brulebois
If all succeed, I intend to NMU ca-certificates-java with the attached
changes. I could have reintroduced the old package, but I chose to retain
the svn to git changes, and to drop the version for the openjdk-7 case,
since even jessie had a higher version. Attached debdiffs against current
version, and previous one.

Is this the only solution? That's a bit odd to retain the openjdk-7
dependency on ca-certificates-java and potentially stick with it
forever. Maybe we should simply remove the JRE dependency on
ca-certificates-java and move the keystore generation to openjdk-8. That
would also solve the circular dependency (#864657).

If you upload a NMU could you please push the changes to the Git repository?

Emmanuel Bourg

[Cyril Brulebois]

Post by Emmanuel Bourg
If you upload a NMU could you please push the changes to the Git repository?

I'll look into this when I've slept a bit. Reminders/prods welcome.


KiBi.

[Cyril Brulebois]

Post by Emmanuel Bourg
Is this the only solution?

Probably not, but reverting the single change that triggered the
regression looks like a safe way to unbreak this situation. Especially
when the said change only happened many months after the relevant
package was removed from the archive (2017-05-31 vs. 2016-04-22)


Post by Emmanuel Bourg
That's a bit odd to retain the openjdk-7 dependency on
ca-certificates-java and potentially stick with it forever.

This is incorrect. See my reply in:
https://bugs.debian.org/864597#100

Post by Emmanuel Bourg
Maybe we should simply remove the JRE dependency on
ca-certificates-java and move the keystore generation to openjdk-8.
That would also solve the circular dependency (#864657).

Tentative release date minus days isn't the time to deal with circular
dependencies IMHO.

Post by Emmanuel Bourg
If you upload a NMU could you please push the changes to the Git repository?

Not possible since I'm not a member of pkg-java, and since you don't
seem to allow non-member access:
| $ git push origin HEAD $(git describe)
| Counting objects: 6, done.
| Delta compression using up to 4 threads.
| Compressing objects: 100% (6/6), done.
| Writing objects: 100% (6/6), 1.66 KiB | 0 bytes/s, done.
| Total 6 (delta 3), reused 0 (delta 0)
| remote: error: insufficient permission for adding an object to repository database ./objects
| remote: fatal: failed to write object
| error: unpack failed: unpack-objects abnormal exit
| To ssh://git.debian.org/git/pkg-java/ca-certificates-java.git
| ! [remote rejected] HEAD -> master (unpacker error)
| ! [remote rejected] debian/20170531+nmu1 -> debian/20170531+nmu1 (unpacker error)
| error: failed to push some refs to 'ssh://git.debian.org/git/pkg-java/ca-certificates-java.git'

git format-patch -1 attached though.


KiBi.

[Bill Allombert]

Post by Cyril Brulebois

Post by Julien Cristau
It sounds like openjdk-8 added two Breaks recently, one or both of
which are causing trouble, and none of which fix anything as bad as
this. So I think we should remove the Breaks on tzdata-java from
openjdk-8-jdk-headless, and remove the Breaks on ca-certificates-java
from openjdk-8-jre-headless. Would that fix the current issue?
From what I can see #863820 is at most a normal-severity issue. Not
upgrading some packages is way way better than a failed upgrade.

I'm currently building an updated openjdk-8 with the attached changes
to see what the upgrade-desktop test cases would look like.
Note: I've chosen to remove the “Breaks: ${jrehl:Breaks}” line from
debian/control* so that I'm only touching these files, since d/rules
seems to only set it to tzdata-java anyway, depending on the target
dist.
I'll report the results afterwards, but as mentioned on IRC, even if
that looks good in the end, fewer than 2 days to become confident it
won't regress in other cases is really short. :(

It would be really nice if we could remove the circular dependency
between openjdk-8 and ca-certificate before the release, otherwise
the stretch to buster upgrade will be a nightmare.
It always much easier to add circular dependency than to remove them.

Cheers,
Bill.

[Cyril Brulebois]

Post by Bill Allombert
It would be really nice if we could remove the circular dependency
between openjdk-8 and ca-certificate before the release, otherwise
the stretch to buster upgrade will be a nightmare.
It always much easier to add circular dependency than to remove them.

Bill, we're trying to fix/work around a jessie to stretch upgrade path
issue for the default desktop setup. And we're days away from the
tentative date for the release. Needless to say, trying to fix this
upgrade path is somewhat more important than what happens for stretch to
buster, for which years still have to pass. Plenty of time to figure out
what to do.

Of course it would be better if ca-certificates-java wouldn't have been
“fixed” on 2017-05-31 when openjdk-7 was removed on 2016-04-22. But
that's the situation we have to deal with right now.


KiBi.

[Andreas Beckmann]

Post by Michael Biebl

Post by Andreas Beckmann
Switching desktop-file-utils or/and shared-mime-info to -noawait
triggers does not solve the problem.

So afaics there is nothing that can be done from the Debian GNOME team
side, right?

You could add a Breaks: tzdata-java to some library deep in the gnome
stack (so that it gets a high score from apt) ... that could serve as a
workaround as well.


Andreas

[Niels Thykier]

Post by Andreas Beckmann
[ added Guillem to the recipients, he is the expert for trigger cycles ]
[ added Doko to the recipients, since #863820 is a possible solution ]
[ therefore resending a full-quote ]

Post by Cyril Brulebois
Package: upgrade-reports
Severity: critical
Justification: makes upgrade from stable abort

[...]

Switching desktop-file-utils or/and shared-mime-info to -noawait
triggers does not solve the problem.

Hi,

Guillem and I have been talking about this over IRC and have a theory.

Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.

* If this holds, then changing the desktop-file-utils and
shared-mime-info triggers *in stable* to -noawait should make the
problem go away.

* I realise it is unfeasible to implement in Debian by Saturday, but
it would help us understand the root cause of the problem.
- Tests to confirm/disprove this would be very welcome.

If this holds, then to fully resolve the problem, we will need 3 things:

* A stable update to jessie for these two packages migrating to
-noawait.

* An upload targeting stretch for these two packages migrating them to
-noawait.

* A "big magic hammer" work around for stretch for r0
- OR a release-notes remark to pull upgrades from jessie
- OR ...

Obviously, the above possibly ignores some time constraints.
Furthermore, the only thing remotely resembling a "big magic hammer"
atm. seems to be #863820, which we are unwilling to do with such short
notice[1].
I know that there was an upload to undo the changes in the java
packages, but AFAIUI, it basically means that Java will not be upgraded.
The user would explicitly have to install Java 8 and then uninstall the
now unsupported Java 7 - that seems very unhelpful to me.

Thanks,
~Niels

[1] We do have an another possible "big magic hammer", but it is even
less pretty and is called "Pre-Depends". As I recall, Pre-Depends are
handled specially by apt so it runs them in smaller bundles before the
main upgrade. We can almost certainly abuse this to work around the issue.

However, I hardly imagine that most of you will applaud that.

[gregor herrmann]

Post by Niels Thykier
Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.
* If this holds, then changing the desktop-file-utils and
shared-mime-info triggers *in stable* to -noawait should make the
problem go away.
* I realise it is unfeasible to implement in Debian by Saturday, but
it would help us understand the root cause of the problem.
- Tests to confirm/disprove this would be very welcome.

I think I can't confirm this theory. What I did:

A) Round 1: reproduce the problem:

1) enter a jessie cowbuilder chroot
2) rm /etc/apt/apt.conf.d/15pbuilder # which turns off installing recommends
3) apt-get install task-gnome-desktop
4) sed -i -e 's/jessie/stretch/g' /etc/apt/sources.list
5) apt-get update
6) apt-get upgrade
7) apt-get dist-upgrade

Result:

(Reading database ... 132342 files and directories currently installed.)
Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing tzdata-java (2017b-0+deb8u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
dpkg: cycle found while processing triggers:
chain of packages whose triggers are or may be responsible:
gnome-menus -> desktop-file-utils
packages' pending triggers which are or may be unresolvable:
gnome-menus: /usr/share/applications
shared-mime-info: /usr/share/mime/packages
desktop-file-utils: /usr/share/applications
mime-support: /usr/share/applications
dpkg: error processing package gnome-menus (--remove):
triggers looping, abandoned
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)


B) Round 2: play with triggers:

Same as above, except that after 3) I did

3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers

and after 6) I again ran

6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers


Result:

(Reading database ... 132342 files and directories currently installed.)
Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing tzdata-java (2017b-0+deb8u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
dpkg: cycle found while processing triggers:
chain of packages whose triggers are or may be responsible:
gnome-menus -> desktop-file-utils
packages' pending triggers which are or may be unresolvable:
gnome-menus: /usr/share/applications
shared-mime-info: /usr/share/mime/packages
desktop-file-utils: /usr/share/applications
mime-support: /usr/share/applications
dpkg: error processing package gnome-menus (--remove):
triggers looping, abandoned
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)

# for p in desktop-file-utils shared-mime-info gnome-menus mime-support ; do echo $p:; cat /var/lib/dpkg/info/$p.triggers; done
desktop-file-utils:
interest-noawait /usr/share/applications
shared-mime-info:
interest-noawait /usr/share/mime/packages
gnome-menus:
interest-noawait /usr/share/applications
interest-noawait gmenucache
mime-support:
interest-noawait /usr/lib/mime/packages
interest-noawait /usr/share/applications


Cheers,
gregor

[Guillem Jover]

Hi!

Post by gregor herrmann

Post by Niels Thykier
Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.
* If this holds, then changing the desktop-file-utils and
shared-mime-info triggers *in stable* to -noawait should make the
problem go away.
* I realise it is unfeasible to implement in Debian by Saturday, but
it would help us understand the root cause of the problem.
- Tests to confirm/disprove this would be very welcome.

1) enter a jessie cowbuilder chroot
2) rm /etc/apt/apt.conf.d/15pbuilder # which turns off installing recommends
3) apt-get install task-gnome-desktop
4) sed -i -e 's/jessie/stretch/g' /etc/apt/sources.list
5) apt-get update
6) apt-get upgrade
7) apt-get dist-upgrade

[…]

Post by gregor herrmann
Same as above, except that after 3) I did
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
and after 6) I again ran
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers

The important file to modify in addition is actually
/var/lib/dpkg/triggers/File, "/noawait" would need to be appended to
the relevant lines. The individual .triggers files for this scenario are
pretty much just parsed on unpack and removals.

Thanks,
Guillem

[gregor herrmann]

Post by Guillem Jover

Post by gregor herrmann
Same as above, except that after 3) I did
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
and after 6) I again ran
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers

The important file to modify in addition is actually
/var/lib/dpkg/triggers/File, "/noawait" would need to be appended to
the relevant lines. The individual .triggers files for this scenario are
pretty much just parsed on unpack and removals.

Oops, that's what you get when amateurs dabble around.
Thanks for filling in my incomplete knowledge!

Ok, let's try again:

1) enter a jessie cowbuilder chroot
2) rm /etc/apt/apt.conf.d/15pbuilder
3) apt-get install task-gnome-desktop
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
3b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File

4) sed -i -e 's/jessie/stretch/g' /etc/apt/sources.list
5) apt-get update
6) apt-get upgrade
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
6b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File
7) apt-get dist-upgrade

Result:

(Reading database ... 132342 files and directories currently installed.)
Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing tzdata-java (2017b-0+deb8u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
dpkg: cycle found while processing triggers:
chain of packages whose triggers are or may be responsible:
gnome-menus -> desktop-file-utils
packages' pending triggers which are or may be unresolvable:
gnome-menus: /usr/share/applications
shared-mime-info: /usr/share/mime/packages
desktop-file-utils: /usr/share/applications
mime-support: /usr/share/applications
dpkg: error processing package gnome-menus (--remove):
triggers looping, abandoned
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)


It might well be that I again did something wrong. OTOH, this seems to
match what anbe wrote of his similar tests some minutes ago.


(Disclaimer:
I don't think this is any help for the imminent release, I was just
curious about this trigger issue and thought I'd try it out.
And since, according to his last mail, KiBi's "fix" on the java side
seems to allow for a working upgrade, there's probably also no
urgency to dig deeper into this trigger puzzle.)


Cheers,
gregor

[Andreas Beckmann]

Post by gregor herrmann
(Reading database ... 132342 files and directories currently installed.)
Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing tzdata-java (2017b-0+deb8u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
gnome-menus -> desktop-file-utils
gnome-menus: /usr/share/applications
shared-mime-info: /usr/share/mime/packages
desktop-file-utils: /usr/share/applications
mime-support: /usr/share/applications
triggers looping, abandoned
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for mime-support (3.60) ...
gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)
It might well be that I again did something wrong. OTOH, this seems to
match what anbe wrote of his similar tests some minutes ago.

It was easier than I thought to do this with piuparts, and I can confirm
gregors test result with "real packages".
Full log, including dpkg trigger debug info:
https://people.debian.org/~anbe/864597/tgd.101-dfu89-smi89.log

it used for jessie:
desktop-file-utils_0.22-1+deb8noawait_amd64.deb
shared-mime-info_1.3-1+deb8noawait_amd64.deb
and for stretch:
desktop-file-utils_0.23-1+test_amd64.deb
shared-mime-info_1.8-1+test_amd64.deb
(all with s/interest/interest-noawait/ in debian/triggers)

Andreas

[Guillem Jover]

Hi!

Post by gregor herrmann

Post by Guillem Jover

Post by gregor herrmann
Same as above, except that after 3) I did
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
and after 6) I again ran
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers

The important file to modify in addition is actually
/var/lib/dpkg/triggers/File, "/noawait" would need to be appended to
the relevant lines. The individual .triggers files for this scenario are
pretty much just parsed on unpack and removals.

Oops, that's what you get when amateurs dabble around.
Thanks for filling in my incomplete knowledge!

It's very possible that the internal documentation at
/usr/share/doc/dpkg-dev/triggers.txt.gz is not very clear on this
subject, and I'd be happy to clarify it!

Post by gregor herrmann
1) enter a jessie cowbuilder chroot
2) rm /etc/apt/apt.conf.d/15pbuilder
3) apt-get install task-gnome-desktop
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
3b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File
4) sed -i -e 's/jessie/stretch/g' /etc/apt/sources.list
5) apt-get update
6) apt-get upgrade
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
6b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File
7) apt-get dist-upgrade

[…]

Hmm, and thanks for the retest!

Post by gregor herrmann
It might well be that I again did something wrong. OTOH, this seems to
match what anbe wrote of his similar tests some minutes ago.

So, I didn't try to reproduce the error first, and went directly for
a hunch. I followed your recipe, but in steps 3* and 6* I also amended
hicolor-icon-theme. And that seemed to fix the upgrade. Take into
account I've not had the time to analyze the dependency chain, nor if
my initial setup (not using cowbuilder, etc) matches, so maybe this is
an error in procedure. If someone else can reproduce, that'd be helpful,
otherwise I might retry tomorrow.

If this does fix it, it might be helpful also to try just amending the
triggers in the stretch "mock", instead of in the jessie one, to see if
such a fixed package would be enough.

Thanks,
Guillem

[Cyril Brulebois]

Post by Guillem Jover
So, I didn't try to reproduce the error first, and went directly for
a hunch. I followed your recipe, but in steps 3* and 6* I also amended
hicolor-icon-theme. And that seemed to fix the upgrade. Take into
account I've not had the time to analyze the dependency chain, nor if
my initial setup (not using cowbuilder, etc) matches, so maybe this is
an error in procedure. If someone else can reproduce, that'd be helpful,
otherwise I might retry tomorrow.
If this does fix it, it might be helpful also to try just amending the
triggers in the stretch "mock", instead of in the jessie one, to see if
such a fixed package would be enough.

I guess it would make sense to try and find out various sets of packages
to install in jessie that make one run into the triggers loop while
upgrading, then make sure that updating the 3 packages in jessie+stretch
(with the interest→interest-noawait update) indeed resolve all loops.

If we have some confirmation on this side, we might then proceed to
updating them through oldstable-p-u and stable-p-u?


Independently, I think we really need to have something in the release
notes; not about openjdk-7 at all, but about what to do to fix or work
around such a triggers loop when one runs into it. Triggering one with a
basic gnome dist-upgrade was probably some kind of luck, but I suspect
there are some/many other ways to run into similar issues



KiBi.

[gregor herrmann]

Post by Guillem Jover

Post by gregor herrmann
1) enter a jessie cowbuilder chroot
2) rm /etc/apt/apt.conf.d/15pbuilder
3) apt-get install task-gnome-desktop
3a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
3b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File
4) sed -i -e 's/jessie/stretch/g' /etc/apt/sources.list
5) apt-get update
6) apt-get upgrade
6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info}.triggers
6b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' /var/lib/dpkg/triggers/File
7) apt-get dist-upgrade

So, I didn't try to reproduce the error first, and went directly for
a hunch. I followed your recipe, but in steps 3* and 6* I also amended
hicolor-icon-theme. And that seemed to fix the upgrade. Take into
account I've not had the time to analyze the dependency chain, nor if
my initial setup (not using cowbuilder, etc) matches, so maybe this is
an error in procedure. If someone else can reproduce, that'd be helpful,
otherwise I might retry tomorrow.

I now took the same steps as above, just adding hicolor-icon-theme in
3a/3b/6a/6b:

3a/6a) sed -i -e 's/interest /interest-noawait /' /var/lib/dpkg/info/{desktop-file-utils,shared-mime-info,hicolor-icon-theme}.triggers
3b/6b) sed -i -e 's;desktop-file-utils;desktop-file-utils/noawait;' -e 's;shared-mime-info;shared-mime-info/noawait;' -e 's;hicolor-icon-theme;hicolor-icon-theme/noawait;' /var/lib/dpkg/triggers/File

Result: The dist-upgrade finishes successfully.

Post by Guillem Jover
If this does fix it, it might be helpful also to try just amending the
triggers in the stretch "mock", instead of in the jessie one, to see if
such a fixed package would be enough.

Interesting idea. Let's try (i.e. the same steps, but leaving out 3a
and 3b):

Result: Works as well.


But. But.
But this is with the "fixed" ca-certificates-java (20170531+nmu1) so
we wouldn't see the failure in the first place. *sigh*


Ok, so let's do this again, this time with a local repo added which
contains ca-certificates-java 20170531+nmu2 (just 20170531 but with a
higher version). (Side note: ca-certificates-java gets already
updated in 6, i.e. in "upgrade", before the "dist-upgrade".)

Option 1, with 3a/b: The dist-upgrade fails with the well-known
error:

(Reading database ... 132342 files and directories currently installed.)
Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
Removing tzdata-java (2017b-0+deb8u1) ...
Processing triggers for hicolor-icon-theme (0.15-1) ...
dpkg: cycle found while processing triggers:
chain of packages whose triggers are or may be responsible:
gnome-menus -> desktop-file-utils
packages' pending triggers which are or may be unresolvable:
gnome-menus: /usr/share/applications
shared-mime-info: /usr/share/mime/packages
desktop-file-utils: /usr/share/applications
mime-support: /usr/share/applications
dpkg: error processing package gnome-menus (--remove):
triggers looping, abandoned
Processing triggers for desktop-file-utils (0.23-1) ...
Processing triggers for mime-support (3.60) ...
Errors were encountered while processing:
gnome-menus
E: Sub-process /usr/bin/dpkg returned an error code (1)

At this point all triggers should be in -noawait state:

# grep interest /var/lib/dpkg/info/{gnome-menus,shared-mime-info,desktop-file-utils,mime-support,hicolor-icon-theme}.triggers
/var/lib/dpkg/info/gnome-menus.triggers:interest-noawait /usr/share/applications
/var/lib/dpkg/info/gnome-menus.triggers:interest-noawait gmenucache
/var/lib/dpkg/info/shared-mime-info.triggers:interest-noawait /usr/share/mime/packages
/var/lib/dpkg/info/desktop-file-utils.triggers:interest-noawait /usr/share/applications
/var/lib/dpkg/info/mime-support.triggers:interest-noawait /usr/lib/mime/packages
/var/lib/dpkg/info/mime-support.triggers:interest-noawait /usr/share/applications
/var/lib/dpkg/info/hicolor-icon-theme.triggers:interest-noawait /usr/share/icons/hicolor

# egrep "(gnome-menus|shared-mime-info|desktop-file-utils|mime-support|hicolor-icon-theme)" /var/lib/dpkg/triggers/File
/usr/lib/mime/packages mime-support/noawait
/usr/share/applications mime-support/noawait
/usr/share/icons/hicolor hicolor-icon-theme/noawait
/usr/share/mime/packages shared-mime-info/noawait
/usr/share/applications desktop-file-utils/noawait
/usr/share/applications gnome-menus/noawait


Cheers,
gregor

[Cyril Brulebois]

Post by gregor herrmann
But. But.
But this is with the "fixed" ca-certificates-java (20170531+nmu1) so
we wouldn't see the failure in the first place. *sigh*

Yeah, ISTR Niels warned against this.

FWIW, instead of going for a locally hacked version of a given package,
you can use snapshot.debian.org URL with a specific timestamp in your
sources.list, so that your tests no longer depend on archive changes.

See http://snapshot.debian.org/ for usage, and recent timestamps (e.g.
June) in http://snapshot.debian.org/archive/debian/?year=2017&month=6


KiBi.

[Andreas Beckmann]

Post by gregor herrmann

Post by Guillem Jover
So, I didn't try to reproduce the error first, and went directly for
a hunch. I followed your recipe, but in steps 3* and 6* I also amended
hicolor-icon-theme. And that seemed to fix the upgrade. Take into
If this does fix it, it might be helpful also to try just amending the
triggers in the stretch "mock", instead of in the jessie one, to see if
such a fixed package would be enough.

Ok, so let's do this again, this time with a local repo added which
contains ca-certificates-java 20170531+nmu2 (just 20170531 but with a
higher version). (Side note: ca-certificates-java gets already
updated in 6, i.e. in "upgrade", before the "dist-upgrade".)
Option 1, with 3a/b: The dist-upgrade fails with the well-known

I could confirm this with piuparts, i.e. making hicolor-icon-theme
noawait (in stretch or stretch+jessie) as well does not change the game.
Full logs with dpkg trigger debug output available, if someone is
interested.

Another observation I made is that dpkg was already upgraded during the
'apt-get upgrade' phase, so the dpkg reporting the trigger cycle is the
one from stretch. Improving debug output there or making trigger related
fixed should be possible.


Andreas

[Cyril Brulebois]

Post by Niels Thykier
Guillem and I have been talking about this over IRC and have a theory.
Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.
* If this holds, then changing the desktop-file-utils and
shared-mime-info triggers *in stable* to -noawait should make the
problem go away.
* I realise it is unfeasible to implement in Debian by Saturday, but
it would help us understand the root cause of the problem.
- Tests to confirm/disprove this would be very welcome.

I can run tests but how is this going to help with a release on saturday?

Anyway, your analysis seems wrong, see below.

Post by Niels Thykier
* A stable update to jessie for these two packages migrating to
-noawait.
* An upload targeting stretch for these two packages migrating them to
-noawait.
* A "big magic hammer" work around for stretch for r0
- OR a release-notes remark to pull upgrades from jessie
- OR ...
Obviously, the above possibly ignores some time constraints.
Furthermore, the only thing remotely resembling a "big magic hammer"
atm. seems to be #863820, which we are unwilling to do with such short
notice[1].
I know that there was an upload to undo the changes in the java
packages, but AFAIUI, it basically means that Java will not be upgraded.

That's wrong. The default-jre upgrade still happens, which means java 8
is installed; that just means java 7 can stay around for a little while
longer while the dist-upgrade is happening.

Post by Niels Thykier
The user would explicitly have to install Java 8 and then uninstall the
now unsupported Java 7 - that seems very unhelpful to me.

Nope. From the gnome upgrade log with ca-certificates-java “fixed”:
| Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
| 

| Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
| 

| Setting up openjdk-8-jre-headless:amd64 (8u131-b11-2) ...
| 

| Setting up default-jre-headless (2:1.8-58) ...
| 

| Setting up openjdk-8-jre:amd64 (8u131-b11-2) ...
| 

| Setting up default-jre (2:1.8-58) ...

While I can't say for sure my ca-certificates-java upload will fix all
upgrade paths, I'm quite confident the current upgrade paths is utterly
broken, and is very much less so afterwards, with no known downsides.

Again, I'm not going to pretend I'll be calling the shots here. But I
really fail to see a simpler way to get a net positive gain on this
topic.


KiBi.

[Niels Thykier]

Post by Cyril Brulebois

Post by Niels Thykier
Guillem and I have been talking about this over IRC and have a theory.
Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.
* If this holds, then changing the desktop-file-utils and
shared-mime-info triggers *in stable* to -noawait should make the
problem go away.
* I realise it is unfeasible to implement in Debian by Saturday, but
it would help us understand the root cause of the problem.
- Tests to confirm/disprove this would be very welcome.

I can run tests but how is this going to help with a release on saturday?

Hi,

@KiBi: Thanks for correcting me below. :)
@All: For testing the theory.

Re the question above: It would not help on Saturday, but we are doomed
to repeat this problem again if we do not find and fix it at its root.

Post by Cyril Brulebois
[....]

Post by Niels Thykier
The user would explicitly have to install Java 8 and then uninstall the
now unsupported Java 7 - that seems very unhelpful to me.

| Removing openjdk-7-jre:amd64 (7u111-2.6.7-1~deb8u1) ...
| …
| Removing openjdk-7-jre-headless:amd64 (7u111-2.6.7-1~deb8u1) ...
| …
| Setting up openjdk-8-jre-headless:amd64 (8u131-b11-2) ...
| …
| Setting up default-jre-headless (2:1.8-58) ...
| …
| Setting up openjdk-8-jre:amd64 (8u131-b11-2) ...
| …
| Setting up default-jre (2:1.8-58) ...
While I can't say for sure my ca-certificates-java upload will fix all
upgrade paths, I'm quite confident the current upgrade paths is utterly
broken, and is very much less so afterwards, with no known downsides.
[...]
KiBi.

Given it does upgrade openjdk-8, then it does seem like a viable work
around for stretch. Ideally, we would also have this work if people
does not have default-jre, but I agree that we can do r0 without it.

* I will unblock and urgent ca-certificates-java/20170531+nmu1 now
- If you do further testing, please remember to "undo" this change
or your test case may magically be "fixed" for the wrong reason :)

* I will write a note in the release notes for the people who have
openjdk-7 without default-jre tonight.
- Feel free to beat me to it.

* For buster, I will do an archive-wide sweep for getting rid of
interest triggers. It won't help with "stretch -> buster", so
we need to discuss how to fix that (but that can happen after
r0).

Thanks,
~Niels

[Andreas Beckmann]

Post by Niels Thykier
Guillem and I have been talking about this over IRC and have a theory.
Basically, jessie's verison of desktop-file-utils and shared-mime-info
have "-await" triggers (implicit) which will push other packages into a
"TRIGGER_PENDING" state.
Once they are in that state, the "damage" is done and those other
packages will no longer satisfy dependencies until the trigger has been
processed. Notably, dpkg is unable to /undo/ this state even if the
trigger changes from -await to -noawait during the upgrade.

I think this theory is wrong. (Unless dpkg keeps packages in
trigger-pending state *after* a successful apt-get upgrade.)

I tested earlier this week the following upgrade path in piuparts:
Upgrade from jessie task-gnome-desktop with --install-recommends to
stretch (but with desktop-file-utils and shared-mime-info switched to
interest-noawait triggers in stretch).
The first (apt-get upgrade) phase upgraded both of these packages as
well as gnome-menues and finished with success. I assume there are no
pending triggers left at this point, and the 3 interesting packages have
noawait triggers.
The second phase (apt-get dist-upgrade) still runs into the trigger cycle.


Andreas

[Bill Allombert]

Post by Cyril Brulebois
Package: upgrade-reports
Severity: critical
Justification: makes upgrade from stable abort
Hi,
Regression spotted by Pere in some debian-edu job, but also seen since
the 2nd of June in normal gnome chroot installation then upgrade from
https://jenkins.debian.net/view/edu_devel/job/chroot-installation_jessie_install_education-desktop-gnome_upgrade_to_stretch/
https://jenkins.debian.net/job/chroot-installation_jessie_install_gnome_upgrade_to_stretch/
I've managed to reproduce it locally with basically a debootstrap of
jessie, installation of gnome, then switch sources.list from jessie to
stretch, then update & upgrade & dist-upgrade.
- 20170601T212625Z = last timestamp found to be OK;
- 20170602T033358Z = first timestamp to be KO.

For what it is worth, according to
<http://debian.semistable.com/unstable_diffs.txt>
a circular dependency between ca-certificates-java and
openjdk-8-jre-headless has been added on Thu Jun 1 01:02:00 CEST 2017

Cheers,

[Andreas Beckmann]

Followup-For: Bug #864597

attached is a upgrade log where I reproduced the failure in piuparts
with dpkg trigger debugging enabled


Andreas