Reproducible builds improvements

[mlschroe]

Here's some thoughts about improving reproducible builds with rpm. The goal (for me) is to be able to reproduce a rpm given the source rpm.

We currently have the following switches:

• %source_date_epoch_from_changelog
• %use_source_date_epoch_as_buildtime
• %clamp_mtime_to_source_date_epoch

This is centered around the sources and not really what I have in mind. I would prefer to keep setting the build time to the current time for normal builds and clamp the mtime to the build time. To reproduce a build, I would need a way to set the build time (and the build host) to the values from the rpm I want to reproduce.

So, would a %clamp_mtime_to_buildtime macro make sense? Should the getBuildTime() function check if a %_buildtime macro is set?

(This is somewhat related to pull request #2880. Basically $NEW_SOURCE_DATE_EPOCH is the build time.)

Thoughts?

[pmatilai]

We have %_buildhost macro which seems like it should be paired with %_buildtime so adding that seems to make sense, but I guess the latter is missing because you can set it with SOURCE_DATE_EPOCH if you care. I'd totally fine with adding %_buildtime override if that means tossing the %use_source_date_epoch_as_buildtime overboard, the tangle of these switches doesn't make a whole lot of sense to me as it is.

%clamp_mtime_to_buildtime seems sensible to me, at least if it replaces %clamp_mtime_to_source_date_epoch. If you care about source_date_epoch, then you'll surely set buildtime to it, right?

(but, I bet I'm missing the finer nuances of the reproducable builds here again)

[pmatilai]

In other words, this all would make a whole lot more sense to me if there was a switch that decides where the buildtime gets set from (clock/macro/source_date_epoch), and then the clamping options relate to buildtime and not source_date_epoch. Also @ffesti mentioned elsewhere that maybe, instead of having all these crazy switches, we should just default to clamping the mtimes for example.

[pmatilai]

Forward-looking defaults aside...

Do you agree with the idea that there should be a single macro to set the buildtime source (clock/macro/source_date_epoch), and then have a separate flag for clamping mtimes to buildtime, or am I again missing some finer detail here? I've nothing against reproducible builds, it's this tangle of haphazard switches that annoys me.

Tagging in folks who I remember showing interest in reproducible builds in the past, feel free to include more if/when I missed someone: @JanZerebecki @Conan-Kudo @bmwiedemann @boklm

[bmwiedemann]

%clamp_mtime_to_buildtime seems sensible to me, at least if it replaces %clamp_mtime_to_source_date_epoch. If you care about source_date_epoch, then you'll surely set buildtime to it, right?

With our build system, passing constants into a build is much easier than passing in a variable %_buildtime. Not sure how that is with Fedora's or Mageia's build-system.
It would be nice to at least have both in parallel until everything is properly integrated.

[pmatilai]

With our build system, passing constants into a build is much easier than passing in a variable %_buildtime.

Sorry but I have no idea this means. What are these "constants" and how are they being passed to a build?

(a %_buildtime macro could be set either via rpmbuild command line --define or a macro file such as ~/.rpmmacros)

[bmwiedemann]

We have open-build-service (OBS) that uses obs-worker to call obs-build to call rpmbuild.
The current way documented in our wiki is to add a project-conf in OBS

Macros:
%source_date_epoch_from_changelog 1
%clamp_mtime_to_source_date_epoch 1
%use_source_date_epoch_as_buildtime 1
%_buildhost reproducible
:Macros

and the constants here are 1 and reproducible

[JanZerebecki]

I think @pmatilai meant having %source_buildtime with constants of either SOURCE_DATE_EPOCH, SOURCE_DATE_EPOCH_MTIME, macro or clock? I'm ok with that.

My preferred way would be as few macros or settings, but that is not as backwards compatible. Only reproducible builds. Build host field in rpm files is always set to reproducible. Build fails if neither last changelog date nor $SOURCE_DATE_EPOCH can be found to set build time. File mtimes are always set, either to build time or if $SOURCE_DATE_EPOCH_MTIME exists to that, no clamping. (SOURCE_DATE_EPOCH_MTIME is a replacement for the name NEW_SOURCE_DATE_EPOCH from my previous version of that patch, to be more descriptive. It also means we do not need to switch OLD and NEW around.)

My understanding is that @mlschroe prefers to have $SOURCE_DATE_EPOCH_MTIME set the build time and if it is set rpm would ignore but pass through $SOURCE_DATE_EPOCH. That has the downside that it is conceptually more complicated. It also doesn't nest cleanly, while we could more easily ignore mtimes in archives in a nested way, say for an rpm containing a zip, where a build picked up $SOURCE_DATE_EPOCH_MTIME for the zip mtimes. If a build picks up $SOURCE_DATE_EPOCH_MTIME and puts it in the content, say a build time field in an elf executable, we have the same problem again. To avoid that problem we can outright prevent nesting by having rpm copy $SOURCE_DATE_EPOCH_MTIME to the build time and unset it. I'm not sure if this is a good idea. The downside is that any nested archives can not be fixed in the same way, if the non-incremented mtimes in those become a problem. @mlschroe What do you think?

We should not use clamping mtimes, outright setting them has fewer possibilities for errors (like incorrect clock). Always setting the mtimes to build time like detailed above and having no setting for it, might work in practice without breaking anything. @pmatilai However are you ok to make such a change in a normal rpm release?

Same question for the other changes?

If only OpenSUSE is using those settings that would make at least removing those we stop using easy. Is anyone else using any?

Meanwhile testing this in OpenSUSE is making progress, I hope I fixed the last rpm that has had no changelog.

[pmatilai]

I wont claim to have digested all that, but just a high level confirmation: what I really would like to see is a coherent (re)design for reproducible builds, where you basically just flick it on and be done with it, whereas the existing flags reflect the organic growth process over the years and still much of the assembly is left to the user. And, to me it's not a big deal if its incompatible to older releases because this is something you set at a buildsystem level rather than per-spec thing. Such a change could of course only go into a new major release and not for backporting.

Edit: and yes, I was talking about a macro to configure the clock source, and the name of the clock source is of course constant. You'd only need to specify an explicit time if that's what you specifically configured. Or something like that. The NEW/MTIME thing I've yet to internalize at all.

[Conan-Kudo]

One of the reasons for the knobs is that not all of these settings are fully useful for "reproducibility" and some of these harm traceability and debugging.

For example, forcing the build host to reproducible does not provide much value if you are able to do comparisons while stripping/ignoring specific RPM header values and makes it harder to determine when something weird is happening as a result of a build host in real-world debugging efforts. This is similarly true for clamping build times, and has the negative consequence of making it difficult for tools to sort packages when they were built with the same NVR.

Setting SOURCE_DATE_EPOCH from the changelog provides value because it influences how the build itself records timestamps. Clamping mtimes provides value because it eliminates variability from the payload.

Everything we do around "reproducible builds" needs to be viewed with the lens of handling this balance.

[Conan-Kudo]

Clamping the mtime to buildtime has its own negative consequence too, because it makes it harder to reason reproducibility and it invalidates reproducibility in practice because every build will be different due to a variable clamp rather than an immutable clamp.

[bmwiedemann]

As long as a constant buildtime value is provided, it will also result in constant (reproducible) mtime.
Reproducible builds requires that you can produce bit-identical output from identical inputs. And inputs would include buildtime here (or whatever it is based on, e.g. changelog date).

[keszybz]

a coherent (re)design for reproducible builds, where you basically just flick it on and be done with it

I agree that the current set of options is a bit ad-hoc. But I think it's completely unrealistic to achieve a single switch, because different build systems and ways of managing the distro create different tradeoffs, and different people want to strike the balance between useful metadata and ease-of-reproducibility. Even with the 4 options listed above, Fedora and OpenSUSE disagree on two.

In Fedora, we had a discussion about setting %use_source_date_epoch_as_buildtime 1 and setting a fake %_buildhost value and many people were against. People find those informations useful, and very convenient to have them directly in the rpm, even though in they could be extracted from the build system in other ways. So this approach was abandoned, as you know, and we still have %use_source_date_epoch_as_buildtime 0 and use real hostnames.

As @Conan-Kudo mentioned above, we have to strip metadata anyway. At least HEADERIMMUTABLE, SIGSIZE,
SIGMD5, SHA1HEADER, SHA256HEADER are "irreproducible", and OPTFLAGS and PLATFORM are often different because a "random" noarch package is selected. If you have to strip/ignore/treat-in-a-special-way those fields, then it doesn't make much difference to also handle BUILDTIME and BUILDHOST in the same way. We have to strike a balance between having useful metadata and ease of reproducibility. Since bit-for-bit reproducibility is impossible with signatures, then I think the current balance of using real BUILDTIME and BUILDHOST is good.

We should not use clamping mtimes, outright setting them has fewer possibilities for errors (like incorrect clock).

Please, no. As @Conan-Kudo wrote above, this has clear downsides, because it forces constant churn between rebuilds. But it also destroys useful metadata, for example timestamps in documentation that specifies when the file was last modified. I'm sure there's a million other examples. And I find the worry about incorrect clocks rather unconvincing. In the current era of signatures being checked all the time, a build system or a developer making releases with a completely wrong time would create huge problems. If you find a case like this, fix it at the source, instead of forcing a heavy-weight workaround.

My understanding is that @mlschroe prefers to have $SOURCE_DATE_EPOCH_MTIME set the build time and if it is set rpm would ignore but pass through $SOURCE_DATE_EPOCH.

I don't think this has been thought through. $SOURCE_DATE_EPOCH now has the meaning of "the timestamp of the sources", i.e. the single timestamp that defines the inputs to the build. We can clamp all the files that are created after that timestamp to $SOURCE_DATE_EPOCH because we know that all those files are created from the original inputs, so their timestamps are not important, because they are reproducible and we can recreate them exactly independently of time. So $SOURCE_DATE_EPOCH has a clear meaning and any timestamps later than that are redundant.

The second problem is that we have a timestamp that is used to clamp mtimes. If we introduce a second timestamp and use that for clamping, we have to redesign/fix/update everything to use the new timestamp. Once we have gone through that churn, we are in almost the same situation, except that the variable name is different (and we also have a two variables with a slightly different meaning).

All that said, I think it's fine if a way to e.g. add new options how to clamp/set mtimes and whatnot. But please make those opt-in, so that different distros that have slightly different approaches can make their choices.

[pmatilai]

As @Conan-Kudo mentioned above, we have to strip metadata anyway. At least HEADERIMMUTABLE, SIGSIZE,
SIGMD5, SHA1HEADER, SHA256HEADER are "irreproducible"

Wait, what? If those differ then the packages do differ, so its not actually bit-per-bit identical. Which is what I've assumed reproducability to mean. This just goes to point out how completely different expectations people have. No wonder having a meaningful discussion about reproducable packages always seems so hard 😄

and OPTFLAGS and PLATFORM are often different because a "random" noarch package is selected. If you have to strip/ignore/treat-in-a-special-way those fields, then it doesn't make much difference to also handle BUILDTIME and BUILDHOST in the same way. We have to strike a balance between having useful metadata and ease of reproducibility. Since bit-for-bit reproducibility is impossible with signatures, then I think the current balance of using real BUILDTIME and BUILDHOST is good.

...but okay if we start down the filtering road (I don't disagree, I just clearly don't know what everybody's asssumptions are), then we arrive at this old discussion that never really went anywhere: #2023
Which of course turns the discussion into "which tags should be filtered", and because I'm quite sure not everybody thinks "release" is one of them, just for example.

Having a written definition of what "reproducability" means would help driving towards that goal. People clearly have very, very different ideas about it.

It's good to have this discussion, but as discussion is what this is, I'm moving this there. Once something concrete emerges, we can open ticket(s).

[bmwiedemann]

Over the last years I just used rpm --delsign to compare with my replication builds and was able to get bit-identical results (when avoiding some issues with fixes like be950ea, also #2343 ).
So I'd propose we have proper reproducible builds, if we

• can take official binaries, do a replication build, apply rpm --delsign and get the same hash for both rpms
• do two (unsigned) builds anytime anywhere and get the same hash for both rpms

Normalizing the hostname is not really necessary, because the replication build can set _buildhost to the same name, but so far the same was not true for buildtime and file mtimes.

We could come up with a replacement normalizer for rpm --delsign but the more complex it is, the harder it is to convince people that it does not hide relevant diffs. Debian's approach with strip-nondeterminism is to make the normalizer part of the build and publish the normalized files - so even if it normalized too much, it is fine.

[keszybz]

Wait, what? If those differ then the packages do differ, so its not actually bit-per-bit identical. Which is what I've assumed reproducability to mean. This just goes to point out how completely different expectations people have. No wonder having a meaningful discussion about reproducable packages always seems so hard 😄

I wrote a long piece about this here.

Over the last years I just used rpm --delsign to compare with my replication builds and was able to get bit-identical results

Whether we skip some fields when doing a comparison, or take an rpm and strip those fields, and then do the comparison, is just an implementation detail. In practice, users get rpms that are signed. Thus, the format that the users are interested in checking is by definition the signed rpm.

(The other end is interesting too. We generally talk about reproducibility in the sense of starting from srpms. This view originates in the Debian world where the source deb is the only common denominator. Packagers do not have to use git, they do not even have to use a vcs, and people do non-version-controlled binNMUs. Thus, when talking about the whole distro, starting from source debs is the only option. When working with rpms, at the technical level, getting the part from srpm until the binary rpm reproducible is challenging, so it makes sense for us to work on this part in the beginning. But what we actually want in the end is reproducibility of the full pipeline, i.e. starting from dist-git. I assume that adding the additional step where we generate the srpm from dist-git will be easy. And in dist-git, we want to have the upstream pristine tarballs, including a signature. In the end, ideally the user would be able to verify that the signed upstream tarball + a specific commit with our spec file leads to the rpms that they download from the mirror, reproducibly.)

Having a written definition of what "reproducability" means would help driving towards that goal.

I saw "reproducability" mentioned a few times. I assume it's not a typo, but I have no idea how it's supposed to be different from "reproducibility".

Please see the link above for my definition of "reproducibility".

[pmatilai]

I saw "reproducability" mentioned a few times. I assume it's not a typo, but I have no idea how it's supposed to be different from "reproducibility".

Eh. All my life I've been talking about reproducers, and reproducable bugs. And now builds. 😳
That misspelling is going to be hard to unlearn, but thanks for setting me straight there.

[Conan-Kudo]

It's also important to keep in mind the context of Debian style reproducibility: their archive format is a tarball with ar archives inside. That makes things very different for them than us.

[pmatilai]

Oh BTW, just a quick side-remark on this:

OPTFLAGS and PLATFORM are often different because a "random" noarch package is selected

OPTFLAGS shouldn't be even defined on noarch builds, much less included in the header. The former is hard to fix for various hysterical reasons, but the latter should be easy.

[keszybz]

If we could drop OPTFLAGS, that'd be great.

[bmwiedemann]

keszybz wrote:

any party can recreate copies of the artifacts that are identical except for the signatures and parts of metadata

I don't think it is a good idea to exclude metadata. One benefit that you can only get with bit-identical reproducibility is that you can list the one and only correct hash value of the build result. (that also works with signed rpms + delsign).
However with weaker variants, you always need another full rpm to compare to. I.e. for our 16k packages, instead of publishing a list of 16k hashes you then need to keep the full archive (100GB) to allow people to reproduce any package.

[Conan-Kudo]

I've been bitten enough times personally that I would rather not have BUILDHOST and BUILDTIME set to fake values.

[bmwiedemann]

When I normalize BUILDTIME with %use_source_date_epoch_as_buildtime, the signature still gives the real date. Is there a value in keeping both? e.g. this package rpm -ql has

Signature   : RSA/SHA256, 2024-02-26T12:00:49 UTC, Key ID 8adc26dbb49c2121
Source RPM  : strip-nondeterminism-1.13.1-33.9.src.rpm
Build Date  : 2023-07-28T16:19:49 UTC

Not overriding BUILDHOST is fine as it still allows easy verification.

[keszybz]

I think the signature must give the real date of when the signature was actually made. Setting a fake date would be very very icky, undermining the trust in the signing process and the holders of the signing key used in such a manner. At the more technical level, keys have a creation time, e.g. for Fedora the keys are created a few months in advance of the release (RPM-GPG-KEY-fedora-rawhide-x86_64 has Public key creation time - Tue Jan 24 22:22:52 CET 2023). This means that those keys cannot be used to create valid signatures for older packages, but at various points there certainly are packages that haven't been touched and have a SOURCE_DATE_EPOCH older than they key creation date. Also, at least in Fedora, packages are resigned with a newer signature for a new release. (E.g. a .f39 or .f40 package, when downloaded from the F41/rawhide repository, is not rebuilt, but is resigned with the F41 key.) So we need a signing time that is separate from BUILDTIME.

[bmwiedemann]

I did not mean to alter signing time - but keep it as it is (it is dropped by delsign anyway), while changing "Build Date" instead to something that does not vary in (changeless) rebuilds.

[Conan-Kudo]

I am aware of some tools that use RPMTAG_BUILDTIME to sort packages in various situations, especially if they have the same NVRA (ie. rebuilds). It is also useful in diagnostic purposes when trying to figure out a factor of breakage.

I would rather not falsify this tag.

[mlschroe]

I think this all has drifted away from the initial proposal. The goal was to be able to improve reproducibility of a given rpm by:

• adding a way to specify the buildtime
• adding an option to clamp the file mtimes to the buildtime

Disregarding the implementation details, do you all think this is worthwhile to have?

[keszybz]

Yes, I think both are worthwhile. But they must be opt-in.

[Conan-Kudo]

I don't think it's a good idea to offer. I am not convinced these knobs are a good idea for RPM to expose for any reason, especially reproducibility.

[pmatilai]

In case folks didn't notice the PR from @mlschroe : #2944