Cracking the Rosetta Stone - Continued

A quest to automate some manual workflows, turned into a deep dive into container manifests, cross-arch libs, compilation, and of course SELinux. Second entry, after this one.

The Environment

Forgejo’s action system is quite similar to GitHub’s, except that you generally set up your own runner instances when self-hosting. The default runs-on: docker setup uses a Alpine Linux container with preinstalled Nodejs, and is good enough in interfacing with the underlying Forgejo infrastructure.

In this very case, however, it is not enough. rpmbuild requires an underlying rpm database on the system, to track dependencies. Alpine Linux does have rpm as a community package, but having two package managers running side by side is asking for trouble, so I ended up with a CentOS Stream container1.

Some interesting moments before the actual build even starts:

  • rpmlint is only available in EPEL, so that brings the CRB repo into the mix as well.
  • golang includes git, subversion and mercurial as weak dependencies, which… makes sense probably but feels a bit redundant.

With the image built, I pushed it to my Forgejo instance, which reports

"Architecture": "amd64",

Oh right. I then did an aarch64 build from my RHEL instance and pushed. Did a pull from my amd64 machine just to be sure, and found

"Architecture": "arm64",

Same tag apparently overrides each other, which is… not going to work. The solution turned out to be container manifests, which allows multiple images under the same tag that can be automatically resolved when pulling.

Some rebuilds later, Forgejo was able to report correctly that the tag points to checksummed builds. However, each build still overrides one from another architecture, which means I’ll have to build for both archs in one go.

The bigger crossing

The easier solution on a search is qemu-user-static, which just… emulates. Installing this on my amd64 machine allowed me to push a dual build. But can we do any better?

I mentioned last time that my RHEL instance is a virtual machine running on a Mac mini. While Apple Silicon is entirely aarch64, Apple offers Rosetta for both macOS binaries and Linux ones. Linux VMs can access the component by mounting a virtiofs partition and registering it as a binary format handler.

The mount is a one-liner in /etc/fstab, but the registeration is… interesting. Apple’s official documentation says

mkdir /tmp/mountpoint
sudo mount -t virtiofs EXAMPLE_TAG /tmp/mountpoint
ls /tmp/mountpoint rosetta
sudo /usr/sbin/update-binfmts --install rosetta /tmp/mountpoint/rosetta \
  --magic "\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00" \
  --mask "\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff" \
  --credentials yes --preserve yes --fix-binary yes

Except that update-binfmts does not exist on RHEL2. Or any Fedora derivation, for that matter.

Kernel documentation offers a manual way to trigger the command: echo a string of :name:type:offset:magic:mask:interpreter:flags to /proc/sys/fs/binfmt_misc/register.

Converting from the update-binfmts command to the manual one is straightforward enough:

echo ':rosetta:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\x3e\x00:\xff\xff\xff\xff\xff\xfe\xfe\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/tmp/mountpoint/rosetta:PCF' | sudo tee /proc/sys/fs/binfmt_misc/register

To map the different pieces one by one:

manualupdate-binfmts
M--magic
PCF--credentials yes --preserve yes --fix-binary yes

For some quick testing, I compiled a simple C program for x86_64, and tried to run it.

$ ./test
rosetta error: failed to open elf at /lib64/ld-linux-x86-64.so.2
 Trace/breakpoint trap (core dumped)

/lib64/ld-linux-x86-64.so.2 is a standard-enforced, hardcoded linker location. It is provided directly by glibc3, which makes it practically impossible to install alongside the real one without breaking the system4.

That file itself shouldn’t conflict though…?

dnf download --forcearch=x86_64 glibc
rpmdev-extract glibc-[version].x86_64.rpm
sudo cp glibc-[version].x86_64/usr/lib64/ld-linux-x86-64.so.2 /usr/lib64/

This allows the same program to move ever so slightly further:

$ ./test
./test: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

libc.so.6 in fact exists in /usr/lib64, just not on the arch we want. Replacing this one will… almost certainly break things.

The linker does mention a parameter and an env variable to force a specific library load path.

$ LD_LIBRARY_PATH=/home/linus_xu/glibc-[version].x86_64/usr/lib64 ./test
x86_64 app running

Ugly, but definitely working.

The security

Now that Rosetta is known to be properly working, I set out to testing container builds.

$ podman run --arch=amd64 quay.io/centos/centos:stream10 uname -a
(no output)
$ echo $?
139

Huh?

The issue somehow turned out to be… SELinux.

SELinux is preventing /usr/local/libexec/rosetta/rosetta from map access on the file /usr/local/libexec/rosetta/rosetta.

Ummmm yes? Later parts of the report explains things slightly better.

Source Context                system_u:system_r:container_t:s0:c378,c561
Target Context                system_u:object_r:unlabeled_t:s0
Target Objects                /usr/local/libexec/rosetta/rosetta [ file ]

A boundary cross from within a container to something unbound yet, which… makes sense. I chose /usr/local/libexec for this exact reason.

$ sudo restorecon -Frv /usr/local/libexec/rosetta
restorecon: Could not set context for /usr/local/libexec/rosetta:  Read-only file system
restorecon: Could not set context for /usr/local/libexec/rosetta/rosetta:  Read-only file system
restorecon: Could not set context for /usr/local/libexec/rosetta/rosettad:  Read-only file system

Remount as read-write? Remember to unregister first or you get ‘target is busy’ errors.

$ sudo restorecon -Frv /usr/local/libexec/rosetta
restorecon: Could not set context for /usr/local/libexec/rosetta:  Operation not permitted
restorecon: Could not set context for /usr/local/libexec/rosetta/rosetta:  Operation not permitted
restorecon: Could not set context for /usr/local/libexec/rosetta/rosettad:  Operation not permitted

Yikes. So --security-opt="label=disable" on building it is.

The rebase

Yeeting SELinux allows the x86_64 image itself to run. So now we get to actually building the image. rpmlint is in EPEL, so we dnf install the repo RPM:

Errors during downloading metadata for repository 'baseos':
  - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http&countme=1 [Insufficient randomness]
  - Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http [Insufficient randomness]
Error: Failed to download metadata for repo 'baseos': Cannot prepare internal mirrorlist: Curl error (35): SSL connect error for https://mirrors.centos.org/metalink?repo=centos-baseos-10-stream&arch=x86_64&protocol=https,http [Insufficient randomness]

Manually curl-ing the RPM reports the same error. This leads to an interesting dilemma because openssl is not preinstalled in the image, and I can’t download it to test stuff because… I can’t download anything without openssl libraries working.

This curl issue mentions the exact error, but isn’t helpful here since their solution is to… upgrade packages. It also points to a test program, which I adapted a little bit to report errors after each call.

#include <openssl/rand.h>
#include <openssl/err.h>
#include <stdio.h>

void report_error() {
    puts("Errors in queue:");
    int i;
    char err[256];
    while (i = ERR_get_error()) {
        ERR_error_string(i, err);
        printf("%s\n", err);
    }
    ERR_clear_error();
}

int main(int argc, char **argv) {
    int r1 = RAND_poll();
    printf("RAND_poll() said %d\n", r1);
    report_error();
    unsigned char byte;
    int r2 = RAND_bytes(&byte, 1);
    printf("RAND_bytes() said %d\n", r2);
    report_error();
    int r3 = RAND_status();
    printf("RAND_status() said %d\n", r3);
    report_error();
    return 0;
}

Compiling this as a x86_64 program and running it with Rosetta (downloading required x86_64 packages and adding LD_LIBRARY_PATH routes as necessary) reveals several more errors.

RAND_poll() said 1
Errors in queue:
error:0308010C:digital envelope routines::unsupported
error:12000090:random number generator::unable to fetch drbg
RAND_bytes() said 0
Errors in queue:
error:0308010C:digital envelope routines::unsupported
error:12000090:random number generator::unable to fetch drbg
RAND_status() said 0
Errors in queue:
error:0308010C:digital envelope routines::unsupported
error:12000090:random number generator::unable to fetch drbg

OpenSSL libs uses 1 as success and 0 as failure in this context, so it’s just… completely borked in this stage, in a sense.

All search matches for 0308010C concern either Windows or NodeJS.

The curl error itself, however, shed a light from a different angle thanks to this issue. While it is made against the Container project5, it is in fact more of an underlying issue with Rosetta’s emulation of some instructions. To cite a contributor’s comment:

Currently what is happening here:

  • TLS handshake succeeds until certificate signature verification
  • then OpenSSL’s crypto provider fails: error:030000EA:digital envelope routines::provider signature failure
  • x86_64 crypto operations (AES-NI, AVX assembly) don’t work correctly under Rosetta 2 emulation
  • Impact: Breaks all HTTPS connections → dnf can’t download packages

Root cause: AlmaLinux 10 uses OpenSSL 3.x with CPU-optimized crypto routines that are incompatible with cross-architecture emulation on Apple Silicon.

Wait. CPU-optimized operations. Only x86_64. Affects RHEL and derivatives starting from 10.

x86-64-v3?

I had some trouble compiling the test program explicitly against v3 and non-v3 side-by-side, but there is an easier solution. Fedora has still not moved to v3 yet.. Switching to a Fedora image may cost a little bit of package stability, but should require minimal changes otherwise.

FROM quay.io/fedora/fedora-minimal:latest

Remove all EPEL mentions (since they will be in the main repo for Fedora), and run a build.

It works.

The aftermath

The link above now points a working image. Two to be precise, but it should just pick the correct arch automatically on pull.

A factor that thankfully hasn’t bitten me here yet is the FIPS mode on RHEL. In a later situation where I was running an Ubuntu container, OpenSSL bailed out on out of memory errors. It turns out to be a red-herring output due to Ubuntu not supporting FIPS yet. Fedora derivatives all comply with the signal to enable without issue. In hindsight, it is quite possible that FIPS combined with x86-64-v3 had complicated the situation already…

On the host side of things, Apple stuff would pretty much be a stone wall, but I have raised this issue on UTM to look into the SELinux problem.

Footnotes

  1. I tried stream10-minimal but decided against it due to some missing features in microdnf, and ended up with stream10 instead but the point is moot anyways after switching to fedora-minimal, which still offers proper dnf. ↩︎
  2. A manual clone of the binfmt-support package reports missing libpipeline, which exists but is missing its devel package in RHEL. I took both approaches and can confirm that the result is identical. ↩︎
  3. RHEL’s default repo setup is hardcoded by architecture, which means you have to manually add a repo to see packages for another arch. The free Developer subscription allows access to both aarch64 and x86_64 repos. ↩︎
  4. This difference is sometimes coined “multi-arch” versus “multi-lib”. Debian does multi-arch by keeping .so files under triplet directories, which allows for coexistence of multiple glibc installations across archs, while Fedora and derivatives using multi-lib does not. Multi-lib is also not really intended to work across archs, with the primary current use case being i*86 support on x86_64. ↩︎
  5. The Container project seeks to make container workflows easier on macOS hosts. Unlike Podman, which works on macOS by running a Fedora CoreOS virtual machine to run containers in, container deploys each container as individual lightweight virtual machines, which should cut down on networking hoops and overhead. ↩︎