Build for Multi-Arch failing (arm64 / aarch64)

[sopeters]

I'm getting below error on a multi-arch build with the Dockerfile from this repo:

docker buildx build --platform linux/arm64,linux/amd64 .

Seems to be only on the arm64 architecture - saw some hints on haskell/cabal#7950 but haven't been able to solve this

#0 261.3 Installing library in /opt/ghc/8.10.7/lib/ghc-8.10.7/ghc-8.10.7
#0 263.4 "/opt/ghc/8.10.7/lib/ghc-8.10.7/bin/ghc-pkg" --force --global-package-db "/opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d" update rts/dist/package.conf.install
#0 265.6 ghc-pkg: Couldn't open database /opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d for modification: {handle: /opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d/package.cache.lock}: hLock: invalid argument (Invalid argument)
#0 265.6 make[1]: *** [ghc.mk:985: install_packages] Error 1
#0 265.6 make: *** [Makefile:51: install] Error 2

Any options to pass in configure flags?

[AlistairB]

👋 Hi, what architecture is your host on? Perhaps it is an issue building cross architectures for your host and docker.

The images do work for amd64 and arm64 so I'm not really sure how to help. They also work using https://github.com/uraimo/run-on-arch-action/ in CI (specifically using this branch) which uses emulation to build arm64 on the amd64 github action hosts.

[sopeters]

We're using x86_64 hosts for our CI/CD agents and I'm trying to use the 8.10.7-slim-buster docker image to compile and build multi-arch application container

[AlistairB]

Yeah I don't even get that far:

$ docker buildx build --platform linux/arm64 -f 8.10/slim-buster/Dockerfile .
[+] Building 6.4s (5/9)                                                                                                                                                                       
 => [internal] load build definition from Dockerfile                                                                                                                                     0.0s
 => => transferring dockerfile: 6.48kB                                                                                                                                                   0.0s
 => [internal] load .dockerignore                                                                                                                                                        0.0s
 => => transferring context: 2B                                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/debian:buster-slim                                                                                                                    3.5s
 => [1/6] FROM docker.io/library/debian:buster-slim@sha256:acba41442b388703260ef3f782793ad1ae945028ab12ad6840e7d80d4abbec8d                                                              2.4s
 => => resolve docker.io/library/debian:buster-slim@sha256:acba41442b388703260ef3f782793ad1ae945028ab12ad6840e7d80d4abbec8d                                                              0.0s
 => => sha256:b5c67df084c3cfb834911de189d63800a696458e5ce17982b15387110efd8a5b 1.48kB / 1.48kB                                                                                           0.0s
 => => sha256:8bd5501da4f4b37ad7488bd0eafa32d8927de3bf1e32747545810f2ca65429ed 25.91MB / 25.91MB                                                                                         1.8s
 => => sha256:acba41442b388703260ef3f782793ad1ae945028ab12ad6840e7d80d4abbec8d 1.85kB / 1.85kB                                                                                           0.0s
 => => sha256:1423af9f04f2198fb9bcf43571f61ba5fb9630e47410060372b2518dd46aa08a 529B / 529B                                                                                               0.0s
 => => extracting sha256:8bd5501da4f4b37ad7488bd0eafa32d8927de3bf1e32747545810f2ca65429ed                                                                                                0.4s
 => ERROR [2/6] RUN apt-get update &&     apt-get install -y --no-install-recommends         ca-certificates         curl         dpkg-dev         git         gcc         gnupg         0.4s
------
 > [2/6] RUN apt-get update &&     apt-get install -y --no-install-recommends         ca-certificates         curl         dpkg-dev         git         gcc         gnupg         g++         libc6-dev         libffi-dev         libgmp-dev         libnuma-dev         libtinfo-dev         make         netbase         xz-utils         zlib1g-dev &&     rm -rf /var/lib/apt/lists/*:
#0 0.324 exec /bin/sh: exec format error
------
error: failed to solve: executor failed running [/bin/sh -c apt-get update &&     apt-get install -y --no-install-recommends         ca-certificates         curl         dpkg-dev         git         gcc         gnupg         g++         libc6-dev         libffi-dev         libgmp-dev         libnuma-dev         libtinfo-dev         make         netbase         xz-utils         zlib1g-dev &&     rm -rf /var/lib/apt/lists/*]: exit code: 1

I'm interested in ways the images might be able to handle this better, but I don't think they can do much to resolve this. amd64 and arm64 do build and work fine. It seems to be an issue specifically building cross architectures.

The experts on the ghc build configure step are ghc developers, so you could ask there. However, it could just be the docker cross arch compilation stuff acting weirdly and there is not much ghc can do. You might need to drill into the failure further to be able to pinpoint why it is failing and to show it is something that ghc or the haskell images can handle better.

[sopeters]

It's the ghc-pkg command that seems to fail on the lock file. I tried to delete the lock file and also change permissions before I run it but same issue.

/opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d/package.cache.lock exists and is writable but the error persists

Couldn't open database /opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d for modification: {handle: /opt/ghc/8.10.7/lib/ghc-8.10.7/package.conf.d/package.cache.lock}: hLock: invalid argument (Invalid argument)

Also tried ghc-pkg recache and same error

[AlistairB]

My other suspicion is that it is installing the amd64 version of GHC. In the past running dpkg-architecture --query DEB_BUILD_GNU_CPU trying to cross compile it still returned amd64 even when using something like --platform linux/arm64 (which does cause it to pull the arm64 version of debian). If you can confirm what dpkg-architecture --query DEB_BUILD_GNU_CPU returns from the container that may help.

Or more simply if you can confirm what GHC url it is pulling down - https://github.com/haskell/docker-haskell/blob/master/8.10/buster/Dockerfile#L111

[sopeters]

dpkg-architecture --query DEB_BUILD_GNU_CPU returns aarch64

I'm using the prebuilt docker container for arm64 so that docker image seems to be ok https://hub.docker.com/layers/haskell/library/haskell/slim-buster/images/sha256-3a79878b07e52d8c487b8ac17e768f78e6ceb89e3155ab28efa41d26a79a9002?context=explore

I'm using the same host to build other arm64 images with buildx extension - only the Haskell builds for the application we have fail

[sopeters]

I found this https://gitlab.haskell.org/ghc/ghc/-/issues/17918#note_376463 - maybe that's something the arm64 docker image build needs?

[AlistairB]

I can't really make any progress on this unless you can provide a way to reproduce this issue, eg. if you can replicate it with a github CI job. This has to be a cross architecture build issue as I arm64 GHC/docker haskell does work when built on arm64 hosts.

Probably if you can provide a repro the next step would be to create an issue on GHC asking them for insight as they are the people that understand the details of the configure step and can diagnose this error message.

[sopeters]

Will see what I can do - agree it must be a cross architecture build issue as I can build it on a Mac M1 locally

[AlistairB]

Going to optimistically close this. Please reopen if you think this project can help.