Run tasks on linux without an image #9411
|
In https://concourse-ci.org/tasks.html, there is a note about the image resource:
Would it be possible to make the image optional on linux too, so the build system can be compatible with Nix? In this case, the reproducibility of a CI pipeline is ensured by Nix itself, so running the CI step in an image doesn't offer much benefits. Using Nix also allows Nix to reuse existing caches, making builds faster. We need to set concourse/atc/exec/task_step.go Line 539 in fe581b0 concourse/atc/exec/task_step.go Line 371 in fe581b0 I propose that |
Replies: 3 comments 21 replies
|
We still need to point to a rootfs for the base of the container, even if it's just an empty directory. That's effectively what we're doing on darwin/windows. I'm not familiar with nix, so I'm lost on how this practically works. If we create an empty rootfs: .
├── rootfs/ #it's empty
└── metadata.jsonHow do you do anything with this? You can't exec anything in this container since it's empty. I guess you could pull in anything you need via task inputs? The running container's filesystem would then look like: /
└── tmp/
└── build/
└── abc123/
├── input1
├── input2
└── input3
# nothing else in the container's filesystem...I just realized that you can do this without waiting for Concourse to implement a new feature. Just create an empty rootfs! - task: create-empty-rootfs
config:
platform: linux
image_resource:
type: mock
source: {mirror_self: true}
outputs:
- name: empty
run:
path: bash
args:
- -c
- |
mkdir -p ./empty/rootfs
echo '{"user":"root","env":[]}' > empty/metadata.jsonAssuming that is what you're asking for. |
|
Does the houdini backend on linux handle creation of inputs, outputs, streaming these? Or is it incompatible with other workers running default backends? |
|
My candidate solution here is to have an explicit cache server and the build containers would be able to communicate with it. Simultaneously we could probably have a Nix flake resource that builds arbitrary derivation outputs. |
We still need to point to a rootfs for the base of the container, even if it's just an empty directory. That's effectively what we're doing on darwin/windows.
I'm not familiar with nix, so I'm lost on how this practically works. If we create an empty rootfs:
How do you do anything with this? You can't exec anything in this container since it's empty. I guess you could pull in anything you need via task inputs? The running container's filesystem would then look like:
/ └── tmp/ └── build/ └── abc123/ ├── input1 ├── input2 └── input3 # nothing else in the container's filesystem...I just realized th…