This implements container restore as described in:
https://kubernetes.io/blog/2022/12/05/forensic-container-checkpointing-alpha/#restore-checkpointed-container-standalone
For detailed step by step instruction also see contrib/checkpoint/checkpoint-restore-cri-test.sh
The code changes are based on changes I have done in Podman around 2018
and CRI-O around 2020.
The history behind restoring container via CRI/Kubernetes probably
requires some explanation. The initial proposal to bring
checkpoint/restore to Kubernetes was looking at pod checkpoint and
restoring and the corresponding CRI changes.
https://github.com/kubernetes-sigs/cri-tools/pull/662https://github.com/kubernetes/kubernetes/pull/97194
After discussing this topic for about two years another approach was
implemented as described in KEP-2008:
https://github.com/kubernetes/enhancements/issues/2008
"Forensic Container Checkpointing" allowed us to separate checkpointing
from restoring. For the "Forensic Container Checkpointing" it is enough
to create a checkpoint of the container. Restoring is not necessary as
the analysis of the checkpoint archive can happen without restoring the
container.
While thinking about a way to restore a container it was by coincidence
that we started to look into restoring containers in Kubernetes via
Create and Start. The way it was done in CRI-O is to figure out during
Create if the container image is a checkpoint image and if that is true
we are using another code path. The same was implemented now with this
change in containerd.
With this change it is possible to restore the container from a
checkpoint tar archive that is created during checkpointing via CRI.
To restore a container via Kubernetes we convert the tar archive to an
OCI image as described in the kubernetes.io blog post from above. Using
this OCI image it is possible to restore a container in Kubernetes.
At this point I think it should be doable to restore containers in
CRI-O and containerd no matter if they have been created by containerd or
CRI-O. The biggest difference is the container metadata and that can
be adapted during restore.
Open items:
* It is not clear to me why restoring a container in containerd goes
through task/Create(). But as the restore code already exists this
change extended the existing code path to restore a container in
task/Create() to also restore a container through the CRI via
Create and Start.
* Automatic image pulling. containerd does not pull images
automatically if created via the CRI. There is an option in
crictl to pull images before starting, but that uses the CRI
image pull interface. It is still a separate pull and create
operation. Restoring containers from an OCI image is a bit
different. The checkpoint OCI image does not include the base
image, but just a reference to the image (NAME@DIGEST).
Using crictl with pulling will enable the pulling of the
checkpoint image, but not of the base image the checkpoint is
based on. So during preparation of the checkpoint containerd
will automatically pull the base image, but I was not able how
to pull an image blockingly in containerd. So there is a for
loop waiting for the container image to appear in the internal
store. I think this probably can be implemented better.
Anyway, this is a first step towards container restored in Kubernetes
when using containerd.
Signed-off-by: Adrian Reber <areber@redhat.com>
This is needed so we can build the runc shim without grpc as a
transative dependency.
With this change the runc shim binary went from 14MB to 11MB.
The RSS from an idle shim went from about 17MB to 14MB (back around
where it was in in 1.7).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Update the dependency and the indirect golang.org/x/net version to align
with containerd itself, and to prevent a vulnerability being detected.
We should keep the versions <= versions used by containerd 1.7 to prevent
forcing users of containerd 1.7 in combination with the latest version
of the API module from having to update all their dependencies, but
this update should likely be fine (and aligns with 1.7).
Before this:
Scanning your code and 254 packages across 15 dependent modules for known vulnerabilities...
=== Symbol Results ===
Vulnerability #1: GO-2024-2687
HTTP/2 CONTINUATION flood in net/http
More info: https://pkg.go.dev/vuln/GO-2024-2687
Module: golang.org/x/net
Found in: golang.org/x/net@v0.21.0
Fixed in: golang.org/x/net@v0.23.0
Example traces found:
#1: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.ConnectionError.Error
#2: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.ErrCode.String
#3: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.FrameHeader.String
#4: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.FrameType.String
#5: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.Setting.String
#6: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.SettingID.String
#7: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.StreamError.Error
#8: services/content/v1/content_ttrpc.pb.go:272:35: content.ttrpccontentClient.Write calls ttrpc.Client.NewStream, which eventually calls http2.chunkWriter.Write
#9: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.connError.Error
#10: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.duplicatePseudoHeaderError.Error
#11: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.headerFieldNameError.Error
#12: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.headerFieldValueError.Error
#13: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.pseudoHeaderError.Error
#14: events/task_fieldpath.pb.go:85:20: events.TaskIO.Field calls fmt.Sprint, which eventually calls http2.writeData.String
Your code is affected by 1 vulnerability from 1 module.
This scan also found 0 vulnerabilities in packages you import and 3
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
After this:
govulncheck ./...
Scanning your code and 251 packages across 13 dependent modules for known vulnerabilities...
=== Symbol Results ===
No vulnerabilities found.
Your code is affected by 0 vulnerabilities.
This scan also found 0 vulnerabilities in packages you import and 3
vulnerabilities in modules you require, but your code doesn't appear to call
these vulnerabilities.
Use '-show verbose' for more details.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.
Signed-off-by: Derek McGowan <derek@mcg.dev>
Currenlty transfer service doesn't handle plain HTTP connection.
This commit fixes this issue by propagating
`(core/remotes/docker/config).HostOptions.DefaultScheme` from client to the
transfer service.
This commit also fixes ctr to use this feature for "--plain-http" flag.
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
Currently transfer service isn't aware of configurations of hosts directory and
ctr's `--hosts-dir` doesn't work.
This commit fixes this issue by using `config.ConfigureHosts` instead of
`docker.ConfigureDefaultRegistries`.
This commit also fixes ctr to use this feature for "--hosts-dir" flag.
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
The new `PlunginInfo()` call can be used for instrospecting the details
of the runtime plugin.
```console
$ ctr plugins inspect-runtime --runtime=io.containerd.runc.v2 --runc-binary=runc
{
"Name": "io.containerd.runc.v2",
"Version": {
"Version": "v2.0.0-beta.0-XX-gXXXXXXXXX.m",
"Revision": "v2.0.0-beta.0-XX-gXXXXXXXXX.m"
},
"Options": {
"binary_name": "runc"
},
"Features": {
"ociVersionMin": "1.0.0",
"ociVersionMax": "1.1.0-rc.2",
...,
},
"Annotations": null
}
```
The shim binary has to support `-info` flag, see `runtime/v2/README.md`
Replaces PR 8509 (`api/services/task: add RuntimeInfo()`)
Co-authored-by: Derek McGowan <derek@mcg.dev>
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
The Server rpc in introspection service is extended to expose
deprecation warnings based on observed feature use in containerd.
Signed-off-by: Samuel Karp <samuelkarp@google.com>
When call sandbox controller to create sandbox, we change the param from
sandbox id to total sandbox object to git all information to controller,
so that sandbox controller do not rely on the sandbox store anymore,
this is more decouple for the sandbox controller plugin inside
containerd, and it is neccesary for remote sandbox controller plugins as
it is not able to get sandbox from the sandbox store anymore.
Signed-off-by: Abel Feng <fshb1988@gmail.com>
As we are going to support more kinds of sandboxers, we have to tell
containerd which sandboxer used to manipulate a specific sandbox.
Signed-off-by: Abel Feng <fshb1988@gmail.com>
These utilities resulted in the platforms package to have the containerd
API as dependency. As this package is used in many parts of the code, as
well as external consumers, we should try to keep it light on dependencies,
with the potential to make it a standalone module.
These utilities were added in f3b7436b61,
which has not yet been included in a release, so skipping deprecation
and aliases for these.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
An oft employed scheme for a lot of our APIs is to include an
annotations field which is just a map of string to string. This
usually allows folks using the API to send over metadata or auxiliary
information without needing to get a new field added (especially where
the field might not make sense for it to be a standalone field). I think
having annotations for CreateSandbox make sense for this same use case.
Signed-off-by: Danny Canter <danny@dcantah.dev>
The whiteout timestamps are no longer set to the source date epoch.
The source date epoch still applies to non-whiteout files.
Discussion happened in moby/buildkit PR 3560.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
As a follow up change to adding a SandboxMetrics rpc to the core
sandbox service, the controller needed a corresponding rpc for CRI
and others to eventually implement.
This leaves the CRI (non-shim mode) controller unimplemented just to
have a change with the API addition to start.
Signed-off-by: Danny Canter <danny@dcantah.dev>
To gather metrics/stats about a specific sandbox instance, it'd be nice to
have a dedicated rpc for this. Due to the same "what kind of stats are going
to be returned" dilemma exists for sandboxes as well, I've re-used the metrics
type we have as the data field is just an `any`, leaving the metrics returned
entirely up to the shim author. For CRI usecases this will just be cgroup and
windows stats as that's all that's supported right now.
Signed-off-by: Danny Canter <danny@dcantah.dev>
