Non-distributable artifacts (foreign layers) were introduced in commit
05bd04350b to accommodate Windows images,
for which the EULA did not allow layers to be distributed through registries
other than those hosted by Microsoft. The concept of foreign / non-distributable
layers was adopted by the OCI distribution spec in [oci#233].
These restrictions were relaxed later to allow distributing these images
through non-public registries, for which a configuration was added in the
daemon in 67fdf574d5. In 2022, Microsoft updated
the EULA and [removed these restrictions altogether][1], and the OCI distribution
spec deprecated the functionality in [oci#965].
In 2023, Microsoft [removed the use of foreign data layers][2] for their images,
making this functionality obsolete.
This patch:
- Deprecates the `--allow-nondistributable-artifacts` daemon flag and corresponding
`allow-nondistributable-artifacts` field in `daemon.json`. Setting either
option will no longer take an effect, but a deprecation warning log is added
to raise awareness about the deprecation. This warning is planned to become
an error in the next release.
- Deprecates the `RegistryConfig.AllowNondistributableArtifactsCIDRs` and
`RegistryConfig.AllowNondistributableArtifactsHostnames` fields in the
`GET /info` API response. For API version v1.48 and lower, the fields are
still included in the response, but always `null`. In API version v1.49 and
higher, the field will be omitted entirely.
- Deprecates the `api/types/registry/ServiceConfig.AllowNondistributableArtifactsCIDRs`
field.
- Deprecates the `api/types/registry/ServiceConfig.AllowNondistributableArtifactsHostnames`
field.
- Deprecates the `registry.ServiceOptions.AllowNondistributableArtifacts` field.
[oci#233]: https://github.com/opencontainers/image-spec/pull/233
[oci#965]: https://github.com/opencontainers/image-spec/pull/965
[1]: https://techcommunity.microsoft.com/blog/containers/announcing-windows-container-base-image-redistribution-rights-change/3645201
[2]: https://techcommunity.microsoft.com/blog/containers/announcing-removal-of-foreign-layers-from-windows-container-images/3846833
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This field was added in 19515a7ad8, but looks
to be always set for endpoints used, so we can trim remote names unconditionally.
This option was added for possible future expansion, allowing registry-
mirrors to get the full reference of the image (including domain-name),
for them to host a mirror for multiple upstreams on the same registry.
That approach will unlikely be implemented, and containerd has a different
approach for this, where the reference to the original registry is passed
through a query parameter instead.
The field is unlikely used outside of our codebased, but deprecating it
before removal just in case.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Before this patch:
go test -run TestValidateManifest
ERRO[0000] image verification failed for digest sha256:02fee8c3220ba806531f606525eceb83f4feb654f62b207191b1c9209188dedd
PASS
ok github.com/docker/docker/distribution 0.008s
With this patch:
go test -run TestValidateManifest
PASS
ok github.com/docker/docker/distribution 0.010s
Note that in verbose mode, the logs are still printed, but through t.Log;
go test -run TestValidateManifest -v
=== RUN TestValidateManifest
log_hook.go:47: time="2024-12-03T13:41:19.308383552Z" level=error msg="image verification failed for digest sha256:02fee8c3220ba806531f606525eceb83f4feb654f62b207191b1c9209188dedd" func=distribution.verifySchema1Manifest file="/go/src/github.com/docker/docker/distribution/pull_v2.go:1010" testcase=TestValidateManifest
--- PASS: TestValidateManifest (0.00s)
PASS
ok github.com/docker/docker/distribution 0.011s
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The Class field was added in a12b466183 because
Docker Hub registry required a special scope to be set for pulling plugins;
HTTP/1.1 401 Unauthorized
...
Www-Authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository(plugin):vieux/sshfs:pull",error="insufficient_scope"
This is no longer a requirement, and the field is no longer set.
This patch deprecates the field and removes its use.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
distribution/pull_v2_test.go:290:3: The copy of the 'for' variable "tt" can be deleted (Go 1.22+) (copyloopvar)
tt := tt
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
distribution/xfer/download_test.go:394:3: The copy of the 'for' variable "tc" can be deleted (Go 1.22+) (copyloopvar)
tc := tc
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
distribution/push_v2_test.go:417:7: deepequalerrors: avoid using reflect.DeepEqual with errors (govet)
if !reflect.DeepEqual(err, tc.expectedError) {
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This was added in 9ca3bb632e, but looks like
the manifest-type was never used.
distribution/manifest.go:236:7: unusedwrite: unused write to field MediaType (govet)
desc.MediaType = mt
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
distribution/push_v2.go:558:9: shadow: declaration of "exists" shadows declaration at line 538 (govet)
if _, exists := digestToMetadata[meta.Digest]; exists {
^
distribution/push_v2.go:562:9: shadow: declaration of "exists" shadows declaration at line 538 (govet)
if _, exists := pd.checkedDigests[meta.Digest]; exists {
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
distribution/xfer/download_test.go:72:53: G602: slice index out of range (gosec)
return createChainIDFromParent(layer.ChainID(dgsts[0]), dgsts[1:]...)
^
distribution/xfer/download_test.go:75:69: G602: slice index out of range (gosec)
dgst := digest.FromBytes([]byte(string(parent) + " " + string(dgsts[0])))
^
distribution/xfer/download_test.go:76:59: G602: slice bounds out of range (gosec)
return createChainIDFromParent(layer.ChainID(dgst), dgsts[1:]...)
^
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
It was used in a single place, and the containerd/platforms module now
provides a format for a platform including os.version (through FormatAll),
which means we no longer have to create our own formatting for this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Switch to use github.com/containerd/platforms module, because containerd's
platforms package has moved to a separate module. This allows updating the
platforms parsing independent of the containerd module itself.
The package in containerd is deprecated, but kept as an alias to provide
compatibility between codebases.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Update to containerd 1.7.18, which now migrated to the errdefs module. The
existing errdefs package is now an alias for the module, and should no longer
be used directly.
This patch:
- updates the containerd dependency: https://github.com/containerd/containerd/compare/v1.7.17...v1.7.18
- replaces uses of the old package in favor of the new module
- adds a linter check to prevent accidental re-introduction of the old package
- adds a linter check to prevent using the "log" package, which was also
migrated to a separate module.
There are still some uses of the old package in (indirect) dependencies,
which should go away over time.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This message accidentally changed in ac2a028dcc
because my IDE's "refactor tool" was a bit over-enthusiastic. It also went and
updated the tests accordingly, so CI didn't catch this :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch disables pulling legacy (schema1 and schema 2, version 1) images by
default.
A `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE` environment-variable is
introduced to allow re-enabling this feature, aligning with the environment
variable used in containerd 2.0 (`CONTAINERD_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE`).
With this patch, attempts to pull a legacy image produces an error:
With graphdrivers:
docker pull docker:1.0
1.0: Pulling from library/docker
[DEPRECATION NOTICE] Docker Image Format v1, and Docker Image manifest version 2, schema 1 support will be removed in an upcoming release. Suggest the author of docker.io/library/docker:1.0 to upgrade the image to the OCI Format, or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/
With the containerd image store enabled, output is slightly different
as it returns the error before printing the `1.0: pulling ...`:
docker pull docker:1.0
Error response from daemon: [DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/library/docker:1.0 to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/
Using the "distribution" endpoint to resolve the digest for an image also
produces an error:
curl -v --unix-socket /var/run/docker.sock http://foo/distribution/docker.io/library/docker:1.0/json
* Trying /var/run/docker.sock:0...
* Connected to foo (/var/run/docker.sock) port 80 (#0)
> GET /distribution/docker.io/library/docker:1.0/json HTTP/1.1
> Host: foo
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Api-Version: 1.45
< Content-Type: application/json
< Docker-Experimental: false
< Ostype: linux
< Server: Docker/dev (linux)
< Date: Tue, 27 Feb 2024 16:09:42 GMT
< Content-Length: 354
<
{"message":"[DEPRECATION NOTICE] Docker Image Format v1, and Docker Image manifest version 2, schema 1 support will be removed in an upcoming release. Suggest the author of docker.io/library/docker:1.0 to upgrade the image to the OCI Format, or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/"}
* Connection #0 to host foo left intact
Starting the daemon with the `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE`
env-var set to a non-empty value allows pulling the image;
docker pull docker:1.0
[DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. Suggest the author of docker.io/library/docker:1.0 to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/
b0a0e6710d13: Already exists
d193ad713811: Already exists
ba7268c3149b: Already exists
c862d82a67a2: Already exists
Digest: sha256:5e7081837926c7a40e58881bbebc52044a95a62a2ea52fb240db3fc539212fe5
Status: Image is up to date for docker:1.0
docker.io/library/docker:1.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
To prevent a circular import between api/types and api/types image,
the RequestPrivilegeFunc reference was not moved, but defined as
part of the PullOptions / PushOptions.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If the daemon is configured to use a mirror for the default (Docker Hub)
registry, the endpoint did not fall back to querying the upstream if the mirror
did not contain the given reference.
For pull-through registry-mirrors, this was not a problem, as in that case the
registry would forward the request, but for other mirrors, no fallback would
happen. This was inconsistent with how "pulling" images handled this situation;
when pulling images, both the mirror and upstream would be tried.
This patch brings the daemon-side lookup of image-manifests on-par with the
client-side lookup (the GET /distribution endpoint) as used in API 1.30 and
higher.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If the daemon is configured to use a mirror for the default (Docker Hub)
registry, the endpoint did not fall back to querying the upstream if the mirror
did not contain the given reference.
If the daemon is configured to use a mirror for the default (Docker Hub)
registry, did not fall back to querying the upstream if the mirror did not
contain the given reference.
For pull-through registry-mirrors, this was not a problem, as in that case the
registry would forward the request, but for other mirrors, no fallback would
happen. This was inconsistent with how "pulling" images handled this situation;
when pulling images, both the mirror and upstream would be tried.
This problem was caused by the logic used in GetRepository, which had an
optimization to only return the first registry it was successfully able to
configure (and connect to), with the assumption that the mirror either contained
all images used, or to be configured as a pull-through mirror.
This patch:
- Introduces a GetRepositories method, which returns all candidates (both
mirror(s) and upstream).
- Updates the endpoint to try all
Before this patch:
# the daemon is configured to use a mirror for Docker Hub
cat /etc/docker/daemon.json
{ "registry-mirrors": ["http://localhost:5000"]}
# start the mirror (empty registry, not configured as pull-through mirror)
docker run -d --name registry -p 127.0.0.1:5000:5000 registry:2
# querying the endpoint fails, because the image-manifest is not found in the mirror:
curl -s --unix-socket /var/run/docker.sock http://localhost/v1.43/distribution/docker.io/library/hello-world:latest/json
{
"message": "manifest unknown: manifest unknown"
}
With this patch applied:
# the daemon is configured to use a mirror for Docker Hub
cat /etc/docker/daemon.json
{ "registry-mirrors": ["http://localhost:5000"]}
# start the mirror (empty registry, not configured as pull-through mirror)
docker run -d --name registry -p 127.0.0.1:5000:5000 registry:2
# querying the endpoint succeeds (manifest is fetched from the upstream Docker Hub registry):
curl -s --unix-socket /var/run/docker.sock http://localhost/v1.43/distribution/docker.io/library/hello-world:latest/json | jq .
{
"Descriptor": {
"mediaType": "application/vnd.oci.image.index.v1+json",
"digest": "sha256:1b9844d846ce3a6a6af7013e999a373112c3c0450aca49e155ae444526a2c45e",
"size": 3849
},
"Platforms": [
{
"architecture": "amd64",
"os": "linux"
}
]
}
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add a function to return tags for the given repository reference. This
is needed to implement the `pull -a` (pull all tags) for containerd
which doesn't directly use distribution, but we need to somehow make an
API call to the registry to obtain the available tags.
Signed-off-by: Paweł Gronowski <pawel.gronowski@docker.com>
The github.com/containerd/containerd/log package was moved to a separate
module, which will also be used by upcoming (patch) releases of containerd.
This patch moves our own uses of the package to use the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Fix storageDriver gcs not registered in binaries
- reference: replace uses of deprecated function SplitHostname
- Dont parse errors as JSON unless Content-Type is set to JSON
- update to go1.20.8
- Set Content-Type header in registry client ReadFrom
- deprecate reference package, migrate to github.com/distribution/reference
- digestset: deprecate package in favor of go-digest/digestset
- Do not close HTTP request body in HTTP handler
full diff: https://github.com/distribution/distribution/compare/v2.8.2...v2.8.3
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The test was depending on the client constructing an error based on the
http-status code, and the client not reading the response body if the
response was not a JSON response.
This fix;
- adds the correct content-type headers in the response
- includes error-messages in the response
- adds additional tests to cover both the plain (non-JSON) and JSON
error responses, as well as an empty response.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Implement a function that returns an error to replace existing uses of
the IsOSSupported utility, where callers had to produce the error after
checking.
The IsOSSupported function was used in combination with images, so implementing
a utility in "image" to prevent having to import pkg/system (which contains many
unrelated functions)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This field was used when the code supported both "v1" and "v2" registries.
We no longer support v1 registries, and the only v1 endpoint that's still
used is for the legacy "search" endpoint, which does not use the APIEndpoint
type.
As no code is using this field, and the value will always be set to "v2",
we can deprecated the Version field.
I'm keeping this field for 1 release, to give notice to any potential
external consumer, after which we can delete it.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Define consts for the Actions we use for events, instead of "ad-hoc" strings.
Having these consts makes it easier to find where specific events are triggered,
makes the events less error-prone, and allows documenting each Action (if needed).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When we added this deprecation warning, some registries had not yet
moved away from the deprecated specification, so we made the warning
conditional for pulling from Docker Hub.
That condition was added in 647dfe99a5,
which is over 4 Years ago, which should be time enough for images
and registries to have moved to current specifications.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Use the same warning for both "v1 in manifest-index" and bare "v1" images.
- Update URL to use a "/go/" redirect, which allows the docs team to more
easily redirect the URL to relevant docs (if things move).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The signatures of functions in containerd's errdefs packages are very
similar to those in our own, and it's easy to accidentally use the wrong
package.
This patch uses a consistent alias for all occurrences of this import.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
maxDownloadAttempts maps to the daemon configuration flag
--max-download-attempts int
Set the max download attempts for each pull (default 5)
and the daemon configuration machinery interprets a value of 0 as "apply
the default value" and not a valid user value (config validation/
normalization bugs notwithstanding). The intention is clearly that this
configuration value should be an upper limit on the number of times the
daemon should try to download a particular layer before giving up. So it
is surprising to have the configuration value interpreted as a _retry_
limit. The daemon will make up to N+1 attempts to download a layer! This
also means users cannot disable retries even if they wanted to.
Fix the fencepost bug so that max attempts really means max attempts,
not max retries. And fix the fencepost bug with the retry-backoff delay
so that the first backoff is 5s, not 10s.
Signed-off-by: Cory Snider <csnider@mirantis.com>
