From e5c6fab7f2955350220c5f21e5d480a07e4f0d4c Mon Sep 17 00:00:00 2001 From: Daniel Nephin Date: Mon, 17 Apr 2017 17:34:02 -0400 Subject: [PATCH 001/534] init Signed-off-by: Daniel Nephin From c4675d9ddbe86a2b3b9db028c05c808032664c24 Mon Sep 17 00:00:00 2001 From: David Calavera Date: Wed, 15 Jul 2015 11:25:56 -0700 Subject: [PATCH 002/534] Promote volume drivers from experimental to master. Remove volume stubs and use the experimental path as the only path. Signed-off-by: David Calavera --- docs/extend/index.md | 22 +++++ docs/extend/plugin_api.md | 135 +++++++++++++++++++++++++++++ docs/extend/plugins.md | 55 ++++++++++++ docs/extend/plugins_volume.md | 158 ++++++++++++++++++++++++++++++++++ 4 files changed, 370 insertions(+) create mode 100644 docs/extend/index.md create mode 100644 docs/extend/plugin_api.md create mode 100644 docs/extend/plugins.md create mode 100644 docs/extend/plugins_volume.md diff --git a/docs/extend/index.md b/docs/extend/index.md new file mode 100644 index 0000000000..61f8689de5 --- /dev/null +++ b/docs/extend/index.md @@ -0,0 +1,22 @@ + + + +## Extending Docker + +Currently, you can extend Docker by adding a plugin. This section contains the following topics: + +* [Understand Docker plugins](plugins.md) +* [Write a volume plugin](plugins_volumes.md) +* [Docker plugin API](plugin_api.md) + + \ No newline at end of file diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md new file mode 100644 index 0000000000..8e2862f6cb --- /dev/null +++ b/docs/extend/plugin_api.md @@ -0,0 +1,135 @@ + + +# Docker Plugin API + +Docker plugins are out-of-process extensions which add capabilities to the +Docker Engine. + +This page is intended for people who want to develop their own Docker plugin. +If you just want to learn about or use Docker plugins, look +[here](plugins.md). + +## What plugins are + +A plugin is a process running on the same docker host as the docker daemon, +which registers itself by placing a file in one of the plugin directories described in [Plugin discovery](#plugin-discovery). + +Plugins have human-readable names, which are short, lowercase strings. For +example, `flocker` or `weave`. + +Plugins can run inside or outside containers. Currently running them outside +containers is recommended. + +## Plugin discovery + +Docker discovers plugins by looking for them in the plugin directory whenever a +user or container tries to use one by name. + +There are three types of files which can be put in the plugin directory. + +* `.sock` files are UNIX domain sockets. +* `.spec` files are text files containing a URL, such as `unix:///other.sock`. +* `.json` files are text files containing a full json specification for the plugin. + +UNIX domain socket files must be located under `/run/docker/plugins`, whereas +spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker/plugins`. + +The name of the file (excluding the extension) determines the plugin name. + +For example, the `flocker` plugin might create a UNIX socket at +`/run/docker/plugins/flocker.sock`. + +You can define each plugin into a separated subdirectory if you want to isolate definitions from each other. +For example, you can create the `flocker` socket under `/run/docker/plugins/flocker/flocker.sock` and only +mount `/run/docker/plugins/flocker` inside the `flocker` container. + +Docker always searches for unix sockets in `/run/docker/plugins` first. It checks for spec or json files under +`/etc/docker/plugins` and `/usr/lib/docker/plugins` if the socket doesn't exist. The directory scan stops as +soon as it finds the first plugin definition with the given name. + +### JSON specification + +This is the JSON format for a plugin: + +```json +{ + "Name": "plugin-example", + "Addr": "https://example.com/docker/plugin", + "TLSConfig": { + "InsecureSkipVerify": false, + "CAFile": "/usr/shared/docker/certs/example-ca.pem", + "CertFile": "/usr/shared/docker/certs/example-cert.pem", + "KeyFile": "/usr/shared/docker/certs/example-key.pem", + } +} +``` + +The `TLSConfig` field is optional and TLS will only be verified if this configuration is present. + +## Plugin lifecycle + +Plugins should be started before Docker, and stopped after Docker. For +example, when packaging a plugin for a platform which supports `systemd`, you +might use [`systemd` dependencies]( +http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to +manage startup and shutdown order. + +When upgrading a plugin, you should first stop the Docker daemon, upgrade the +plugin, then start Docker again. + +## Plugin activation + +When a plugin is first referred to -- either by a user referring to it by name +(e.g. `docker run --volume-driver=foo`) or a container already configured to +use a plugin being started -- Docker looks for the named plugin in the plugin +directory and activates it with a handshake. See Handshake API below. + +Plugins are *not* activated automatically at Docker daemon startup. Rather, +they are activated only lazily, or on-demand, when they are needed. + +## API design + +The Plugin API is RPC-style JSON over HTTP, much like webhooks. + +Requests flow *from* the Docker daemon *to* the plugin. So the plugin needs to +implement an HTTP server and bind this to the UNIX socket mentioned in the +"plugin discovery" section. + +All requests are HTTP `POST` requests. + +The API is versioned via an Accept header, which currently is always set to +`application/vnd.docker.plugins.v1+json`. + +## Handshake API + +Plugins are activated via the following "handshake" API call. + +### /Plugin.Activate + +**Request:** empty body + +**Response:** +``` +{ + "Implements": ["VolumeDriver"] +} +``` + +Responds with a list of Docker subsystems which this plugin implements. +After activation, the plugin will then be sent events from this subsystem. + +## Plugin retries + +Attempts to call a method on a plugin are retried with an exponential backoff +for up to 30 seconds. This may help when packaging plugins as containers, since +it gives plugin containers a chance to start up before failing any user +containers which depend on them. diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md new file mode 100644 index 0000000000..1578579b1c --- /dev/null +++ b/docs/extend/plugins.md @@ -0,0 +1,55 @@ + + +# Understand Docker plugins + +You can extend the capabilities of the Docker Engine by loading third-party +plugins. + +## Types of plugins + +Plugins extend Docker's functionality. They come in specific types. For +example, a [volume plugin](plugins_volume.md) might enable Docker +volumes to persist across multiple Docker hosts. + +Currently Docker supports volume and network driver plugins. In the future it +will support additional plugin types. + +## Installing a plugin + +Follow the instructions in the plugin's documentation. + +## Finding a plugin + +The following plugins exist: + +* The [Flocker plugin](https://clusterhq.com/docker-plugin/) is a volume plugin + which provides multi-host portable volumes for Docker, enabling you to run + databases and other stateful containers and move them around across a cluster + of machines. + +* The [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) is + another volume plugin that provides multi-host volumes management for Docker + using GlusterFS. + +* The [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) is + a plugin that provides credentials and secret management using Keywhiz as + a central repository. + +## Troubleshooting a plugin + +If you are having problems with Docker after loading a plugin, ask the authors +of the plugin for help. The Docker team may not be able to assist you. + +## Writing a plugin + +If you are interested in writing a plugin for Docker, or seeing how they work +under the hood, see the [docker plugins reference](plugin_api.md). diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md new file mode 100644 index 0000000000..e9dc1ebaa2 --- /dev/null +++ b/docs/extend/plugins_volume.md @@ -0,0 +1,158 @@ + + +# Write a volume plugin + +Docker volume plugins enable Docker deployments to be integrated with external +storage systems, such as Amazon EBS, and enable data volumes to persist beyond +the lifetime of a single Docker host. See the [plugin documentation](plugins.md) +for more information. + +# Command-line changes + +A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: + + $ docker run -ti -v volumename:/data --volume-driver=flocker busybox sh + +This command passes the `volumename` through to the volume plugin as a +user-given name for the volume. The `volumename` must not begin with a `/`. + +By having the user specify a `volumename`, a plugin can associate the volume +with an external volume beyond the lifetime of a single container or container +host. This can be used, for example, to move a stateful container from one +server to another. + +By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. + + +# Create a VolumeDriver + +The container creation endpoint (`/containers/create`) accepts a `VolumeDriver` +field of type `string` allowing to specify the name of the driver. It's default +value of `"local"` (the default driver for local volumes). + +# Volume plugin protocol + +If a plugin registers itself as a `VolumeDriver` when activated, then it is +expected to provide writeable paths on the host filesystem for the Docker +daemon to provide to containers to consume. + +The Docker daemon handles bind-mounting the provided paths into user +containers. + +### /VolumeDriver.Create + +**Request**: +``` +{ + "Name": "volume_name" +} +``` + +Instruct the plugin that the user wants to create a volume, given a user +specified volume name. The plugin does not need to actually manifest the +volume on the filesystem yet (until Mount is called). + +**Response**: +``` +{ + "Err": null +} +``` + +Respond with a string error if an error occurred. + +### /VolumeDriver.Remove + +**Request**: +``` +{ + "Name": "volume_name" +} +``` + +Create a volume, given a user specified volume name. + +**Response**: +``` +{ + "Err": null +} +``` + +Respond with a string error if an error occurred. + +### /VolumeDriver.Mount + +**Request**: +``` +{ + "Name": "volume_name" +} +``` + +Docker requires the plugin to provide a volume, given a user specified volume +name. This is called once per container start. + +**Response**: +``` +{ + "Mountpoint": "/path/to/directory/on/host", + "Err": null +} +``` + +Respond with the path on the host filesystem where the volume has been made +available, and/or a string error if an error occurred. + +### /VolumeDriver.Path + +**Request**: +``` +{ + "Name": "volume_name" +} +``` + +Docker needs reminding of the path to the volume on the host. + +**Response**: +``` +{ + "Mountpoint": "/path/to/directory/on/host", + "Err": null +} +``` + +Respond with the path on the host filesystem where the volume has been made +available, and/or a string error if an error occurred. + +### /VolumeDriver.Unmount + +**Request**: +``` +{ + "Name": "volume_name" +} +``` + +Indication that Docker no longer is using the named volume. This is called once +per container stop. Plugin may deduce that it is safe to deprovision it at +this point. + +**Response**: +``` +{ + "Err": null +} +``` + +Respond with a string error if an error occurred. + From 994a581bc59129026a89200734c995ee23879081 Mon Sep 17 00:00:00 2001 From: Clinton Kitson Date: Tue, 21 Jul 2015 21:00:27 -0500 Subject: [PATCH 003/534] added REX-Ray as volume plugin to plugin docs Signed-off-by: Clinton Kitson --- docs/extend/plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 1578579b1c..4f3af15466 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -44,6 +44,10 @@ The following plugins exist: a plugin that provides credentials and secret management using Keywhiz as a central repository. +* The [REX-Ray plugin](https://github.com/emccode/rexraycli) is a volume plugin + which is written in Go and provides advanced storage functionality for many + platforms including EC2, OpenStack, XtremIO, and ScaleIO. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 7c372cb6d39f041cdab86165733ae7805654a4c7 Mon Sep 17 00:00:00 2001 From: Bardia Keyoumarsi Date: Sat, 11 Jul 2015 12:35:14 -0700 Subject: [PATCH 004/534] Updated volume-driver plugin api docs There was a mistake in the plugin_volume.md file regarding the /VolumeDriver.Remove call. Added fix and a better explanation. Signed-off-by: Bardia Keyoumarsi --- docs/extend/plugins_volume.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index e9dc1ebaa2..3553971982 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -78,7 +78,7 @@ Respond with a string error if an error occurred. } ``` -Create a volume, given a user specified volume name. +Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. **Response**: ``` From 9c456e418d93bfd1acd1b73487512f03d49d4af5 Mon Sep 17 00:00:00 2001 From: Morgan Bauer Date: Fri, 24 Jul 2015 10:24:25 -0700 Subject: [PATCH 005/534] Fix Plugins links Signed-off-by: Morgan Bauer --- docs/extend/index.md | 8 +++----- docs/extend/plugin_api.md | 2 +- docs/extend/plugins.md | 4 ++-- docs/extend/plugins_volume.md | 2 +- 4 files changed, 7 insertions(+), 9 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 61f8689de5..b32daddc91 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -15,8 +15,6 @@ weight = 6 Currently, you can extend Docker by adding a plugin. This section contains the following topics: -* [Understand Docker plugins](plugins.md) -* [Write a volume plugin](plugins_volumes.md) -* [Docker plugin API](plugin_api.md) - - \ No newline at end of file +* [Understand Docker plugins](/extend/plugins) +* [Write a volume plugin](/extend/plugins_volume) +* [Docker plugin API](/extend/plugin_api) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 8e2862f6cb..7109aa7def 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -16,7 +16,7 @@ Docker Engine. This page is intended for people who want to develop their own Docker plugin. If you just want to learn about or use Docker plugins, look -[here](plugins.md). +[here](/extend/plugins). ## What plugins are diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 4f3af15466..8208d9581f 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -17,7 +17,7 @@ plugins. ## Types of plugins Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](plugins_volume.md) might enable Docker +example, a [volume plugin](/extend/plugins_volume) might enable Docker volumes to persist across multiple Docker hosts. Currently Docker supports volume and network driver plugins. In the future it @@ -56,4 +56,4 @@ of the plugin for help. The Docker team may not be able to assist you. ## Writing a plugin If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](plugin_api.md). +under the hood, see the [docker plugins reference](/extend/plugin_api). diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 3553971982..2752039d15 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -12,7 +12,7 @@ parent = "mn_extend" Docker volume plugins enable Docker deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond -the lifetime of a single Docker host. See the [plugin documentation](plugins.md) +the lifetime of a single Docker host. See the [plugin documentation](/extend/plugins) for more information. # Command-line changes From 7b45c34119abcc9d967cf580bd94692eff7c5142 Mon Sep 17 00:00:00 2001 From: Stephen Rust Date: Sun, 2 Aug 2015 12:16:07 -0400 Subject: [PATCH 006/534] Docs: Add Blockbridge volume plugin Signed-off-by: Stephen Rust --- docs/extend/plugins.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 8208d9581f..847d844967 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -31,6 +31,12 @@ Follow the instructions in the plugin's documentation. The following plugins exist: +* The [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) + is a volume plugin that provides access to an extensible set of + container-based persistent storage options. It supports single and multi-host Docker + environments with features that include tenant isolation, automated + provisioning, encryption, secure deletion, snapshots and QoS. + * The [Flocker plugin](https://clusterhq.com/docker-plugin/) is a volume plugin which provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster From 4d6dc3f409c9c16fc81590bd01b9e1bed56745a0 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexandre=20Gonz=C3=A1lez?= Date: Mon, 24 Aug 2015 10:44:08 +0100 Subject: [PATCH 007/534] Fix broken links at plugins documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Álex González --- docs/extend/plugins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 847d844967..d171feaf23 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -17,7 +17,7 @@ plugins. ## Types of plugins Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](/extend/plugins_volume) might enable Docker +example, a [volume plugin](/docs/extend/plugins_volume.md) might enable Docker volumes to persist across multiple Docker hosts. Currently Docker supports volume and network driver plugins. In the future it @@ -62,4 +62,4 @@ of the plugin for help. The Docker team may not be able to assist you. ## Writing a plugin If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](/extend/plugin_api). +under the hood, see the [docker plugins reference](/docs/extend/plugin_api.md). From 1a479857adf5d3ff39c3701a3831ac01223002fc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C3=81lex=20Gonz=C3=A1lez?= Date: Mon, 24 Aug 2015 11:45:31 +0100 Subject: [PATCH 008/534] Revert "Fix broken links at plugins documentation" MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This reverts commit 354147df012dd35f82e59e3511d9be4fc7610352. Signed-off-by: Álex González --- docs/extend/plugins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index d171feaf23..847d844967 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -17,7 +17,7 @@ plugins. ## Types of plugins Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](/docs/extend/plugins_volume.md) might enable Docker +example, a [volume plugin](/extend/plugins_volume) might enable Docker volumes to persist across multiple Docker hosts. Currently Docker supports volume and network driver plugins. In the future it @@ -62,4 +62,4 @@ of the plugin for help. The Docker team may not be able to assist you. ## Writing a plugin If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](/docs/extend/plugin_api.md). +under the hood, see the [docker plugins reference](/extend/plugin_api). From 30ba067991da02d35a31207524e3888aec99f433 Mon Sep 17 00:00:00 2001 From: Sheng Yang Date: Tue, 25 Aug 2015 11:57:17 -0700 Subject: [PATCH 009/534] Update plugins.md Add Convoy to Docker plugin list. Signed-off-by: Sheng Yang --- docs/extend/plugins.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 847d844967..30d6ec8172 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -37,6 +37,11 @@ The following plugins exist: environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. +* The [Convoy plugin](https://github.com/rancher/convoy) is a volume plugin for a + variety of storage back-ends including device mapper and NFS. It's a simple standalone + executable written in Go and provides the framework to support vendor-specific extensions + such as snapshots, backups and restore. + * The [Flocker plugin](https://clusterhq.com/docker-plugin/) is a volume plugin which provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster From 71d67fc97652e7cc6c216ae435bb21b1eb1fd17c Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Fri, 12 Jun 2015 09:25:32 -0400 Subject: [PATCH 010/534] Add volume API/CLI Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 2752039d15..41c4dd2194 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -52,13 +52,15 @@ containers. **Request**: ``` { - "Name": "volume_name" + "Name": "volume_name", + "Opts": {} } ``` Instruct the plugin that the user wants to create a volume, given a user specified volume name. The plugin does not need to actually manifest the volume on the filesystem yet (until Mount is called). +Opts is a map of driver specific options passed through from the user request. **Response**: ``` From c405760d8fafc4b116243995083286d7b837fa5a Mon Sep 17 00:00:00 2001 From: Peter Edge Date: Thu, 3 Sep 2015 16:32:35 -0500 Subject: [PATCH 011/534] Add link to Pachyderm PFS volume driver plugin in plugins.md Signed-off-by: Peter Edge --- docs/extend/plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 30d6ec8172..d482318306 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -59,6 +59,10 @@ The following plugins exist: which is written in Go and provides advanced storage functionality for many platforms including EC2, OpenStack, XtremIO, and ScaleIO. +* The [Pachyderm PFS plugin](https://github.com/pachyderm/pachyderm/tree/master/src/cmd/pfs-volume-driver) + is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) + repositories at specific commits as volumes within Docker containers. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 626fede05686b710d4115435f7f9dff73ff1e12e Mon Sep 17 00:00:00 2001 From: Peter Edge Date: Mon, 7 Sep 2015 00:34:45 +0200 Subject: [PATCH 012/534] move pachyderm PFS link above rex ray link in docs/extend/plugins.md Signed-off-by: Peter Edge --- docs/extend/plugins.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index d482318306..37b4f2f0c4 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -55,14 +55,14 @@ The following plugins exist: a plugin that provides credentials and secret management using Keywhiz as a central repository. -* The [REX-Ray plugin](https://github.com/emccode/rexraycli) is a volume plugin - which is written in Go and provides advanced storage functionality for many - platforms including EC2, OpenStack, XtremIO, and ScaleIO. - * The [Pachyderm PFS plugin](https://github.com/pachyderm/pachyderm/tree/master/src/cmd/pfs-volume-driver) is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) repositories at specific commits as volumes within Docker containers. +* The [REX-Ray plugin](https://github.com/emccode/rexraycli) is a volume plugin + which is written in Go and provides advanced storage functionality for many + platforms including EC2, OpenStack, XtremIO, and ScaleIO. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From dd275127be45d3602c84606bf667a2cbbcc8ad21 Mon Sep 17 00:00:00 2001 From: Madhu Venugopal Date: Mon, 28 Sep 2015 18:57:03 -0700 Subject: [PATCH 013/534] Networking API and UX documentation More doc updates will follow Signed-off-by: Madhu Venugopal --- docs/extend/index.md | 7 +++--- docs/extend/plugins.md | 6 +++-- docs/extend/plugins_network.md | 40 ++++++++++++++++++++++++++++++++++ 3 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 docs/extend/plugins_network.md diff --git a/docs/extend/index.md b/docs/extend/index.md index b32daddc91..fd847dd912 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -15,6 +15,7 @@ weight = 6 Currently, you can extend Docker by adding a plugin. This section contains the following topics: -* [Understand Docker plugins](/extend/plugins) -* [Write a volume plugin](/extend/plugins_volume) -* [Docker plugin API](/extend/plugin_api) +* [Understand Docker plugins](/extend/plugins.md) +* [Write a volume plugin](/extend/plugins_volume.md) +* [Write a network plugin](/extend/plugins_network.md) +* [Docker plugin API](/extend/plugin_api.md) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 37b4f2f0c4..c526773958 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -17,8 +17,10 @@ plugins. ## Types of plugins Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](/extend/plugins_volume) might enable Docker -volumes to persist across multiple Docker hosts. +example, a [volume plugin](/extend/plugins_volume.md) might enable Docker +volumes to persist across multiple Docker hosts and a +[network plugin](/extend/plugins_network.md) might provide network plumbing +using a favorite networking technology, such as vxlan overlay, ipvlan, EVPN, etc. Currently Docker supports volume and network driver plugins. In the future it will support additional plugin types. diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md new file mode 100644 index 0000000000..c2175d2c4f --- /dev/null +++ b/docs/extend/plugins_network.md @@ -0,0 +1,40 @@ +# Docker network driver plugins + +Docker supports network driver plugins via +[LibNetwork](https://github.com/docker/libnetwork). Network driver plugins are +implemented as "remote drivers" for LibNetwork, which shares plugin +infrastructure with Docker. In effect this means that network driver plugins +are activated in the same way as other plugins, and use the same kind of +protocol. + +## Using network driver plugins + +The means of installing and running a network driver plugin will depend on the +particular plugin. + +Once running however, network driver plugins are used just like the built-in +network drivers: by being mentioned as a driver in network-oriented Docker +commands. For example, + + docker network create -d weave mynet + +Some network driver plugins are listed in [plugins.md](/docs/extend/plugins.md) + +The network thus created is owned by the plugin, so subsequent commands +referring to that network will also be run through the plugin such as, + + docker run --net=mynet busybox top + +## Network driver plugin protocol + +The network driver protocol, additional to the plugin activation call, is +documented as part of LibNetwork: +[https://github.com/docker/libnetwork/blob/master/docs/remote.md](https://github.com/docker/libnetwork/blob/master/docs/remote.md). + +# Related GitHub PRs and issues + +Please record your feedback in the following issue, on the usual +Google Groups, or the IRC channel #docker-network. + + - [#14083](https://github.com/docker/docker/issues/14083) Feedback on + experimental networking features From 448ae809b2fa6749fbcf4fc59068d5658aa0f2b6 Mon Sep 17 00:00:00 2001 From: Mary Anthony Date: Fri, 9 Oct 2015 16:50:41 -0700 Subject: [PATCH 014/534] Enabled GitHub Flavored Markdown GitHub flavored markdown is now supported for links and images. Also, ran LinkChecker and FileResolver. Yay! Fixes from Spider check Output for docker/docker now goes into engine directory Signed-off-by: Mary Anthony --- docs/extend/index.md | 8 ++++---- docs/extend/plugin_api.md | 2 +- docs/extend/plugins.md | 6 +++--- docs/extend/plugins_network.md | 23 +++++++++++++++++------ docs/extend/plugins_volume.md | 2 +- 5 files changed, 26 insertions(+), 15 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index fd847dd912..61afb720ca 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -15,7 +15,7 @@ weight = 6 Currently, you can extend Docker by adding a plugin. This section contains the following topics: -* [Understand Docker plugins](/extend/plugins.md) -* [Write a volume plugin](/extend/plugins_volume.md) -* [Write a network plugin](/extend/plugins_network.md) -* [Docker plugin API](/extend/plugin_api.md) +* [Understand Docker plugins](plugins.md) +* [Write a volume plugin](plugins_volume.md) +* [Write a network plugin](plugins_network.md) +* [Docker plugin API](plugin_api.md) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 7109aa7def..8e2862f6cb 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -16,7 +16,7 @@ Docker Engine. This page is intended for people who want to develop their own Docker plugin. If you just want to learn about or use Docker plugins, look -[here](/extend/plugins). +[here](plugins.md). ## What plugins are diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index c526773958..e04de760d3 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -17,9 +17,9 @@ plugins. ## Types of plugins Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](/extend/plugins_volume.md) might enable Docker +example, a [volume plugin](plugins_volume.md) might enable Docker volumes to persist across multiple Docker hosts and a -[network plugin](/extend/plugins_network.md) might provide network plumbing +[network plugin](plugins_network.md) might provide network plumbing using a favorite networking technology, such as vxlan overlay, ipvlan, EVPN, etc. Currently Docker supports volume and network driver plugins. In the future it @@ -73,4 +73,4 @@ of the plugin for help. The Docker team may not be able to assist you. ## Writing a plugin If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](/extend/plugin_api). +under the hood, see the [docker plugins reference](plugin_api.md). diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index c2175d2c4f..335ee9d04a 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -1,10 +1,21 @@ + + # Docker network driver plugins -Docker supports network driver plugins via -[LibNetwork](https://github.com/docker/libnetwork). Network driver plugins are -implemented as "remote drivers" for LibNetwork, which shares plugin -infrastructure with Docker. In effect this means that network driver plugins -are activated in the same way as other plugins, and use the same kind of +Docker supports network driver plugins via +[LibNetwork](https://github.com/docker/libnetwork). Network driver plugins are +implemented as "remote drivers" for LibNetwork, which shares plugin +infrastructure with Docker. In effect this means that network driver plugins +are activated in the same way as other plugins, and use the same kind of protocol. ## Using network driver plugins @@ -18,7 +29,7 @@ commands. For example, docker network create -d weave mynet -Some network driver plugins are listed in [plugins.md](/docs/extend/plugins.md) +Some network driver plugins are listed in [plugins](plugins.md) The network thus created is owned by the plugin, so subsequent commands referring to that network will also be run through the plugin such as, diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 41c4dd2194..6ab4bfbd7f 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -12,7 +12,7 @@ parent = "mn_extend" Docker volume plugins enable Docker deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond -the lifetime of a single Docker host. See the [plugin documentation](/extend/plugins) +the lifetime of a single Docker host. See the [plugin documentation](plugins.md) for more information. # Command-line changes From 330302e8a55317bef435e13b5178853462081d6c Mon Sep 17 00:00:00 2001 From: Jeremy Unruh Date: Wed, 21 Oct 2015 08:28:37 -0700 Subject: [PATCH 015/534] docs: Add Netshare volume plugin to extend/plugins Signed-off-by: Jeremy Unruh --- docs/extend/plugins.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index e04de760d3..fc2cb97820 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -65,6 +65,9 @@ The following plugins exist: which is written in Go and provides advanced storage functionality for many platforms including EC2, OpenStack, XtremIO, and ScaleIO. +* The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin + which provides volume management for NFS 3/4, AWS EFS and CIFS file systems. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 8260760b40824bf642d967b2a318538b0134acde Mon Sep 17 00:00:00 2001 From: Jeremy Unruh Date: Wed, 21 Oct 2015 18:07:55 -0700 Subject: [PATCH 016/534] changes from review - move up to conform to alphabetical order Signed-off-by: Jeremy Unruh --- docs/extend/plugins.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index fc2cb97820..c422b4011b 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -56,6 +56,9 @@ The following plugins exist: * The [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) is a plugin that provides credentials and secret management using Keywhiz as a central repository. + +* The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin + which provides volume management for NFS 3/4, AWS EFS and CIFS file systems. * The [Pachyderm PFS plugin](https://github.com/pachyderm/pachyderm/tree/master/src/cmd/pfs-volume-driver) is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) @@ -65,9 +68,6 @@ The following plugins exist: which is written in Go and provides advanced storage functionality for many platforms including EC2, OpenStack, XtremIO, and ScaleIO. -* The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin - which provides volume management for NFS 3/4, AWS EFS and CIFS file systems. - ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 5723d13bec62e78d0fd153504aadd563c8ea555f Mon Sep 17 00:00:00 2001 From: Jeremy Unruh Date: Wed, 21 Oct 2015 18:42:17 -0700 Subject: [PATCH 017/534] changes from review - which -> that Signed-off-by: Jeremy Unruh --- docs/extend/plugins.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index c422b4011b..2ea373a2ed 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -56,9 +56,9 @@ The following plugins exist: * The [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) is a plugin that provides credentials and secret management using Keywhiz as a central repository. - + * The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin - which provides volume management for NFS 3/4, AWS EFS and CIFS file systems. + that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. * The [Pachyderm PFS plugin](https://github.com/pachyderm/pachyderm/tree/master/src/cmd/pfs-volume-driver) is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) From a7b3ca0db82924e7b20bbb2ef07799accae23dba Mon Sep 17 00:00:00 2001 From: Mary Anthony Date: Wed, 30 Sep 2015 13:11:36 -0700 Subject: [PATCH 018/534] First pass at consolidating Removing old networking.md Updating dockernetworks.md with images Adding information on network plugins Adding blurb about links to docker networking Updating the working documentation Adding Overlay Getting Started Downplaying links by removing refs/examples, adding refs/examples for network. Updating getting started to reflect networks not links Pulling out old network material Updating per discussion with Madhu to add Default docs section Updating with bridge default Fix bad merge Updating with new cluster-advertise behavior Update working and NetworkSettings examples Correcting example for default bridge discovery behavior Entering comments Fixing broken Markdown Syntax Updating with comments Updating all the links Signed-off-by: Mary Anthony --- docs/extend/plugins.md | 5 ++-- docs/extend/plugins_network.md | 49 +++++++++++++++++++--------------- 2 files changed, 30 insertions(+), 24 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 2ea373a2ed..4d73c8e310 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -18,9 +18,8 @@ plugins. Plugins extend Docker's functionality. They come in specific types. For example, a [volume plugin](plugins_volume.md) might enable Docker -volumes to persist across multiple Docker hosts and a -[network plugin](plugins_network.md) might provide network plumbing -using a favorite networking technology, such as vxlan overlay, ipvlan, EVPN, etc. +volumes to persist across multiple Docker hosts and a +[network plugin](plugins_network.md) might provide network plumbing. Currently Docker supports volume and network driver plugins. In the future it will support additional plugin types. diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 335ee9d04a..b1c05558d3 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -1,7 +1,7 @@ From 18b303f010ee28986a2effbf4518a80152171208 Mon Sep 17 00:00:00 2001 From: pidster Date: Fri, 13 Nov 2015 10:06:04 +0000 Subject: [PATCH 023/534] Add Weave Network Plugin to docs/extend/plugins.md Signed-off-by: pidster --- docs/extend/plugins.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 37147a2b50..6c17b9072d 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -80,6 +80,8 @@ multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. +* The [Weave Network Plugin](https://github.com/weaveworks/docker-plugin) creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 1645728db153050eca7a8bca268ee05da1a89811 Mon Sep 17 00:00:00 2001 From: Dima Stopel Date: Wed, 4 Nov 2015 15:49:53 +0200 Subject: [PATCH 024/534] Adding authorization subsystem documentation Signed-off-by: Dima Stopel --- docs/extend/authorization.md | 119 +++++++++++++++++++++++++++++++++++ 1 file changed, 119 insertions(+) create mode 100644 docs/extend/authorization.md diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md new file mode 100644 index 0000000000..2f722e6fa2 --- /dev/null +++ b/docs/extend/authorization.md @@ -0,0 +1,119 @@ + + + +# Access authorization + +By default Docker’s authorization model is all or nothing. Anyone who has access to the Docker daemon has the ability to run any Docker command. Authorization plugins enable creating granular access policies for managing access to Docker daemon. + +## Basic principles + +* The authorization sub-system is based on Docker's [Plugin API](http://docs.docker.com/engine/extend/plugin_api), so that anyone could add / develop an authorization plug-in. It is possible to add a plug-in to a deployed Docker daemon without the need to rebuild the daemon. Daemon restart is required. + +* The authorization sub-system enables approving or denying requests to the Docker daemon based on the current user and command context, where the authentication context contains all user details and the authentication method, and the command context contains all the relevant request data. + +* The authorization sub-system enables approving or denying the response from the Docker daemon based on the current user and the command context. + +## Basic architecture + +Authorization is integrated into Docker daemon by enabling external authorization plug-ins registration as part of the daemon stratup. Each plug-in will receive user's information (retrieved using the authentication sub-system) and HTTP request information and decide whether to allow or deny the request. Plugins can be chained together and only in case where all plug-ins allow accessing the resource, the is access granted. + +Recently Docker introduced a new extendability mechanism called [plug-in infrastructure](http://docs.docker.com/engine/extend/plugin_api), which enables extending Docker by dynamically loading, removing and communicating with third-party components using a generic, easily extendable, API. The authorization sub-system was build using this mechanism. + +To enable full extendability, each plugin receives the authenticated user name, the request context including the user, HTTP headers, and the request/response body. No authentication information will be passed to the authorization plug-in except for: 1) user name, and 2) authentication method that was used. Specifically, no user credentials or tokens will be passed. The request / response bodies will be sent to the plugin only in case the Content-Type is either `text/*` or `application/json`. + +For commands that involve the hijacking of the HTTP connection (HTTP Upgrade), such as `exec`, the authorization plugins will only be called for the initial HTTP requests, once the commands are approved, authorization will not be applied to the rest of the flow. Specifically, the streaming data will not be passed to the authorization plugins. For commands that return chunked HTTP response, such as `logs` and `events`, only the HTTP request will be sent to the authorization plug-ins as well. + +During request / response processing, some authorization plugins flows might require performing additional queries to the docker daemon. To complete such a flows plugins can call the daemon API similar to the regular user, which means the plugin admin will have to configure proper authentication and security policies. + +## API schema + +In addition to the standard plugin registration method, each plugin should implement the following two methods: + +`/AuthzPlugin.AuthZReq` The authorize request method, which is called before Docker daemon processes the client request. + +`/AuthzPlugin.AuthZRes` The authorize response method, which is called before the response is returned from Docker daemon to the client. + + +### Request authorization +#### Daemon -> Plugin + +Name | Type | Description +-----------------------|-------------------|------------------------------------------------------- +User | string | The user identification +Authentication method | string | The authentication method used +Request method | enum | The HTTP method (GET/DELETE/POST) +Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) +Request headers | map[string]string | Request headers as key value pairs (without the authorization header) +Request body | []byte | Raw request body + + +#### Plugin -> Daemon + +Name | Type | Description +--------|--------|---------------------------------------------------------------------------------- +Allow | bool | Boolean value indicating whether the request is allowed or denied +Message | string | Authorization message (will be returned to the client in case the access is denied) + +### Response authorization +#### Daemon -> Plugin + + +Name | Type | Description +----------------------- |------------------ |---------------------------------------------------- +User | string | The user identification +Authentication method | string | The authentication method used +Request method | string | The HTTP method (GET/DELETE/POST) +Request URI | string | The http request URI including API version (e.g., v.1.17/containers/json) +Request headers | map[string]string | Request headers as key value pairs (without the authorization header) +Request body | []byte | Raw request body +Response status code | int | Status code from the docker daemon +Response headers | map[string]string | Response headers as key value pairs +Response body | []byte | Raw docker daemon response body + + +#### Plugin -> Daemon + +Name | Type | Description +--------|--------|---------------------------------------------------------------------------------- +Allow | bool | Boolean value indicating whether the response is allowed or denied +Message | string | Authorization message (will be returned to the client in case the access is denied) + +## UX flows + +### Setting up docker daemon + +Authorization plugins are enabled with a dedicated command line argument. The argument contains a comma separated list of the plugin names, which should be the same as the plugin’s socket or spec file. +``` +$ docker -d authz-plugins=plugin1,plugin2,... +``` + +### Calling authorized command (allow) + +No impact on command output +``` +$ docker pull centos +... +f1b10cd84249: Pull complete +... +``` + +### Calling unauthorized command (deny) + +``` +$ docker pull centos +… +Authorization failed. Pull command for user 'john_doe' is denied by authorization plugin 'ACME' with message ‘[ACME] User 'john_doe' is not allowed to perform the pull command’ +``` + + + + From 5a57e0ee7a05b684ea77ea46c83445324c5bf66b Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Thu, 12 Nov 2015 13:06:47 +0200 Subject: [PATCH 025/534] Docker authorization plug-in infrastructure enables extending the functionality of the Docker daemon with respect to user authorization. The infrastructure enables registering a set of external authorization plug-in. Each plug-in receives information about the user and the request and decides whether to allow or deny the request. Only in case all plug-ins allow accessing the resource the access is granted. Each plug-in operates as a separate service, and registers with Docker through general (plug-ins API) [https://blog.docker.com/2015/06/extending-docker-with-plugins/]. No Docker daemon recompilation is required in order to add / remove an authentication plug-in. Each plug-in is notified twice for each operation: 1) before the operation is performed and, 2) before the response is returned to the client. The plug-ins can modify the response that is returned to the client. The authorization depends on the authorization effort that takes place in parallel [https://github.com/docker/docker/issues/13697]. This is the official issue of the authorization effort: https://github.com/docker/docker/issues/14674 (Here)[https://github.com/rhatdan/docker-rbac] you can find an open document that discusses a default RBAC plug-in for Docker. Signed-off-by: Liron Levin Added container create flow test and extended the verification for ps --- docs/extend/authorization.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 2f722e6fa2..7b599b28c7 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -91,9 +91,10 @@ Message | string | Authorization message (will be returned to the client in case ### Setting up docker daemon -Authorization plugins are enabled with a dedicated command line argument. The argument contains a comma separated list of the plugin names, which should be the same as the plugin’s socket or spec file. +Authorization plugins are enabled with a dedicated command line argument. The argument contains the plugin name, which should be the same as the plugin’s socket or spec file. +Multiple authz-plugin parameters are supported. ``` -$ docker -d authz-plugins=plugin1,plugin2,... +$ docker daemon --authz-plugins=plugin1 --auth-plugins=plugin2,... ``` ### Calling authorized command (allow) From 2073d6ce83388a110c436d734bfcd90175d2242d Mon Sep 17 00:00:00 2001 From: Dima Stopel Date: Tue, 8 Dec 2015 14:54:33 +0200 Subject: [PATCH 026/534] Fixing documentation according to comments by @moxiegirl and @thaJeztah Signed-off-by: Dima Stopel --- docs/extend/authorization.md | 223 ++++++++++++++---- docs/extend/images/authz_additional_info.png | Bin 0 -> 45916 bytes docs/extend/images/authz_allow.png | Bin 0 -> 33505 bytes docs/extend/images/authz_chunked.png | Bin 0 -> 33168 bytes .../extend/images/authz_connection_hijack.png | Bin 0 -> 38780 bytes docs/extend/images/authz_deny.png | Bin 0 -> 27099 bytes 6 files changed, 173 insertions(+), 50 deletions(-) create mode 100644 docs/extend/images/authz_additional_info.png create mode 100644 docs/extend/images/authz_allow.png create mode 100644 docs/extend/images/authz_chunked.png create mode 100644 docs/extend/images/authz_connection_hijack.png create mode 100644 docs/extend/images/authz_deny.png diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 7b599b28c7..d16ba350a7 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -10,40 +10,192 @@ weight = -1 -# Access authorization +# Create an authorization plugin -By default Docker’s authorization model is all or nothing. Anyone who has access to the Docker daemon has the ability to run any Docker command. Authorization plugins enable creating granular access policies for managing access to Docker daemon. +Docker’s out-of-the-box authorization model is all or nothing. Any user with +permission to access to the Docker daemon can run any Docker client command. The +same is true for callers using Docker's remote API to contact the daemon. If you +require greater access control, you can create authorization plugins and add +them to your Docker daemon configuration. Using an authorization plugin, a +Docker administrator can configure granular access policies for managing access +to Docker daemon. + +Anyone with the appropriate skills can develop an authorization plugin. These +skills, at their most basic, are knowledge of Docker, understanding of REST, and +sound programming knowledge. This document describes the architecture, state, +and methods information available to an authorization plugin developer. ## Basic principles -* The authorization sub-system is based on Docker's [Plugin API](http://docs.docker.com/engine/extend/plugin_api), so that anyone could add / develop an authorization plug-in. It is possible to add a plug-in to a deployed Docker daemon without the need to rebuild the daemon. Daemon restart is required. +Docker's [plugin infrastructure](http://docs.docker.com/engine/extend/plugin_api) enables +extending Docker by dynamically loading, removing and communicating with +third-party components using a generic API. The access authorization subsystem +was built using this mechanism. -* The authorization sub-system enables approving or denying requests to the Docker daemon based on the current user and command context, where the authentication context contains all user details and the authentication method, and the command context contains all the relevant request data. +Using this subsystem, you don't need to rebuild the Docker daemon to add an +authorization plugin. You can add a plugin to a installed Docker daemon. You do +need to restart the Docker daemon to add a new plugin. -* The authorization sub-system enables approving or denying the response from the Docker daemon based on the current user and the command context. +An authorization plugin approves or denies requests to the Docker daemon based +on both the current authentication context and the command context. The +authentication context contains all user details and the authentication method. +The command context contains all the relevant request data. + +Authorization plugins must follow the rules described in [Docker Plugin API](http://docs.docker.com/engine/extend/plugin_api/). +Each plugin must reside within directories described under the [plugin discovery](http://docs.docker.com/engine/extend/plugin_api/#plugin-discovery) +section. ## Basic architecture -Authorization is integrated into Docker daemon by enabling external authorization plug-ins registration as part of the daemon stratup. Each plug-in will receive user's information (retrieved using the authentication sub-system) and HTTP request information and decide whether to allow or deny the request. Plugins can be chained together and only in case where all plug-ins allow accessing the resource, the is access granted. +You are responsible for registering your plugin as part of the Docker daemon +startup. You can install multiple plugins and chain them together. This chain +can be ordered. Each request to the daemon passes in order through the chain. +Only when all the plugins grant access to the resource, is the access granted. -Recently Docker introduced a new extendability mechanism called [plug-in infrastructure](http://docs.docker.com/engine/extend/plugin_api), which enables extending Docker by dynamically loading, removing and communicating with third-party components using a generic, easily extendable, API. The authorization sub-system was build using this mechanism. +When an HTTP request is made to the Docker daemon through the CLI or via the +remote API, the authentication subsystem passes the request to the installed +authentication plugin(s). The request contains the user (caller) and command +context. The plugin is responsible for deciding whether to allow or deny the +request. -To enable full extendability, each plugin receives the authenticated user name, the request context including the user, HTTP headers, and the request/response body. No authentication information will be passed to the authorization plug-in except for: 1) user name, and 2) authentication method that was used. Specifically, no user credentials or tokens will be passed. The request / response bodies will be sent to the plugin only in case the Content-Type is either `text/*` or `application/json`. +Below please find tow sequence diagrams describing the *allow* and *deny* +authorization flows: -For commands that involve the hijacking of the HTTP connection (HTTP Upgrade), such as `exec`, the authorization plugins will only be called for the initial HTTP requests, once the commands are approved, authorization will not be applied to the rest of the flow. Specifically, the streaming data will not be passed to the authorization plugins. For commands that return chunked HTTP response, such as `logs` and `events`, only the HTTP request will be sent to the authorization plug-ins as well. +![Bad Address](images/authz_allow.png) -During request / response processing, some authorization plugins flows might require performing additional queries to the docker daemon. To complete such a flows plugins can call the daemon API similar to the regular user, which means the plugin admin will have to configure proper authentication and security policies. +![Bad Address](images/authz_deny.png) -## API schema +Each request sent to the plugin includes the authenticated user, the HTTP +headers, and the request/response body. Only the user name and the +authentication method used are passed to the plugin. Most importantly, no user +credentials or tokens are passed. Finally, not all request/response bodies +are sent to the authorization plugin. Only those request/response bodies where +the `Content-Type` is either `text/*` or `application/json` are sent. -In addition to the standard plugin registration method, each plugin should implement the following two methods: +For commands that can potentially the hijack the HTTP connection (`HTTP +Upgrade`), such as `exec`, the authorization plugin are only called for the +initial HTTP requests. Once the plugin approves the command, authorization is +not applied to the rest of the flow. Specifically, the streaming data is not +passed to the authorization plugins. For commands that return chunked HTTP +response, such as `logs` and `events`, only the HTTP request are sent to the +authorization plugins as well. -`/AuthzPlugin.AuthZReq` The authorize request method, which is called before Docker daemon processes the client request. +During request/response processing, some authorization plugins flows might +need to do additional queries to the Docker daemon. To complete such flows, +plugins can call the daemon API similar to a regular user. To enable these +additional queries, the plugin must provide the means for an administrator to +configure proper authentication and security policies. -`/AuthzPlugin.AuthZRes` The authorize response method, which is called before the response is returned from Docker daemon to the client. +## Docker client flows + +To enable and configure the authorization plugin, the plugin developer must support the Docker client interactions detailed in this section. + +### Setting up Docker daemon + +Enable the authorization plugin with a dedicated command line flag in the +`--authz-plugins=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. +This value can be the plugin’s socket or a path to a specification file. + +``` +$ docker daemon --authz-plugins=plugin1 --auth-plugins=plugin2,... +``` + +Docker's authorization subsystem supports multiple `--authz-plugin` parameters. + +### Calling authorized command (allow) + +Your plugin must support calling the `allow` command to authorize a command. This call does not impact Docker's command line. + +``` +$ docker pull centos +... +f1b10cd84249: Pull complete +... +``` + +### Calling unauthorized command (deny) + +Your plugin must support calling the `deny` command to report on the outcome of a plugin interaction. This call returns messages to Docker's command line informing the user of the outcome of each call. + +``` +$ docker pull centos +… +Authorization failed. Pull command for user 'john_doe' is denied by authorization plugin 'ACME' with message ‘[ACME] User 'john_doe' is not allowed to perform the pull command’ +``` + +Where multiple authorization plugins are installed, multiple messages are expected. -### Request authorization +## API schema and implementation + +Sample code for a typical plugin can be found here [ADD LINK]. In addition to Docker's standard plugin registration method, each plugin should implement the following two methods: + +* `/AuthzPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. + +* `/AuthzPlugin.AuthZRes` This authorize response method is called before the response is returned from Docker daemon to the client. + +#### /AuthzPlugin.AuthZReq + +**Request**: + +``` +{ + "User": "The user identification" + "UserAuthNMethod": "The authentication method used" + "RequestMethod": "The HTTP method" + "RequestUri": "The HTTP request URI" + "RequestBody": "Byte array containing the raw HTTP request body" + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string " + "RequestStatusCode": "Request status code" +} +``` + +**Response**: + +``` +{ + "Allow" : "Determined whether the user is allowed or not" + "Msg": "The authorization message" +} +``` + +#### /AuthzPlugin.AuthZRes + +**Request**: +``` +{ + "User": "The user identification" + "UserAuthNMethod": "The authentication method used" + "RequestMethod": "The HTTP method" + "RequestUri": "The HTTP request URI" + "RequestBody": "Byte array containing the raw HTTP request body" + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string" + "RequestStatusCode": "Request status code" + "ResponseBody": "Byte array containing the raw HTTP response body" + "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string" + "ResponseStatusCode":"Response status code" +} +``` + +**Response**: +``` +{ + "Allow" : "Determined whether the user is allowed or not" + "Msg": "The authorization message" + "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or nil if no changes required)" + "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or nil if no changes required)" + "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" +} +``` + +The modified response enables the authorization plugin to manipulate the content +of the HTTP response. In case of more than one plugin, each subsequent plugin +receives a response (optionally) modified by a previous plugin. + +### Request authorization + +Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. + #### Daemon -> Plugin Name | Type | Description @@ -63,7 +215,10 @@ Name | Type | Description Allow | bool | Boolean value indicating whether the request is allowed or denied Message | string | Authorization message (will be returned to the client in case the access is denied) -### Response authorization +### Response authorization + +The plugin must support two authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. + #### Daemon -> Plugin @@ -72,7 +227,7 @@ Name | Type | Description User | string | The user identification Authentication method | string | The authentication method used Request method | string | The HTTP method (GET/DELETE/POST) -Request URI | string | The http request URI including API version (e.g., v.1.17/containers/json) +Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) Request headers | map[string]string | Request headers as key value pairs (without the authorization header) Request body | []byte | Raw request body Response status code | int | Status code from the docker daemon @@ -85,36 +240,4 @@ Response body | []byte | Raw docker daemon response body Name | Type | Description --------|--------|---------------------------------------------------------------------------------- Allow | bool | Boolean value indicating whether the response is allowed or denied -Message | string | Authorization message (will be returned to the client in case the access is denied) - -## UX flows - -### Setting up docker daemon - -Authorization plugins are enabled with a dedicated command line argument. The argument contains the plugin name, which should be the same as the plugin’s socket or spec file. -Multiple authz-plugin parameters are supported. -``` -$ docker daemon --authz-plugins=plugin1 --auth-plugins=plugin2,... -``` - -### Calling authorized command (allow) - -No impact on command output -``` -$ docker pull centos -... -f1b10cd84249: Pull complete -... -``` - -### Calling unauthorized command (deny) - -``` -$ docker pull centos -… -Authorization failed. Pull command for user 'john_doe' is denied by authorization plugin 'ACME' with message ‘[ACME] User 'john_doe' is not allowed to perform the pull command’ -``` - - - - +Message | string | Authorization message (will be returned to the client in case the access is denied) \ No newline at end of file diff --git a/docs/extend/images/authz_additional_info.png b/docs/extend/images/authz_additional_info.png new file mode 100644 index 0000000000000000000000000000000000000000..1a6a6d01d2048fcb975b7d2b025cdfabd4c4f5f3 GIT binary patch literal 45916 zcmd43bx@UE)He)x5RM?JfJle5lt_rQlp>N6($cM@(x9NyQUU^sbT>#NA)(SC4bm-w zNPg=w?&o>uo%!bb^P4&MJ;U}~wXeO`TEAH5*`3>RXU|Zc!N9;cs~|6?8mEj#}{hMznYINBu?&oe6G$A5M8Mr ztp2%?yOZk~NktGSO(E}t`um}Kj_Q>uvLRE#Vv_j#d5(dNzJ`Sk$N2q`=ttiPm#IZt z{P`34KQ{W{zd!!}_0JMA*zHq|{>*G_?Q1M*>9Dh#VM&LRcFg&rSx zqV8j@3jHsWhM6dpm8o)fUw9v_7<@n2sL#1iD7KdpOnpr=U%!@JyZC-ld^Fc=n`(Vf zM)_vzcntLu*&9zJ`cVqiE^F_shw^Vnd;OY=O}q7q&(Hq!q}M^-d*1oauSeN#GtfA# zk7;ij$t%$jUG;sZpjTY`IUxJC_u*EcN>L!KmzLE~zAaVT!R<_nr_a?GDsE&_@Fi0) zP6ZiKT(Jn3mgzU2sByQuS+|h%I9TZ5=Z4@6OjFa=YRMOo=jYgFK8Dl#&*oRJKRBG@ z_I8}AvEDoSIXO@!_2i5&pJ~T8x6MWzY6I`1vQ&e5FZPXG?Gj3b_pTduQ9oglMuoy& zCkj8zwnkoMpS!nhmFGdZ$xQRJ(B*-gYj@1vJ1tQaX2DAceSOW9(*3ISU{|5!cH?dv zt+%?PzS}hR?0d05y!T=k_|2RWuepBSI^6zp?fH3*`3A?ij%ezK^Ig0h-Pc%@lbn96 zj`T%{dwa>naB5Q#iXRr5w1rX=e8P0zWwDv48NsZ6LB_Q#pDb3(sGKAk`rdg(gp5-w zoc&h>qa4-vd*_GWI%B^?2tPE_xU{U2DtlIg`{84{qUgQ#VN{3O#X6@YRZ|m~C5;oz z&gl<<*Df2><)&uZ(j6Yn&FAaieo{^4_T0Wj_Aoi_AbG-jDJ?cpaXVJIGlomUYtLZ3 z(8xd4X{mSFTTob7doH8%1DPk$TFmvwE<2JshbOe6Zrwd;H_7GJ$Ev7d(LNOY{*f`% z`*=I~_U=>zZNAGodH(rc8>fqld3JaPHqKth<O{E%xZw@1v&Si_61$gmMUD=>Y9uD`AKSVZ);YO=~aOVtu2P{+n;3-nzPj= zq^!!r-LRq5JxbRW?`9|`52jQ4bsHROR#xT2suU(rD{t+Dw9#JN`J^&**M3L3o1ONk zTgD);yZhJ9VvqEIL@bVD@X41emc5x_p?8b#E6Dj`VXKRb5&m!w+Q%cmOmXgT&T7cZ zNWHl&&e>YFZ^qct{jSbU-k)lV* zTNPn%C^&YXL{>-d4t5*X^Y{~RrhF+$*>*De@*=JBezPuz6q>?|e%P3k1dY-sr|Py^FNW@{3|+gd_x@e} z!J}b1>N=H074qPY#Tp0x?nch=&9Fk3H65ID0(Di(d&jsWN6G;>XZP$z%RlkjjR`Rl z2ESk2{L!a;alv{ZdrqswEQGy8JxjIK{o!M+kd08K6j68qcEbTZ5%ouF_MvQQ=|Wb6 zxt59cJ#U}n$J5In&e=ansD5+-yLx2noWZ2qVUgodzFz3hN$<(b6*cASb9>)AxzE0_ ziw^U$cx$tX>V&N)a=aDhok?jyQ#tWYfs{G!OQP^><-vOO*^6h$FRu>QKCYN;4Y^2g zNK1cN3^p(y>!OFpj$HLFUMfR6wSn#Qi@bsIUVMLBJhk(|dIn>kJkO<;zhqI?Wc*4& zJhWzeT)+L3#)e|btZms-c6)1}?M~GB1E;#i2Nho@%;t|puZ_{+cV=axNydA|Y*z+y z78*j()BRa$trh76)FMKX-Gq*~q(>oWhX|JDhKbt8A$#>ly9x1pjRM=Jx;t>6o!2Fx z5;8pGf7565aesejr6Xyh{$v8P(U@SiElhC6`+Gaf{7nLHnyW<4B4bq#km9;|J0cNy}-r55^xow;zzj zB^tk6P=EYhjaTzm9BuqY?aw4NeIcI7;7>IWvyRQ6T#jTPRI$fAio%&1dy5V6QJO8T~s7rs(4oNw&7KK${cDNFc4aJ^?&w&$A} zyRzC(u>4)C=$OU%YA)UgN&aZl6f&wI>PDEp@kOIUS;FE6x^? z-S~TE@uMFv@6~x|6VQleex4pjn|8)%9PFvRce0GGcVzIP<|v9pD1^HXtjq1S1ybu8n#%bC+FPfSngyMl&03W^N?PvBi+_( z5*&97vHIivoQ^cO&)c5N(ogZfckzFrU^Z09wcc!@-GaCu@0uBURMexU-S{0-&HjyQ znqsTz_G8Pgs|z0^J_My62;2^MXq&M>p(Y1UV|1#mYa@@C+>{(TpmwMCq z*VVL^o*2&2thSTo{AB(%Yfp>US|?G3GddBmn40vpX~&~qbEGRlP`R-8G&d&9L4wXY zo=TDXwwWbtI-?AIzaUaK;g=cYSEh}ucNJZKPN24>hglm+{pZ@|Q)qMd6VK~9Y&Txh zxFdM9@ z&)XYnrlw-9@T%2^Jh$YqysOeWL}geGVU|=Hzw^FO8RDbcbMkHx822u;FTZf2CbH-i zGVAc*Z25%Qfl*F=#)y|*hC~HSRFUv-Ic_O|2z6cQEC2bl`Y4?$oP=%O4V(u^+{Lys z;2HC`w0C$WLo2ZTvz&H-n55=)nWX<|D|IH5*5HoYH9rlz;&`8a8YwXs+}%pQd&pRQ zAbPwv8_92ed#6>LN$BBxaK!BmNLw-+l)BqW1cYkTc9V5^6ZI)!MPccZ+oEC{LKW79oZM>U8E(-R)DA}A5 zIV7#!Y{DysP}?Z|9s(#k-jRgl<3)$#L-$&18|7p%+f|b`E02t3Nt!rp&N|7e(+?*d zS@J!$Dw;l(t=^$NDDKe}Zv5V!C|uIET04?<5Sh_DJx^&>h1L zIK8w?y%dESx5y$JrRU*tPJv>**U>kBLa{q)bVNdrhM~;VGxiEHj57X`bX_E?XaEb7 zrHgQm*88w0j=eV`Z`iN+((Oc6H?Hq3FUYaWy|!A&t-ikVI@)KyDf({b!7`rYaalX3Equj=e#2 zyz3Re>5s*p@5culoxvCQQO2n0c9vvy-EUjNCSmPwZ@jp_|6^%6dUe^alQ!6R(TBXG zanhva`S}hxspKJD2x4WU6xZWK-HX4AOVmNFs_xuP=!CCq6=6sw3z6|nFs-=$?#5)n z3xaksMm}7_zDh?ENnuEQC0A_kC$+UDFztMr8+V;Ne|^S?j3%yFssUFqS6^ zTLv&bd8wSDQPeSDcf`>B=s?lDqWnxib+rg=vg(@!U$m>fXf2t%DOMg`+8DH3TPqmP zntAd`Z&0bVZbBMM*{tT{t;fA4ujrZ}y=3cuOZ$>gq1EE6!69<5_R+waj!O*==1ts) z!J3Zw;UW_P&N;q?XP%rlTkv`c?Y(&8&+DEhq)B8qcqpuem3wO0uEjwPC_vi)r*fh42NZj=S&#j7%PinPJBl)_@ zTY1AlqP2Oeg&l1*Bq(kUr?ZcqVCsLqPoOoP`YPkyJBrxCbNb0`FP4dNAC&3V^RO?N zwdv~a8C#d-N%fG~Wg8}bZyg_#+o^u#B<1>(m*c15>#&u<4$oFA+7e@f%t~Ebd;W(8 zKQG^PnHDYogo}WRypKNR|HTv+B{qp?mqo3<}UUF>%+W3}v!+a0E z$^ZKHQn}wkCW7Hcl=MX^MWZhZ$=>-k{ktgy_O&}Pl)S$7vQN&OZ_e#E=-heL(}b=HqOsc@BV32x_V&wbFhQ`mPk z(E}?lLnGg1h1_>=^sY#CnFih1^?@EJAk=68m1rJlv9504ugKVTv9?!s*%wJhmCbUf z*4%5`c+lXD^&xOxgJnxq++b%9iqdGhhC<`^NapCmKFdNBxfhdwRXe?IUm8Lm{%u_bi0U2D8)xEB_HOGbkY`631^n91a87 z8ok*X92KA66`tuMj-!0&X{7yw)2#guL-S0kK~s$B8P;{3oo2j+!2$zrH*-7>olOS* zy&S{+(_(^?-Y3^vC=cmOpwG3 zQ3{p&Z33Z!>!3O7P!{k_2L4E6kJ%SS$X$XgRZlr15C_8>7F(Q@9Wu zL6VWDt@ih#gq0_iVr? zg7^pCZpg5WWZ(yS^2_&v(6b{$#&2KQSl{Q|aUPYzFEaW;kLJB&Cijm}GkgtG5kR^Sh!qLnC{QPZ2t{N-GUd=Eb)r^9}DN>495(<;>G?qAQ1$#TmS`Y z3FF^aMpAg&$FEt0KmGR$M5Yjz7xPRc(J51x|Cqf&I7Z97_1u30^p_+gNsLpvLjTc$ z4I-FlNLISy{QeElHN#tx!jBsH`=T+>S}gG1USClAyUsDt0deqVgM9g}{e5qAm@6n7 zHu<`fZ*Et8!H3A_dzTLp0 zXX!4rR2x{JdAwQ_KDV}>)B~GGkCkLYsn~!tMUxzU zID~`?)S3%Iz0%L7>-U<3~m&gnk-Nb>Afk*2&C{j z+PTZDD!#wSs9kCifmD7m-|yci*_6RBuz+1$P)Fo>{A=OCw~xfBC+>=~D;})t7l5`* z9(p`EWZ=yuXg3xPsb1uj$?-|!lhF$MPADYjXixS6_g6xXYYNGFy{GlT+s(Xa^FYHlhMcRn}pjL;ztUac{38b`5xJzDXF`2!QRwD2UHjU!02xKa=l!|beOYEu z;l(P8AE_%%-bxl5pA}aIj`ZD~H&T8^o=^8K`KS~(`r~xMZvEB~&9R(LIVoVa#&E+B zYX8V-AWmG{w5TevdW&vSNzX3oh;G`6^R08&FJM4T2==C%0EKXbOXxBE?NQonauV^O$xeayQs;++Y`Fi{4g(R_qA@UzcjR*bqcc_)IUZh+>ROBY*q zy=N~QF`*xWk5`HH&}Ga1%!sKIT3d&y2FwaV8|7m%irNpJn3lwcRpy4MG}n!Z?2QJw zq|i&y7iLz4uNk zKYMLPiqX9%K#+Fie=W0Kv6y*OU}hlL&9oY7i+biY^@00?t`8|=^9`sIYHsl7Pq$)7 zV$%C39OH4He-b=(V27l|5_2yK4>Q%hlsT9CZ8Wu(6LCXhyTEffbo{e+HbiZ;c6Kn!ckPb`v6 z?b0hD$G1>JOY{5)5mT-A@)UO4Oo@LaWAI^9$U}GYta_kyAb|qRxas$A?W)=8Y)hjR z1wejDrYj{}0tTsm>Zu*U@ehTE9_u)A7^#f7)Yn|pbwyW+d0L1pHW!jGRjMD%Df;qr zc`S6_-Qqdf9ksts_QI0+)?MTG*nXPcD&ON8K0O()cFifd@q*0x;lNk6A9{!O9*0|P zR#6vsTGiPT9rZTK)*qM|xXLm6eip{UXP|h8tM@TkT>Pmh6@-JrsHM~}vFu*G+D95` z)}!TlB4S>Ll_Klo)fFOM5CC3(*DEr9A~Kn!#xm;x#P2{=M$)4Wp}^P37J9t_kK^bC z1oHA2;OLNtZ@!`hM(>QwDT^`6g*xHWhz=)bdbI${9XuoIb?8>G_4vo)o0+_ORjMu4 zW0ggWKp1WT0uW9EO19{^x!vrY9}x(%KQ~%vB;5wwc`#qH_p!UP7jOivRr8+`avmal zk0Ze~&}CE>Xdf-o&(=hXSu&k5X6aue4`35}`Sin}ZiSujne*)5J||*1!Y8am(GCtP6p(s z>}^8PO&_r}huXW=1oJ0KGA^zs0(@v=7OEo_+=p@f(+$#3IQw$2?cv!axE;8SY$?PH876@*NJN5K7F;DadLuIT-8jTRVqJ85cKL~!M6tEJy z5Zpt8y7h_<^;M3C-IxFoBSf0r@ds8MfI-UC3y!BV)bfDUw>-+2Zqqdl!CQmz`b9xI zbv4wy_3BGA%IQ&{zchyp&z)s7#NE_DD$8Y#M?aXT$PAvtrG!11!xxR^i>5q1;?2*`qS_B0Z52 ziQ?2&-N??XQ<$1=frpO0toOhw2XbWg?8Al6?Xc|LH8V}z^-G(@rC655)u8n)z{E-{ z_=L%D5tGB0Xet5Ygyq&+C7#1i_ zE%Z190HY^tiuWess_j>GgE*q4&FK$M;~HWw4;@cO^?29cdR2dC`h`KA$1l_g3sGy_ zXZ)*%a~F>by1W=flsQ$qPw@5=GxDtUhKH_4Jg?s_CuVZU^`nEF8fKVRo8e>6O`+77 zSY3cIiGkYlsR1UNCRKVQ!9T;BN;$N2zH}^%0JHYn-N~-0jIG|A+*z)Ji*qBJ^u2^0 z<1xOw#L90+hH#T<8n;T%NFB&F(za3!vwXVJR6?jy?eXhDwD6(O7}x68u?(Y1*U6=Mo+~~*b7w_W|O1lEpwC6ek*li+0V@pI!Vptc6@7?XQcw8#f8bmLBA8oi# z#+bc#l`z9($cOu;gKDcf9hD&Qj~|INj(wK*u+zk7pL0!wUYJi+SkIA+K7TWikoNLM&35Q_vNnge)(uKj8>bq?aR)yioT0#K z5!A$`zV)fHZv1?X$z~qz@oBBMmduOZsvMRSf}$uM@sOn}xjYgVO*&2)qr9Xj%mi_Q zW9S#{Ur(c#$ z&R4JnkB^TIIzeze^m-OvC3^kJ#?q*d@a3N$q%jy!f(yJRt*pss4p%Y!Tx4Yo_;5Jc zJX#Y(F8=cZ%thbrXRE~*O!RENS3VKg-D;oO5Q_?ZI?wZX*Jg*({)hKngL4l>*8`I< z5i2V7=&Nk7qR9^hDO_2fwRYbK?a6X8AH?0mWa@6GZNBN>Whzo%vz;8*=nP&8}XQ$w6!Siw;ZP<`+ z*gE&&ty{?yC}tZu`mN3$qm7%FYI+21hEwdWQk1ue<41`n_GA5;$Eqfp;;c6NM7bk& z#w$mlFwg^s(EfUTSOj*k&9p9gVhX*)rcZ+vJL<x!Un#tu4m5t{jkG=h0UOt6(V=zo0CZPzLcJa8`e2Qjb(rs8m#xUU2q|WAJ z60PO=_BY*n%y?v+nO@{LsoX>iZiSMK6&5{GXV9UNQ?DBP`ifhyr5f+I)3We%3n)92{igaPb0467W&o`dy%5KLz}{`G!>m+ zpm!o}8jvf!p07LhuKr;CS|gu9|0z!r-CP%+J8XCJ6Rk;T0$*XE(0=--SF(hTT=Z*l z@jY?Xyol)Bo?v)=$GZ8RS0m^kCTY3b9sBW@ZZCGCp?&YhKC%Lt}pFRpOwu z-51QdCIu`gGhWSVqpvrWm6z@n`jMo{QW%|SAw0kA;XhJ*f|BMIwX_wGB1vNByZaj# z`pOeEuoCIFW7HEr^hqX{`tbDZr;I9PtlKD?Iq8}E7YP4=U*qQ(6UEPm_zZh6C&K(9 zNcN%((-@nBex1$d)!?PPNDMHTxUowL`7+%x+K^J_feg?m>RIsRexA|=4%<)dMPFaOdV_kiwr_QLEw^IynLR}Rp++2x?;(Z6%z!8sj! zf^$)S5qtAhK;y%9;yTKIc>qUAIOpZ~i0Hq7-FFxP;KM)?VI9t&wg{N(Gk-(8kY<6a3h)*2o6iSf3DLCE9s=5EDa3?tW2WD7!bMGnA_!VHxc@M(F zqCcaqZ4rhl*PPyEJ}9-R`B`SDP*QOEX801Ll8+cu#HG*Q|06n7iGenH62S7`00T19 zz0KM*{}CPRFp*i|oB!r9T;d&KNQxKg{EzvNzX8XbCrSE``3SlKb1hG5?*1Rqkp)xv zPO6FYzkvv3vMpyg!~P>W&ciY4@^1gbeDuO>Gbkm_{l|Q~V~~)HJcCvq4rxfc8T$>; zl?(ri-9J1?KsYQJT%-LC07UnS0RyV{Y;rV%xOZnkx6F!JK|#Sz94f9<}b z_L_fA)Q)`l(iCdEtd7tpq;2DMI~Wpi#0gGdy-iH=_mZ z_1QFY^KqR+;*b+bq zK*rMPw_j(^0KF;yVBs^@TxV=7zygY`p*&p{jcoM@QTOlizy8q{<8Gs!Euh*q z48W${3r`u-E-^#PYKvmuA~t$=AHYr|z4Yk|eD{OiDJSb^Yv!?o6D!)XFTEY8pm*R< z@;fBv5m5zNN?Dt70sI?tyvoC_-@hpcoB<$DgWxTRcQTAltHUa+if<*8OyNZ`iTFA_ zBcSWu;Z}VY1aw+Y)-cduBGiF1hTvPWT0W)5`f)OWsAujxxBk5~8d0}9q14wdIW!GS z@2-tnEuZdV0fae_F;I@!UE1bP-D;zA>nzc)eZ=Rweox@}_Wnu%hwkK~6)q5fUkK*} zt#TXG-P0+xpx2SO@$L?p)bD^rM@9dv?>2^aaItCUnPxw6V-y9QJ`tS|yGAw>u(5(- zaUG59)VV)?{j5J(lR4WbglpM*m~jpKJAIY;)T*^}Xcqn9_{hk&>AoPgaDu^!KW5*m zI{4OyniA7aGG;{~RT8uAuHPZ}7VNxRJe3EuSQPXPR*7iROSsM0C>#&P)-c+FL%Gk1 z!l|>`nVaOBx&{PNnqi}#D)_Fl4C?C; ztun94*Sif?adZjWp(@x+vQ(vHU_x*3=mltqSr6u_j*uktt zOe&1#`aAEFV?MC6I8!|nKgS%A(#3~FyD4u}B-NwH6Yqr?JNy1jl?T8dT@Ze7e{A3r z!TF|aD$a!gZr2ijCEu@8BG&^{D23Lnl1X;cB-;7(-k}Q%WFY|J#bC(E`!Rp~;i%-p z$m}>?lS_pT_X&YeH=N09@%%bE?gm=(BJbY-l_b%`d2qB|y@ARRq>xfhfhq_~ZpdvO zPo>y#R(8iL$}Qh~@iETB->d!vmnoP|A@=g?M`z!AYf-#FsggSEa4S&zoBxT_Sj^!( zeyGWrt;cNmOj98~89jt+W6xtN_3AuEbjfP|2n$3QnET-Jbt+w+yV8XEzTADo=e?th9G2+s|^GX1gdW8$?xColpkpTwm(f*a%J-OeaB#)8`2 zcuw8&TM$0XD2hPz7GQ8(`zkPd4`jdlz`6CND_s=!+%I)n?pHAToM?G3`KoPv6uX8Z z#Dq(bI${yx)VkVrBL-L(1>jxNJbr!WGik+lreWl$dN3E+lPY%;?jy~j=dG*=I4_vK zR@ieRu297n#IM01+YYo()MY&#voA@u!$n|mj5Ywbs`@d|iNyUM?j{*{?nL+J>E3-X z-^B)Q%a_kcS)PBGCbD;-eDWah&-!72iDmE;E`D+g*hx^q8C}PnV-f!%B9E&sUS@7S;{MS#8`;%pi)&EidcuO z50||#Qs-H{I##7tw>N_)Nu_K`_o?dP!aE4?y>Fy(%IXapv6hAkk`FgO5GsmF_(*)C zn}FE_nd1hSeO3j&K$2v;mh3izHYJ>4B;=|T9m!hx%=KqY+=o>z^Yme}Vx#{fNJUmr zr=HCC9s~2584(~6LcwbsV5t5UOg&cbrV%&Hgqg_wk3~KYtAhko@H*SBf@~z;KR3^A_D=xy^`i zf5PwOWXOWGw0a-k7M+A4D>oIO1J-Ls{#!2$OQ)gozWVuIa#vXTu1F(W7Iq9;E6Tj{T*mWVjC-@Ufm{ET$D(Xvh}Pe;HR_u@3Ue?$nlCO_8x zoXA;T>f8lSUG~zYhe>f>O&^~TXs)e-)D%o5tjbbxM7D-CT@Ju=>1$ce73;xUz;C6< zB(H^J_`dNj? zagBGIsSGoO^M2`wu1n+laRu}P`Ng!!xI9_!Evgyq%z#M}WCZmc@0eSD`s|w!%V2~c z#UY}R+B0OwCVUIEkpjT>=!eU|9Q0rLeE4fSIxO%r9wS|xj&J1_86@X7bW;v)YhoH0FX%P(u4lmK+rdp-Qj<-a&HTXY-CI1*IiJb(GiT(Q@0Dz$3bS zDgkCATZ}^3@Aam+`@(D2cfh@htp&H?I*>i3{RAZ?7dJXgjwY8Ba*8LCQ!=>x%2ABC zo1{RcFgI9H1OYk5j1nm;V=3MZSBS9+a{J(SQn61aQ zg0iXMJ`SOx(5I$ycnPW@&k;vl(xYWIjnkSFh;C)okE*+*wd6m=rW_nVr;N*e>l31y zeII-HB`a(sUFK_>6=QI`l%s#R1Jv_h2?PJn<1eZ`QPk3?lda#$BGmgujqKkuxW$ye(fBSx*=xX6z56JGRdS-_>u%$UcO`uDDKaGXbD6|u`+zE?@- zW=ZDd5b^t7SEXRx|30r8%2*>9338xE(IMRiPzki0dq@dsnP?Ioa;Dl-yPAin!Fs4D z{!$E;mJz?#N+z(a(2*d#bIRXH;sYm=`{)d@5{uCk)TEc#F}>48KtH$f$H}n-e?9{e zIqM@66BBJNj#}U%lWGAE6_2W5nqu*V?cMo(uCq2YoT_4^mPzh{M^o!bnXb zd?Zt3o{=EI7|azf97F%+szyMH=ulcU7FNt#-glyyEM`l-!h9LgLkc6y?YcsY3xiS| zC801Ffz%dky__tU(RZSnO-qVZVNit*2C)}8`Rd@C&37LcUxmRQc;CyzebLWQJY$D|gyL%KwmnJlavmQJWt)S{{JSfBxf3%W?2W2;#*qM1bDh5|OQP(<7)s4LijB-R8w**>n9vlgxW zXt@#qRVt(=c=zE0mESwS$w;!IW;QD~|B@7PLD!NMFcO?XW&<*tA#R{n7D)=dIR zr9dRuqi_EDK4;n*d_nfmd1X)mJkU(gs?nRP&A-(B#vkdE2`7|wy%S16anvY)idws@ z45pC(FJM<$*Z!&-M_YpD`kotm@`3_g4{#bxj|dHqOrA;rZXJj^ldkc@93*6;GFmJn zplGs1anY`rsYD=9#2y{|#FLQ$H{QS^zK0;srz^nt3(P%)CJCh137=X-9rhRtQjY=S zP}$kLGzP5q^?9_fzbKv7?L;B^ftH0TOXtV#=; zi4hsTIfGa*G&H*cQ8ow!Ci^78DBmE68B8`cTK~&`De6ujz)L(?txaD-bjND}>lnkR z7PW?R#IlOzcH`^-fx+APR31vOG^>H^-U@qzS5Pe?t_gPaOuCP>n1l-=&dY8cy+ZX? z1MFe$O?600SBV2V>&G}=6VwcSe$4$bKv{KV!8L*QJZeZ z%-b&}q$E);+Q5o!2pR;aKW??0yuZj2~{ z=k@2e%={}J)Qq_P`fjRGGe29vb=)fbnoaRAt;NlSrGf2Ybp_EHqb~(R_uCtIWt8Sk z0cb2o%tp-Q$gSBnj*|%R8f5kaLt<}-h-(UkY_N0c;z=JWwU2bAehQBihcqKlxN{ngWfFr)Q-rM^E)Dt4D z*L0`8Enjvk`MoM0-n}jKStTmKpM5*{iZ4@O5>tp%J2B_~M$)L8rYc+VKAvo)dCxV3 zwLa$t18N5>-8t~sDthv^ey&V7-Sjbhm(9o^r08^7RTD9Z3l<5TZvZ`p zcuc-tb*rtXuqsJ~QUcNWR62zqGc#rg#}XLa?jO`uMRB+kH}xrCAuuBgpC_&t0FqE-LNHq7`@2h~{*UApL6>4cO8cx>>ohtb0jiuz zG`}~J=5SB>Mn#>Qwuj)arQbMl6HI>&;ioHZjVQFregR^c~IHvSq#vE zggAfb<7|u>xa8!)LubMC`q=?HZE4UqkZs7zOvz=p73Hb8% zAM_^mg`NRL#M&??L&SWJDd`c`mwya>6j~yRQ>Rp3n|R}GfFD!G>yVbe|KuZxwMnah zQjwCJ$s^yS{F^w3eIyuyNY0R4bh;A%FD=&LHdnJw7aJH{_Zj+oO}aBE!~la-(GXRx zMeCQ_XlsMb0_h~7 z{tUWf0C)oh=u|$W!ItC-lGwnbfY%&K@UZ|oa;(~||0dsM60c`&-6@i)l@3PjV7-EL zcgJOQczQej^IiM#k>#vg@3g^127L`Tz^cOt{K89UmHG$)U*k2e4(hZBQkv*}D9RA6 z9Z3COtaDeQu$uYBpSouvEA5Rn!;zyEUS&tpz57eZI zPT!2-R3Y34kE1-|Dgw7$wvGMJ7%|W28DdP`@o4w|mA?(Gp*D#=yZ-%3GF~YSlBJDT z{$}a;lHe6jxMe$i&EKv~shcS<(~(@dS9c%}%c9Lb#WRDqn(xeVUq%=u+eoEjD&oLI zhk(nc^Dm^ovn778%HL?w`GUKSwes}VA5x$g(r}}7Zy^4KxA_`>83lMcH7^b~e>T}$ zAS^IKBa2EZIV;t!+j}#Lh2m7(8zV^_&AKWakkK=*VaduZ_h-#Usx5zzScwqHXcm%| zjHFoW()mrn$>SP!V5>1*b6r>WVx&lG==T#o2YfZP6csQ?9UcnQv?5OO;RI^p^LGI9 zw?}itA`AhLC+WM6!Cydv1SNt|wLt&+t81?Lxq8)_A)$uZ3*dlK9dWBRyWCb&V7sGv zDP8$F8JE@VOQs!Bjv`lS4Mk2j>L&RPTU}6Wi(o*ac#6BnxCsfSwSHJ`p8)w zd-HS!1;!s7?K_($R9j=8Imbh8x@xUUFC-d`P%9bnZyT@NhD^w;6o2ATQ18B-9{*md z^(*}VaK!k#>V(7DYkc@hKGcANBp6GcBb$%tEEa(B=-S6W*%vfL3wg6$E_43+=LPuw z`Pv393gvMXOqkqID#burXT!O>=Xbw7@&-Z($QmF<9{IpP@xyCClyd_ar+GW=Be~~OhouiP^F^on0jxExMvFhDxz-q{zM2Su~` z>ZVwv3>s|Ty}h|CzXNj|hO87%nP1?QfDf=3FN0Ajao>>g^bnZYGd&cNJiaUZn(`;? zzk;t>oNG|856$Up*mz`7V0Xgxvas9RobIi9Xg;U@+yoah(Vvucw^nak!`a@{xP^$y z%!osCx4}FeRxkU_pf}LTnK>Z6`3$J15Z)(qT*?OJGi}@ZKR39c>fJVdGQAhdOS`3rtJ>BA(}xI zyEC{7rH@ED|9Ra>cmVuRuvSX{!(~KVSocgxxxRD8A`1n`XHbuy(3c|ie4V7Oe>AT= z9R>ttObHW`{}5g`f&;mf=l|L2nScO46N!<+i~gHq!X+#q23aOLSRTAO7Vdn4zY~Wg*0a5mpSi-3{XSCPu_;?2F znVA(4zV;}#mumY!W8Hl4pg32*Rzzc@$l)#M-AF1gNC(UI5VB?q92uOqAo@H%0eQ)j zOc=x*$$G@12pTybfaRr`HSda3Ndljx`}7lX(bj2fk|2aX+UkwWmAggS@B!eK z@$ySZ4vqfuw0|YvXu&oJV2MZzAyl_Wt{zkg&hw(!T_4wlT|^JFsj&r{mI;XM7{Cok zhgt9*D4Q=q>yrmQ{vB*fgzdg?%_a9yrty88xkaBI+I!LxBkSLtgsAu|y6M=!_Vw*c zvH_^Ftk55R#}bHLM&OTQz%$Ioo4zezk92TAH5&tbb~|9vbE$$8Zqgv0b0Q%F8UWNL zJb&tC>6B%|4`4{>dmlM7{<>@6$qYC=8fN{BkW!+MieZ?>CE2G8SeOHKo{xV)Op3Zr zfqCKjqf)Sot3e)M2MkOelqr6+r2M_W`{*VT6@h@)&~_`vA*W{@g5DXVrM60F{iVbh zu{^n}fbkP~x(*cRRPSVky37o9-hg#VB%pWgc{VmqOE253&@h$XIfV>JYW!RIeTitr#kVOPt7@NDSJBy(4G!U+5E`I@^ zKo&7JE}3Wz4gpoB7?BvHdQx28AcCL^u0}9r>;|4^`zWD%RRzle{z=h6!e$K++x4!E zx&u4+{iTe7(la)XK)W!7xXA9cJ^u+1KWFy%p(&QiJlDNS2T7$io5?!8XVH&0YVRtI z8^vcCw>($ml(M0MWXt=&{0sjVQAQ9iqQRxf?ieJ5&qP#!1iAvQ&PO+%U^L^_EmfcX zAHYDW+=Z?cqb+Ysz9@|0?724r9Iq)y=$@FF@7s!$!YJ}Tn!5^@n z(v17PaZ&!Yh_k4vWuifuRdk!E(Ik-61O6_cbATOz!k1!;#qiwBsC|7j4vkv_n+VMm zt`6%oMXF#Sbt*j$A&vT53_lD8^iB;tdH)Zz(9#qkDfQ4OnQ4m0<`X6OT@$7A)WF zHWj~43U~sIbE3InP;SU_IK}^wByJ@eer=@ch+;DlbNbr$1LyzX2;y%0=5692lDV4x zI)NdJ&8fQ&8`Mdq;;)?i%74;BOYK0w$O>qCo)?G8l-bG`@67kYiQE+*v1hIdI{=y))L;HhedUl>9VKZ zC7TjT#VA&KXn=Q6e9FdA;?`&SxMY&epN`Jzp+G3E!xR7IB7;geql~uGNaCAtYFQ3PrL)jUeV`^xfSLZOE zviBWe#3dp|T;?1ag&jBqMOxIe9Kb9X&!PV>UEa^trSfzXnwCl^NarjCTD#gM??K!X z+(+QTpQ~C(QllNJso`}Dxwu#)zMGOn6bSp{3!>W>3x-+?TlKU0wjDrTn5~S-)nUSo zJBSqIjl?+7^h>MN!b}Ca0ew8rH)VsDq3tmvJXYx_z5agXvOpR3!d-9ws)GH5r(5!L z@yK>l+ULAxhIS)m>>?JM#)>g{&S@@JxlRb_ox0 zgB5;m`m^#=)^XY7{3FKgmnRGlyib05^O}@sNN+8>?X5MmJ@G}&yvDdY()<{&#EYJ# z5Twau_hk)AF~(LmRBj#8tdgq#0<0l7#-BP+`e?{$o|^C)EiqTlX{Sg=y+y1;R*TvW zxg9)vR~dTM_e1_~-+*j{>&t_ig{fqUqeub)IYJ);da zgx`7uz!V!zP4+R=rghCZ=IQjVL%^#f&!9|TD7(5ssfu-unZBjH3u+LOL>TmQ#n2QN zg2Bcu6Dy>sKJRXhg-YZ)uMYF7YM2WDsnJf5rp(m*LW>>-sKJA4 zcE4H|+l=rDyR60;jYHAVFZc^G;)R7D^3uQ#e*;X}3+*MZgde6mdP2TJ`ecyq3a5<; zJ+R+c-g^vPmUDoH5YGDIg2x)C-g|ZM&agrAR0RBV1_;S|s9i&u(`^W#9zuQ)NqqvO z$26$jA_c591^8wyzsbhZq6wgUiiOm5AsHx5dE^%Y)+#1rAOu;fRsJdDgNS^ch}i>M z_9P7fghCdgty;;i{uUue69L^i24HZT=4Woc2|odF9}ED!nJSFTMB$6jD-;DkkfM)p zI!I~p1dqJqrLtJocYle-%ri1AkkGSaPuHAB_4ro)Kh(W>IMjdpH_SAav5rwuqQTgg z$WmFyzGlr96{2s6P?jOvNTTdZDN9*eM4}``HKG*SktoKRJ(N_BV1fEka{0G!w;_A6|>^ePesiA=o}cnETh1WUiM zx!6w!fTSf7W}~v;&9meRze8{p*%`4pEM@bS{qHJ4AEwqqZC|MAy>r+JNzCX}f@w@n z!$DRZJS&QUu~*+5F+)XafL?HR&j;QgJ!-Sc^j`)q*3sDkBMBXjwbDco3vTm#d|+We zB0iukk$?i~Fsr1En%#B0u?GQ1G2Zw--}D&mAa|1~9JF45ELV)wYCo;E0$InWy|)L= zk~|xvaKD*ju}d)hyPMb_0-~6VK~7e)!Tr0Kc$*0l=7Q@zu=^WgS>_laK5d#jry3)2 zd~~6|eR>bN%9b0hca!!{(hD^}h!*zkjS0&;-CoRz}@JR($N#)Kly=wmjvo%PU`xNflI%Y#~nejihwyS zFzMIaN_`6VB1xTjWtx`wLVEH}vt0*x=1g^d4)HgzQ&yHE(%( ziKV>;A5+8W^+`|NhZ^l;A_Z+f)Q87n2)9pwM=#E5H5*!SOAk$Dok? z%IfQZX75_5XjZ=rt)D+?J!<8_SK*q-kyJj}`=H035G?Yt27KZwC`Q}sCx=Nh_TtHltiqKwBYOO>XOi(dbehjQ< zmxD1JkLVd&f6PX_KQ`iLN0V!$%VtzV*jZpOx3TdOAhzqu3d_2Irk? z*gBnOH|XW|zmEcVeUF48V3m1zUFn?B$n1G3s%0j#F@!myhbU7Kae?sh&V zN@)hrz%-PRw$B&Bi!7X6s|U*lzVK;f7Q zykO|LXR1bQ6>q2UKeS3_oY03D~KH z$S01x^$@&r?XB9u*JQXK5N?$XO8L@KQJA=c$dwR(sej$+rs5hlo?YI6&DX4u{5szaIM9C9O0fJ7o2I{SN)pOqP?SqwKSJe9VOy$Qw}Ko)UDXGRpk zH^7=(fZx_@?FVR>)a;NFxYW6-h_Dg$ThN+VBT2_VG<0#XZ_Wzri?9vg9ze-wJ|$k@ zwnI{PHCgIm_-C`BIc$MD8tV-JkzH~Qe8eW`h{21o_f~9pdOn00L+3wXn-5x}=MY<$ zElsT*e^Y#_#NwP4BKT00i;Tm;5R~V)Fp%^6=;VFg^@1BFV7i7%CZUSY;Y9CaHEN=J zhoyDL)SwcY!HTG$;UPd5)UqnZ_uJt zB7ghNDF(mMqtmn&o4JG7XUL29J*dK7irR*sobzA$bl+4}F2miWj_&N+tW{@rKwIGO zju>#{$=yRq*XzH&)HElfK_W33K$li%y0APIGb~76MN3~>jDoLh#!^JIuQLSph&MP3 zIDBaYpFr`Zoa+q_?;c*>N`2J&N_Ryb5GJG^eAc(J96G$O=-vK}9K2q_5yBIc+3J(2 z_$8{6`N@#|jH9v+^iEY47F}&TNpbW-HL#&dY!ZoH3JjaR_qmv3G!_o&yZT9Gb?rsn-b4g3f zwIC76q*z5TXUf$y2!|BPbqx;`T2JhvNW>8-4 zDfzRT8=nUQD+FdCQt(1B4%OvZ-9*2`-o_Nn@ssa=w*s}@k>C9_buYCof zLGaKsXCwgotj1P@RX{m9zRUY7!!Ni1LePOT_?wyJO@tf(DK_auI70%> znE1j;&&Ybe9`)JaJ>uuJkJqFh0W`)uts0P*l+C&o#Q_YZOcYXCaPo_%lOy>4m!iUr z|8GTwvJV^k(;y%cDZM44mzUnc!1)0np8FvX#R5%JSv>@^A;KGg$m3q4uM1;U+5gi3 zQF#d5U*j9!dVD(wvTM8B2X@NruD-Sky_8lQm*ovB(wUyz#WII{E2iuvoMwt`z%$Yu zu`qATLyHi63&-0FV})iu=P+~JGX}e11uDmn2O_^ux*#*gWZ$EGFbv2?|9W?^1k$vD zf!YYT;l81&W)XWa68JJuhbMt0im@UB13ClypL_2eHNstdk-M!GhCe6-={FH(*U8UJ;;`_7D*ij*2H#r?Bz=+MfYy} zL$W#o(ag!dMn&@A*h(jY&BxT{zy=cD0|T+z4&oew4zD*agvM!9`}gftdK}9eVQMoC zE;($md;tbH3Ol56_z#X^%YXDNyY`|8L~+zgrh_3rVZFFU9}0e&(x-nL@34SW?n_NZ zke-VaLP4Umfz8#c>R^o9gBt+^kPhCv^oFLbH{ej^LsMd_inkyC#xd-lpmh$zlT z%We6=5F##DuT%Z21=w@{+O>ddPp-xD;02%`#wX+ejW+hq=gc4gh#NlRolc?Q3Is>( z?WtGi>OYizP3&ARt{NZ~&x>D!YeHICXMeqJYm6<4X>%&{gS&4r4X`llY=SSJu)T3S z0F0SbY3d1uxt(Tae3q}s@{C3#pQw1{6lzaF?Ok5A(;uf5;SDs^FR}HV;5alXRh2Hiawp^cAWt zwZbbfq1Hh!X1z?#{LR?uRw z(K;l&Wy{;occlBj?JRkIyWWOidUW*8(ZaTl17U!cnbY^^LJDH8i}5kfWTr*CZ?0Ue zp`wPB_gP<7CvN=Ke(v6~px)b}xX2F6_{AzR6r-xw=6`&@OT*;zEVcsWS?R550==R~n zP;d;H;aVI+X2=aP!(aIdzc$@P6bsbqMF>Qc&})!G z0%(E*!@|NyXByp0k`I+`54t=?yuva7W#Hct1=`yOQIk%{9^6)RFt_6 z!HR!hVt@e=)Uve~o^lHooOVG{h&FX$ZjdgRdSUwU;=Oxo&OR&WzcG!v2w)4Xeq%yO z6EESsczeJF&7khbFbm9-D8jjh1U~U0xJ2Go?KimxlaJwXITka+z}tUQ4=cC))fZ+{ z)7OK(@cWcN_1`>0F)OAbFNhbu{?YU_fg>Yfn;Pd|#r<)9kTn`K8R1wGA1q>TtZF@| z*YF9|(uKrCbCugRe+n+4QkM$9t+;q6I{MN{^xEJLo(}`Z5!zFMR$1h?$iWPORMXRs zR-!)KIehNj9UEc5gin*cGI|~8qXeaFpiORAJP6=S{&h+1@Na2Am8*69w8;mF#yYc+ zztR?Et4nZ9#im}xTj}XYIlT=~;dJtoF@0=w_9sIqwuw+1K7So$OoxA;ltE0lsZY+oU5pOHp-`N} z_q{*-lx3Pi!k4v4n`)`cY=Q9*{?rm^{&_HR8SgzlrRGxOdwvtjJZo9y6wKX%Mqlee zLGyL{5yXut0rYhaC;-#{!e!*#6fH*CJmZ}=il=&VS}#L<=I-{|$B#609@ zLNN4a9Tk;*ZjR;7qU&UL*;v8|f4LTP6B9f=cE_l(B(+1g714E2xt-aK0yR_yaO|@t zB4G^P>L6b{2T>YDaovmfd5ey@d+43yoxUC&6J{~s)FFaK zL@@r;iJdu0dfrb?ZpI9PiZ#|l(z>P)brCr1#I2l7 zAc+leJ$*;4J2JU4OqKH!SI58Cyt*S?PB$d}jN5$#C~AZfoqTvhX_b{;#>Olc z#@uhn#QZTIT_2x}ig(mrgEq3a{}5U*kS$OMa1KFOSOFGb$DW5JMGa4Y@q2r{ z7fLnE{|+VaMiJg_!dHpo{T}ZW<8Lx1&a2eU0usnJV2*Z0_I^Lq)7Cd#dx_`U zkkh%@=m?8Z!@0{UD#p(Q(d>(KDW7oz+tcuOsr`J%Gy(_WUPX>9Ime%B-1GR!yE_qF z=Z0S7zTXqltlq_K%@esd31Q~82y28B(SCLFOfq)mIFVrpl5AZrlxD#g(%3WeYqRUe9Acs4r^IvB>3n! zFm&?V8Ur=w_}aPZR~_os6A4cmY}VJCxI?%)L`e%-|2**^KbF)L=c0qoT)6Tnf=Y&i z7w8*kTzqae{lT<+ZOk5JPS(9lZb|4ZDDW$Dul#%!jq*_gx=(IZicpUH6%-r@FZK>_ z$R?3_Tkn@^&9~dzf%I2x2uWXt&y%&-VvRC0@Lp#M3z0q6WefXq(9T}pMjNilWk`g| zme%*>rg#dnHtM0pPH^g&1NNAePigoB8KBsz?Kv`W^Puf3^UL`V=k34f=J&NabKmDzs*#<+-litkb{DXlBTSc~dnKFTKB!ome_B`dmkr(PxpWU9! zoZwc(nN8bp@#-be?AP$&58>=<_!eX7?)fm*l-Ii;Eo|@5H{rHpaUuk%cDL2i%zPY= zRug5LVBe+kE!Bac9VN%kQBKB^gQR7>^H)ng8B-_`!}h?#xf}|T&j?8r>95 zRsT~L(_&sA1b7MS{`IeCU=yO24Ys^qH7pV;cOXFfB9kxUONC#HLq5TPQI_sMjR^pZ zU^?;q${+J$RVPXfAkx^1PyT%lc76=S3CE9%V<4cbubbTflfJy`t&*WtrX_Z}*Z$ZB zqt)hzNPUegk=&u3ep#L7=AXf&-GqzhrWrLi4Oq;b}cGGQ=tM3p+I9W4=By7t2Bh3E&(ZWtM;_Ur<2F|!4p@5wDP@_ zL2eXbnWq)D)yx={NVyCQ4oZM7!44J)AqGP*UKA<`q?=y3wzaYV#%L5;THFRcyJv8l z2~W78GOY+-^Tc;y1zvj$%b;HKN<4P37)S{8_f)hf)E0Ava z31V$wiy9eVh;oi82pK|CAYNra0YC$5^K*&{&mdT0cPRuD&c)JH-ok<+mj~f@=0Ie1 z#~c=UDMH7U4JGwC46Y^68lbkx|H-k<@X;Ewqo+5uk3>9C#8? zllw56l=p;IYbCKCR0i4qywWY%dCLZy7AQwUty>=L8eqm?E+Tx2g7-g5rU7}?B!gCS zNdVa4<7fr?O>a;c`gB@>w6_GvZSy?Xo0AvYl(-T2=$KXkYaaNK|*^BU(S9bU|n@jA6^s(L_hAfocV=Hx((Q!D8VSx}&on zr>Dh!pw;~v8wMhbbjD9uZWno%*uOy!xZ1O)W%P6;;qrVYiWlCo)5F4)7>u(?T1x!^ zm+s7?b>}3wZj8p2)|sNr;UaziQ_s46 zHxHFd^L#n#Ax5U-8y*)e?dSf9MCXNHk3Oip1(#q7>LM>768377G#q1Tro>%X zlnVuFD;|e==xLTDX`(`p(r20K_-So?T3K?El250lyQ0wf#4ZSDUreEwUWgb20w!n% zZ|{L~3LgX?p`J+_hgdeS%O&6CdmxGi&jAk>UV5n^9uE&T1xW~j0{>6U`!tiIejyAoTVde9!uQY}G^z7|spJQnzq$mx!X9-EOH|D}x;j=VpGMu1GdOyaN|=$@)W` z`#C|Q1~=FpBi9e6j2B0O7e{{UA6-FKKDgB$qM$gnZvgs09(x?AYQD*lAEeM8V8 zAsM_2fbry#rsn2W$gOCsYd)BNg6S8mS9y4J^tC9uMfYn8{E^m`o6u@vNzkv`fs+); z$sr@j06gFdU~bAl$>ML$ATkR4lJ5n4W~Xx??XyUzrFa&8oii|9F^6vxTl8`<6Nqp(BF} zS{69y{Pi)2ZP3Zn;%WoVH;LMBQU1ZC4#()8;l%u#A<)`c27w^z%p=k8Fd?Ig zGEPa`(9IU2u)F%T(Y@cxkXz+Ni(x+UB?TPSKhrp z>S+(+K5uvmwD}X11*ZwDGvFN#X3RjK)y{+{{XN3muT0=bMVhu2{@CP|g=XD$9m>9M z8lH|K5i@$!ZCC??+RFYS4v!PNE}O3lzMG;+q3lbm{n4ry7d_Gfd^H&|~*52+U(d_|{lyG$}YiY4>#j!w+au)Voib0J2 z?5^_tnq44<=&L%CeG(=bWXZe?itV`$r+sE{8Qt*Im;%hKl9{>ZXs( z?%D>CQ5$rl>V3N^^>A2<(cH+3GM*EQb!g})zjq`ke^Ua?104XXlnAqO5$ zOOT)z4oEJ^u-WhT*y1)@%Nc+CAzDznUcx4LVSnnjklq&R4ZIhOf4in=uQ?5~A3b-< z1(V;6_)dIK65XY#CjuNFvM(`>Zs`TDqpCN276!>BwCSA8JsgB49(PCX1101nas++g zR2N?fiur|-edW72`?^JZF_rAlcE3S*9&{re^4vowgd(7>Ma|1;i6*$x%6Ygv==yI! z12R8*mDu)FZwF+z-y_&0JctRs3i{BN9zue~QdP5%rvGpZTfL>ABRP+|UJLKKEV<8( z@8Yq+r`*FE^dNY0%^Tcuh*<*51@LI@xzHK{fSz*Hx_fhmE=jcwB1+qM(47e`J4kIkFIBGZ3h*xHLs7Y&Z;@*feUdhxwhsRZ~c zy?ihXk3O-k^U!yMOh?veyJM>b&0eYw$82a|n5$`zp+3uh3{%zCV$XS!;2@$NALID- z-HFB%q#LL=QGw<{^M_7PfhWdst@uIJUbVO%tc({!u0_Mlxq?f=XT{welrgUx;M_z)k#NU)Hc#MYJ1)H|83c!DI=;sw;c{W3*T52s@SQ}VRH87o#Cv%s_IT- zvaZv=^&7-D$jpnN#PAcr&`0ZgvdJ;P2WxnHTIu~Egc*Gz(5ZXBpZ0oH@M&VXy%)zk z*M0b=8FZ-jbv~U8O}!aRV;LmNHbcyc_zVD9tzZ|c=lSsMOW#T~TL1&2H>F82KTTUO zY=RU(OCo?h>j7&5A+`6Wc$JM6=u0Bsg11Hz66UQ|p#t1xo7MFb>yEtf7?@Z|bW@j0 zd!Q5Qx_tEmjJecy$elrmnn@Gkl5}EMg?A%#19qTtG?QKPPjI>kDE{YuWhnr`U=e|(O_v6reEiqb>Y;ER>@|6Y0iPi-MXtx?S{@Szkr7RU z4(&Vyjzk)v&JCPdWi^Azk71ck;Gzr!iY|1g>O;dv4~hW2)@A4ofYHrZ93Nq26jzj5 z?9PASC;9-8=@_B_%B;a)a2bf}lzEU6oxb_IyX2M5Vq4%xs4&^YLlBLM3hqDs+t8<5 z!U0dPtD()@lvs&Hv3+3g+jS8+E`f#@o&dYTx_ygx)J7seVwD9=K`V5S-thbyX9gDm z`a{yZG0=%26*tPz%)SXN9RntJxSnNsLDg({8;PgcaD?1_XU#LX_YyME+zS2Z%d3y$`5X*l6bf#6%g^7(#bRqM~}e$G>PJqqN3D zXwwP&{KeQOkkLB~FgjXQL{$Jpz^>61qK#&p`#@T?>-_YjFYM~<36b;`?;a>k?j1*c zeD>6Z!M^3Q^^1?8>Y)%gil`xu96=RWC>k7>`|mNLdT;gIorM=i4%;E$5XyVr3`bTW{Ha6vPck z(REO{s5Nphd0V@fVjEmO*Il={WTlWJ1f9JA4TE2Nti9{IzZ24qdh&`Pf&IwI3gvHNC4i# za~SbA4U1jsj1Y~lC4Z`=2`KDtPv_-y)`iw}ztG(H+I8zBAhF@=?u*+rgLf=POcY9R zZ42|+Z#Uk)>33;QUMNhVXr^d`;W6SxeY^0*+jO&?1;rsCPeE^**n;Pk@>9iLxVWx*nF z*$)s&S?B4X#T9>h8QMizJ~hCBz4^CZEwQ?rEwNv`sbIatriW&%WD$ZRl&Z(qHNi&Z ze2OXF4W$WyRwN6hO~W%WoPQwh)w5ANKyw!s2=dJZU~G4%Ler!WS2Gd6egLaiKNIJk z!zK`wkOSbl1q6zqjjoSK+JT-wBqAy-&l5|a*#YeWOJOO?x#_>jT$>T*`7=9Mq{3GN}>P3G6$ z!Bkg#V5f#mW zQ!g(Jy|=VBoBJ^PYzCd{BEQLUaHi`UGVz{hOtUAEZcFP0DD&f-6Pf<`;9Pq5;9+O{ z!s;~S;c>&@?TJWUMO7g`-@R)i1*y0Lp3%(GGPy%YfPW!qL zzlN_CIps!lI8Mi3bi*Cgx%K0d-5dk~q8sBy%i-Cp1@&_8!cYnWOO^iDL&b?Sp!d+o zN)Ql`f*?bssj5yfmc%>1&U7-^X&W3}+rY9@%v~@ZL9!@A5u`eI%B4NH67Rg!+|3Y< z!8iPaFD4ZBvKC=Q_~j(wbNar~?{B9dfDx>yx&=~zz%iQ&^rQ%QalkpdWC3_NUtaua zwbIk$V;qo(Aot}EW8y;}$5v)IMBwQz23F@Y!W(Gcu~9#V1vY2|T?75r)3JA;BHIzY zTu<-eNQ`abcwi{B+Og2qP$4dtv)_=j_2BtcOcLU0tsf~ z=}^w&t~OM+*rD&YV~^@t^h5ds zHZcP@L%LZ%cN*!%fti%@fB5rCadJU->LAJ%gl2;QKtWpD5G2WFA=mhR?K^zmkKeMt z`{H`v8%MRgkUw}_+*zhLMy#E|4j|Ff0L^1q2N@Lg9N6B?z+Nx}*|wP|3yt#-fW;C3 z#K1~Jwb*!5ghZ2aA?Dc$~6n2 z?gLLB9`FHlXzT6*FZ~?$rIWw_BGMBGk?E zFr;d%+YZH|7z{X10mDcF=iv6KML_71Ae;G9j!xu#J^_Kx)o9U$F;EuC06qyRD=R;R z1~D1hr;QLX6+HB{=2iOGO%^BDOxMq73vhP?09PjKHb;I-w*?VpqwC^);6i69=xUy7 zeyJb4liWMmK3p5MIPUa*sc`q_wB+DwaRqcLP}BHnX`Jz<&kXvPt z?(6e=6*0LQ$*BGG?OeZe>;Y!Wem}U{O5EXsPJ*JUp`|Fe9%Wu3Z5gZqxE>#Vpe?8H zG=mrpbLqM8QNwn|Dn#K#PR{J?>DXe!;7jy@U9UA-KzA zbJov8CYK|3HaU+ZDS6!^39vu=4=#JRKkIS~AQct{~?k9`l zSdbhWxhF7gux4-p*{Wo|v;w=ZEA2C!Kv{Ib9Y;%uR%@yV(gI=q$+U0#<&XD#KI43J zqJ3D5eI8sh=7ypbSAgUhxbgef_RzlG?*126q24ZsgtSSH;|kpjx^TT1lQnz1vsy^RHBqNoxJ4wCs<($4uC=d>*r3I zS|7$rwaR?0Ptezu3Ff+KDt}KnI07yq(p`2D*m=eDN}_<>Q55cobz%*y=i*{QL{}RI zL)-0=W#8mZ`97YVEZn(z;Jd59k*Yosp#CR$2u{^>rv0XmtDFupfF9J11DO-o=*B!Q z6K;yZW<2>b!oq(^7op}t6Y(#wpUMP$01j`u{;&9plokQc{`{Y2$ZKR&4j91=;fyB} zT^LSv^ixoBe3J4rDmdfDX+dnGUmAy?eo8WhMFL*ECDzNL%8Jg z1elU7A|mP`)ZYQc?1EfoeWOyzDx2WI+W8eOe$WyGXSu+pk@IkX?$76hkGd1t`#(J_ z9`-P&Lr?7C<1+IC1PpoKCO_D72q?;@YRx4;tZnB%mzz_BGvng0Xgf=uZuDMAF%n1B z4!~p!5Mp`3k#{`_(sf!POVAt60-xXT8qr>tLoivs(m?Up#3i_v;-WXd(dUE!?6Iz@ z;*;_`d8J6h3obTO7;zj5(d-LJrJSC|^=S4r1B9OCCrprX$aS|DPchzo_lN>A5%*@m zC}su&2Ez7_j=hyY(msG!U7;hGzCoFWPP397KH4)uYhQ3hgE6MIcpvaa3qnFz&VJ#0 zAyXGXwqyp1uM()dErX~I-Mo_zVSbPjG@ivaF_VYV$BT-Ju%2uMgqa(TH!h$EvsENJ zP$Na|fB5ta$!F4`633@Il>WmJZwDv4C|YMr;Kkz?(CV=#Cn&eEm=TzltX#kxDG%?> z;r7wpHNG-H$2i^&K?DitgzW~r047*VvTCo@$8>_E)^lpPc;16dlQ*u|_eg|w{pE)^ zym(br6=v23-c9pWNO+ioSb-L3QMER90J@BpmeLxJf`14}Ls6s?0014zf;A0E`H1ZT ze=eeE<{*OK9J-Ve({wSB)?psF#}32sP~{l?rdvsztUR}(iLDY7EDGE?6NQXoF!T-I zZ>{37W*^W~a%DBp#uFLmtJmY%94oF@>~_Fb?6SnoA>`ALhdTP~5QnQirqm+Fb11q# zdZGW+%8goikAp^1mYN3)eJ^zuI2`gLDgMkP2^`Lv!$?qClDrs|Y#k1eiK8 zn%O%!bYSTRUF85beA?fU7Ubsgfx~OabSPs!HEh$PVOK?%F3B@iG#SPjsB3YX(yU=E zJMX#C(SuO#Qkun;>*5lz`esb84(kz>U}{3arw-w#L*myl*K56W9ca8?|EB+8JCW%P zz?zO?G?h;C2S%=qu{2=t%dcDs7rHeD44QM7x%i}oJ9sXRLh|k+#=TX+Bu1HZc4F(s zO+FaP(a|EKGbMoZz=zH6bIe&69c_kKIIt2%jn&2O;K0_HQbf8_cpEg;)C8oJe)FmI zDN81>!%AbfP$qaC`uJT_Z$Ds`Alnjb^5j8RO#8D+|*Ihce7U!q>-L zwxCz&jFz^0*QLatl7pP`)sf-==z;{F;5a>eV_w zU4;v6;IJ+Sa$LNd1+gI#E+9e0bXl$tv@_@r(G49CmH$Xu7sx{(h0{gIpf-mHhfNcJ zKp|9c>|*5i^Mkc;%*g2>zwVCU@vN7R{r?t-!%#6qCkr4A47mk6q<{&{0Hs`WI4re& z@;=J%{*HY>OnXDqh~&9wv**i!1)cw9dKiWsXo-M>?}2t#0#=>a{p4;;131nC;!8=8 zI>9jJ0L9gf0u+$C;;%6WC7vQ{LG1pt|Cm6iJ5iX)V!)<)lmBi&_+2n=aQEil@9ZbL_p(j}$j0{a!fiT|#KTtQBNTgRAt7&Z_$_`QL} z^J$<~eFoV7Fnq9R*$3Lyh0;}!`53B$lNd=uV>eh_%i@>}4Iwb4{U2bvuYUx8rCk4Q zk4Dyeps_})RB%ZkjWq-IxE@G#WbX+^yMod1fjj*lEHK6u!rw~b??OOaf7L^f{#@Jo z>B9#Q3*Ma?GXC?8OaOLdXh<{$Kd<1--s3C3e`K35?`8S@{{$DBCp-W#aRyTSfb|$T zZqUMx7L8*gNpm?!CItD>io26WkM@E5=?**fE9eQ(7WwR2X&^WVJEU*J_a=LURecQW zLG9YmWdlo_(7E{j7myJ{Z>;u6KKj$~lv~igix3jhp&|VgTF7m9ZsZR}D8+RQtKqR7 z+bB~@D~|BjH+;VbfrAtXWZe-qSF+Lg&Ejqp&8v$1i4DKK}k$9e>VKq`-%MWnAE2I|F8No!428W8F6 zR%rr_1|ZkU?&@+5mL9Mt1`P}hc+HP@+}}Q*+%LLM|2e|Ag03f^ss(?}lH7Ylg>Hj< zwG+fXqHRw%z=8}8d2LX_5;lP#VQo`9QD;q~Ou|Zx(;5SP=T4m7HHciciDj*lgS+n> zOhTpm@k(cM&I7#n99$DwId7WpEq-?dxbsu?6Td(I3Sc4@ytpUNR;=u}?v=x$XtdgB z!}fqnr{9saUVtQXm*sPCi6i7L_R?F_(*Q*6P|gu+m#(zP+NjJPb8`C|y?VI%GEsbl zp8?qyVRmc=TM7uVG1?uc_A|QeBY(t)l{{3{_=2m4gmZt_fT+JP_Pj7GiMC}fukCrN zlw9^t6RY&nfzM-1mVGDj4U-$O@46+w#dMsA&LV=ce|3|&*zgeG?WSY7%i4xB`unFdx!K!_1T!Qq#&7mKynUuvTvgP@2 z!P0)>m%FBBm6t2J)@Y39N$#}Y^Ma|Zl@bK#8wGrv(D0S$vT1&x?frQYGDm^e_RG zFcBlOOV*-i8FU>yE+gd*J1gDRO33IQa>g*835c%?F;iF^lU)|}m#XLAm56CuF*R#R zWMkWmTl}X?Ry9LpBCg9Zjj6~b3rmn?xOf=d_)Kcd4HlfuA=|e#ey%Ielo`YhdX;1v zJ85w^9AzDV#et$vk^xx#hK?Z;g|#}^lU^W2Bzuw64p=h@E7780`N*+3EkN`LBz^7l`R5N#PdX!R@X zii66%Q{%#m)wzrsfO?9T4-9%%Ybi|O2 z)Ai9Wz1mU*Xn_p0jL)FvbbJTUsB9kXX&>f%+5+S&O3!JKzTmxVL20pJ%MKG zMaw7>JOs_t=$l6%5bPX^&H$)^ku3sBCySt?cS@BvpRSH^6lx~m4&D1==HPL>S3@8VwVIX1+RjKH` zK+bC2`jLO({iEf}^|mvnexRj!xUziw9x!`^o6*ZZucAx^9uck4W%nZz%(^)8&_L0r zoBMRim8_SW%E6*gEwtRr3UX+f8lC|fHH-1n!FkyFw1~D)a;4LywbAPt*&RW-D zZ%_O6zP@nc+kH@EhTr^zbiE-en~13$9}b6`>)N0C9OZSJNSm>-Nb^URt-)@MJ&nsP z`;Mg6AY>cAC<~3<5x&o`-Bd}ELoHQQIQ03a+95=og1z}HkbW{wl5Q(#Km~Yz$2w}2 zTf%vm@r+P}h1$~#u*9qsmi5s-UAKpg9Ycvy1vX^~!3{Z+Xv=-*SIul3%FavK?3rhv z0rRm1piV_@u-m@rrHA$$Lz5y<`!FyCZ5u&BCh;PH6%E=cUF$zm$3YdO zTnahD5uiUtVmIV}(psRO*y`~Z8#Y>a3adLhl9hGrZQjLJ8F8jB>~;WsbZk@nZYt<$ zdc?fNpVe>*SbLU+>V*8pd#BI*gw=Z!RlB+5JZcGz3i#?@#?J?tkA$GS8$Sh5TjxGl z_IrHq%t|tpE150Zx|0icVF&Locixvf|M|YAta|qWl1d`YjIS0vK1XV`&~lca4v;>h zb3LUla4=p28odyo-*CD6Jb<`0pGeTES`A0R;sl@L0GSB)^i&@zuF(doOp2egGu%eq zy}++Ge}@d;Qa_N61^IHe>tFi7+EP)Hr2^{#_k(EC?D>TfEtry>@&LylQ;UYL(%~W4 zWv)lSyJI2uhn~v{r((kCb#PQ2Jk#Lqz zNa!$3#xJ)+SDU$~;*oCnXYk=w{386mnt8R4>r->^c)K!4Kd;{wLW>5xwZL_q;#a|m z)>5|EgEMFlD4>cu?tLq|P-C1{q#!py-CZT)aIV|?V2SDOE@)4@d0RSgey3AYoq9lH zRk}^6;!($i6F;N*&wu*8tifT{ps)1u;@X@>1MWt4Azs|HM!#coP&HY?Fn7Q7wSfM| zn7a>m^jEA4eU1G0z4)&>6puDsg*}ODSm~q`vvwIBpyU=Z*Yp0A8lY0FSQ}kq<3#yO zh6==IPUqNz=D1z9rqSUhZ2B2)7ps>v4LGv}T!o*sJ)b?zTLuS|to1AKjke>dVVhr> zsU*`C8jrQsuTsvlT+&b2S`M^C|&sF~Vo2C^aDi1dA^|r*2%Uhu~ zPUpH5f9*fsmZRJ~SNMsST^I?~56@s8l_u*dL}Tctn5GiJw}kTL!lmWKNG>j>uiVNsv`p%FSK5 zDgGZ(QHrWyDabpmp6DMmG&1@W+8AdEm_TR(?qkrsg3VSezk^l09~M4N-Uk>Sc0~&R zRPd$cp~r38miC<=Gz>bQQ*voV;G4R2Uxi=)z3IeVLw>wMda*QF_6#VZ>?7UB&1;WS zx3TP>en{|KlZIb|mMKu$c)b0s3YG)4Sl(Vp9X@)HeN^VruV${mjawdk%LqS^Tu_iK zC>Gi9KJxs>V@;U{zYq7n6WL^an;7J~cEF=8@pH*}msXDA}UD($H z>!kXRM4Q(fvYpr+uj4cy(j+eWkfm3h5*7L6Q-3YUlENe1Vih$d`hGPWnI0IUs|DTt zv?o&Q{U_V1<{CIq*?Zvw&J3SU*rpd7T@FLJb#K(79~wF9zh^jh*4807_kI1xLwhg_ z9`Pp3xA!O+#@4CkSZ#c-a>sAwZk0LVD{Q~GIw6_btMdGshZQzKAUULO*n+3bwQyU^?Tl%+|yzjMR``uVB_lhL=YYDM1-vofuQSS#;`km_r zGr!{XA8I{<&Jmw}9d7-6MBr|Ej2Iq9rNPT+-5FR=t{n`l<=?{})ikTcv~w4yzvpSMik@;EUO76(hx`uk(x^^oJudl%E-PU#*t+I-M zs$|YIQ5WXstL)$;6?#)8VCr!G`-}csmp9Iez))LhmHb&j;!bjHx}gwM>+pS<#F6T4 za}V=Bk}=VI?75d5xjxWw<$_X708YdU-|tdI&%pxdMvVA?aSB^THd+V22m!y*v+4zY?kG4)Q2#cv5613ZI!Z= zi5SjyYM0dJ-j?$@;ogx4T!qP@#l*8%YW&DIXsk<{DT7O)$wtbb2anQm#8JUVIKjDh zdXp7*=;xK$Psjf-IR0Vgp+X6yZ+%P`{(QbpbRcW_0R2#hwf210>BJM;n%{lBuqhsU zP_)#qDUB=sKHW$^?S@rCN8^kPKSXZe)kO$`r={+Q8Dm3b1I`aH>n`xo;FhOw+?6hn z-wq4g_e~w0Z2rXaF*m_@AE<|ZRZ)9HR+Ma(+3hS|=5tQ!>sL-9XZu%aiqiy5WQN79 zettnmi*v@ezk*ZmHJ(^Yc2{3Vcq#PphG@(>gKaquF|&RjKRmvXbn5K+&6suN$C~)D z6hC4ubk1S(Y3gmS8-6B-UZI<4OT4O^*(G)NxKs?yCeLZ6J$z9+hDd8{d2K>T*l@t1 ze+jwQx_Q!Lq73EJ%G^!I{8{76`V*I8M!CYJU_%o<6{0Tru6ao zfLZARwb|ea-H(=f(Q4sMAm}j=rZ}8*y~*D4GiV{WSF( zG{RCjBWFVBc}?36Xp*mG$-BIr$5*hOXTst#3RovfHGT)})hg%ujE`R;l4sbN>*p}` z_my`g^eJe!7q*&@`NyZ&Q)@d{I13)m>hfJ1vK9)Q_S{b3nu$K$=Ud;!+iJSJ6vEaL zb#}in=Fp7l*`b^>oPk?;cUlG4XgqE41vcvX0log@7nYsk?w6NuXWZ!2pXK=JT!BkJCkk$+sx8VZ! znu2Un)qi9wZaW`)nIJ)`6^0iy#JbC1C@r!gtubb76devNi3FJjOOph?ul#Nb9ZDN> zWno&iV^NVqHd*R3sTdFZ?;=x6D;oc4{g z&H6)sUHz-X=SjJ|7`swDN$h@l@?XyIFjA0&ti1WVJ0yV}25-JM$-ay%CaM|@eV^6yb}|Sc!~cVqyO6Q{^nwcAuFq4kT{?%e`fQ4u{3{2bv{Ja6$yP{|2sy5 zVwQ0D$L1S1sYYzMG~`JEodj=)nv5a<8mIZXA2OBY0ibcsi<3~t4iLnG@>%^G9My3rQ}2vgXj6;DwJ?hbi#iP!Kry(pDXOf4+|yv^0{9 zIQ>`kL}1qeNMe=TA>qGc(H9MZgI9>fNc|AjL3cfQ~JDl zEP(ab`+4e5nerZA`rhO3(}lLUr_8lAigk_Nh|vD13M00RV|mV{6d+lHlN4vbqA~~L z5=xI2gE{IPOc}0U=60)Z1$(R$04@&q7Ic9A7<7IwnhRMKjwlU6u(_#8S`+4-@n?^~ zomc8~vp|;fGv{`+L48UWA}j_#*SpyiELSDa&HB{c-5T5j`(g@zu?C00=m$zDap<^h zLoVr5@00TXeeLV0zyC+Tdn93WJ_q(yY2G&g+HFC*Do;r^E7oI^S2h3E82D6FP#>5A zmZbvdX?VmnG`efSZ`J+%|LW=5t#}G@+>|$LSDvMH*M402Y);0B6m&r6( zOGtyQXz7r0-PW|pamiM0p;;wLE-4c!!Z2j`*RGLDa?_=BZufNRuK9~3Rz2bW8 zC8z^O)}pW?*Kxh3KzMO?Z^%GZJ|F5WHdIPXu*Vsbg-86=etI#yr9ruu0U0`f-=7w2 zP`^wlo6<2Nmn}j7y9qe>UMOffC5TtdKql9{JERjBmbxpW-w)!G)fm8cI$#tUhPDkC zzRkPPanrn>Gvcwf|F#>Zx(&z`3|0{ zZqR#Qvb_>@RhpF|@jf$%XgG~-G=f3I6hKs!69b|aJ|fz74 z`NA{EWP%InF45)F=1ab5LSj zx{P#jvh=3LSNFFviTg1jV(}?stXeK&81e)*EHZ<20Jp}CMN~Dl^R@4OR9Ao|bIy5Y zNkFxr<%Z@#Qan#A>y7W8&?b7C4!x}+adaz$ERA9q8j#WQW%uDKeSKjjT7V5>4+8D~ z(thm1Yv7u*)74ERfLIu_Ej@`xdH_~TkvPE>+gKseg`^n?Uk*zyewgPdDeSTGw_clq zidYf_TYC%a0M8X`pK@Vz#bN||CvsOzG3ws#prLQD>r&CJwl<47{Oa-0RjukHVFG?t zEtq~k-W20mbutk=L-fEw2{37;x=lQpq>7s-GwFmh3Ax%SFVKIx>-V4c_Z~%@T12H(Ar-^1rWx0h24RG zuk%>N1VUWeAqNqf8i5RR(cVLQWJ-#FiO7(IDPec&W32xLYdE#~k0|LzoE%0w<4eBH z>*r@qwN)+Wh#yxR4!>USLZ`GO%Q9bxo2>#k70Y}-6j{bD=j&0|QkmwOL=b|oAh(8W zO-sKmA+&oFxpHaMrsz7TetoTE?%=e5B;mCRuWsI#saWwddqf<35JW`wB)bF*lR(kHd zr{0cl*g^LN2;55T)RwVGffVR7uf#H>u}~i$70ATCZR|+#k$UG|=!z4jFU`rAHG!KU zGraC@f4k2+i^7tKICDe;8DR}X?zI_XByEr9`N6c$nTIulE;th&8>Xr(YkE}AG6)Z? zoH%*qn#B||OTCwK2;Ogv(vVHN`nbX+LU8TjWTmFxt3%_TeM1=ve6c~kzqFLW-B*8# zcaAFwfkFgwXZGW1RGpju9QRbniV`ViW z0?9ptTbPE)Ya@x>JMn=10T_{?+Z$YzOu)Ap#6SCd6sG^1fo!<_ zB2Y%N?(aan&;GDAkVB>QAu3kpzc`no)wZ4u-yp~a*kLP|vkwTi+^?>=rngg%ACD3x zic~IdFfnn@*SwRp2m{T|7;#ZQB#Gi6jmC}x19J$y&;TPCm6ON`vz_hF5=B^DZHT%K z9O@66i*ixgCY?-$mL`V}L&ZTPrcaA(z=ROO7Q-|c+sM)0r8bgQ39r~*htA345(yYnR5`czI#;WUf zbz5I5a~FFHQKnr#|ErraZ()Ct Ol5(_nwX52}O8Ou06wbc@ literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_allow.png b/docs/extend/images/authz_allow.png new file mode 100644 index 0000000000000000000000000000000000000000..f42108040bbbf9facb9fff0c320428323f061e10 GIT binary patch literal 33505 zcmdSB1yEIg6hBBjN;;%dQfW}S1ZgBh8UaD1rBg{Mr4a;_loDxCN>UIMkWxZGLZm?& zY4%+8xBK7!?9S}Y&d%)4ym98ryZ65PjdRW?&V8hn=q!g+ci8JxciMc^7!KkZ>@vJuSY|M72eJ0*RBf^rxl2wv9?-oiI(lUN6 zFTW}e7?N}zG8p>)$&@ceomwRb_4h-s%3R0l@IfsB4KM!BhYN$k&kUDM4g>j2Tfh;H zC||MCpF=t1ZlOD;t;3&{XH}UwaPy)c z8F$#SHy>}Q*9Dx^o2<#69lQG;Z+3qFK~_~|J8;3M>4C4MCgG9zOY@PkTYBQ(s%(fI zd{4!$4aQdJC#H>uqm_$~U)H-unBT8V;e7}Hxr~4PB?0TMq(oj5kvgBv7d;u021+rs z>~>!YVoC%_c}jH)HAA%CJk(o_6v27QB#zNAf1z_slYPL`xp4U0`{6HzTHRxB9foKK z=|qwivI5PV%p3jmP7nG5JGzTZYVLJ3>{VKIPQ==Lex}Dze6uL=aYXm&+YvM2L8+tF zs^!|6gN^&Ye|^fn`Y=x=;le9-ojNz`nk5t~IjJ?3o zd3weAucUs$)B9aCYpBgreWGMPTv9aX`@GP!&MjX@w()N3Hl3Zv{`_<2uV1gVhG4OK zE`LpARVFIZEsX6+6ISNb={#}WxEtZ>MzTBPz}PQ-v^!!xH)LR*EFX66Vy4Aqs}2J9 z>VO)|qLe8DGO-6as|I>ZH39bS|!itj*|`F#IXAkVzVyWC6cZ?&Caw%Y;ATj zsSmx5OzTOK-oHM|o^BGi9-cRKuXCHbu|}X3VtVv)>6}}T5L?W@{#If z(FotI^;#RVUKO?NXEw`t=e1@}fA+iiY|Tqb{VC6~;fKfP^&WgFxE}MSp5Vdyw<=|Z z^`YVv*M=MUY8}<$Yp<=lC%SwN=2999+QjF&Q@n3A_zb8_y|z|Q7IB~NDbq+3qFL}< z8NQzHyE|kMBj)tw@p!!pg7rBp&2*6+%F(#dTz!igB{D^8XQvk30Ps<&+?rWfmv zT>t*A=CRgOp0pL18l{FB!i?dGGC#VJ?%z{^&gou9drk&1v{#~qJS9#DB|XLb58tdL zC^K$8tJcnuFN3AIlvidk&Yg;kS2-d68-EYK0Lh3TvBFCwkJ5rDLp_!T>&ncV%ZQTs zZjJ4?N_nlOiq~;d=S;oxE-$v}%j}4oJX&vFH?F!Hv(gbmcS-9+A%dxAXX(qkjyR_E z!@Y6mTY(4N=F#rYM4j&?$=VIFvC1gN(w7Au&oY&1$)0@GD>c5tI+<>EHp&v1r)=!u zDB?7hx3$Bew9uR3&TClFvSxLf5V$j-OsILqVff~pJoVkr&*sLO{NFjWN0Nr~*5@XX z;5d7=v{Q$?+sk~j_3KguQ-Hv#5(7!V!5kGU z(c0n87cIJ>i+Zn;B*Ir(&1?O3UvVowrnux3LBwFXSn=YanD5pdDha%gR1D}0lfDZK zxjhn|%e74qIff?d+k#_o+p@{>*Uwj!S#SKE>CpX@w+xo@yLjw_YNv618wXtJ zADDLK7Gvy!BQgZ|1^&^GUD^auTsyILU1xfCI%63+hz()=MPo8O*a#+#W8!wlRa$hN ze5sQg-LQdaM|Af#?Pc3|*Tu=k`+Oc!8<@DahpwK*&no+!ot{XH?ZU{@u^OhkPq(0Z z_8eRNhAl+5v>VPZIM}%vqa(i6&gn2*QkE5X`c3S3HcqzwZeQlw){ev?>8)?Ctz+Jt zitWE<9X)*J8*qBCp!AKHN#?T83#$}u+yb*E$KU6f95Z0$7L2YCO(b2^joy=cey1mO z!1L_XpGqRHKBFi1fwSk8!Z4pQlN!xi3M%jJ|Gq~hfiWM?Z!VK8Dy0w6Q<#X?=ym&0 z#chQVhw{b(DKsiRQx|rvOw)X)siwvRBD(rE{MHd))VNve8+FU_6N~ZA(!_~MY4qL! z$vfvdObHutiCyOv?J^|2MG~(+7a`1$vXV8Wd+RVF^19=NT;n;$np2i^ad#zn0Gqcl zZ&KZd(cNc#h=PbxJ9h}?kEP=d#-=K*B0Sb7Z!u_QO4apbN}H%X<*{Sq*DZ+lSRK`Q zb+hQg_CjAulmAf{)9D6|c))S3nmarqt5XG?YU#RFob+9aN4^f9$=yHWg}DrI3wpkn zAP$Y86FoQV>NxgBb?t03E|B8Qr_awt=iaS1J%Qj#Gh{#6P*=I;gz0`7Cv*5x;*IBu zzFmKIu!TQfQ{~gPp2gK~RUyO}^V`BUy^eRjy->tAg6$JcFMi|m_co%Q6V~aynr5HIj|<1)Ci`vSV_RZO*tFRQxMrAXbU@*JNX{{_qMePtrDJ z#Way@z&_~?K_){`9xh#g?Pqk$xa>eNe)A?r`i={neHO9r+~yuH+c60>z@{sABJ5+0sLQS*uS(xNu;6 z=9PivGHQTP(yL^m)>T2;MHOFq3_EJPqJo^^ozr;r#ch@!=ehO2>TMpz#Pqdf$j+jyrz9lD+@x}N`vdhSbjvXV*gS($mmt7n z?oKl_GOc>czettzzQ?LQadDbPGeg4q+HJ)*HZ5vPa+vK5vRi&^usS7uw?B)1*Hl)< zdrzDYp^VNwQAx|B5k<-SYLMe5XAOb0SE8}GyQJ@(HE+@MT{Vr2<>6B9G$HF7B3A5; zdr!X>8)z+$l$ZYybsS~(YftaDH8Wt$Oml8N(cfDeFZ?wsr!1lnPOvO+?JPgVjLh^4 z)II}+A7Ln>ogA4Nls3m!^48ibYiq^k@Dw_`wlTGWY*b!0CJz3a<32x^sY#UBiyH5v z4^@+^)n_Z_XRq#2ZE0M-^O5Arix0flsJ~4sYn@SF;8kkOS4+N8xc058liL6JwqR5S zW5)A1ocat-Eyj!bC8_k%em7&L1?mTWX?lniXlB^o!E)UoxGO0Y=lg4Rn5%Y;bi;yF zm^;G@M#XLGr|R3IF7EP=6}Q`VSUer*yn19O=eICt;=dl!cA)A=9UDVF$q0Iec>jb{ zq+fHh_t_Qo^AY3IYAJlYB=@COW>(6ZMara|FD%>5<|srY6*K8x?UVkU-mp7tviy*U zu>duFrDZLDQTx3@yr{F;?oDM=a-WXSUn+b?udiMIp6%pXoLT4ieY4*>m*A&Sjy%?G ze1S!lRzi}Op!xkVM&~Km0L}*jy-g5E_nhAOJT~%8cVa zEyPu_oM>uf2&i@Xf3JU-C+TKOCor-}tutG%!ZlIv zsqSkLPt+HmDrEgQiAzVGh)KrLV^SbFZ=5>!h*$#nHh9644EIOCy zD+!cKh)mT)TlvjNG=SE6z~8 zd6tLytLtMjcgjdZr9IO~Fj4CpgNn7pR0k$6_5yzo|9z)GjPzV@Ih*SSt*eg#BQ(aw z{*KXkY{nL$*Y1+ih8cvVp2~DROQzAnj5wE){&)2v4)>rS?fXSs_irwB&y+bg&+U?< z8;I#QS%pa|X&TaFQE_GJhYJaq$C3uHVJIcKeE(r|U=c#K*OCb}0NuR@CAcG63>pLx zwqHd@{jj{?rjOFinuWz6-GN+ubRqO6?xMbOYl zI&#%gzII%3JSlZ=U}0L_}3y`&iFiyC%708ZdKcj z24t-P_X)YrYwYqJESf0blPRFBhjku}E0Fp~hlW@>E|7Mu2FH{*s+^lyQ88K>qC94PBvBWm+m+x+UExs zNYH^wOnQ4Z02fzoagDYsnY*wI-;8e0@oGvpJ493bw9N)a`EVf);d8sY)K5qhL)~{O z$F@pD|6q7I4H&h~Yr<8Df2XZdEl8fJn77aG$sd5gP^=eSbGMmty`@$hu1fFg-g!s~hjUoFIQbmU^(?UwWHX^d8TrMD`~D?OyIfx%}$H zeWAB5AHTB~0?wV;j+mK`&wNi)S~h0d^zOfi+2!=z`kBb0fPGQtMd&f0fh8=yuPD7?G6Eh8&-H6VG!IryE(U%CR4zmYV3DA;Lq+x7=Vo zB$nBiJB51mV{V+xtI(59robeK)5aTd!BGv6>0Ud26I%LG7}r;SJyXi4H&dFs&V6Ao zje`QGzv}l6U+4w%`I(HCsnb2tIq6^=>+Si^Ti?B>J`EKc=oM}NoJ@g<=>mMynAKyb zOeTjzY&lwPF1y<-27s1rM3~}igema2(zW!5-<80zH+H$#nr;@|7|c~B7QO#@M6CA! z2!*5?``zWCLvk&rv!q8L+O}18J3ki-MG@%PTGiNZ(=mwZo@9pFMVcmpW@O@ zm8QdG?zs+xDConN0XUNgB}4H2l-6&k>7hqaQBO`!j=D=tYQ7TZR5|pFrQ!9Y3Mv3Z zF9>D)avzXS-4^QE7ehNbhOfU@!vh&d&ir)}MRxEBBm zOfM5QP$DoParT$c%k)jjH@I>d3bO)?`ycHR0JCT-hw=MxdHVf+?c_1Pszd!Y%4mbGN+^T)5@xJ#^lOgAgs=L>X zV6TR>_hrfQBc5a1romcZ zlJe31w)^vm*7BVn2lu;H2IQ=Y3A=F{DR1=LKf*VBfHlyYCEMJ5r$g{qSmEWs)t0!W z;_{|fdxABm?c`dxr|6&2-J4b7)}d}ZS3oX6HBn+D{0*oEHDJgLfH9033y*Kniy*>K z@W7ppIOlXQ?%Yh{6pE#nx*SZkSX7vKV~t3sV8bNZjb|;)7Tp!uT{gP z;nL?VjgKAj?tQO!JKlG_f)(aK$Y)r>&|<3*fv|?jm-Ox&?{D6CW2>|Dt*S7I@0Kx# zW;$2kej5`?_WeA`$18T4O!(u}pIJQ?`>*;dTqB*f*L1}Fc}B}n4`uy1N_5K;k}}T7 zpUtPc{8=9?{m^yQUEbvi7vkW_Xl=Mnw*+Hes+J4JFpI^e3R}S+SqJEvv_P@j@rH_$ z)bEthuRCodSM(BPoIATvq=hy^eLM^}!N4sbLkTZ^K_-2~e54Vw8*V&tlfhQ~GF=Bl&HIsho`9;QWU#;|J zNM=Ud!n-vKEC1$*8?4A%6*R3?nwdxSB+j0WL$8uw;~XppNeq?UGWj42b*jYJ+`;`< z;;CnW@uTC;c?qgyBJW*K*jnkwGwstzrIl1&%6~jf)W#}t{E*bb8YgwZW&IhZk`eVL z5L0q7lb<`MuL$7OE|5$Q+Ej3$H-;P>?yhvnX?(WuFnDXPC;CvZUO8xmo~3dGS7~!z zyV6qeZIn;#LrcK+yU}6s zL0U?9;}x?-qZ05$H}5YGoSyaKey7xLYi+L?Y#k88!KywCgGF%2QsH6i{-T z*!U_Q0816}1gAF~6>6eSCF2E%IRq9z0lPhItW-#%tEEaki0Bn2_LpArDf!fLi0cKGa@n&@aZYG!W=499G?qbYi&siSU9mD?Dz#Vx`HOS_4 z%C;#p`LiKBW5dy}cLAT4%{crHVWICle~PoFjwV|`jc&CoX*x$;Iqmn}_M|-Uq}Xlg zDZjbdcg5fjN`|3%3VG$Eqmk2f^=9L_k{KH=g6*6^MIt^+0*CD(N_|1W7?g6BW7zn; zBs(`B@BLVGmy*<#kfE5Wtzp971sXD&awq(PR-iA+SowQvyTvCHIpxddO%EoC1baJB z@LN+?XvJKz5f-adBdohic)FAHch`>>9sc=vShe2--pk`)@cH!>F5P}BrG%T>_n}m~ z6?GDKzDHbxju9i(r2Wk~ec{}gnZxjpRt(stb@OQJL?Mj9VcWEVDaAG1H(zF#nMaG= zez@QndcTx(qn|5$AN6LOv6(!zZI58%=d@<{!S+ITs-V@)1+6$47Ry?8O&sSq%{@or z43Si4sXFrQrBrj}LLsug9Y|c%*K%TEea9KFdnCSz;GVpyH2N&ansT<)Gmdlew&wZY z54;GtC%Ls^<1lKl@unK@6B*v z>wG)yB*9;FHsTYq05SdLFjni)^hz3YI19@dJetdAt1e74VjiNe=qt$rLm#F!FrLy3 zVDe{1M~M6EjLkf4Q()hSi?T7aYNa=I_EG!Fne~9o#UXovn2J`v2_bV&g6Aq{DA#hGQFQ3cqgGs0lqk#UzE0=&^SHJ51l(HR zOV(~whS0b_p`L5?d=&95l}U=M3^ok=jJ2HdUb2Ci4)J>HV}H{rA2Z2-$R}?6wdFz? zBym_yw)cI`j!*jJZ8u5iT-w{5TsTohCj~S;EEPXgdxwbXz%f5SNT#hN%dBeQCsmb+w_$8FC zvZ;Bh1P10Y|BUx}{;#bf@&e7}Ij^%0LQ~{~q|liK73HSMeC;hCM-{oR_IQ1aXIU=8 zZ3vEU4M53%56J253%FKlBlt_ZbtfXTpr7(b8K;lu<-LYtU5WLGoXk1iB}+w&N9bRDTo z5ZMMk-g|`}Hfgq|=%5PnH?u0L0((&_@bJkY+eOw)M>?Q)YOd=JX#e9Ha5K@uWPI*Q z{YPy$Dg?tPdFb9M+=)2xvr>yTpYQ+5B#Xv!DZYef+X)j1R+qUV!t2?0<=n;AuSV#fLT>!vFCk zB5-cC-6dMJf4Y1td3HZeU$o6%4L?4}372gtU&PhbwYrFlFoqtzfuVcOO|eYwDTNS* zLIhE=_L=#%AFk1MyQ$JlEu$6Di%2$i_AM$JLW>k2kCXtx#a_NQG0MlG4orlTRq?m} zwegzng6q$FbYjP5_)&SWFXA{%^PmV7RTH zg@peb61hbfLWRC0@qfb~WR0u6$oqxja_B!U3Vwk+lV3qk!oOo4WWLi1vRVF{Z$FsN zeqrm_|K^JZ=BxWM7U8DiG-#A=Gw-p?BWSe#{JiN1SPR7I+Du)~t`zf_)(V(a`xKi^ zHTmm3DAq4!1J2$|7D|wW&{)-mO|WjEAMb;aX?{tsMuz8HaQEAVPq&x#OawU<25q*scI(^@%wuvnwHPk zyMYMp(Eqzc^jTrF45NO2Oi(yS$@5YM)asEcxysRCE?9Y!m1?sPxBu9kd>{RyX*$RJ=;V=w~n zgQXz_7Kqx>-kSCP#!SeA=@>b2i58P!D_pK|lT05yPP6;zpyGW3j`eG~x%u9&@5Q}# z-|OF+Hu&gBc&)ya5SK6y_{&kDfahbPE{xA#{#x|Jt+cJNOcp>op3&)~P{>9sov3Cq zPVgS3fPHV$#V3di!Q2b5`^F@yC&sFNrt$ZrZrK@PfT-$|;l{)!0KikG0!YIe$dk#} z9#QYWqR}Tc;3~(sh?_3yt%=BWCx^Sb3;f1!R34oHe) zDf#A0?W=x!s}}(cCIQEz0*DjqfYsu%d9gmPxe@Iq2)x^AoU4-HJJ_P7m+(*l%)~)* zW*bVxAc6ZqhW4is2VT5fEp8rxK}XMsWbpC9wmJaz38K3n<6E6%DI~2rWA*r4Y6laV zWrIiN&I7;U#3&nJ+M6!UYAE%x4H)7ie5QLl7+B{PZFj*~;9BFqO*FBEd%9JEm@k0a z@MW0v|9#7P1=MUVL>IL=pRfjiNgdc4MQ@D@H@NUwAQA-&kWEL*M7!jsU73HUp2WpM zG9!15vHFE>5E=`=Np*>7quSDFdf?f3slCerE1TA7R`(K!%^j|d@xA7m zh7WK}K;6nN!2I#!2iQpXI(R+D_*lOPI6HoOt@ub16`3MeYsD&(ta=sBoWUHG3>2()v#Un<9W=R&y`o6|)3Y>ZMdL&KSpw0!@o(_7d~_ zeUN;M75yWKnVy1wpm+(hcDphr2sh4j$Su zWUlpCx`E{Jo2ABjh(-r{;%4!dV4#fmFVnXwMBAreE783YDi(OSR2X81S#oGzX&GkB z8~5;Zs~tm;XIX(}kt>^?=Y5-qJVeQ#;H`N=%BuA8MCCYVR5?Vb8`4@ND);1U^CtTt{3+)2o)1YD_hsmm*F_>#8%qpf)a9ROVz>Exf(6F#I-its%2y zghTU9B^`#XE>h#|t2-t-$z%|Jr4koo!86J@cL`naWvG|idB}pUG##g|f!fyEa zGl?GtkP>7({q92Mk+ASrhE7g2%WM|QiIC_@;)Ef{x&y9}n{nQ``LDlee(ZW;P! zT~Iv7{Y>RKlDk?sAu4NKmXgDG!8m3G8j8lB|Gfvn-pfI7j7|_^K0(TCtL=~d%!>_z zmR0eNd}PQ-;PFD`s;G}hvOkU&mfBu~t??dC#}l@EOwHONen9QuqMw4I z*94_t1N-i72y6cmBY=KS>;)gGr-=Y2+*5xpWV2OCkvA#a{F9}RXL13mzqmF5Ioeejwbzv!a zORP*Y^HI2jBvl-V+}9$ZijdD{y#+AoV6(&7k||=Pp4=hP;y^AC1$ux<9*evk_B~Q=FlRMv_e#D7 zQ<}K@e00JI1o^MHhd^>DnF2kIgzi5YI(W_yD?@=snk9XfnClA)UgMiWcYC;Qz4J;2 zXIPOH4iSAk1S+*^$8UMS$O?h4sLO#U$N*73S6I!w54+S1Ola{jkoG(;mF0c!jaOf{A1MQy$DITaNxFbn z{4nSCNLDgNzV0fLU`3!9sPV=06ApYIiidE)zCRO`5#ji$mdruB$!j0iMG9b8)xi*u z+kh_7nkjw%4v;}x0AJMsyz+r2kPIx7dahDTo>`-xy(ZIznXc_6^LxU*i!o1`v7U9dTl1FppX3(w9qN#^$q>dp1vFv?i9Dq_F;vd+ zYE|FYk0Y#hE$<#;6Ou*0fRs?I)B%3NsLh?)nyCO2u!*rlV2_6F{DBz{}?6T?*{=(?Y6CrE0i-r~1*|~45yD4)g zCy@O|h;h*>_gST`dh~pyPnpZid!_st#?Hlkq>v`HnGB*EHxJrhjt6m}F7T|09@5Zt zycWgsze&h^U18c2{Dfg3)@ZN$oj@#2Cu7@gg<_)U)@Ytjf=qhA*~JdjdA^&Il}DVGscHJNl<<5KRg`pJtkbrrQ`?!w|gZurWiIb%~-%S-a&jnC+e7}4@N{5~8a&XXnA{6N%0lmgbGa;brtR?`OY z7R~Ltc`MBcB~Efe4&fk8kBNhr8H_0*850G26+& zRQq)k6C#EZVn;N-De8adG1r?V?q2XrVZ^f|l7Y|5gZ4OV{`2!AzCwDnI<^1l-h}OoKVSC~A=lXSff{!~TLXcYh`#bNG`ELc+ z>GYU)gHD9d07WLHb0N%`k*OJgd% zHDn;x(M;jXcmUbWK9o2`YNF22x?BM=*;iX@xBuWaEIJpN;5%iZ9O4Wu7JxVH&(~(m>oEj< zPtbGedK-G>TSS$-3@W82u zHc?Ir!!SEyXrBNANiPHv?*O{0a8iyS#W>`D*euRN$S;0_T6006M?Qe`$e=@|9|Eb1 z2C_`At@7}D3>5R04>cj?pUV6ZTvo00s)?hzT+}g0Mvgi-J$+W0nZa6$y|(jg*_GKu%Ccw94X#Z9(Bi0B#JU;TKsF%Db0np~c00 zSoV|TZFuy0ZtwQkK7uTQ`W8dHzZbE@yku8dfe^2s7T(Gy|w;FUialCeF^!i^` zC9Ou^^T=Bq0#hKyf+nctcee+|Aqe$e1J1tN(leQcn#L|1m8^0y;wE#Ug<9Z?ankN1 zYV^nc?b=n$P+_t`uHr;z)M7_}EM1NyuVO&9CXHdXS#x;WtGrX!V0&9WgPdCMJS&;_ zJe>GaMTiMb)Q0axJmcDqGQu!kwW37h)`Y24g>hCF!H;>(jUIUd$!gVJW(`uJ3p3mq zlP+2BF&}!Q^xt_5gHgN0dsPytM-cQ*0CRUcsEg-6JbDR^_3iM+{^TE6C%2&zAjZ6T z_pgdBhoO()Q8o2`BKXb72Nx5n3F>Si^?%0|=ir`t#vxuXjmSri8t`WuS~4ZS)}kG!3I1kAZPs)4rW+fW;+P$TxR5b zglx*#rnTX@BSd(Lt(`Ey1elCa_`LFDSUq-YNE-13)%zi|qta}TE zVCc60`jQ0-ia3L6h6|Ywwp7102R1WkKJ z9gKzhEg^?!*pqjAS!8%R|_l@Cr7_i9vp0`Ld!rNuuZQab$e>A442M< z);b6U+R&$ggTuo;n}^MA$KMuW;>x!G*Z()cv8k;6;0>8Gj<|m-5UL$=eIk-c7~7&P z=Ke_xrO$|z3i{=__BycSbUhEMbJY-S_&s^QYd7$4$IchzR_m4^bcns@m&V=}^MDuG z}*0iT@*tzpJo`VH01tbStO5$b2YdQBmg02w+O$lRvDt@yG zr>O=^Z}E_3sgCze= zM_!>MU?U-fVX@V=Nn<0qijlg`r}N^PHv9hySI5=7ai!V=AaQM)?z$|ba2seQa-PLi zZ2(rae}?9}6y^a~&Qf8t9=ro}o-0quPQU@eWe^BPprMAq?DLy{SBUra{B^yCdGbi! zmpC6}NN#A5^I>>=kBH*gW=F>%u?5?%afU3EC&9J~9|dh>?Z0wekqv|z^U7fplt?^g z4Teyy723!2VilY#(8?;K{P{UuTo#vXLL0|Zy202eh61XP1BI4)IIbm;hGt^~io zu$J6Nx(n}@aEpLfq=7Dp|5NnRBqh}yHBC=|UCntT8MMM#Tc z?gbM;k;NEV@u@+HVUB+HY<9C$r5tJ0p*FU}IWC;2jf$B5QWH_%wYMX8pcJrY>Z5P@ zA?4hfMR?k^X_Hmsh7prHilg>;N&Y%>n!1*8n1l`foGAE95o0a^qtvLp{s4s}+4S3= z%|jLcQ0~4~#!l|kgYB5ri8@`gz?0oP8_IC{hPwTkjDGjMhRpY98ActfEu{WZY|#*1 zEAY`v$Pw$qnq}GRC#?5PuB5O==l4s#pfSQDi@c8Evrk+=u}b-7bk+z9mccz(1}@}x zPh(jHpjV(~p|ef8yDyO1BJ`w#&N;p_upBRtO17PGgbWie^f{VyIlk!a`E@Z2{^nz$ zk^5q0dTqKa&h9()Y&}B}%Lix@S^-ajL^0?G{&TwFvZ6OKmxF2JHcyz*la@e;iV%fN75i$znEaVs$zJcDnZb%vPyiRU616bBVK)*X0A z3Pg*HKr#<=5sGU|mHp%P6Nvs<6n;`ENH!p2mP^n=BO#8+fJC?&X@sN`9`gRUcQMx3 zCHFl5VB~gZ<-?7{Qqjz6x`%d&?pZv)8AIGtsj4ub0SW1d6gT4I(S6*w6I}7O9}uCt z%Wl1618d0B5)$Q|r;e1gib#%W!kf;|JvMxZK3TH?T>&(Dr4nnhv9$xWd>9EPk5$e)E*p)0&2r@57@=53QDeeO?!B0Z@7fV|NVx(r( zyc*SATnqHC{O2&FWVX<&EbKR#{*jG`j0qHI%od833u~Uewbh3Be^O~M7YWj#<2coH z3;3f6HGXhz@qowobR$CcBNza_$13_o$TQjf;EDT_IU(#L^o>vTRY0mghLGl`tzi@A z4^eR*s4$tqb|c2W`PT~fjs1|?#^(7F=AHE1-$N47f1iLE_As~*vq_Dx{-yE_MJVc{ z2xk8)4Tyv3MDKlsBzv4!(ZO=A@yb}R{-4=b!OerBoR_E#L-4_WmYoK^6qYwANUon= zmcJy z4nh|?cPcyrTZcA77<3k?R$6{QW#BTzn}fnQRBHV%r4}prRG9()&kllYBA7;s%$u85 zzrPc0wURJDS^KPSZv7)2Y+U$eG|#2)dm(N$1RRGGGnFk9tYcFz_Sd>vc@i8oZS5Gu zA(kj06!YQU2#cLj%HDkOe_LodENHGUhUqQ&#H80+R4Cs2?=0#BA0ZiZYr?(n6Zl~6 zoE{$tc`RzPn<1V6j_c33THC>CSCtQG|Bs;zaL|x1*uZqRzv5HzB|*m3x=t$RLd9d4 zjoJa=3kdrB(nxvMrSU02=PM%P=Ul@}C%dGebqVQs6S8PQ3oZq6Lr=t~`wM-_s2>lI z!Jd}DAh7Z`?j(wN{j&22L@K;_Vsdd`59LSMs34L&- zBNiczbTOS${1)A{sH(I%2Dvo;?D+Rtn>fU;8IND(r{eDPwJKRP?;V z(5txw-Q(}=!>j?C!!bGiCN)mBG{}--dyHnf?vlI?M8@ZCtDA&WRpN;IZB5<-8WQYf z^7gdS#aP6MbGNcJwg_oGt-H4f=*~$XP zVOPhLpu70IlT&fC(2Np45IqN%k-oq2CSRO;zNkQ7jt+fy8vSqcPc}>bt7+x0p+OtX zaD8)-xI^gsF#PU%&y|V9_xW)Hvey{jI}uS=${sG}s@(+cW2JhT-tq`)fw#CMbIzj` zR{>b&Uuy=a*|i23yP10&1g#9onmR5klksybzxC$9?D(T@d8U`&pe!9BoCVjwfcKRG zTWA=6>3HrDR!q|}y2;yH;W?Dwrz0m{5Gqvtsq)EC;Cf;Oa#%lVi#c=y0|Cj*J*ni) z(6<%MBy09-s=1tU{&tzP|KZMHgRlD>FE!B6(^8R_=KGYP9rL-UvwqY}JpZ8OCX?yj z=$H_GItZX~k z(I=wt>X~c)Hfphw{{IQF#>x9F>4wc1hJg^?$4M)?S9F|k zhI6oW7a7JXN+qiS#;-xC}ooJ8~Jcp8R_oHWF(nE4W0_qW&d{U>9I&BcAYW z|6yHWGPwYgD%YVKO8v<&Xn3JdAYZ3@r2X}2LmiL?$6bUiKmKJB=5%|2(%Gh7LSwoEv6^&%exs3-r3b*piU*zhlV0kaOvRKKzx{ z;^oAkHfU2#@%`^O38@t((0DRYt8&@UJQ`w<`)jna?ehfKfM4zd0*mw==Oy)qX?cOT z$n}nS(orQ^)Bh6@UD?$723WV27{JhkILNo9@DNY(V=8{W^@inN4e7IR#I4 z9?-#v_x+;L>%3?NN&OAJT`+=lL8uuLAMFFl1)}B1$6Vm;5WBrq79EVYDhQZbw6dAk zF*b1>Cu#-F(lb~A1~Nm!7?uZsbYc?Y3j7Wdi628m2h zzs&R^xOd}0?mFns?1bS8LC6FQ$|s3IvKS=3g)!FzpEdh#7nGd1Y5wO0px`q#ber$F z2uWU_bq7e?BLx;uy-&!Pk)57=u=e1^s$U(S^9f+kA{=etkg$eo(krgDRdG z5$zv6AvhH~wC!@BFbYCNsfjlYw#$`^qmA{ z?q6Wcvax#qp$zk0f&v$mlw=MJ#G8|^t)nJAj}RjC7;N)Orq3?hwX1$%RQgbR6@!ww zg|rciHke~HF6fKmWB{*x6ztmv&;fIFq0NVE<0=|=XJvgRt2R>*HEdXmivS8uUWwaA zGvKm$IM6HyExZ?wmy<^U6_C%!o31Uv$Qhi6JYN`NUyLdSShf?PVme|NSn*v+MYXHp zXdvvT8!V=vFTyN%Y~g6Pem~0Ta0fa}KV!BPA>88Zh)N*X1W5y~SR2rMGiZqdCQrGt z#_)MRJ;a1{h$oYE0#8w7Y;u)aB&5h%Hw=PZixDG7zN#K%={!}PJI6-CjRe%e!4jKM zT40H-Q7KateBM z^P;SMk$5=`v2T)-mXSHQoiu1v7|#EHMtLFE1gci>;tr7idgY`aim&PBsjvV;{{&g6 z(C%iD?3KfC#*S2>G%^or4H#LKqT@@9s?SnL-vLexzSs$-GuRpgE zSac4MAd+C&cR@{o`e6>1b?D#qDKW#ZE*2fD0Q1;?1@_wNpGRgR!NU;b-Vt~E86Sd$ z--Q@@5f>Q}0MDJj#Kz>mlhEpqL3~xP|B{xHsLp4C3`gjJt607 zko^V&G6#x7J?*_I*e5AS9|7{d39wJNNQN_*dzKkizR<{!(96fMc|$CLFY&d+NcV1U zdXe|RpJY_c89%sp1y=`a`n@xFI>;W}!Mhq#0hm}G2^4RMSu$h;n-G8nKZOmtWNvf@ z9&al9&ZpV10lL^4wTc5@h`~?j0}1JEfi~f!c(7jFf;__suhOWJg2HgCQ1(Q%?pXq{ z1Qi5igC)VRpx^1aEF>3kZJ`DJyM4?SY}OYjXP^RKLlZgDD!puI zk+)JFE$5^d99$3QnqmrsGAIuW{W%R5V`@tn^~ zJg#kK%236I&^3BNE=TmOH*k(gsVtZNDRW5Mk!f?lEj&J@S$OY3GQ3_y6^ObyxVgTK zRo*06?9)JerLAM#f)AJ8S;&?2-sF{;4ob|+`%Dc#5LWQu&r#V!!;BDe~7>jS%CV8^sHR$ylQpr)kEK3G-f#I3pAit z7Z$MeYywYu<6UqPX=JIyy4>KQRpIugKEKBcfecHc%m+Q451fvQg$pcOTtQZ{zjL-+ zTz*LlUkc&4Q(%!Db`3}3j00ytcjWc07PvS-<(%MYy9L7YBPnq5dkJQEL5<25EZ5UJ za9`+Q5Fe0rbq%hLP(*Bfv&s4dJ0^L|Gu#{Xh1J}D zH<@%9uY-p!gINtIB1`>6!39Fs z_C*@}8qlvfscBZij7VI1oGdG8l{QoPJ9tfDH=Xa+h5eL&m2=@J%#1AhC?~MUFUQ~CgolF`neL8y3jZ^(1R~Q&H zJ>uHw0V6ySwwnu3<-EK|$@8@Rg=~q<{_%F-kd#P)WICgE;8&Uv2GCZ5w(;6a`mOF! z7Oz&yGh86niFj#ok8uWk3`r1RU(FgyMOdX`c4A1Dyzj(y0dJK3kjP*wgM`a;OLiGG zBy;{~y-64ASsbVrCYQ-IlngWC9w2x~f+KZ2T=WHK$6b;nkA*@ik?}kT!fCkR!f(ZZ z*+90~j-n&`@pzZG#(7Gg)%R!2`Qyc&;J^}8$y>t;i>oC!Q+C;cQ#s;$!py}YBEUBB zGI6y<*#AMh?a?sKB>GO?RAHjM^0`kq0(>H9^X)b z9QKOJv@EFB3(0aAEX>)8GJ-Xj;n_ZS|L9FJaDD$5GoUI(2O0fb+ajUmX&06)6|8tB zbRI^S0sg!;YX9b&5NefcD=5O7bh;7ae8vLpTsOu10kruHli zsaisyYN4P=Q|=*E$&k46L1x}K<2(oYeZ*5!_1*R@78E$V{kWGwrewYfzX?Suh>o|< zAy@hPbGR~0O377F-D145&x^qS)ws?Mc#uV+!}q9d@8}C2qk1fvP6rJ!5PXM1Fsc}L z0ULTV>dk14{ZN;)1S1=Q=~a+tNEGRIllser(wl4PBXCIus;oUbZex3L43hiEz?1#I zDiZ#ynEr~}a5Ll+EC2;(1p1{al+M>Y> z9?jRFLAoc6K?_eM!g)E(#(0CEw0(%A9X1d8G$2HW*N)-+KdRu^yaST-X4hR5(ATSQ z-%d@Fssc(5clT?rSuupG%d-uj-f`zRk5`wbOW)V$oB=Q`7Y#*sr{Tk@ z`eC+sc%>9#S~-N*%IFN{samZre12{KXFFJH8oa)vN&`K+zy>N4(Tin2p`L=b1|@*u zQw`if?C_o?q1*3K$z`|O!V}@WTgY2{kg_b_yUuj5IWQ1$Yap&7cs~a}APYSxGbA}c z*NPXNdY}tgn>9{)FO`5=K+c8e{r>=SgRdimYo`li0|R2c2L44g7y~O0df(jEC#5|n z417P5?{3~Lywu;v^pz%)Twnk_GZ_jY&Uy`Jde6yY4s>o36+0|eTTDE$47ozhjGqS! zS>DAQs8V>d);-spiQrU_0X6G(AgirkS%H7mp~C$MRu(r9ri#oO>vxui)b&dYS!7T5 zbEUR>gw-ZM!o7%3$)k;d@A?Zk(Zdoj@y^{}8Rlir1WM~ER5dE4#&45cMC}HzAr%`^ z#Fm&hn`KNv>!d2Y?^jg=RAXz>ku#tIg z>L~|WKLxh(TbVeA)@k$R7ci%UmGcsX%6tiZ(sEYz(dFs;236(`JPVMMDh?hVY->jV zM}Q}Vv2PK}I0tW9(VITMab7?#DB%Q%6tBl{S_$!6gEG=i6zz2!p}Swd^=;Gz@x&qUQc2R&y! zg&7ik?GGNN{4dpgbx>9NyEot_50XlVgtQ0>qJVTtsDN}y3W79%g0}m%;Xd40h(E4rL?|o01jPQJ z9HBRTe7Bo#q1o+6|LJeBPyb-4NOlaVU~$0E({7obO4A)<`oc-~UKfm_Ec60}z7q|E zEF@Nc;NeA9R@r z??quvcY{KgTmzft8jEpcC14E!a$dqnYt10)mPWOCY7~;;2!__k8W5*RE!=CGGzOq7 zukc>$mC2(p5{&-V-F5bnCuL=*f&>HTn~OSwFta^A9BIOkgTC8^qYY-!WB zhIssd>D;?leSu-gXfX7|y&7Dhh(mq~=sH>65;BTHaqMu>B4J+`&~Hl(!JPUq8pKRW zyc>~Q_7xxbY9g--84Q0OzOFJ{ zQ)PD-^bnjeIu4704pR68%C{rhNFPi#-A8`Enc0nW;V%3C%MHj=?xo^gr?< z(24M>vo?-k0+Y!(qT91Ma+wY9Oc6DRvS@7pZ!6>!RLm+5Jm|3+UB~2;7={-MJ%k~5 ze@EnT?Sq_8>`P9A^GyAb01=8(x~HGi`;$4@i9vzQo|O2I6_ATUSWDc>&+BFXIynf7 zwrLv)k+)5mp!F4;NDjYXjLDC>=XKn_bY?#Qa>!n*xiQY(zDqA}3{rRsYU-H^54LRS zgt%a^T3p65jlw4w*Tqym!ZB;o2nAJ?il|5`Z6;A-M}RBE<~i(=53A;amE6ogt1+JL5%45iJoOCL8haJFyvZ2o}3h3rr2M96K!yBw0LD?W9 zC4WU7B%V>BOM|{J;9(;?weVqXEMC9@WCLAPY+M%*5+*h?yWsMwTko*mCR5Z}GBA@C zH4{jv`Tza;2#=U1thHsd0y4CD*IXV8)ou`|vy!3zB46yH3%K-e-B3vGrEP_X`O|Ba zQ>cYRe@TR4zE}+JN_W&{{rxUU8}dyE&A16GP%O9Z`EnHywS9vx7qH}j3m4+Ud85N~ zfotKf7{^X%SG2lL$iewPMUoHC<2@1|D8Fq_MkmF5l?iE#UcBvt-~zYFVPG>^YzQBe z{98?wWC#JWEJC1Ko;Xa_+JC()HX%yJ0(W6}E?2ki(L!fR;3o-M=PdX&f2jZaiq7;O zl=H#Yw#ofq%`U##!mJCj@Lb5qbnXVGD4u zA5RTQ-xr(*L6Hgq=oZN#o6tU>`A+rr*W_2Br&mG80!nKPDfB}d7-@q~bZ>oO~?0666%~7@3SLX=CmL6V5*#s<%58O*QkA0vcLNJWe z){42m16Uq3;Ir8PZHNcSq{2XfQ32*T)QebgZx3Xh0Sk|8=mg-_NZs6 z2W5$cytEGhMsaqDu7*==K;YhU>T=0>rg&s`d5XAOaksS>3 zfRBTNmF^`7A%`U<`ySo8j9BGI#*_po`g!wxLLrNwyAwU~gf`|IP&o zpkIrm0`cBJ2Ff__d2wfet;>gbt^yX>S&!v0|Cr8EK9(L|K~dXlHi-xO8#S6KD7)_1 zPl)Au2c6sGingyeL$wQ9BR>0P*Ka!N;ClM+3&?IP?q5<=Q z`BgK%4JpaE=zN>oza^BHy$a=_bg#?wzg|I-P%XK)`*HM%mi;q@7-O zN0Y4&VpUH`hw`Y;JZN-AqkIbGOZ{1h-Pn9lBi{A3yvAd*q)*>)JaRHbeCS@H1X}ZS z60j~WIuzCXY@)lJEo!g~Sg00bc$VW!;J8+N4>o*bA;j(vm8ywiMtAp?d>^Bv8(_P6 zpy7)K<)IREXj9;biiQq>h$`p!IYR?LJ6Mr-4xY*_(MM$L6(f}D>?0(>ej-@BJpPRV zWP2!Uw{hpWgu!yPwDkJ2`LlxE_q0!mDYwVDYl+;=MhMPu0L*2B>uebq0gnZ!GIhJx zfJP-7rP&-Lv9cZ)xI`eT^)<7^q>-%iZuZkL}jWucKdh0|hlxWp>dhQah} z->8*-?XW(aVu2)un1s|t=n-x{ra*R}-$UITj7Rzf*DT5Z6pecz5f;D6nfUj$P86VY z9=6gC{(LUeO>`WYmhivF@qpI@)|7g(e|X1Jz{Zi%Dp~w}J-|}CGyCcjd+Bd+#4ZJz z{ym@IE7pSU*FLAt7M(_+5Vk1e(>iy|`@wurKi@d_5)SVdNI3S&8WdW@kj{xO^S}yW z@#r41fFlZ6^Cn&SbHqBB5C6aT|4f$&5#9>?COaVS4lJ;GkstS+>7?hgf5$dK0>i2X z;#|X?2F3ZU6Ao2->!W4O{d@Lf7+CoR1QVzLaIAm}_HKL(-tEVMOiGH-L3~Sb9f<)= z07!doiN?7NgVpn2bq#bq@!%v1Yx@@a-(pgSsyqe%HbgH&}2pkwa}SrgwwGaV}p=VKaJcnV2?oP#?-c#Ie{uWE&vm9}2bB~=IqZABq5m<^D1yW#9_#1XuT@z62uT%bME zN2|be3dhX{i;DI9|5ZDPJi;Ow=O%F=krPXnKH5*?YJH5 z7XeFx?U37r;S(6nUjvr-dVuoA36|SFtTz@_Q--PmZ^mOh^R>7Ree}l|7D6~R>3kwMwI`tZQ-~s zkgS?OAkm!L0Q375WNjDKx;4pTIU@dF7~7F_7yb#3PrJJDO4{#;}kdGab4IY8~l)&Z$ zL}GmuoNIG+i~<21=dXG3He{J^?MqI9xa@U3mCJ{^ ztq=#U>ApkiJzU};R9B8U>V;VRO z@I!t9BacZ51PRU$%k_vo@b}X=D!h<^0)mL_+eEe6zcIVz-!rfRlem8Zl?TrXL_LP8 zG+pLPkOAqktSVI|*ypu9fA{5^zx#5@`65-aEjy4sj^zXsxPI@Id($05)rSEV6+zc8 z)cFpgNMr$YFdu&FA}q2VVHI#sm}$Rt_0}_esUc+42bzh6tR&yFBZV)|BqXl z@&~@2w{Vus9ya})u^82xzbH+R-bmFb7cLc}Z+ZUW{kvTJ>T&UjRJSdUab+{}a6%na z(`C-W9h5HCxDa2K|IKN7)w9XHYGd7K$@}2bf$!+(>ejOFW9LrK&W$G<-kt89&Ykq> zB#sih%Yj@g@B{oaJ;HZzw8R6Nmyhj{dDtiQ{Z|bvEOPE18>8z6W*a5%HTh>UdQl)x zE?o6K<$wt4AA-HB%-{Gm^eTV5$yt|uar{sAsJAfM@AFn6*cn)FWCHBG z@GVel-iE-{>{{A@j9*_KRsq-leewjkyL3^m=UOb^~EIZcg zM&+?ennm{z%Ck?$dnaAcuf~KObHVJB%r~RpTookNv0haH5H(a664D+Ii1aIFj2{?q*Il8i^H|V`2T@<^=FiF1g+<4r(rk^Fg089hWw+VD z>FR3K`qRtDj)ZK?hf~6wb=cII>C7Vs~?tr(I|v4 zQ*2(^PSX-?JJf5E5#wmA)#cuLbZF^gO2g#N9?|>BEpUj^@e-)^;Af`lDjoj5eiUgL zq?U~(pHY7S{Th>`9AmhL#p%Nd2}j8;cQ+vln`rR}P>Eb?x=dky5RfwIWL1^8tjBfl>ctfW^KkXw4r%E}7jDzU3Z zAazp=ItI_z4xvIwY5|+u2$C!EcbH*Nl=v%^{-QnC*3-`v0PD$uGUe2FO|S(jS<9I; zjPRY=w1o+%2pbWSxf{YPEHY0^0EEfyvI3h`pO$U%(e<4VEK!zv>0gB2Qv(ar8h@Gv znzh(ZhMqiMovTKxYy`bUTiTZLq6RJ07;tdKW;oW5Dea;Nbg(Arg!4V0?fKveK^afI ze)uY~3cx-i#-JzM@ zHFq6D6^~Nx9vk_b1{v&t@;EGe*+6g>LkrOz<9^WyrSi5G!g+3=1$BDu}9o zmhw96*N^>AAequ!K^UdJebq&MpJCgRj^bNWlx14p%XRo_6%Ho;xLBjFE8{D-Q>V=n z12$h~U0+lEfjr<`vyefd0I?G#ADcl=!vg@2Hvc|=LG=Qgz@^*(T9K1?No_?CTgPR4 zW9b~2Itq{wbNf{LD4WIiA&jB-y5^l8(*f_-#wnBD8LM^SWz0~PIsh%oA z8LhlegHm~}?xeH!Kk0?gg{f4}#Tb|e^WG4%k3l@_4QLBvcnI6;)8m$5rfI$S$slvp z4AFTI&Y@x4TjI30{;BQh!R{@wpmW&MeUKwzZNDlQMF{+G?Uq}{QC&1u?tWMCRdYUj zSymMfNT=@ajC4)=O?{WUeN;!aZ@4_n(S5FQ+6r1JWwzS>Fahib6sa4*61!3m%i9W< z-l&csAFyd*J((n$@x9H-sKJ1yFYG09M74xJ?xMbMXY{~uIWhngaLLkK?;TbL3kk-D z&o(=-fphk@&`%XZZnkZv>W1H65~2wK9jx`rJwbI&{7S^ZMIoE|R}_?GAu$GmW} zy$5t}$q`Kt)wdDLMUdc#=b&`e^^?wX(14Ox`q|YDoqIpIQPj*pd6mQXYz6S-NWgJ) z`*F>#oX6U1db27k@5l5&$xD|o%jwcL_Q61-OkbByq%Ky7Q)V};H%G68-6yHqo$$4c z!k$j*8J2C9WlzQmM8hh=YK%LrSsDW}uU2UPT!@HlkFx7*k{DmGfccFFf#pbjIQ7K2 zGY0&gS742pf*9p!SZ@f$-`j;|Uwd1RKTebB{({K4FrDCV zNN$p*i%BU0dwOBsjIYWsk1b)iC$S{(ULm>kkPhs5Ww7S=k>U|y4wc@ewIDS z7)tmRK$*VbNyBCSx>;2tb~mXd9a4E0o`T?$MCOh&M5k0x*uo`mYHM%rcLQB?&nlR= zzHjlA4`s~2GvEFQHPq`K0QjfOL1i1T1~sKq{kj*OcX;*kT z`IXUD7+VTF(#as3X{NZ4!wV_|&CdE@(kPm0&>WsXU*|+V9E6>Rr^5?okiiMKJc{*i z<~~OW^g@EfwT$8Wddl6lHmKh*(Y1tdO@J_mq%b`vx>eIQhs&B zc_&)cy#2$6?p>xEXE@b8Jl-1GLsm_2^6_%*A0*dR^z|$*EdcLtj$tURK^(ybO#M# zk`#*J{REqs*w8l3GT`)GVtra2pTu2%F=@*9U`PK7(mIssHzNNFwWtId*l! z?#;sMul?r{+wg++i+7amiqKIVvoS}~HsY6DWSiD^U|+AfzQ|uu&C5#!hD(J^y8d#c z6J_r|?-XjhucN8PfZJZYu5aXi{StWoJ}O~d!QMBy7eOwXbBqZ@&1CI~GnWek2uV5$ z{p#-64? zJF8o6YCYF_b(?hkQ}t(6qWCFRByzTK`S`~9me>Ihhf~0`3@D!%_-Yo)j+xd7F|x(1 zHjoKLyv0q?CeXr@#8r-<33g9gifBs&76t(91$78C4|TJKz+la6MRCsJFa<_(K)FH z<6J=2UIbG}y-8r^9SK8_0$4lz7G3+{*`p;cjR<8>$naf-f z`vfTR`A>8(oAZ#2i{`&tBHTSX^j=`g_rz>>6{ID31h}uL29OLxCi?T$?)1f(>7<2j z_rAh~7Q`jj8%d;Y2r?@6Oqy?ksF#IQY%N-Jb1aa&J2F%_Bke$5`SJzFWUTpW_s+ z)1Atzw*>nBq&r;_2VOViuy3&loziWY69^EMt{hnxka*ok_z(3rS)7F1fe%SCRfH~h z;2yyhLeaYHt)$T%D<3Q+kT&%N|=nd$J4bhu>3H1 zS1s4Ap<3r>gUROxvf3f$9L=$gaN+yk>693JXn4Q9#T}pzq4#Gij;MHQ9d)i^49B*H z#*vwH{Hi1*(sVeK8*q1?b_mV54doi(1w4^zn^KQYH zG+(;xgYdW5zj1|v+vxolS8m&S0-mJWN+Nvvmj&yQ`#+`jpi-VKs9aHMr_-)@dh1&0 zxJT1sp{O!3Hm9qXD%umrdEm1+ttowJ_oJZzb1Areo*EXn4O0wW9_o7CJX>pO1gX3A zTdh1`885bT3H!B_GUpon&kw$nCw`8YdNc`t@X$3j!VxksEw$qj+)Sp8hJi-7u7^tc z+RV3KRZS$RNE2Vfq{H}rfpFqx!?Qr0HiACU98u$SKj*aLn`ca(KZuneWbFuHa4X_x zm>%n~4%Q_}RvK$;WEgNr2>gEZovawxSouETdF=D1*R8I@W(~chnD%nZsaf25M-GlE zx2b<{IBI~rn)SFH`Q9}DQN`;IX>a^4F6&=c5fIG~eH z(Misw@eu_qZgd2fVG%J99jO`waxO1aFn;TthVY2>vR>nT&dsq}jgxln5rPa|&X=)9 zKKzREG}Z`C`M!L;$%3DTelpM;myydR{zx0C=;C;ne@&uahthI$%vf~r+^nQD#mCDw zK9x0_g=j~+fNys=@xQS8j6>LN?VFj@xrq19p+ehJ&Fg}}b9=meqmi+{j*m5U z>4nkOE#XkM===0oPAdeQ6(=uuv`pe0=Xf|sLVI3TTW;(enVMRZ-{|^`t4E&vg)OJ> z<6H^*g?+~af_2t0XYje1eRlnslI`q#27R1lmfVK2PV_K#QAHij_7qZBTK{mXG*wpW zkImt{nJ;b|dhGs$lz{c)NpJD77G?kyq9xTRYxD2wWDo~iyj*Jjz~KB#Ck4Z{G{Sc- z@1OVWjaOmrw9p{_9mATZf;H{1BOgDu8GeJV=m#N5+j3>sgKvI$Dtu*1l5$V=)*FWq z=pmU;P>Vl%pU$q^_l;6WjQV}7nOXIm=Ly#jF&B!Er!c+lw$uCt2Gs7u>sF29{D0$Z ze^RDpL_MW7L+6J+uaztE%CkA(IbG)<5~p{$}ke(8BA}>YDAGe;CgO6*z4l zi}1hcJ*Q>hwfXZf+P?w2n}n)8&0XPqe;7|DNQdotc2$!v^KklRC6j=OdNYylZ`yAb zvA@CIb<|QlyDF8~Q_1326KNja@(zDRv6>tu@i#1OBrVjH{~$Dw4z-N(59DOFVh=XD zse#4jh*^Zjpn)y5HXhh1ZpinzGDewjCxh^U?H3K?)oU~R%-~f=Gb_7m6D5V4XeP&~ zVa)y4Gnz#KOzAT8@RSLp7TsU{DZ$tumW!+MsrbXDM`^QC7TiB@X0*Sq3Ak!u819 zWIQa2Bqy!5U_B5N8lzVC(N0yV}`& z2v`K{v_KuEHu9D3xaTR{5pM3t777B%TJUNFHs!8{T+ zI{pOzq)5m`Z_EKSQr}=ne?-BiqyS)WHjJi_b601hSPjBZj{-=!LZN_zVDfZ6fM+z$ zx$}~I+R*Y4NM|l>WMo%abetG2u7JRf+rOGdsS2l;&>jOFqp$&vQ?GM!vm>9{_T4Wg z^Ph{4U~!oQq?n)5GiCW4&dATJ=lP^tkkkcx=-2@I~YXqb9;bZoeAbhb(2<>*t>(m zjSQpWb9i>Zk5Ov3+zy*tY9XsAaE5(eWA2pzbw41vs&88Hi>;d&vLi5zJF zt3)Kld5uR!siHfW$%Mzes)KPR#eOk+<(W9u^W!?d(|nD>P;EA3B;~;1n#7(kF?2pk(8!JTcvZ4^ z6s%0j!q{G)JB22Rok0Vj$)Fkr7|8`V?cPew0>fdH3H4lKmw1N9?d6u=gr6GXMxU^z z&Hr

n2L|QNjuH0gY?``98dz!+rP4sz-~>n0h2UJ|yt@5G3@|T*xn29kD-R9|CCI9twHC4;H|uCd8~Rv$ z1i0gpqA{5ZL|$$I!iAh?9>Ym}j=j0pSuQTv4^>>h*3HW|X$1PVY;dwU0R*Q6?qQ6D zdxA}U_R*+2SIvxDYn4!7UZqVJ%iiTHaQnecp?sd#B-4k9_pvqzT46?A8mgI2vR>8V00Sy@-%Q0`L7Ra6 z((r}bZ1IWwl|DS#t(R%aLo=YMzMEson@yh2VC%I*7y1?M$%`p{$|4Mf<<@lJMiO$xC-czcN^$OG@a?3J(6b$buiqzR`N zdk7_ik1UJ>v>A_pU-wg@ZXTCDi@xnES-|_0k!>|O__lavyp=AG0Z>V+%sG2jyXC%R z@%l%&ts_b=x1`;KCv|V{LgiIBfde;k9AJRC8NqQ>`w;jl3jtVX5vsrP-BpNV+zh`c z3&mvnKg4F71ySRygix%BRf7K3OL#IB_t9Y5Npx}p03HqiG_->usp~%Bdy{*SJ(i`V zd|`5KqFt>J5;vc&)A;6A7sJen%RyQh+4l88JB=Y;09dSm#au*cY#HxanWfdI7uqVT z)v!czU%d}wdKlabCsrkIgUd)hA7Gv-GR zoj8S{4z4PdYf3tc4&3Ut`Xi;L!mjMtZojd6Md>lvr{j7vw~)9lQfNXsRd43;>{xPe zn0UWi{7|Pi_*@-;PnNxXeI0PR?EuAE!5%p{aWOIJEuSxa1fuzQ;4pX|uK`xtA#t+e z$tf%>95+Q-DUE|Im8u$`d{bubv(QiX^NkWeyU@NWe}%3y3@Ud2EtAND4#Cm(>G0qk z2b_6GoZA+uRV3+*xXU1_w=J;b5K?j;0?E|D^e85`olqdJ`*Rnr2Tgw;Y65~fr-k={zlyfOERWx(D8w-jvNrt5Zvf@F-*aRyT&H&wM4sOv8lGzq$i7_aN|&5s zMhmxh^_PglegVeNKCSZ$kn_3blSB-R>#&UVv`A-_l3PNn^<$l6MqKDYUGgZ`m7xQ` zOkqiN>E9w9`Pn7DWAl0^rnlUk^x15Txac;BmD0j0prYX^J83azNASWTaKV`5Srt5` zb74o4UCT3q9}BnQ&Rh{ap^7rAz9z+=u$5>3_f$}T34aRks0DIq;19X9bq`AmyVmsx-HtJ33JiH40FuiB z1<_X4y*Fp@surD_qlyljW##kOZj)w-lW9_cS-^rh0SNkI@t)w2L%m|Bu9IkL&dyXW t{`B9d77A&U7VZb{!qJ#((6j~HnNeK-%)Zb~CNyMLr_wfZ6Ib^eZ5npA9(WRnkv0@}@nx)!0 zf;6vg_@mLt5TeTB#2`4yzqU$=OGYp4&il*s(vl-A8W)XAhMxBQR#^XS*Y~7vrzzd4 z(rM_O@zCBorxj;c-pDVr1Wbgqs26`<2xQjoykxnG9fk7u&zDY|u_rRh~JfX_t#|MQt`Kc<G$S()49>g@OyT0UJI2b91^yU{%mO~_LA@D+atkfKeEk+0*|Ngl3r)sM6y?)go{-yGX8YgKBFP2-_ z2zgq|Qs%AtB|4S&di={1H6IH!_J04!E_5skC-eLKima-1 zjFnr*=Weu9K9y%478= z{WzJCo5a_gWBKico}t?o2MZ}rb96qK$ua2^3I!E#iMefviyqDqy)&pOqm7``EYOJI zG8VP|QI`9Ayh@pjtK#v9mRQ5=oAZ;rm>T6bq8^E6WS762tUR_@`AGPrjxp!Ft!n*z z)%aZP+)B`CLF1ggT6_AuSvaZx*Oxqe!Og~u)tRx}U{+c`@6#i9Ny*d3lP&qC#ax-_ zdsKa~%#jYBFs}5$cj}ZQE5bXD3*E`>H4PrSYKyrF$ph*I8mtv-&;P(q2;1&U6`W;N z$q+B9=dp~S3NdyMi4-F&WV2K4K&CERhyIG?1J ztOi@cei30-PQTR~{#2im?6nq!fZgvZj-O<&`5(WHuZ*2a#uT;mic) z$D3bhC>IYuMyyXYl`M>!_?^KUyxu2nvnMZ!m?>5}j_1C{+E8vw7iyOzNWCIuWgqVC z_+>Ire(O=x$Y$Q(nwI=f_jw_toDj@C*~x5^`K}IxGEcihZ^>{?ulFW9Cz??RaH7YiU^o z)!ny81Jf-5maD_>ZM;d8u~Y9j+%x&QWKryNZ~W*JzLWDmqt6k2D z`e%P$ZyhMSz9sttkCKV2>7;Corl8StcSW=R^e8vcNUfgJpqlpkTSfIJw2^lAJ97Gh zRHE>PMRe#s2fwQomqZ)Vrr9@S$_X?1JG}=AX+oMWG1spbqrZXuR^WBKzn-XVN~N>>#O9-C*BSn z&bFJ|KU{o6c}yehHZ7aTnMT3$srA9pcApqt7yd9W$L%x|mq*FJOLGFEF)bH6qVLD! zb{!7n%IjV4YxFu6E~SfB=1y>K^gIl6DluxXnfv(U2d{Df2A=TKhTQ~H-Q;9EDxtwU z`;!gNF+TEIc3wLsvXRB?YJMWyw9$(D;I6omlaS9@J+u4P+=?5_X-}}Q$n!moW7CH0 zGi>X}C6^T`&D+CxEcD7OUM`1x$-XH+xErf1x+(M`TxEAYTRM`0w?%ET+97)}QtV7+ zetoQ>%W1xogiE*abWeZwL?Qr#{L8OT#AKf}jp@z5%XxNj@Ap-o<*6!o9(jL_VI;wH z+t?6$OEhY&e0^-I+3yne%6$jjftEAnQ5WNHyH49Ne&!Na`Zf-57UaB$9+XkzpFKU} zif7Z5+qmm>RL-GOD*kM3f;C0psTL`5Z|kE$V~?GF>F}SWg9I+&hYa1?H%zHWvDA8A z^P_L83Q1JLeMxD#box@aoe_VA}MGLJBnvje{i<_=DefQb{x@S5^~?|Wb?~VwpLO}Uku|b zQu=9LQH7rE-+n8rB^hmT1ls2>m8ZU(QwmG#Jv)h94y-0?3tDpbaHH7pAr3v3GH@wd zg*tW_+NZ9KE@?k3ab3*-@%4}L62 zyh&ix@FEZR2D@7>Pbn>otjNS9Xqx||`3l!noBpgI($DK~wO)K1+Qw5(rD;jLzgEOH zmybQfFDG+{Y$`5zC2|?>;n02ji2qrWxo!=&ao1_J(#Lj2i{kbmrBlD#G8*x%2UQ`% z?ZP+DtdS8`4l5_eVYW=}Vb`zDc=iuUpHtJ*SUF)oe>EpwTxTS|rivNcy}OmB{Uw|- zL4mgG&Q*D*ngtJ&s2m~;ZNIId#iWv<f~O%-RCQ5eeQM~V&2@q8bUU1+j+v=u<~EE-7+j3klMt zanB4J&yH4Cm`=O-`tqd}c0z=Um8fMHgwwOmRX07PCnQ~jJP%3Go z`thjIVLF<=ewi&P&?9-92mf}$P5p<$S>{)$ahQy~A{sZ%$A#@j8=hCqcn>c!J;PqL zZ}vr9QDvIrH*3Q(mqRJ}Flas79`2!elVJLLp*Gjy?~i?hJ1^2~ZK*pDTioJU+wP-R zER99oHZpJ`B*jVnt5weNC~vc9?XC=|D`e<&d&<IV7TaE1K9|P_pjO0PE>Y?Uw)x zs^Xn=@1Fl%N~0@j+y|*V<}W6RxywJ|laph9FdOl@5t3!#fD>Qd)AJ-<`Llk^j-tav zwL?~Xq;5@nm0LAtv`O;*Qvdy*gBcl{d)$Uvp4$rDho#S=RPrE%cP_s1;S8~e>{49T zf7^>mO1$tOGs-l`EF=57Q=dql_H0Gq=eOU*&NlJ+F>&UfM$`^}A*%CS$_ySX$4H-- z^1n-@MMp?Cv@5)WFLsjgR&ts#Nz|)WZ6Tj8%XKqMIK~hzKPSTrN@9BHi0i7BI;Yr{ z#_fe7GnV%(L>JU}xWX5sTBPtDdEzrMpZd*(aeWs*)cJ$&LJpUGt>Z=u?oJz#GS=I1 z?$S@^-H>jwB1~0cj`2lKTmEF&+Uk`h6S0PH@F6QU^-Z!ukaF&!N>ZK>* zx|+f~*S_69y1p)G{Tu&*IsUr92AZb96Q{f^4YJwHRvVC#Ke?fpeM?arq${t-esrr%Wu!y^47DBD{RMa(^f&T<-W0omiB8r*#G6)mA6XIev&zhOw{Kv z+HN1Ep@Nncy7t= zJ}5leU1etIk~9pcZNagbYHYCRN>HrX-PB^Qo^`&e!@aKEenN}(!o)9u(@?Kr7pK^3 z&>D($qo+NgCC!g8=fdl0aktRLBy}(;g^8zld2;=%;-XCyo6}w)3aRa$xk+q16l}Xm zJB_I&@|L@sVCGo`|5hMJshsL0eWdU&5{VMT_BQjN7kQsKJq0!OH8X-4wuyt2!|m{9p8Ds>FA*}$A_zC*(vtVq zvg2O{;sxL9e7LvK?=LR#*uJ5q0k`lq@(`-Ohoq@rZpRHwmZ9}Sa5uf|P!r~KIz4jP zJ-E6+$}<(}sxuozY#+cEje=brjIxfqSHXYN`dEjT`5(|+8jq4)XZ!hUw~Q|W`PVFp zrZKSh!9wXDkZi(^%4V(az{~gV@sTL;WP%%`fNubfapN@D!Qe1urQY)4Oc#HIb6 zKH36o6_~CCQJ-bPV!n$5qv5f7iw@pGlmN zYN)5H^I~o#dEsaoJCRPMze{m-2xdCF)?HWd;%FJ<(O=(7|B(H=@L8`>;+-`FxY(sH zj{XWq)7R{>+`YIA4di94ELhH8Y!kX(fOWEOL>piNrB(*t(SOfV zPQ=8gcH6WY%#S|eK25vpI(SBA?5S$|M&$YMYQ4%6&-2s$GWYET6^kC>)2r9`ZQ^Xk z;oIb0fLY{UTqe(P+x)dv)pV+etj7*&V(RFUn?Lr67LFa)ET(^aPjPnU3dLk`XN}vY zS?WBn1mjBq3ujApqRvYy@rg~1ubiOpV-$3pi$ZRTh2FGP&EoZjp4E(S%9(nrF_y~5 zWpc|oukWEYou3`^J{je132x5J>38=zKXv_4q+L5Zai9Hw+7Ck+}vL3lzY^u z$-9A~sKl^NODSF0y4>p?>gZ;Z{s%LuuIJy7n96m1GX9qJfZI#!wYobSkTg4?D4Ux) zKdU;T67|%y#W(@< zMhQ%L@|hGWrrcKSO%TLHpS_R5hHc#%Wgc7fat_dcVZ1{ z9HsEq&JJert;Z|hUl(#_SqpP}F^g^T4P9!u>Z$SW`E39fl3h=}oUbvz@ig0!*tx%5 zV%e1-Cy~gZtLCw@G{aa@v;ONf0BtRIpW*lV%dfX)hmvxhk%IqV%2zBOHFZvh3u1x5P&lD2D*T3e!(7 z2C}4ZYh2gUx2_{xq7|7p)2)d*u3FAsQ{0Xqj^inVBtLCsZTKjW`WniT8re-1OA>#{m*Q_Nym>#X+ip`za_ zg*>S3Wsr!SX`z>=3f5^`?1(#2Rwl#e{Hz_1@KiRh;rYH?-LpS&g~X(|b_ei%&F{=d z*Bio=eI$(UjU!s?1^=I^DLj{!ho%G?<4CLlR<$rF&`rnQ{XRTbk#eE zNJoJ!Yf!?g^zkqxZlQei(d&3m7SJT(JUIUxNWD|G58~TZ8>Q3dL?d+Q+fBIgbwEuN z{JgrZQmA$FN2%%kREEB^yUey@<*NH1zQ!`k|0uJZt#9{4TT_-<5;#*Ubr~F?7Ic&x zt9n|pW=+obkl5fHur7Oz&0v0}GSSrYx8q02sI8gou&R34X6YkrH|}8J>}Lj3mkIUU zdEukRqRZksae^dsD#B~#R)z|3&pvcxZ>QX$=^~(ifMu|ewC;I!xMY=1+(hBGy3+;iDDwEfE*VWP0wTs7oP_LsEJ)F0xFoQY2qc)gk?bI) zx;U2GcEjRf*AK6j7D{Bs%iZzP}j`B6;Yoyd96>1D6dV_ zh(KW2GpM$2VZS?d>??Y3yxznfVUnkwacR?Nq%Ld+ws2k)-xWz$OePdI`-w?Y$-aea*(?Dk$ zgP&RFR%tOQr$K(v!)+d&)dhqr8R5jVLN9X$y$yu$w}GDrF^Wn6!9K^vFQmX;dzcfr zN2b%appkhGz2fDmeZ|TSgy#ZHoe>%P$HjUTa=JqHt|{Lkn!C;uAi3JOS0kR?c&9{@ zc#_XPeMjX&C;W9q59+@$S^XamIiJ0!uce?SPn*7Ze{0ufX?6pRZkoht@fGRKP?G-l zeV-g#1F<`Wf`cMxIP`UT?LX0nQ{C0#mQGCy%rm`ef*VOV@lz;MX@dnLW#U_(zh!jX zBI6$?g}1L~btS4pukHj`W&e>$I3c$utBP-5A{24!7A+*gZ5ZFwNKp)6=iF`7=Q zs6|*w$l+IDritm*gUWF22R?W^Jpo?V7=d{v-ch|g$c+~}8kk4*Ew5+0S4bH#x8j|@ zkFIw#%t&#pn^Ak$y2(85mOoQQ4nDOoiyH=yO6wAZmcYWFUvf#tlC7y*m^7YOH=M(S zdstPNuMihKY)n{{RTdZX&V9<8wC}Z;<~q2&d-OrDjrkE{{y2Z;DdV>cUe1!J%AHG6 zD67@D-6bjN<@?H)%K(oQ6ok2(`d|kX{TZ(+bl3LzLiAA?zsz&{+o|Nb^b^#Dt{DQq zJkDY_3IlaHYyICvH7XZ2;u+h<HcHo!>o%*%p zDAG>OPn*t}12FLufQHlM9ikTtYqaW1w{_>le}IWP!ENeJ*GaKvsYAS^Um9cVwW~SL zc~iSMHFQE61rP19Gmfi|Uzkjz>VzJ(h+AOdS4C`fl?~sPmHR4bPg}*QCESu)#qEh> z(zU)B5qO2Kqe!n;nZLBEBGA|yR9Ak7qJDR)ypw)ZXhni1j2y1-6F|$uyKP=q7^bf= zhP%JHasNffQh&ClGlPM6BenpaF!RgaP%E5SIeYz<*M3{U`s4dIEkion^QoK?+$5Pq zWzRpgEmEXYdY~maeJV#}J?@{hC%Z2}kAZ5Z=HwrYE~_t-`I~FOan}QsDNW-SQds1k z$HeJ1QmtKH6LIB*l>2DiKaQB(9$dw4GHF307qCmEh_Wuk$H9tJT%KF!|InDaTeGSA z>wR#NSi2`LhR2K>*ZoqLz~}&>wb2jsTJhY?{nXhym^GiG5!Z?CV7LlQGQ4oVFWgl7 zeBX}C#7!7AvSRwjL*X;TW0qos00Wxh#*^HVp>WD+{ih9#d{(`ekAf~OY?ZU~eS0+I z$vJXg86}ilpa+n4fuLT!5veRDRi<$=5jBtd{!iB-&Qlo;mg847PZ}cpSSF>zYpfB( z$={*W%`_rX7N2`c1ByV3~cFgDbmB`5m!wn>I?3xoz5HESR+n>rQ(N#_f8C z%TB%5WIy!;C8AE#5U#IE*^?RQy7Be3y(9{F8`givKrN&Z*qZF4FTp89X=L>3hzte` z3%r`Bt5KC6yEeOFMdKXl8I&yxM48`qQW-f^-0qv)pk2Bu2XijDL!XJ?QYIdCZDKMc z|8pu;5qCBEW){JA7yZ2rD`z8@~u72eSVh?JQ^S-FNW6v9RkE|H$mzm1iE~6;%L-$T!@!atMJo z9-QAc4}M1~Q}LL5_QL0~^mhn2$CT_|ZHy4V^R|O_`kFjN^>lh5I%;n4Bk2j}OdV5f zbB*x*yYYD~i*=<}iy=^@pS#$5S=KIO~Y#n}dteXV79fSQl z3=iQ?`$Bfi=_py|yO`J}E;HfA)tkShXiMcB)CaHy=9of64Q9^G?KWKnJ1kIA2W5%XXmk^#M-$omte z5x4)|Ck^yCUh2ZO$3dHqDBs?(KmTTDmZl9?zFRMV@w}D>LoZ8gEdH0e^bVy$wlC_L zwEcS97m?Bk;y2Ip4Y&B-1QN)+V6Gx^WQ|it<5HAyD6VDF8@)m!Y9RG%w(y}sv^4GL z9`0U!2mRF=!dYxg1@cEDjm6mM8Gf#)=~#C~wX(JqyJJqjyiU*NuzsxS9(Ms6uc2*_ zPV;DX-%lYjyB)PMQu1`N#|4wfelWE>Kv4A-5*&tHLvz!=My)C3doz9Ng;cnpjqoxW zx~(^nZktADuD_aWUHOczEg`Mn5+e#(*_jG1n!{$oU15m;6g^3|;MwR8OSzPjuT^D( z6Xo^pm;dVp5RgY=UcV?Jaf2@QlKL#~8(d-NfpI8{Irl>olP?f%lz1y(TIN)$5@1V^ zs=i+o@CR2|jUHZrSaCFLW^u+e`)Bnr|L_g>fPWCmaZ48bhmT-L1svP@V^i+$*HH{-NIYg(~pIzBbm3@ zf3;GCUE{gue@bI@x2`l`RAc^Nhc6{GEdNQ2JE0pU`lyU7QV^jLdpek}niIkSB*Uv~ zx|gtP&A(hZ+;2LkTC#6jk8JZMF<#=g8I1DSUC}in{qyIE?jh}!+RQfyzm}~ylcQuy z3{_9=mIP-8R)Vnm#YPPjB%xd}M|5YK8?T1jymy;!QvUgqI(pXS9i!vF)|l=zux95o zoc?$6np@z~sV43ahhMz>2`_*CJS!5pU}gefCGJ)`uuxpQtb>?E_qKX`9RS6+XC1VGsk?&pCwQGy5^1kLWep_wKryJOVkhx`x4Qg^gm@l*H zqjph*vT+Wiw1G-neSUEs-US@wcqMt@eqgYfc=l(E)%BLaz)=+(A0>AwSJ1Q-p-CNA z*o1h@4^_M4TnB-lneAw4X4x+uvdd(rAm_%{oq)2x)r!m20gz`NfYCsWla+3@eP$C! z!=7Q?vzsRADn}icJVZ1x0TPpqO%aqKl0ox$-++gSMhE??A3 z3}9?fYxHzSIwb%#OCiacNa3vzS|0|Te(*S4iOc=q1GWLw?Lg1P-Zb^$qm}?n%ejtd zw9jc8%b+F`vaRG~^*O?R6sKk6VZieXruL8lx_e&J4fv<}%0Hvm#JuYrDLw{(M1eZu$ zh1jESw2Z4zTC+geC}2fWxr-FsVwn^r@i|(xxN+%_5v~dATTH}OvG3u}U!Py$5m>4XjB){C zyfcWJh=?B04=;7yfFpVG3sa>2E~olE>&m4kq!MThS6@MF1_RKKI@c1Rhw+Or*BU5e znfHSkypDD{=dY068IHax9Bl4pBxJA*?9V+ZD$X&EV4#u{;YQ{nu%kp=tq3=%1Sf}l zy0CkW<9w%DiIE7nkU~Pw+C4V`H*1QMEz}fhy)P&ZGE2K<9n(A%v9vW*s9l)<hCI^vOPT}HI76rue!V84U;u|7N8M*W!eAm{F1+B-1D_c3+L5&S-Sf2%?! z{l=(yB)0$8tTaC2UEtrc(`O>~YF`vraa+OinqXBwkcCaE)-c9B0I$*XLo_`J z`qsNrp~Q9%T52I&2KvE@UZedOvP5g5_026$@M3&OE;k>W?>wJY9G!}!r$a7Eyo~?| z+-j!m<{2h<&+ZpTBtk|Ug7z>&H3O;`B0u9)e1;Q-TMEZrp|l|-eu?NVK@FEYWaYz- z61PCozrXjgcWy9zt9N&r75RCC=_Bn+%VQOVVUz-#MllAp z&a6g@%ySk_s@f` z&hIRLe{n4spGGTT6)5j8uW$Sifg~6f23nK74y@~=bgiTfVR$XQp0ixt9*x`mMN>AW()?{AG zm@KJRod7#bJ3c(>r;9fRf<*HSLJrv=eX(%W(OT0NIF!Xs%N0`Z@UReMd;_ltBa+e5s&0G)v{c#S z5za#(!Ym-I^7@>4)_`;8){{|{bhlsF{K0tWLT(!-F4sp48$2{Qjp|t-Q^f)M*$J93 zViCc1X~dw>Q#ajfPuKcqZH3;OyROAMD?{?$N6XB)Dw%Pa2M&|zBF`-#o|ORr>=KTo z5i6d1^8Iai4CW#|z6cgJLtgRS#J?+Phg?Y}>h`US`YY7J`9-G0xQzQhtlqG`;>$F< zOeLgY(twl_b#t1|jtV!micv3N$P{S2%YjUZWC8t#-V=4U*Si5xo(llPap~I)ua5OC zz%``RO+0TRgn9bvK?b!{72ElCe+&4}F7VX6!@#y?g= zW}ia`@|0>&txZ?uJB7CCB#~5j;npm=byt)IdH_pVQ zkc8lB9Gw)7`^L22rp*dl?TS^3{b5cAXb34BdR6${uJ`o>2)pw%NPt z&lBWd)_ESXcbi#fq4Z3J%MAgveHQWWP!e{Ar<3*V@1SfkL3ip~KnV7XypEKb6~d4) zdDZe6=st7@Y_LUX-O`G8q4R?f#t0$oS{hEei)+uxsEsCVCf_%)I1 z4Gv0*pwnANKj1?3d&pNOOjtkN#0z>F|N547RuP|$g~|cY%W6(dkmf$rDp28SV4#g= zY=HKmZuy=*r0D5XqrbAWnGez@diVM;-$IY;@6j8e%T#oK1ev2ibc0)5sXH=~T0~FC zWra8|wcia5ZHt?$3F=Tcv2?Odye$K5_1RDT`Cqa;VHt!9^eYsD zah7-jw+%8v7KYzJR`2XQI+%^z|#6<0$XgI&=tks5CQEjG=UBSl`V57BztY+mu)Yjik{WuKM}|o9KPrYLokK!H3ixlkg&=#Qm|w zC1#~=C?N0UsB3U9zs&sdShg7-^?kO;Ro2Gq-C&dX}@@?+>zr7t7eEpEy*`yrOro|$B;D=RgK_pwNbSmVN? z-$vw{h1?3Xe*al`KZL>mU;ut?{~}J& zO>!Pg!VqrwZGvGFVu3Pji+?dEAu_FtdL8ATLJm#@_2lqW3hMMus7B@hupJSbyB}yw7OR=pgt z%*tAe{Exngmsxf(f$ulY`ukh*MdUpMv55J{N2^2t0`h<&Ea58rJ*Uyd1%+CWpp#|X zIGs9D8tLc(A{VJ4%L0M%SI>J5(7XdmryOtt+xTW}_KoflJ1@lbAcw_;{oky_-mm#$Eg+I1aSfGz%C& zA=Mn=GOE{MT?5~NELZ^>w!+q5XbNg=WnSl_xh&^AR`34NXg&t<1PCVEyJ`<*(1sBv znyqrT&Zk2w?^f6h$(g*I4HmSHyKCNl*^i%^rKRD@D-Xb=F+wgj5v#s}pa|_=ZzwyMNpRJ5kJk>#A6@B-%5ihbNY4 zn+_F|p!viO@cb+z0kIHaoIO&^p)1Ht&CG`xrR-|JBXa{OnbCdg+@%-H>dH04(TNuj z)!{EhbzB|3nOl0-WhG}~7W6!=?THqxHaQIB)z>-9zF;%j-i$}*RaZS$o2jINSXdcA zw@vHELjNsCmWTvh^x1D6zt$ka_UpSzJm!O*JJ41VKlJ?k%uBb{`Q4L=YE^>fW|XyE z&`%4dKfvzo0z>r>PoZ!Xlpb*t|ADL{xBi8!6xN?!;2~?fLYFBqt&7WlN`6*Zk@a?U zoQ7&m{=dS})z1j-#nAs}F!qu72PH7FPuoa4AA%66sd4*?-uz1wAw)*zWfu3i0AHRW zdY9z^sY&SQID}GQ=yp1*G;MN|1!DkmM;@|&nG%II2aRej9z=}$j)kI`qr*YPr(pa< zq&K0|^wV`t*3YjP(0*&5kLZP9e96{43gS3-Ww`Zt%KP~J?}-}qN%zIHZ^Tz44NhyC zXHdU64gO}{^!=)CPLRP!g^TTqEI`vx7Czr~UYge7u~xlXX-%iZ&9)r!m={L@jlq}f zV>90RTn-6;`dur!{}$8fctK zuh&|GWr;vH?Df!|E&4n21L;QW5l1>8Y zqV$!oH+sI|ArB4VH;AbG*qq25LyL;oa94cbz>IR83p*6W_#q)NQ~F;w+ZlG?Jqi5= z+RshI6O=Y+04-x_d`?u=@|37nvg)_;82k%XiL2p`abErl%#yG7vu$TB4x zK)44MOtB3J@SYHqb^I}5$c?%3g0K(PnHDve-~brfIb;$I{^Ii=mi7}v_!9?c%ZbY& zfpe>!aHRfo?te*IcCd#}%PT@4js`}y+vyt;0YtX zsi(n>@(ueCnLS0kHU0tZLV;Y)(9@y}+$b}ca$_Jtb^)CICH!`KNZzLmgNZY0g(c|h;JGatPYm!~PBSN`Y**b<^ZD(-|z!KKpFP)#1IDPT>$q*78{9q?#B04KY$yLA0=7fX*$@Shf*y^ zhu`IVdhr?^x!Zlc5CE|mfzH(cUiZ`MR6xmdJt-qBInt5TompqdHa5s_a))mri?V4I zu>iNIy6AJdJAzht(jzcMd_CYE(tuKy4=MD98~Or}z=I!PmgNETSpuPALw)_}FMms~ zjcEzsZgR>TRqxDUJ9l60wUX)zExQ!|vg*0WRtLR*EqzNU_!cXbl|Bd%=|zqT2KnH} zJ0B!}{L3%N+(NA`oH3dNs`GX53x3?HB0&-k9TnSqrlhQDZJsi=un`RGsU{W{c7XGf z9WK(H|A!NL>0Vhhc5Gwlg0wvQO+C+OW#mTg8+);9=!VT~`KWYYNW6*~ggD zWDfcEHkoD7W8KWaOYy+jjOnb|RyTv;i2ZBqc}b5u@X7Og((OV^U2MvTDf9wNiy=pH z{}=8-3C>A=AMYTe3YLH$1cOdUq#7vXao390!EA;b$F|^GVo*c(#{1|UlBz(DHnkIk zfVIw#e)bcByet?6bLq z_$Y5W|K`oo(vS659tJEKNJ4Qwwto zsw$Ij$1kW&SIQ0dMz03r>@Gtzc7 zU5#=glcvx;(#kw<>}6*A54F?7PjF4#&^AD_orz%!Dr;C;Q!3E1toau`VFH@EgK~kq zA+9r!pgwSACUH-Y{~ME|9-+A*I9?OoTz+I{ID8=k(q;jml6Kx!bwjQ=t=}Ugfa`hR zRr^oUx&>zpu@l9)i0W5+khrqOMOWbB=n&-5{XNn%7l|n=8JX0pI9DJ1{k^OGa5R1G zA&c*CvfA&gOjZ)C!B>{XfW#5>3FvlIfSfAtV?;1J&N^nRUn_=V2U;x zLbeZ$`7HKQ@svDIYr}&Dkc2Byx^?fNI9k8#Gyushpn&8cbX$VEqhD8cI23VR3Fcx3 zPjXX+_8k_Ig-4o`-QjeWP^yDqwVWaOqtUAlv@_{4i?4J@SP1X|w=vQHY->oqvh&h7 zl79kJd(C~gKvNzZv9?c~f5ZNBIUZupj3k8_XYsGguFN2dQ3{G z*R1@$MA9U}7L@~Q?;O+>NamKD$1D)3s{#o}2ev+K_aA)2#GBmIAKo02L3}WXjfsOkq}UyqJ;#D&>G* zkAYN$-K!kW7OzpHqpI6gngM5KqZF{y(AtG@L4!8uUjjFtT^kvdk*?jZ2{%(T_kGVU#UDUlG99bkY49KyQHyv1N#zx4UvS3@Q?`^rw@f#e> za?sg^xP+oJSX#ucUCe=Ol|@R;$d37v)oLhC#a#uCK_XTXXI}!J{Eu|kJG_CE9&lDz z20~!FH8x+pe5tzKrNt}$jz+{|a5WB8WElYPnzs%BlW@UVHTr)+g%A(MYcJ9wHW-y`mlhOVrj*1Q#-_GyeO>m@du7W7~d#cG;<%{?`;4hK9wjZN6 zgm-&(mUMf?`a zBkq<7zKQWhY%NE?jTS%?HwK5<>U1X=KZv_MF{jcA9KV)WYT)YTBQlOhIbUde40=>& zg@Jwa8%3zKPkt7GTWgc?>P;MqU4Uh-%`2a7z9*v2zQ<-%U#(W|Ax69oNo)7*2rt;G z9k&-0&5dE}FsS9BhsoIo!4?EQ5^IL{vFp_kCiFiMq~yi=L zV?czlD!-H)0OfQx2KJpmnDP4*{$`T+sB9}s*7C!Fh!~`pr&fj16)A_Nrz&kZ zufqyc`mankcNeEO-DqDfR2G}RPsaFvg;A@dt#Vn_vL48J z2_pJ^Xi;W>PU31X?+;Y}0)OBNUm&teb&v7cJhX7I^})tvxXo@2{U7jr2e$r0Nk@zW zw0^MrmK1OUh(|u`LEGio`s7Vyh8+6>VGd|y9LOvvH$T|V+_MJT0g^U@(w7MyY!L-f zPafo93+Ts3E&gSfN1*fZVYu-Agp(Xck@=9h6BN*S7 zM?YkLP%f$8Y^UV)*m>0OF2O!|ZLC5bq5$UZ*H);8o>T2t`9(HoUyiGAT{nXEfdCG@ z@{T(sodrO&Ii4Ii=+-@Zul81Duyk$n_aFGe&>@snT9&PflhA_UzTU7uh`BKHHW-su z1-$1mjx#jSTA-Wgw-sa-ppDwO`QOYU(SP8%B5{m3mLV-jY>EgV?Fbi!hni%sV6R`o zV1&*h?GcQECQWy(T>-?=V{`@Bbu7TmsRfX*EC6L$=yLhGk^4N!XR1pX;$J72(b6v9 z=2baJ?{xktJRFi~$j1mk&qE?Sx@6TbL0)G9NpnK4K6}?D_}nXmjcI-l6(*Kdnht3d zYAJ#&mV?hKPaR4*%(L(*S_{wznAj8k4^m<-@l0(H<2aObvY_=*m}o$YXh+|a9B4*y zQh1isEK^7!2rBTn&))wc6q%mGWkKT(q!yP9>4ieP`I4W|yIBK_;*#NFI&fMHEC`A17+sf zg6pn}RZp2py@57+9AcIXE^Ybh>Gx`I`wf{Edd{?`^FL0Y;AjrzgbtbRWWFyQ@aU3q z;Cfv%luD~JN4>hf++N9zZ93AAw_?oeRp+|#SV1AobL%T}==Zzp_1PGFCKy3fPwq`+ zWL-%h?Sm4nG{v_tzFg~@{g2n`@Z5mGlU306y6c9HsvTzIG_e@%;!R%Cn|kXpS0@E1 zmWCl8yy90;NZ-KYiuagm{0P%^7t--y{p|&ho@FR=7J776;kgCs$3IVAa)CBjksph$k0<6bXi>yuM*qSiTw2VTToZ#vm} zWfp|m3X>sR{um&f1GLFT%bK1%g(u_0wYD_k ziynxxD!&iB#7|u0fMD-Q6;$u}(Th|k{V;^4T*W5(u#5uAp2O`qWD2^eS$npm%9%=Q z(teP^P5nw@mKNAzZ1U9G<1Kn&T8C!E6w@vjkB;0V% zsy3T7JNkRSSqFB+-I;Hv&y}|p`Ll;^|AlOTz-vgtKGDN#94h@|_B4#Cx{z0rsUNO75_z!(AG#is< zAG8T|wq#tgyb1jK`_(5`ck$3eq|-7=fql%FYw;AqV77k@s*;As{@Q!*BOxF8i{ay5 z!jN>Mzp;e3y*uVK{oIawTw}0B<(DAT5zmNZ7B4(H z%3_*J+^uh5F(M?&kbEx>$=HAXMJkxIPpJOjg^VE+9f#3>7hHNvx9K}_F zXn85B&Nbpw_s8hrjctGV$V{ebHb%T8jls+{wf+<_66c-Dn~xu$DKSn9F^0d$$a-eu z>RfD~@vYaEQtkm-ze01o90i`76jjg9vB_FhiSe%38psL=W!T3QA^G_M>nwI6u{dMz zZ}xe_`to5_>&a$gmuEeD;G}|+NARr##pSg@9MaZBg)k_|>F-tt9dns!4H|%EIT!jVV4kjnLC5QLDX4mYZv0WS9jd-#fywVd`tvnG zM$%y(dnDy{3wAMNZYJoNA^m-hq$a;Bi@Cf$_&aa&069;ceWB;8v#_L*4f&KP2rcsO z)N*s|XIi($9+zNqd<3Q6&5+-+6PV$g5R2|4Riv;5rkhw`|Jh$1K(AbT%~)9wI)q>? zB*gkpU6RT{2YoOqmja;Gxf77W;El_q;3%R6K_(Uoq)zCS3zN@y(g}nQc<-X1*p{co zr74^5PFAmVUFQT^BmvT5p+rcNr>WuI|Cz(oNY2OX^nkX9T)UsOThQjz37R6_D`zlX zAt`W(x3A(@RPX(->b^Uk%f9cMpFNVj_g>keNcP@QRz}F?M~F~<_Ff@Mg~$$tl#!KP zl$}k=3`N7J=X2;hulu~N`?~Juy6@NPc|Ff_{&$|8bo`FraeTk;&wG74li4bh%tvNR z<*8jwZ{S44m=qf-_29=dgrBH+1^1=^^fw8p=0Z5c0ss(cf~JBXmnDDURr0Mm`2TNxCYbt*J{h9)m0ld}8J0*s`VKxFZpJ=hB0qy7L+ z!ltDC$@*)%rGhw)8`BLJPP&h?!+0p6ANe?p5@|Fmha9fJIeu4uX89LE(D3E=6MU_v z+31@pZyLYN1cf#J1*hyH5y<}nr{LmH@T-to1L)$VuTk`TKLNa}sX+jn7(Zh3dVbRI zu%h=#=?jiQ8zI~0-izwsYt3XVM?jhJGatuVAQ`=W9DEr_)DbIg!zu!Lv{RuxYd>NP z+Y@2n6|>-#`Q;}W6u;&=*JsmjccYvD-8*67J{ z3;q~*tX5Qd)1numK6e98{qDu3m&96kPuG&tiz%9i0uy`-_-MhxdYC{Rd+OZA^o<|t z3ef*|5DO9AXTVM)SSVJ)VTw^#u-T3TZvlEv&^biD_Od;u?AzyUdw`;_+)xU}Sj*ha z$ET;)Y&nOO`5{6lwv)*};uH&Dsn!N!-bt-7#%Y7r1A^~M4 zRkligw_{W9*i+c~@LkX0TYflNbQfh<=nf^=MBCUYJ*k`Rt7lBzUTgJYEPbX7% z<4Nx~P&AAi1H7fp6jNxZZIFr{Y*MS~53Qsh8uAV(#5aVFJAnPbOZZg3TzT)4w4oe= z(Hr2P<)?8gl_r`FzfP`- zeaN;GUo_(6zrk=8S1=dX<-}xn*v6CS zcbDAEACj&UPrdHzf_#C5QQx;p6KaQ-XK{5}DNr|>K1M*;ti1S*L^gcVlw;@U-#sR1 zN2n$|)FlfnTIuT;d?S*~2h*6Nb(c7K=-bF3hF1HmZ%8~}*rwSErm_d9ytzj#U`u{%XlG=Q zBWt8~jw^uV&HGmFvS9J9#2CNya3z9Xi^hOzi?KkiRVHHcb9O}BM;Kb9X@>0m@ReXK_3PqWa!6vd*w1QN??kBB+|;>5-Z86GYt7*kk-Isx$IynPt&?BVL%n97bLC0o9gKU&@JdM_M_us+YS_ich!udyRdQH? z)FoAn?!1^>PkQKdVv#dc0n^-ibiG;-OxP3s+{nOZO6TkOY@O;ijPm3FwlFjE@|oYr zg)<4*nGR_Nyx$rtngnQvr!`V*TEF>jZ-noD)Wd(|d`me(L6$W;u=N&d-Q-1TDtAwH zB1{>S0J>Ot?<^bb78U?&Nxx6o;Pd&tR>ns_IFxrXq+EwJK^e?g4(2=+j#%6Oj!ChA zE}?7)ls!xV|FhMZ4{=D-Zo`hAaHZCfXcLC%3uKUt7!SJ50QVIB=vSYBWa*)c*aM9Z5# zu!3Af>Px`qno7x|-3`3~;2nV(Hd?u2)hm=d6)IEtCxGb5$G3F81Gh&D5x!^Qo;VHk zBCQP~n0$R0{*kqK?C6CkIbvH-M{LWbf62B09Lxmvy9a5xQiw$K@XE!XCxLz}?Nu~K zT1Ub8)D8abgmZc25TFr6sCGbif{w&$$JST}Ht#_-#9gTUv^C|WHh>-OebcuYs`tla z&d6h7fYrE_UiJ-?r{3qJ>7vw3jl_mYhVC$7r$G*iabo7VpN<$gMxtX5&(nd_f?eh_ zVze&hYdlV$j&S!EnTnB3ab%W!xspa~8q%c0pbm(c{eZ^DLa;)*vfvPq`lPVo1Awq? z0U*Ip>E0jGwuU8hG$!f7)31grV{aC7`eHp=DUHuS1SpWv2H{&(^>Eij;Ofq2E>^zy z=^EIqFb*vnjvno$73V8%LkWo{n{lQ*wM_XU7+p&ak!F*B68$lb13t_s(;X?+Fpa@h zhr+YOgM$jh!)`uo_yO}-52_k-!2E2!4uv%7km41LAI`ad?@MIP^mD(Pa51IWxbo!1 zTn^B$laju z1I&!(Ri%oNw;w-;dt-lcHoZ&Hrt;P@lgaRHe8g2h&kM^XEX`JRA55SfL;BZg!0KN- zYZW}FWF2XgV=Z~64pPO=R!~{>xAK~GvW4uVw!9ljoyls50tx&uyn&Go(<0u#dwU2t zjQ%^-qc?eU9)2}n3R>tO5&)*3QP>uZdO22dj?W$o=fczf;9*{-5j=(t2boOf4=l+3 z>GB$94XLL|TfAj+-KbnlG4v&=C;CSFV;R03?E01vs;@^jv6Fz5>j~@6YY`X^VkNWZ zL&-&PD1ZG`iesdBF(GWhTNvXbIKywYh%N%<uqiP-4X~aO2EoK)6ET39WxEpyozPY$Y!n0uhfHDWudl@;+ftQ*Y z&XUrzGd&cmaZFaC$H#HD?2c*<#geaIJ>)s^di?>+N(w5dYocbrk+)k+b1yz>B+|*1 zLNOAHpE+JgDHWt`Ej6ijmS%`>LtT%D)vZ95E=4SDx-W%!(LmZ=q=-|0HS5^{Pm{hj#Tu6z46MR6W`o zu$B@uvT|DRAhO|XM!(h@??4f z?!Upoc)GZZ{*m2UK$JJDn?|-rd=B0(qfK8WMgjQAiJ(RbznBY5#TlwH%q502TFQ){WX5 ze+hsM7m0ryxWDN)HQNakuqgw-8@OH`+qL7HgPqJJd)&B;-WPtZZjWzH+g%;QZyo+I}{tPwfUE^ObRsB z8Kp+g{};?c<{LM5{YWXI0nsslm8WoD^13FwxM$Oy!hYR8ML33t%ovN%!QUQ2R8 zvfOkw;?4nFU6Vgom2MFjmAV1Og92~dfZ5gOce)E9#k}6($tqPlkJ~mj*Vf0y!Piye zlhJ-LiEV1LDjzD`x$uao2XB?}SZ$pz0a|pm%Cs46KkyfL0$|DBf#iY&K0h=Q{ZnxW zT49kZ-VN<)oNY{M{tteL5?z}%O8oBIhmA=r$a@=osr;Fbfqvkka%Fz+GF%;IqAx?I zTVFD}hdCa+(O@GgzsRzZ_{`00ZG7}>ec84VA#=M`e@mY~3H^tE>UoE@dlXW$UB;>|f6a z=6M+!SyC*B1BqowiLOW^JFkSJF!d5iQDYmSH<3Pc?Rnc-s2|ErFFN0bVI&!qMfH#u z^6~ zIN~x;rEu)P@L$xRfMG&J3{sW~6zc)(s4aO^_?{I*>%wI&Vt< zqMW$vRU)$P<6mmB23R21c-tm|XL;b|28x=)<5>MzUJj(}J{8ueSU3|(j(xgug<^|- z*-Q7%e#Q^xM>j*WvG!xL@y8Z*T1+#2m4%nTqn3WW(F>*Z^L;Anrh(Gu0^%Z(e?gJ9 zW{5HJv+SMF^TUL}zhiEOl$s2^{dd!4?u{f0{kY=Q>31x-iTsgyZ?F3x8-}VLis^nsWY(Y^RKGGZd) zz*y>1V%_w7Z_mtG+xAf&ckYWB>I)3+AL|LX>uE5*W@r2qHB5cy=LwW`3ZglibT+Y7 z_+!O=l`{xA)hJM%VVsN^BK)RG*Nl`m^E5Y`v^QC}+3#WzX>}ibY={p0=s2DEN@U&h z(;2?NO}%GcAMONRar)FG>E>}(va9{>qS^Gu>xw=K^-v153$B@FgYOsbc?RTP zZE>d_88F4!NEZX>^^F6U=3vp6F{=9{l4nzYU&wKK&~}|PM7AJ6fh1H?>OY0xy&RD zxCj%NPo=%1-Kh8pCb?g+kVTIHjwL2pVNWv$7HCaux>^`u6DMH^jNP7G5gK*g(c05fQ+!)}VPGd|0WHPWnc`>WG7lpLugtI>;pJ$A_mAWA^{#E7vC8G8!@oS#RL5m*WImI4FSwlk?FM)dr$$b z22^neDB3GHS9qDHn}i`dk0&n}x<(LB3XyMs?+jdEJg|XraAeGs!%@Mj zs+t;+mzUQ~mvsuvMBRMp3~O+9v?DI4=z5H#&f-Xb0otO3SrvjcZ#Do>i%&@*_Xf?Y zFF`hPh$~c!4%ZwukCmW1;`ZuseeTI#SHV$g!KQk4_*bpx^le;UP$oOTc$ikuxYryn z4}>23#sCbMkIH8Q`0=w*3ue4uWKkR_M3@!lY>fskxOPcmaeZf-C3F;Pi zf!$XSA9!6Ra%wxXq^|#1x>fVp=CMO;z+$3IxuC19ZLaf3X+}(PGT$qVY4&=0OqZ@4 z_6qw+eXbr(PEJ*2Wt86*s8mc9o{%5Jb(hQXP-gK=NSL_5dgdB8bojd&fk)Q(tkO}t zt2eN7+`_gu!$Mzxi0eG|h-D%U@wm-g-}k_#$2!L`6j@sHs&~uNI5H zIh#Aux<3>0?INb9nBx$*T$E5bI8olX63bVk8X(l3RxtW?fIHjjJGO^EpssCDp$LPa z+ZDe51Y+@XUoD-@wn8{XyKj5Z32lI{>|8_ZZ0p$A1&F}C;Y2rVFkB5!2d&If}! z?U!XL+hA1QEIpUoTJdM==e}CL%Y&FsDFXuo%mIvJ1wLR~)#0IvvcK%Q=EO*@e{E4p z*@8%EJn$7oPB=RnFuRvXHf?k=wK8oROZY4}Z`B8?c=o@4Ck!^b78XLa;(Z9kpm-)* zRae0ZWWs#PCn60A7UqMj<(rIQD-|WY`fw~63!kfI4NP3MU&C&a>+%eCeBa+c$JWwv zT#`6K$o*bi+N2Hp;2Rr>%l-V<_cvtTK)Vq3$nRR$4uBp#5#iSfc6r^G%uWaZ6?(fE zYB9baew$T$ldq-*Z<-o_pqG%nR0SRo58vX9`Ow+jm5V=E%bW*;Hk&jq5Gs-nKdDQT zo)zIW2S$+`Z5D1ox<`aN1(T$1Gp%4?mU(VzC$5g(g5G5@N^fASh(o@-l1c2~6(u~x zKlA}uSdgLGWQPiCzcZiZlyJp$)v0rJ)5PwrIwdOJeBTHvPOGIdy}iAAE-}KF{R7rV zejAuwR!#{D#vx^|`U5ZqYPZQqNTju?C@a@&!31V#27OXBVpHW7wQ+6m=$AxlsHz%{ z!nE1 z#Nq`@oNk&I5b^2yvol|pMX|B5JD$SSCm}9w6y#)PMjr|1zvz&-y{i$3gyw2}lJ-}i zHVRD5&f1U|=>Hm)D4gqj_w~uii#~B_DJkqgatk#zHWuvFEiNt=K7IOhXKO137F?~i z*Jx(LRInVdGh^;@o|}EtHU^KNx`syCdoTrVcW_BbnUF{}5*S8YHSVT%@f%o0g>&&osLo57T|8k?6LbF6o^1K( zF0LOPp(~pAqlRMOu)_6&EMk%&{PQ-*gKiv9UaDj$6YjntYIbG~RypmR7<91(4ZL1S zp4HFsl`5VAy=JkZI9*W$%SDPh;(jk!80jjE`bo5bFW->25}Jv^eGv}+RAadk(>hkV zpy~DoMn(p;!AuNkEu$c-g`#X@n}0L(nEwj!E%U8UVmo{9{719A#p6Pjx--` z`aS2|x|7`N6;aca{*1oPYT#Todg^P+CXDY5AC>*|>C;*bwa#zC?g3LR3V*Bq%@gZI zotX;cCK>EOyL96QN@)LKD&9QyfZT9fuShSv;e}2@Cmco7C_ zYedQf53zHyu?S9R3_>|(M#PB-glPUHnXQozc2zg%K}NOLX|*jcsoI>K=*mYl7E}l| zm2{S!vG({zs6xe4IiY*#Zqq}<$n58gqc2(ZvrC@KovR-rbdl9l)I_4$w#V}Y+SVD% zJjXYLDc0lJ$J4>?>nw5V^1W*kBe8m?%2vL7K3vmK^mt($PP*OH3c-rLLui^TY5K9U;WNJ3;NZ;?FJmEiQ{-=A_cEGymq_|W!2x)BTBR~ zIgXDQn8*f`*osE0j+;oaPgDutSZ^}N2@^f_$##?4+=&|NGYU|-rQA^pH?Y5S@P7zCu zref8lM<<~Ko`k)PE0&LnwVueTZND6Q zLZXLbx0Rdkc#)qxs_I#*F!rWJjm+>%^1&^o(Ay>kgH@r!#678pYQL=gY@n`wwa<(q zpEN3jILK2;c8X6r)!5HoIP}$MN-hJ2?*JsmmmuV5Q%TgWhGVtW38-}@#Y%UW7wWyH zDPnEAP8uY^d#!T83g2xlMvZK>9<;+>CPY-sh2nb#+yt^bDE&oQd_Ng|%J6CO{R>FU zk~`Ec2vt{W>m^xt)=t_XHa9Bb0nyjpRC7*9#0?dl6lAEMaHxL5V^{iN56LiSpoUgXnw`X$HmzkU$q!APcdZ4YYm zm$Nj3NuugJLQY?8h(5>Ctd}L2-u}>zrTRXz{tYu8Aw9y^KL}pkKWn3tbn@io585kF zYV#S2q; z2Na=mvb(G0w%IhHzdj-6p?jtoMwuy$29-m`kuHvIf4~-@5RrkG(~Et@4`@he?u#q%)v~hl=5INUNCu{ zCj2AtuXuG)VFt+8V%|A7wtZ$DqtuP{*BJT7G)|7(qhrp>1iXVP~B-vrs~X zhnTnRfuZZAmqLZJj~)lvKQ7wKD-qgz(8PbrPxS?c0grx;sdYS-?OV$k{!{kFO}zub z?keoZfV)qlY3u1MuWJ-BUGk=L2Yp-r-uLgOlDuOoEyzZCca3B2IZr1UK^AGctIY_m zoOABQP~JAyv*+`_KYZVN*d>2Wzc%fpjY*Q@?-{cSAKRIvN{`0h`An-5&Ty_0PKem7 zoH$C~@uvT!oOXP^EVu68zho{SxrP7pUvZKX4Ivkd${olx*m6Y=-}HclPCj(RnasnJNa+ct zIIS~Svc(VK_TdUYIeNs{YdKS3=%Zj|6#t4e&8i2)wLFQ1O)M*dAJUMF9;5&&7}USU(__WlvsO^ zdI0K8K3HWrL7&fullJn#T(JPj5G^wi%esUG`7>wKfZgnX`k9*V;Dm_j4Z5G{JsI;k9hIotR0*q zcp2k^1Z;I~K(5^g&KO#V{E1=)vO{GXj^^TI2HOn}BvE*km6iFYm<>$7L6paEa_%nD zt_N}od2@5K30dRu-3tznSWFrHq@yRnoS_QJsVdB!nm5U5X?ZW%+Y6MPSfZ3|xD$Ky zI0vy0LacrQ!ZDXJvaqmd2;S<&)D6+4Zph5^A{G+FwF0xW@jVRTL1Kcv$l=4F&J6~^t#O1&#W9+U<5-L%olx;i` z62{FCo%#${PMs_dxVT7Gr!n<{=wNFuL_TllKH)Anv3&t$hrDJJ8t^bdj!k=@JNX(Y z{Fd=HRkoNM8822Ml`Hk&LOeL@7(Hl8McR}CBw8l8V2Tssy4Eb^Pn`Ofb50u3T25$ zUgk?1Q0r9T2UOu@QiY5NuE~VqmD>x)2Aqn2=3Tz7Xw$ASlRf8w@xPf{L%d%3ujR)@KbCwuKV$pg>qTSl@tSNE`xph;OdV5zu%( zJdNq7kU~UNbv$PRiT-L3BX>2&Ad>7h0QV`P_&#S%?9Ykzb`?oC#Y4|h7Srp(DTla%x@h_oP1tQCucwC+7T{Bf{y z!~^w6K|#R*UU^aI>f{pM#wUx7jg32T5y+9sGxlq@I50(=ItQi`yYi_Q4o{1VNqi+M zsP3@LP`ZJ4)*jbV8dsK=WcfIdkaY0sa5ko}1~4w<_7;CE+B?=JylwKs#InTc+^+Ok z5_AR{Ppr6-$$aObZ&n$>3$sYAT~fa0&wQ?U8~N_n!xRk-Q^p@Vk`TGbQqRLW9dZd~ z_tms~>rbO~NFbdqu z#C>g!u6o?j0A!WPS8%ww1@1bIp(4rAw78jr<-rg-$OlI?w?oEOTO6YW2Ik-sOcC{` z9(N2`S=)Pmc=txE>C{veqsGM@k0>oGm{Lo;7(PE#sNmZ~c2!Ts{^x zB6mdPq|#qQ))dh7em}CiM5sOtIIIW<}7H~s&cTNzPdDF7ZhEo1b34M z4bDD=iA$yvJ119Q(@7Z`1*1`qof{OrNdOCNamuz#{R~uTsUG62P}qP(DOlvgECyyJNo~(Ay0)hN+>j&hZ6W3U#qb7q^u$ zCh3x=A@U3*Uom7yh7?KX6R-mzTw_7ezpb7RWmRccrJ3K%jpJ5&^DL9U0BX~`&fVO{ zB%JOmB-he9?kraJ9^1Kl9t#VbPfJzF#8X={{H%Nz69@syHHsJ%CE3L|_)WI~VQATs z5j+8deTq|qV1u=0I}_t>*aLRvCD@qXLuPXMZtoh~%u+WAskk>LoQ_wA-Cp?}rGdSl z3}3-E>y-`XarpH9^KE!-)b+5g=mgZZuQ563JbTOjz*NUmzvB-uW{6vL{2H_-IDGV9 zj?alXwcdyBBqk&r)E76T%SZYS5qap-Q^7RoWnwZ}Sf&%R zWWDp9;V3n#^iR8Vn5G7s8%$`TBUq(bOW3l;WI2z2oK;v^MMVvqsmsGpp^oOEh3*cG zliyR;W6gg9+c|{T=da2c*`tmAfMoR8XHdTgjE9x7LPN4EvEUyqHGS1b%C=$u3#2zz AiU0rr literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_connection_hijack.png b/docs/extend/images/authz_connection_hijack.png new file mode 100644 index 0000000000000000000000000000000000000000..f13a2987b28d70f81bda7096a54ec479f20b2690 GIT binary patch literal 38780 zcmdRWcQlrN{5QHQWZa>Wd6P|cLS$x-WJmTWTSjDrtVC8;86`53k(E6|W|FMz>~$kq z;dx*6UFY}5^T%_Z^PKaX<9p8c+}-zeeXh@Vzu&L%z8+mylP4ymBE-SLAy!n7y@7*+ zrw0GA;-7#|9MZp9Nu8EL$=dljUBrC8qOcZqwAp>oGp5>Y6`LFzYZtTam6~jq zl(O->Yv}U0x4S0Q8_JNEaN@t7u9aS2%^r1;MNd*kcE>vaFz z5`!y2a9wzpAnfBm*HrLpe6;=dC}A)=ihhkB)vv3_{WAtU3bpE!|9-z-9X{|4FysIC zNnSWjiBAaQ9qex>ynipe^-RF}iRHnK42757b01?{H@w+Mb#iW8>t~h;CXy?v@!A#| zE;Mf?No$QjeRCr6Kf|fhR%So)ZE^ao@hDl4(_ChP5&vrOXw|)at|5biD%SkSd0{MB3haB+ zu?K4qm?#;BPH3G{z`Q2VdZVK%dT%w4(xM~&;%j~3(skto?zsLpx&}HtpZ9wV4(NS% zOkdSl_2f85QNz5RfcZRj{IUXOfuO{3+z6X~jm2MKO~P^O!D_`Eiie6I%FhvxCU>(d zl$|lO9+v)9us$IL8z$#QX3X+XvGUuSr51<3b{*b+Dq^_vIU~C6=+MB(bLI@YUf~Oq zI`7?$jrCUgs?65z-dFN*bIWjPD!8EerNH~Oyl8BNWO6$H|*flr<}TKpI@tT-lk7%*fg_D^WJ#G!72ZQ zhC|$g_&mRCzG1~wyx1q9*nccoaHfEc7Ig9x8!xA zC@7Ur?a=r2^(cw(Q4WjYpvF~pgM|&Q@z0qiFnHWl1j6Bxb-o9NjOj+j z);(o;w>G|i?BD%X;X*xk+2Px~QXEH|^V@Ejx}Im3Iuu5Vt<~eWuZ2%B@`*f#Vnd303ey;0#-cUPNOQTi5y4&vQlKu2>sck;hC7b3B`MJy9J5J@w@I%jnI~@r;Yob1TCDB)r zc{-f$eU;xBbjsjsmPtcE{ss9bly2+gDG^kc-gT;PzbrqLky8BFayzs&*edw2^5}5? z;!^G18Kr3s$E~$*6~)dg#t#lGGf2M+%+VJLfZ~CI1p<%M^Shc4d z?RzOawF{Dwu@7ar-jJo$_o++Rch>d_tYCd%l|3GfdrT8ocy7Ml`wIKk@X`gFFWDM$ zqMSURS4PTb;3@FbyCSGBTm0Htm)j$~x>G+WHwF`;u^&pVUu+e@rTPhIXBvy>}WvM6J z>3f|sjr~43tQgCp*q^K2Uu5jNdo9)Hr+%sZ;k4uPAD0eahlDInG(229q!hdzWj9u> zFZYCoJ<-{72W#qo@7nXa74PlQdvPa~jkkY}ICfOJEoI0)BHN3a@yZ_5J9~I^@@Qw8 zG)dSw-Fttfe53Ic52a$3T8fR`$;#7h*6qeEscwE*d6pgV8w17)%-X%>A{|!P9bi&cq}6dT_zs3pLogx-*SUu0NOc zUmaGtuiTxaz5)kz`srmyqYEoRr_P0Y$MYDk=E5<31LuUh_Utd7+fq!0%3apN-`wuEdD{_mRDM(phX&uM^$Bh4wZ~^a{#+TcoNkF| z=k(XYTIdbcY>(N}1YbT?Jc{lK^dP1z<1u{yqRjS7Hbdx2Z*b)qZo@$~8G7~3Dwq%- z>z-8YYxVnY2CEHwK zULTSo3w*BCk@5o3z(h(HYNt=tjvw^qGRJ)~j~24hzORl}afs}lU$+{bun%}yH+lRR z>4TA+P5kfTU+k*;i4W3=jV}ApH@$jxeREsjOk2%&{c|27rErK99XC=?G|G^sif%Q9 z`Afdhd7EM0`efatK4neBFl6Eek%mc|oI(3j_TxN6Cz+J3#Q9f54yZ_J6VF=BzOY8A zc~J#Y1cwJ$)-Jm2ZdkrzBTDk5TFeev7wbvdE~RYdE8gMH>K(b6ZA$Mym}+&^ef672 zPv6uV)zou4N|AoP_V`vmiEe+LyY47(!Fc7%!0`<_8=r^ak@z~pRWug^nyCDPXQ zBv-@w4eHN!CPnnk^`u=c+iF?p%i6NKKIYTU3R_lAMCX)X-8PS;u*fSBnlC z`0H`oW9vPiJvnN0=5$_waqjLarSX}|O_Svbl1AHZ1q^USxl<*bC_D}0%$;31{24zs zur7I3^O0$I4=akCX#4A6k-3j{#GE1~4{DIcu(Y6)k8$wQx6oZ3Jau`8yH^i)rrY~o z){xiIwTvysz%e+3da}7gKq7X%1maK+KGi^L%pNT%`+=swxX*UV{lR&Dz94E&<9+kn z2_p2xIEJ;}6;!nk=T1s(plvBP0!-ORQt|Dvvt7v|nrlic!=(``8N=__YWKtQ1`3QF z?aYtic8Poae9Er*I(zb~R|gx;4eQED*#+NlS{*B4x`V!0UP zPB0RT?=V~~6<=>Aw;?w}1v7jWbz8&_R%TFh8ZD?h%gn*}o;-G-a=dZS zFa@fEoXH>e#ysU|TMFsF9)A_>XHFn2em`aFu{hqWzT08`bYPMrsTkIe?@E?JG-FF8 z?|mPZPRS~t{W3h8)QG?&8LPfxEH-~->ceC1PBrQI*xjJ>R`=z!3$6NhHR(Gu`j*BL zmmY3C#x@q^=@&09dim7d-^!)F>|!)XCw{-g`Jxp~ne9OL+k@F3aU9xrO?}wy{rO1* z59lBKnru%J$g=d|eT8kMfAB@y`S@owLu-d~WqcjzS$mzEH;VbFrix6hURz5nGU;p= zmG2pJ<(~Q>a0cIeGfz}p`3w=KR`!d*X@RN-N4}#JwXuG$@IOs|*Hi2WJ1&jW#7Sus za1y`iMCgGOhPRsJf?W7@eTM8k*o;I>1{W@x*$CMWGmTcdX%;UQ-rcznDVDBtIp3f> zKbl*M`u0ftMS~2P5R#ZnwmO%X9%<+Eg}*-bLPOp-81t>5c25awUk8c7X3M^L`GqP3 zkGlrfsAW&*x0#sxILYo3y?`z5d?S#8iBL!TZ_#pL(<~;cpl5f+t8GvYwI2 z7bJ6H^cQXuX4MIIW5gxHs^O%cXvCM%*?B}PHc@KoiE0)xedI5Bw&}~jrua{ML7OiM zSNE3%`klNNc-19C#i{K02nNX~?xG;q#iM8xO@7Sxwm5p0I*i_Y`!Z1P&GA!bDH|qd zjjQijT+quYB(LXKhYBmcN8-q^|G4tK!~{otujBf-WKI~|`^hwe?0IAaJB)=;uiW{X zmo!lCN|aM2=Iv5Cpzsj+@~?`*zB zZKIc^(4*AoL>s)qTeSeYAmjGk`>dibd=nWS318}b(G;%I$(RB4_aOO;){m*AJu&~F z3_mG9y5D8>-@rzpJ{2b@R-8VS`Cnx75I*q_K#K6c{!G*%G=7w9ApRGL6tYTA zTJ(fiEB5p*2*%^|Zmy~LKUYS-aN@u2^zdJ#wG1$6Xme8k-z)RtXYiIu zzt?yrN1UVmMh9Ox{B3$LG4)vkVs{VQb6k2Dm)XwfyEE+~)vxnXFXOo_eu+LoOyzNf z_LK7%Dek11*(tn+YWj=s<{0Qvjf}~)2R{w&4433Y=B~R+cQUJCdN=TbL;kZUvG6n2)2siv|2aUFM)g?Uf7i)_41UJc=<}cZOT!L0MM=l< z?;sH{nkxHV{onf=;a?Z_#BFN&&ma*<8;igFpZlMJr{yv6Nc?w@Vg#}&PJ4jHW_IV& z5}M!Th(41mp;EDz?7$K8q%p=t0KES4-Uy^C9*{hFP zQ#~3DV*DZv4(e>+AacxgCJ9{K^xByWiSW`MFVr>F1T!v-q5W1mQcO{dp`JmPI78C`ifRUv&~k7mC=;-Hzdt)7JD~JRE&`Am`y#!hUPBQ!l=*tKhco;27k;@Ct49EcIw0J? ztYzv9u&QT_j@{TtdxGm-OL}Wqa&PsP#1NqPq#Ab<{KLIp{7x_ zFFo(q1w`{~hF-+usreDJakJ5-;!YpSE>dg$+TC32Xaby+0oncCy|uBy0J7<~K_2t`ajnHaT@T~x02n@SBfK>B$%OO=R6 zRG$lVEuFlm9M2WQs!rEXcrpUT7>!e%gU}|v2YVlZh3J3COg7RW${Jn|@stYyoOUCD zDIstYFE{F0xym!@2m_Ajk9I;fqI@-Tj~TTez^RvMlcb?V-x+@Whdgvb~u9P0`Y+bKGqbN4x6mI(dnA zFPPuXfJ*=F-j6%wT)bnXy<2l>50st@*?$rC2H=z_7eR%mTVD2r_DVaT)VJ=Z9|t+_ zkNfVcrp;a@H?#+n<{mT;xV|K-X3i*>ik# z8aD46F(`-FA|(#rQ;9ZzB3yOzxT`T{9x2YU^KSYM;6E}CO2RW78Px=B-B&Pg zZ5PkFYY;8JaJ;uN%)GVOPyTBSH|y9*5oOA zgQixjt(@?G%uR-&a_4Kk;%jJgrghq!nHYW$~Rzs2a4C4 zvyUfbiI~k3iKp})B>lMW|KSmEb8lV&cT_G+J;?CQ`27O_D+O{~pB#u>E@Y2x9#w01 zSJS`EmR;x5rBp$wrA_DqWj85(ceL75;Drjr6KA(kh&*51E*g|+?YBnK>+U1f-2Phi z<`aWjh&j5gBc-<1G>(uVMSpe*^iZ}I7TNT3xK+zX(zUDuK7V80_gYop$#L5YgLV(K z3-r3J=Rdv7KRnn!N!$>6`ofn3*j|()Vn80Ie}1c1B%S#NjtM6>clTBksl@wR<+pno z@K@OaoyDrD`<3W7LPEA*Rr*iv+Q>YIl#%m90$YD;8Q@849Osa>lkKD8#~VGoh96 z;8flCu;5h)G>y#>WGbMA>e%=qw^9qWP7*b-7MhIihZg&r-VGkYGM@}idcfN}gIPt( z+}8!^-ca=l`{Ui>=fbZ}kg2?oDO^*r+s%yLHIJIjtr%TrQM#XL7W zj5(s`v6D@~^9|=Td#<{6%pcV`J*#t_>TrHSBZAuImlXu za@$18%m_(5hTa}yMCK}IP7xBNm z@Zp@XmW=RBM}ls##OYsQFZAqR)^2OaC(bR4+@e32L8V&W^v1(K?{|-8>@;f7i>mfI zBZj`I1m7=s;icw^<(C1EFijsZ$~8R$LjUQ=t9#kcgvrCjRyMHZ5jP8`l}qEK2Isn- z4=hqdeqck=Ii)7Om`7nao%qeJPkx*OeOUGLw$CP-0=Fc*&wuOLc}*l0%plU!-@->t zGX4X7Ie`7Ef8wF!$-BgTF_IV*`nf*?XQxhM&EuEcp)AdFrt>X!*xbgF5Dfd8k4we< zD(N7?MHhld{Kbjg-TIpnyj%{-kJTkf`zDX*&HTiOQ{_6f3`=fDBh1C4J1eyQZ1rS# z!+E~Bt;G_q!;)j`kBLL&rM<~y@3Vbv-z<;Vc^4t%Fj98WC>A4HdQxaikuFppPvbPr zlb51-_)i9j^AtBXdnBsd*j~RiyD9k9sPP8Q&g+#9-etdfrM!nz3n`5f8&`ey+JFrB zWmMQ@m`YcP{Yl)K96!Q@#>Wk17JDHVPWjTz7b_skOUoPH6FS5jv9~o~tQ+!U14q$w z?Rm%7k)8C1DgT#*TxxfW6nJx1eRxzgjCj2#J$Gjk z`puFPqeFTw`Z^MmmNn;H`~J4`w(t`i+aKc9tL{gSF@}X~%)(|xN}aB<^xG>Vy0jsQ zxVFu)o`lbv59_2uQDO$@wwoDV=EiStUOJ;Tx#Ep!J$C0pt=+@}@v}TF33lENyqX#h zSkTLA7jRWE9Aav>iGnX+S16=W0^C^O4*gs4$D6UwRTEDvUG&pF##aX-g2BNmZM8mH z#pC*v*VRSO@TSN%W z*~q}{6QnryJFSsHJd@5J!bO{RC(CJyb}+suYTPN|Cp=Vz8I{{^K4RnRFZL!w=usV{ zFKBo2CTRt-Fxg7+TnnBA+tv_2k ztrr=%Q*_Oaj2DDC@ZErg@@n;=<~o%T@xG2%i|E7-*QBREJ!Oj!SJa$b;bM<>vNDF` zjc!ooeKe6No6wgoB4tkt>a)vbK_t{dDqXWQqORHYqN%Pd{wCYTg_|xcrER%qtX`PC z*CbV595yc#s&>K-9-lrL>sI^7bM7FP@_s{wkd2sHLqmh%AEDDM0;k$W@AV01RV$f1 zOdiyevrXTgIOA!MPg6`Ew9Wnju1rxlItMK`E+q;FZe>n*N_Uc5Y0C&E=~AehoUY~y z10Z@$!}=ZjA0!HR37c@JaYu!>G52TU+lWU}iTdZlX2|ghE*0RP!zIjeingx!^VKI9 zzrjz|X+AnNG#U&#jlBq3pINx30W3dRrPF!xH|(uf1hAg`g*xxl->}>76)+znm0o%x|G;k6(=VLv-f%4B z{s&P%VF4cC|IUC+_oK(5YP&*3#3>K?$= zjQ8}f(3&wq?A>RAk7nn06p8@QwZoy#gai_O!*T(b@Rj|lDVEfu^f%Zc7O zPf#KNLJc>Rn-k;<#TE{~g{n`3^jfPw|p^VrtPp zOhWRjohI=LdlXFZs&hEe+kSqsM!7TH?}L){crtvp%vsd50$U|11xU z3M@}UN>lDX4glfDusp?k4@(*TrY@X#VR=s8BYE?W&_2Nh%hSC``Zf5UkoS4Z@SO4vv@EKi?58;M{{`bbdg}RZ_UrreEm^y+nSjZ09~83Hzn{AhLNL?#|6X zQCq5d?jq|!#Ys@MxnOIZ3+)Do(gqAbCNRpME8T4MAs}jj&4S}Uy{{93AAjBVz_&-=eC8@ zf9J=3tSW!At`msq%s0a z)@b{1<0KI!MkjCL8c-RzQ~p2DY|k1%K+ZI*xSPx6RO!9@PJ_o$Eb0Xe_jBEY2N&LY zHhnAC*{%lz!38yQ^z*#Xuf0vmj(y+frNf1sjDQ{kq0+;2%%` zMb&b$F-T6$N(U5iP3b#3*hw4VQvr@3PPe~#=#BYxxK;4rBk06WaUO1;v}!#zCm%t@ zoFeMR-%ey6Y8$IAgBdb4U_#d0lTi)5R-@-_r+l4L4x?8+2zg z-d=wh0Q!K;qcC8S43!eN-v)a-kr+Ki(5yjx%Q}z^_q)jPX1;%nU2Gs)>juosq|bj; za%K4E!S;9ua2O^}&$P8XcdX5ckr7O%2g-&O!BBLi09&(N)Wx@+ssNvL(V&Zvt`d|w zg-2w}PbmXF!LSmj$PPfwj$>7BHIiymNfy?aF>gI)R1N#3D(+kxiJ0p|$o(Dj!*84u zSMGdfDqpTY9zeED7=c7RvlKA&kd5N*Zp>)Pu+A>?Sy0$fOZZk(Z8Uy5zfn%EQn?&( z{8Vn&!;_@#Mdq!jCbi+!ZxyeCm$BEo(>3Y`##4A2(~-BO`!MfJ&~gw0<2L7d zsCkGcO?Bu{&A^M!$l*2J<6U8vE-VwG9kfqirEAOP41Ul1D`b$4NOOT1-?hdyb@tvZ zmgZQY&@(+AH29AY60T%I>L!u$0)JuT4s&wr(x;a)qF1(7*{NJpX1h~3HD9ZoCfc5) z3dQfDwwL3I8xrKaLO1(^U`WoJ!t4WY^}-z}3}YcCaF}NH?79^ArCGf#XsSxBux8fk zT6nLDq=8;Fzee_9uff+oSjKt-e)nZPfBeQKIzDytmT--ARmvyQonL^`?Jqe(D(!Dr z9J#^`fD%vHq(1!w)~jEnN{m62SFH5jnvr4xcM27EGa2DvFoPuBTw!W#Rs(s!Nl`KN zVP3&tN(PT2%nLNp0Wd$=49d-miNsP1l+T(vWl9K|Tf~ zpAKTHJTI+9Fg(QhHr=oR6T@SUb-dT3%uF52t*h$Vne3lf}QwUp>ZF#*b} zUQ>@)zjR(j1zdoo|H(&wdc_Y1ru4NX7b)uej)~5=`1ZF~va)MqHF5L)a|A|}U;&ck zx9-j}YesvJB9D##M81OH5|K+fv0ga=9g7N~B+k_*y8@_1eeNA;F@y6TV$YnRgQ79? z5}q?RZ8Dkd#qHvn#IF%aH#~$}E2W4`l5lH6t+PE%SZIRVup0vt{{AKb=af$g`dqBm z#n%a$`1EuBRU`)~+3VfbXANG#Geo>k!!n6W7ABQyNJ%GwZL@0@TNgOBRnP1NF8JmFX}y85bhh5YZ+AP+@e_EIR;e5eEwmFYNqobjSv zOc6s!a9!}4Y1TVMxDYPXfrOOjc`*Ki7vRfYKHXu$)au?iXG}<{4tC@OhNUXzweBP+*y*fD6E0uQ`J4kM~@%Jf}X?F7{aWmBw%v{bxbWZ^8Q+e;l0(nfFv+9lScPzSpYssM+P<`+XuNQr_ zQ_0~03t{!{?DKc73pYgEmlOR>#%q1rn_QrLo7RIe*y4Lz^k=Y$h=6Ud#CAZ4vQm)*K2U0l4rRv^V{Qa#(#jkn#ncxuN_t|rGqe;yLk6s%m8*O67hcgp39hjVnL~AE{H4rE zsVm+y72w{>fw*ANovaV2n|-=9@*K#&?puO{Gry(pW`G+I#eV78>_N)U^x9s%@!rYQ zbGfMXJHnE@*+uJ%xXeC<$>em5tDT2RA|C27joPC_Fa4VPrCfT2l2AvnL10e+yGwQ= zTj(=DT{)ZYSHT|%#pZcLU0nWFp;`NDLs^Q-(Q@LM%Uri2R79Ded)at_)$uc!IT1DP zX`BD`07{ukC{5cyLAU5lzgFf+C+e!uUksr=_o_mqSPAbB9raOnB>YB6a={i3n;DCY zE|a50xbTYo1nQVJ1*Yi)&m0ycG?fXs3#!Sf}Els7@DChpHnMYMC~~P$Ek$ zs}GLZ?>p<$@tzA0(YBWDPc>+wtRVwr0`TE?nN>-VO*xg33Oa=5O5>$ytWq>%5Cz+H zNsQQK$1%H-=hVpQo_H)tXiVUdiegN`saD0G<<`oM>ZKqGY4F+vU6ZP{lbW*Zoj~Y$ zd+-RxkCZ!hG(0@nkH!RlFBt`;lz3>t2cO0MV=QX|Y@Wqi9oyurYS|~NcfUiCR^YB7 z-)!HY`4kl93OFTNSA6;pU~z2{dy`p%X(u0jK6_gnv9+EDI;8g7`r7o#bwWrv|HPRFt}De250CQ@3?J_qdae5W={%y? z!k}6e7B8yfD>U;b5pM|^+g?2vF@k<;a;jk)D0cj*Z^#*pP_=vxC13!6XZAM9q?BTO zNkfyJ#gD9MTBJ&1T7vtreqo=^LWaeapJUq*)Uy=Lb~tQXpa}Z@mvzBApR-?zc_g7j zN`W;%jJEcvh$yw<&q}f-Tip*1;7kApSC`I55TIQng$sa3I8m*2O>n~m8SGy^ks*9m zoJgW6JuzaVbDh2q+Z3M*y;XT8sOb5E7Zox|-I?C^HBA%F*} zEhaGa-2lBO=T=o25uMl|f6eZTa(>?BkdvYTJsRa%X}&i-lQ^_)F!#R!#tRTAh zOpr7eh1Ji7>KTF_vW-B=<|xBh7S-SWZOeR3E_-u#hUzGEERwOv3NN7NHqM|;@3|2r zgC?E$jQEV=IyK*?i+1S`@nt)#QG7;v>9|EUCD_xU@%!xFDE+%Pz^}ueLm=>g4C5b) z@uR#%RPY%=xAjke!=R;2MT&*zz;>5n@WXij{!sQ9#GBjm z*UJ1+G%7ghh$2gp&+v!rID`Yek#b88$OxF>c-^B)9*kg~LQ0r^$#y{bx8h#uP9cz)oEArGIyvX@U}uLjB1C zp-v!t{0Qcq`Lv#uGG39A-xRx!SLzCj{~-9LD5wuQbz&qH2(F6-QCvO+H^*my;h`I? zG?>uJD7I9yobn+ZvQS!NO8N%u*a-rY{m!a0@FX5b%wCdYurrhx3p?T76Lly|s*v5F zBn=W3(Py9*NKdC8!Y6YQ!VLw_84UYBTL#%~NFl-mUy}|<3i^xv`v2}blAqCkmpXU& zt@Gj0*I}+d$TS{<|DD7{F)N?de`ldsolY%y8{@P8Q@i!~C0iC}XXp3=#%p1r1)FV5 zV7GA?$Wnc-1IB*_sDhs8x--#a8`V@U50`S1<+h8)NNRz6~C|b z?6M;_xJsTPF@4LnGM)jDHDG7qt0_CxQk!5x^7ogj(=6SNW>^9-;)@VhN)>4TN}Dfg|L zQ4&OqQ=qu~jL&rTo@tLW+LS#|OBKthz6nsODx8u}8gnpzn=bUNi9^vP!UqcBk+EVO+L=P>my@ z6-_Kw!j|Pfyd~nk)py+X>izLmTA4)r7Wr^0us}rsZmZXDOdjOSBZr7@ zP9W&;O|Pq#`igs=Bua{sVfGH3B#znz@J+Q}aRm#NewkfTo^hQ~>2Wh+nTJ4o-@u(h zJfBeB=eoFiIt-U6J_?8CHYKF4aAFAVFC>=n6YDk=XB|v1_GS0_xvVjg`e1p{1lb;9 z9oT!uk6B4Zc9p{yk`Y@YyLiKWBnYB#o3Ve02872}f9aC15Z~@J@_LBD#1FFvTmYrpb(3srZTVQ6lYrSZNV_!d9U?unK+7Ql~!; zmB(=FGhO_%<@{BmgK_4%5^wJQ5|>;5&cpuNh0I?#fo{V6##3FSBox0>jHf$_aU-aLRh}cQ!D@)9WOc35~Y8 zDQ5bX3tGgIpwfu-3URyOkI%qsKP*^2{-DW;$oR+-hx8L-SKHV*@@ia-5B6xRfx>_l zF-5Db(td^A6j%>CrAN3Pf8mXjZXZXB;s8OBTEopWrMZvy$nb@bVW~dd{iDmER3uM^ zp5xTvny~f61lzXt&IDW$3PPguWW$k{E_)MG~eCG-p)g=*2~zTQ4H z!mRlPG%)<#N^u>$)+rC>LQq2v4c zbJC3sA?5PL4>WPJx>7x#ym1yiO|Sh%m*#lJkV7)E?1^;E-WLiV|fR1s5; z8rc~gWOQQw5~dhKaGLu7_h|>cQWet>zSiy2_Utcrw`4O~(pyt7jFm7ewBoqF#_!35 zS>wX%WlIJ8zeHzY|Mxx8@nxTTOx;DDcH1v{_vd+-ChYJKs~!MBMd*qc<+zzHOMpZ? zc%W+_U(C33$GF2~a?15||GKo3us`I4Q(wQfL*G&LGxvqh*Ty`iBRpC!EWm;?ZJk|G z`aSECcqFU{(b3TH{92`xS$J%D>WuSW*MP~vLcQ}HH?ZojaUjqQB2(;TelfMbe$IqO z`@?Da2uKaDlI1g*rtr0!kwvwCet>&gA(kbAP)Dww-w5nOaZu8=E%v`T)*kZ6haA%N zec7JG%~9#ogsczP_=tbYd6Kf^J3yU%2kgHXl!Ha9LG+9P;4bf4m8`Q1)_u{pN>T-+ zL7JbU`DOkLH0yL5NMZ=ctU*vdID?~?1`0ZWg|-AzBIbfT$->vS&yluE7WGsvARXdh zPL!aX==cJd?0CQod}*m_tyZ&<6DZpod3t>QY< zz|IWqs5ipEIaJ>#j}k#0}m;Q`DY3!YE_JN5_(lG}x?_4h9W z*L$|h;C6i0ttt^yhZIl8;n^%UDj-`v+7)hnIeT1Bd!%QDB)l zSS6JxKuj>MbYHRQ=Fcr-PfpLhGDP`j1vv5E8kEOFRg`_sEP28WkzanJ+p!w2do!MU zV5>rssS-L9;U9GJ*pQ-b0ihcFw{yXF9N>c1Hz>2yNX-HPhS1l@qTUa8(eOXn{futV znbJxBJ;3*t&7r5;sRJ7|=pE<6+db{(E7R>9EzROqL7qJ|O!BaN0m=St?1v80Bc~;@ z=EbBZDN+vFb^gq4BECsBv5YxnYb|G0?4F6jM>{atXjP&45YaE~#(oG1Etz2mgGBKt zSQBHQ5*6{lzM#3{u3~RaUw|Zm=nL{0I&W_x9m*YW!s5yaWW1p$(jFR25lAo}`|4K8 zN#0}yjL~IdmcOci`Iib+V&pe;UC#FwcEvTldU#GiRHzzemIbk&+@RkXC#L-_?T$7?WVLUHI&+>S9qRGom&8izm-eTV(2Bnxt@B60EPduN&c zu0=y&W;(B(AF0eo%Z;@o{W;UbU8Xzm>D?Hvp$7*n;8VLH6W^wH(tTv+5ssJRisUV~ zn(#2W$rIvg8iP4@WPeUzBS8rW1wU7-Nrd11?FATp6VS}SHIWI*T#4j7=Jh<3cKps$ zvK8R-#9XH-Qi2T<3r2rVN2F`+_<&Rf=I}Pu3hzKKJd+&Z9euW8o-)0uY*fVfZ6PQ5 zx@u^zwAi`DO6XOSK}#!`&EJJw%^~Rxj@aB3wjOSEV>eZ>$D4(0C8zfTDk`-e;{H8n z{;1P_)&w#(XmYeLpnzxyB01-R$DJII8sO^!&@=JB+v6^G8p26W$pe?`-MQ|Iu(8hm znplB~EfY2$hcNX%noBs?#KvwUWwRfril0vg?LU{zX|=)d`VGKZ4&~4?#M?R)5IS@d zI)?dN?<8auR8Glz@Bdi10CoC?_ExAk{oLn=&rXj7X1o<{$%0N%3xxSw|AoO4PxO85B1>yKrzVQuF;PP5d7t0fPOU5gs_dL(lZj z&H$Fd>GVWi<-hh9CbSt|Xiqxx6C821N7Jou{>m3nt%gEGCvp>d{&z%YiV_Cs`Ty=a zhByMwzdU|}>?pru_!HYbAR!RK5MI9MPr-`;iv1d8{>>;R z*Pze~BaHlWLXjH554ik4ZB_q#O9!SWFB?nwdxyeTfWrrT_wN6r3;B`aK^+_qZ8E#7 zRcqgFKo77wE%f|XCYFzqIfnqQH$#XByMdx9cXJHNGel*g!{TV(%Qb=~S5?1iiozJpg3yce@)sc1vSEzBIvEu@`?h}BI zM4UEEU}AJ1+63}SG8BM(5We7J27f&!oL&n1R5Fj}+{U$BNZ5rg9R&#dPfvnL!n5p_ z_r`Zc_M0Wy22{&Ub^n2zZV2B|}%6By) zj(SV*OxVO5Ay5%%a$sD9f<4RW?yDABGJGCnRW_^_n9y;kbPd6muQ1O5?=hoAfQ#TY z0xmxadHcX82gk4PER~0IL~DigQm6s!;Xy3E-q1>g z2rKM2GD6GkJdlt3{j0D+NP>-@)=Vk1s}8g=mHU9 zx%4q_#%p3_!g{5rY%iN5uafYUcy3zNEd5LsvcC!K0b$NukkH6fo08?4-~|w;)%@tP z@Yf`D(oo6LnBi-&n!?3&$k|{963*z5D+QGvfMXSG`jbYSNuxK#4h4~e7-+zu`bmT} zJi_0378YrUt#SZU>@^IOZHrlU-kH;$Wd9lq(brJJ<&He?+LPif;f42UY&JsAV^Ao_ zuPnfWDg-1aF%zId01*c?MOQ>gs*u;`@pFAi1h+@xY<&v|F|rb{IP+`fWl<<;=@-yH zO`*=f&?OBI-U$75?8!g$g$QKNlFSeEgW$59V9gJsHlKmie`h0NIv2dhwa9MP+NgUX0(bJX@cA96fgU& zOMeT6|G9YLA7&BV$(3{t373HgqcHfX*aV?k(CWX*;m{Khq20}|kblN#xntS}&CA+h zDj-b3FsNl|^)cpJFeyDZ5F&9fk;*{X&3t`pfY)&BwC7Cw8MmZc&!LqhJ-;XZ;&Y4y z{`IE=|IunOR^po_=96!YDv6OMXCyI5_)|<4NPizNLEe?oH2po^ZoKvu68V2SwrpzQ zEi>!D?Mb3l5_zSFC*ydG$Gny7)2`f8gLY0fggAtk0NnKbwJz@ojYV-gP)3Jr+&w`0 z9FXR7f&$hw?;DUTSdk7kL(n2`xzRL*qV*8&6sV+jP+b|ezqv$;?JpP0i5$L;&{gd7 z2MwO>tO=f}3nVszKy>?8s{xi!X8cT0^L~q!&6HwyvBMW{Rcc|hpnvq$aL7whdx-Pz zEaBPErjsQK@9UaAhFHQuYd#Ac2-(oMM>bHam!m6~4rZ_cOv>>pp3cA6iBr#cD8esizq>b{ESll$AN7h$ypgYhOk12F#g-awz=qTyd!C``BV|Ps3oH&rO(q&enJ(`yE}W4500e*pjU3Qc9K@JjV(7nBDc}|pjk6tt-QU)!-R;8$5Mg!XpndCrtrSN}ca^ zNKnE3g=*=8C;Q&8c2BnRf}fr008J!R9I>+$_LSZR&f5mh1%Jg7-ck~sEaK7*8*=(g zcp{@mKquaD={NA+n;XzuYtS3mi8ipLi)kzZE`cz34C!%1b|)yc*(>KF7}R!vlx~ zNkK$(k#v-Mc3YZUGedy^kbX40I_DO&ZZJGJ*h(N16b_zPMtW?CXP9w9Fjy+h9ad-+ zEO#6~i_)$`oRLp*z+7NMX$P-n;=rLUQUkb&$ zi6v*0g)n|}W2~VXsDTSO^uhJsL2k12rj^*h4i{xo9>V;Pl2 z3109!za-Sq6e#JaQrF6ttekBsGzdV`67Y?Z3I>OkE(z?U?CM0_Qq?Auxd2i=BWYOa zwr6d=jLKIq8h=?952Z#qpQJLeA9!t@>M6&eo1bSz%IHCxda3!Ispm;Oh}vLmy~=O% zg{?f-?7hGju5TRZK-#b@jHJa`NU*0>G2Jugf1U2_BJiEz)21&@fu`x)V0KD;4;D z^p>2<5`dfh)5kCIlRfp9Uo(@2yk9!|V&zj};2RZiApN5xo0uuo2~e#aj4=p7>Q2oG zPQoY@lwV}H+K;njEgF;+foq&k(|r`${w@X?M^qr{*WTMo@VlxYZZLKb$jEd)BLU@c$D>noQ&uk7Vw4s90q%+(^ zqp{nqvy{Gbwn?nw#S3GKYn_(zZIk%c6OF$m0?MA=?c|?}`Ezl|Rg_~(=~n=Ihhx5& zoZuP<9+Y%!AciF|24QsjA1fWI<)yO+x-7IlO8jAC+(|IWV-QN*9RXj<{KgDl;;fIKt>NOqS zGjt`~DxPDF|7kf0TMwsPK>)p=fvD7{+;?kmOWmok+25);&tEr7Aa0N}8?3`b5+`H) z9MwiAM6`n*H3hFHYuj;xjb9vxQ^!#z|_4I0|0bD}h+FH7Kk=ge6 zW?|vWL%Ls}C>X(;G2e+59eg%3xp`(d!p8vb3Gslus!ID@#%Rx=6Ik^XRydps8n^1^ zeSWi9?x`3y*F{tH37w+JK3TUKU|CHMhxWa^X87y-2jVQ?G!O@4zSA#{0bQLFBp`|K zM8#zi$qpw45Az3yw(<5~hR@8wwg*`UE)e-L$%o=)kvdI%+dXUUY}Hu+`M%b2*b4TD zYwi-Ud3yDsSBDnt7NXvxfwe0%SWJzjdg)tmqwde5UG`W$)%vf-1ust*3SQDpmH?fFY!pO|ABVyN%j%oJ}zeScC65P8OOFVZKp&JO%9On?F4Tm@X zm6qY3O$*~0TWL)=mN8f>Ugy0I>JX{@EZemkJ8Z|%rh9RPzA#))8iIIhxo*a?Rte>w zsMkYteiUfFZ-2B8oSYH7O6KMXLhk|w46cmM(-YhOu(EVGT_^Ue@Sp8ElYHXw$-$2U zPR-fzvPj*s0o6Z7+0!Wi8rC=P!4EgLLr)-6C+|@C2;~)`m+Tc;WW(8$W8a>fndn&} zI)Gh~-tbkm!y8p?<%H|uUeyVUU|{WOCly^rZ8_l#=!Rp#QeJyZIYsqSy|87wWOhEveI8W;W>VvZDC~zN-_K*Y-CdH37JJj&sa}s~ zG#h=~0z^^Aw#(prJOO7etyVzUH8BIQu%2K^$(xzUZ*o>X(PQAwXXGP0`c4RC&S=m- zC>)sliY|dnJGlJs=|5J_o|C-ciiGj`U8dEhN*&v?cdA$o;s_`p+r&*SaTYySp211< zqtm8Px_={LnBmedtUc(Uto^vQS2S*cv+P|w!8PE_7f@U#ur#|6Tr4~~XBV*iS~#*C zqF1Z8YKzM#L$eEJ-vH#2p_{zgGwA*0ia@D`a-kRDb~S1oNvq{BgT^Isk2KNZbLsG-11dZj^xd_db)r$_|3(ZlHzTE!HsUzwcZw!26ExZh9qT?jCxsSRijP!h=sjf5>0FyzR_}O_=?u-1=^R%z-|zB7<2Ij2{`t$hO9m^F$ED=-%jE-$+62ISly*D^&*!*i%(C? z)-FcL?s#=?Px16zmu9_}Ymkp-gMF9b0Y1`WN?q$#O{vq zT~GL#G>IH39B55E%Uz_X+1n+!@%iLY$c0ps^USgj{2#`V5 zksRMr68o`>nX)Hvo;Hnr);KwVeXB~n31HSIz&{Pv8qm3dP$9xQ`BVEmfdK#sot=hVV9w71n>8xz`Cr7gg|HbiO^ zEyxMW0>1@|(ru!F+S#5H$~bE}Frp>FVT}$;G-^Z3zPDMj<4?8AZA5GrPKPPW$ZCg0 zTo}xAPO`m5_ciN9_NvK;A@zq{PTFgy5Dy@Sa{sTV1wH(7)?rOVd}gNdLE%L!EsH5G zh~uupnx%q?;l<7_o=fv=JC(})M2Z;i?kQQA;JnJUY}dNWbD7~ek&bqY)(9>fSO=3q zn+VJcXLVP!8$bE|Aj1l!>o>Nz0gL>x=kRHcVis*RVU%oHMNOiKe15MP*xr^B7Z$p@ za-1toO)XV}LmjQada5C+iB4G{*?E*Sha1a&1Hj;zihF@>LeuCE{x$pDqgOwe5S9;w zX&`-V1{lXodQ}@}^7;ovm~uE93Q--Wn|y5T-E7x-|2EQ(O=25|60bQ8cpXPR1Z4TwGAP$W{J4ecD(duP> z91%gqXFx1my{Ssm z<88_61n_hVMo^tnNTof!5EQ=9S6PI+OOH#U(jMc85YIHw(g4K$9=Ak zt}cA|5^~kOcS;7#I2aTw89y>6e2&m84lu9F_md)2-r6NvnX&vqY9Bqntx8(CDvpCE zLr9Y_@7cwZy@^f9J=NZa_&QfdCkc({`BPoS%XGIIYxLfCZK#?sq057xwDjWBAD$^$ zC{I+KOY_$kHtXcMcT=@|H}qo0gwq5j%tgFGn%yo6t%KA8zYo?Lts%nZE5m{lhltWD z>r=p(L++)!O5OC@EQ^Bl%}kO77SohR3fKqE2{t%>1udBqi>HlaZb1|e>Nt=sM%#W8%qGW5*EDL~rg^TyYhz*I&g3QLVTZ*&?nrUC zrmXfQp0n-`D+Le%H-mvPdg&{#ZL@j6>X$h_y@66>e|w+L8Hbk&qRZz+93k#>w2YD& zZm}zF_(m**0%#vgg?*cMV&}oXg-VDD9Sd8;-&{&8}~13Mg&D+lamIKjO!yn3sg(Db~S`2Hzjb zhh{b7|3uDoJF5X<$EO0wRHQo_VJoKsi))k8Q)HQkt^-MEPmLuc{pkOZm|2kh`Gol9 z%h9O4s?l3_L;<6EZn)@3yE*|NPkei8jGmGhN%g;?oh}LTd<>q&8u$}gc|s^MJvkPI zu)hgF=WT@AeAu(QeFrLg2|1F-^jPz4gw$j37_Y_e=$nfl9R45!>@gat2y>qznr?1v z%_U2Tk#-vN!QEg6!mN5FX~KVb8WXJcdU=LtI-Sc*;>)3azjI3NGjPYd+M$r8+*gPr z<06q2Hk^(elk{T{uP^dGy*4_iQ*jy!m|d41EbOu?4P;C&L|wb%vKtJUi4vy2{n>6| z;lAdaoWfnO(lReQ4;_xVw)bwY;j=4k3vC!LlUfJ$U-$iH+chJFFg4{8MZUr1UQP78 zfbnCHk)^4D87qri{vz@N>6;XGfqvATeV+b7p8vQ2<0vd0fhZG^VfwLw+Bp)arvUG! zVm

#p8!y=e%VSEz5A1!{}AOuQ0US)*&9cEQaDFTK4dXv*Ss2JHEZmqF*js$nh!g z7(PsTXuHQ67S_@q_Z*)uCN6R;d{j;IJ4VvHGM16B>XM=jEF?5eHM4mdr{asBpRXTD z-^GxJxM7R1ouXq#q|^xLGIbCUq(55SXt?xM33H$-@wfl^{X<6I_pCysXP#LpuhxhR zo^%;aF?uNv;)&ky(2Ff<{K}6w{cymn$bDnuNKiS@#<&?6elVBqC{bbw(29dI_9F@R zU|=cxcNhVVYnu2gxNRKb52_ym(j7}$7px`TQ^QsHCkM9oxDa?B7J<#xOKcVI+W9hU ze8-_dm?jN-7W^L4IjFt;pNggR^8T%+@utFqo(7y{VaOFsuU<>q{|-IOTka!)rPCoI zkt^(-YD4E3coMQ|@J;GGd{0n*! zxwy=q?<3)U7VsR^;NT)nPdpFb`Wx2=(GRHE83by29O-%2^d?LAeEi;;ncvg=M=d!{ zQ(5koaBcsL5*Z9)d1= z(}}O&GCvN~-p!C4*%;RdGE-*0RmlKfW!zfl%Uq|4>6hEauJY)E+hV^aE;e2EV2xJi zJ{{5w3ps2rj>EQ8i$>dp!De@!{aNJxI{Z@OrL8uoO&``;SD2tcH+@vtZA#ZMWV6vq=&r;om~9#2cM?*N|YfX zDc%r#TYK`?;9&FJ8_QG7Mwy<^0bNwCaWH)7Kdw#@*kEch!2kAnD2~KGvQ=R}sTIDu z8uE?3d*5_1`i;P37kXVu$zN>W^bmIKfTkXP#u(fgYjfsn&%)d@7AueEIz9QYigh^f z^f{PvOBDAo#Jyjixs*|dfp+EhYQh}Nq;BQ5ue~nvHVvo5M-!0CmHn{=F7z(6_9qk% z`|AW#-FWN&_$(au@|r+XR|oMgh!>omy%MMBsa_9KU_{730g!3FWtoprh>XqV0Wjb z+_-do-4VSP;rg9C^1CYL9#N9YjB*%&H$b48wNjVLxzAdRz7IIW_%CbWp$$m^p*USt zKh*t}YiR0@u$aF#zkao%LWmIO@vfFZQPknl?14^%J_3Ei;*MWhrhCgbq&qH&PNs83EiZkbbD&k2@2Mq`!nf$? zJPl=p{Y%9enL`+)m<)Jy{JAJ%Wn8~~iVEINZ74IU>pwhr*tBkI-QD#cc3GvEJhj^3 zUC?J~oZaU15h7N|?|NOD<6ZrNqPVyFss2mC0y%rZQ}2*7b4~Bm5lYcqRr&4j{gbpo z+czkRo#V!zZ(?o%gy|IH=q`4;{Y||{W+0+jRcd& z)BjldMpEhEUV`3sXzDW<*wqrB2z7N^NHRoz{~EkPat0(sB&Sz^m%}@3y2hN{yNMXY zKx0oSMV>{Qw*bUAM|UDuC1xhn+7rHoNT1Tgf0CjP4X|{nZb5~;Av`jc28i;K*lCx+ zv0NDb=Ga(@#x8D)&p6bO+N>nDA%?_v2E_xBUU>BJrOs4eS+n1SU*%Xb=_$`qcfj}n zE8S6|Nk#T08FFB?AcHKOUH{sqX(QRAgwT(Xl8A23VyaoWt zqeTKIZ!VwdV^11FfP)K<^gBTLUDn{{T-ER?!dbuGZUfkV zkvQW>=vf;+k*wiW71WZJIkq1%bP?a2Av|b{1;KU89V7mnM+s*pYm|?F4wH9&yE4fp z&))VZKoctVwq53~CvU3(PIE3@>I@0+n@3PlO2%8oNxp7hR8%1G;03X#XqrH-d1J*# z?_ax;pd1dens!t7>Z?z)ko@#=UHiSvq?jAqFAVO_JvnBa&~kiKC7QSI?hOmKSN9L5 z{h4OuYfPaw1*PSI8ZQ0p4tUa=kzW=jF9GHo|0zWieCV&d!9Rd%F^}x2ty@Nfl54%) zK;KgF98FcNOy!!fcW`K8SbV1p3T+X@<0vG;8PY{)CdkTLAN_pnMb-A^^2?0d_>PuI zU1W>R)|t-iJ7SU%@g~_&l%%p2XZ-pO+)la7y*eQ-o%zb;TQ~kZP3I(N+E*Ce-bSDK zNLvb75G-2!`v!dd=cM<2ZC|`7`59bVaID0^cu7a${C%Z9>1N*6}@Z^bc^$>cIFmHJ}_FfpH0dc+h*```U|`HCJ$eu;mC(9Wn~(-F@dP9la#bewn5O|XFu6qM>@Ky>DS7_F5)u;qCsInqBDpB;5y_OEYH3mVX-0Sr zdZ~-w-fFjlme0t>c+sbq=R|I|@_MtiDd_0y2ajfcv5ej+IMa?1-HMFXj@15r%r#&_bpC;s2G- zhB&ELl*ligFQ2$DLxXY zcuzaMCQUV1k<#?%*2&$g<5ON8fjQFr-E-idE^FYhoHx@io}e61R(TelD)R4^e6?}g z#|*U1{xk0X!kyUJJ*9LjX#R59NN^Kj)mr;E*85W8Oc4`*w1o)F2Sw#T8dk~w`xnF} zHiTVkU}+hBsP$gW<|k;MW_%kS9=2Eixv88RFm$dsmjBT83!(Umk{A2QZq2SnWqcdN z1HxlUzAUg3y>ek*-1R2<-jz_%2p2NWdHiEfdmtMojE7^W3L<6hfkH~?rhx3Vp?;G^ zw~wQYnha=^xJ8IB_(C2V^tVbx@1jgsz)@VM!j2U(kHD5_bf~*zC~VIFguK$qEgsS^ zG7XI|PKbE25Jn|h-%i0CjTmI9jZ}-0=yDar&9G%E=IG~0z`N%PibffJbV9C>cm#w) zbru{@GAykDf{a8*DlO8z5qw$EQ-sf|qe#nVjDUTTH2@*I7BKv6FmlsPh-?uLH?!BZ z%u;Y*Zex^*|0vxU-~iFX7sh9?IzN;v~?`7_Jc z5T*B7vTh>vD}#t#dYHt+@Mybxq>oGVD>c{Gc)c&ULhVXy)F)tnGoX}5CM>%Bn*xXZ%S=;q5C z8+H`I=iRZpxV~ifi4*rmw&tfxv1FkR_7qovUmDL!8?&rZlz8z3amZ?LMyrrCNcp$G zzB;9|TQ^)%EV2Mw|MqI1%O-T$VjX%(jI8uY(t!HuM9I9C8)!^T*5F)UnR+U*9<9iC zAh+3-_n|GCkLDLYl!&8!O`9D{P6%4}lDS6&hXTSK15%#>Np*0lFo<1l#@S&Wi?5$& z;j1o#;sMHgC&il|I-UonskHmPPDZLl@JsI`wJ&_z{GV}ISM3R94{De@kGVuN9;g%> zaBoC55>v>FhHhx(r=yK?PlQSO zHLk$3cE9@WPA*qYKky@-1G8DBA9t@k;(Yqcbg>0~EiE=Gn3SeqfWjg-m?ovKztcck zjN>&WH=Ska+I>K{ZmHys;GFp`i4$hputQshr^{JbwtS=27HnO0LT{EOw+E zP-SQ>=lc~s{^?lJn?H+VO;;R?dMc4X~uHZS6alTo6a#>JCv{y4{Mh)u(!I3GgOTEkS?>DsTIB;n} z)_Vo#4^eX*nJGLaF9>h<&(C{J9IH#W%QL!FLvQ^eL1R{lktosTy6A%_QFL^5C#Gf5 zUZR{2js&UIv#0KFc`^Oqoy*xDbbrrv_g&(ULW+prZat9kwM6H5;oprV-ix5bCxlwg~>ze4X) zs-(+p)r+s>SVZO-IX>=ANC_rg6YmMX9G-`hKL=z9@s`M0P7q!$XcUGZlRPw6{8-xt^0G?LsobyzoIsguZ^Jj75n?Ocx%Vi^b z6c6KQCL=+l(eYQC>`fvzj=VytYSlX4Q(x1Vx;5q&-!E4pQYrKPsgMz3E}TSV+ag)w zMAQYIKl{JNurJ&QaI!G_od;dO!LIUiPE0wVxu4NZfW952g~TGZWs&f)BXhC-=ei<* zzVa1m%eCW*qu=AukzL>dPQWoRjL7^*g+O8!w!_r+Pa}Xq2s`sqi#7Ecwe+Sw^^(+TtNdrv zKii1uQBFRgpX3~0ZvwTP45kxoxdpUKj6^9gjG6Va=;)1D?Yj>GV4TbB(0aFQfS-v0 z&OS18WCXy@bExzh87v9G4M+L0D#4EdX{6a-yGWbG?K)4TC1u?(Sj-zjk3X0lT>qD{#j<5%lo3DcpsIK7gfHg_xF9If?6(ggMnofwW@HW}_-mn#lqW-i7N@Aoui|@=cvq%wlWC5>`yZQM zj?0w7FKbHknWS?R!|z1zzaXugSwtKZkJZoM;fuxwey{8|$fY+!4PQ7b7RI&b$%19y zX!d_c(jryHTx3O*J183RI+Mij(@(i*oQL$)ErY!dd-*C19<^snB5Vs{xjL~1vQ`k03UusA{>r^ z9Zi`jn>QK=sIO(;@&*TfNnd(wEMK0&wG|hQLl`X9@;~79EeUKoK~3iCm$WO=@4sSl z?xLRWv~wK4jX{M^cnTk>?m5w22>oPqg z+zhOA`#XSsZNH%Wx=NZ7Y1Fleu2r%kCdT15dBb87puBv?*H@g;0K_r8{o{8z!5q>b zocnon9j7%%+e19UiZ?cbut_#Hfrm(f&Q+hUc!LX9_P2N_K1()y?v{gU;v9TYf2#%G z^CWxHacFSgL{Z%LPwTJ*aUr@|FL^lH!1cwyrsEzXTrZb^=~7cej*`s3pQK5*M~D1+ zDRZR-%xk!AQ`i4X<29A>E_=0QOWf`MD+n}D5PU!4@D|;wKLMs#0+)J-&QN~-4gZU~ z|8BDd9}#6NEd8JH@iYl;&%(F;)ji`(y^j0HXB4aWvoQGoqXdMTUk~M-=W7o;ZhSaI z0)M5t_(#9E4X#JZ=;`ZAianNXIQ;(678KYr7i~^$RTP%J*pJFjY<*XV3^vBCYi*n{ zx(V9Fch#dg9QlGcS9S`XdM_d(#w-yuJb&q89R_rSiUvD0;nc$-;mYCsYxF>VTZL`S zCR6Xi2x+G%DuV{J@E%%)kBQLkU>I7SM=yGI%Gh`M+%A#Zj2-;XKzs)mz|ufBc(3vX z(ELRhk2|agy9f5a2^c(Gm3?_8Vv&343iU6AMba5jS`v(??!5+dC)iHyy=yk-5;V8o zl3WPx<<>V%<}dGp7y{_UmmBA{*4*;P8DM3(cbrw7-%`pb9wqgam}CAcVOL<;Ob+q( z&3P4)H^W_`hab~l+-X%U(_-q}>o(DFe@TQ(-`1x49v|D6(t`NRhE=y0oa zY)OB28F`ejeWR`}jF>N_4T}A7tIWz_^LHvty9T$qPe5(HOcthP#;vaGpw8c^FXaXH z-qnLc|J!V^m1OVvmP!5h@l@n|AW{6kyP!v}TwC)U&#PeyrFoH;2jPa2nh`xwIWYdB zklJ|MJkXRLgXGQ7%d4>9BOqS4GYDHbCu)uosWLJv<6|LrjIMCIgVbiFq1)c%bL#!k zV31sN5tv11r@m+qJ<{rw;y(Hn_3)`_l0!3vTguQs=lAa-;;|z01sJnd4|pkcZ??K6 zjQvuvFJ};czFSCt5(-5~2|b8WHH1FPa*laFt9!b_Ma{70OJKN@MRYi&p;b!OO1DJH z6ZNmSgoGYTnOP9`;cUoGs?p(eL_<0sGkOT(1I<)HlP%m=hz1kB2AM@P=Df?8*>}*9 zU;<=XYWnpoTpFdGO^uxQG>nCkloZzuH?@OvvxIm}m1zq{vvxicT5beS>9K#%}ZBI^SO|~JwB#zOx9tg5E z#>HzzO%Tj5(dT!&`OS^=j#2}d^-Ct1Fda5{zmv4#h|=eV$LFiQMrcnHscIC$_aY+) zeagprFa2lzk(>E1>UKcEZJl(P@9|Ekx2HhJCGgzXi0fFN6>)LsZ+j@0pQz*RvWHss zh!2F!$qQG}jlXoRW!QTRiI3-`%w)zy4x}Nq2hjeDBelNmv}$7@ugy641jCZ49pRS& ztIrK7Zw>IHlwPFKvq(xxnzJp($lNhI#wt&MC2WVO?pf^k@OS%EoH&7Q3M+wOv7NXHWVD zDql?Sa!-SoW&7cLft@MJ*Z1}&y5l1eK2zH9rAwH9{KQ4PevOQ52Lp?sYU2-T>C>BK zNN$oLz;c>gc%sx1o-fW#Zm_Xkmb? zo_2>#{=R?Fg^N4qv}7gEFB};MS(09q1uJfm#29&wGX#nmPjS{U+aI+&w2}wffUInW zV-Tf%L&u4s4y|WFy23ufVK!as89g{RP85}fSfYty^}Dn#8>r3*D2a>2&%mnO0}9#3 zQ@@~eA$C(*KVrE%*7MU&Ki|!8|AOiu*oedvs_(QKY7GA=;`HE2yc|@oj;l+y2yHm9 z;xoSLC?%tgb$fX`6q!YP0Y=^1^!Z%b6$Ta|zMbP-v(3+DZolfh4%Wb`lJFyJL-Jal zUZ7Q*Q@t4y$e8UK1C*uHW$l0y&_*?c9QXvr53Da_?&m6T+_n8PWR`I!PbO%1% zBpJp~0P zP?5Ko9x7dZ)92>w8K@qdD(S{571zbJdO!OzZMe^8RKKF4In0t_1>b0%9O^`NZG%cY z&|G3*Yh0FT2Rde1M6%-_uro^g0ptGkAINq+YIb2NK*2`{>gvYOVo!G_0`e~cY{Z2m z``5STy5rvHA@#K)C+>Z`ui5Xzp@9b3OFe(vH4jAi;5E^wfg<^CG*rz;V!q>kEWg5u z>@d!6e$t@bTJ-q%m-G{#i=6e_$yBsUZJ3Yn3180*1#sMwvwd)g-*863XwghsEOl#_ z5&e3fCyXK0nz@hVcqoW>7JvVFcgOPdlhDj1!PQF-oL`n{?~^ECMFC3bA$JW&W5Sbj zX!`q`i=e7(DJ-!G)@A#72_ML(tk%4dpx zZ`gRrHFDXsBx9e>UhA!4%AGDt-muxEJ8dmyVmG428t$YH>|t}?QF;A0A=vj`QejKl z{@HhrGsUP=K2mcNSG{aBT4Fh&6+CP7gdnf&yuDB_frDxl6xmV~u|?u|Do9DH6>dhe zy=I^qyM=6#q8QyLZZS{4I;6i^cEhtzdhM|X5bv!W*n8pA)1R}7)a@a>ZhR``sX}y9 zA5hw+w^~H#P5ZF@zL1;FDjKJ1Cni{OD%z~dA&7?AqvJe=2u!tEH+(%Y9*05p0u`kO zi)W~Gft(UXA}hj#G;BMLM@Rd{G}|Z#a&9+j`s%B?=YdFL@eH3#+K}m$f^Q=(UBTWP zsNKx_ZTdg#)%Rtu#71j04wqin?)x;MW4*tM>w8yKGC@=*-E{h_e=z3)#$+*~3Qg@g zA%yxYX%43CCbZH@wV$U;tpyok)wOan2FvOe(w<-Qf>ut2!Db?u=Gn4wLHkX0yNlHi z#qJAakC>QI5E4zu-Y@pixs_TvdPSv}qj*&EXI=pJ$zy>q3ToTDlI0dq&G$de&*?yr zGt9TaU4G0(TTaUd^U8Ls4j+qyPoLKApo`&YpOY76>gl$x`z?R)s?CAq-x)=Vd*3`< zeeu1}0{LUl-L~?XxN-${-h=sj$r%wpx{uXVRr>aGcC_lt8HToaB4jO)sA>yUp%m26 zsU%JI@>;b@ za?Wi3Tzq=KN6}6g_9SNU#dX!YefIL4HFz6H(t_ZoLD!1*{eh?n#y2{;!k4O309oLtUKF zl;C>itVJf9^oL(JMg0f+RI(z@sFYMIlnqci6ntMF4ZBcW@^M|qSuVDT*gIzzDEi+} zZAuDLjmSBgp3^pb?&<9g<&XCcTHJT7WV>puurVu1sb$!FK(07f4uIA(DckdAblKE6 zYKl5Icmz4#e$(&f&URlp$Nq!<*BO8NP?-77R9;tLQ822R_^x!J$<_T$Ks&=_8%eR= zrK%zxv6m@a8)=I8WAjU$#7B1-xs@F7_gQ}D@C(|cbOz5rVU^!Sk9T_S#P~?J-S#>Z z)$xx)E#DT=UT;%)Z=z}Xrt>$ux@HlrXt+(9GO$+YC$5l2q< z-ea67ms2VhVw^FgHdxr_6ql7U`gi_ zzYPb81~6w~MZVR_^OYYrw=mHhX)U3j}izPk*X~ww&N_5|rl+mNp5iE!nnf(RiM{({(?J z7vm%Qa>kg_SDiz9O52tMW#nBID54JSFm~;zj8r^%-FtTWmp4B}t!nAApv!r5djWa% zoo9R_@aXZ7(t_aU<%Qu|aunrwL|)5*(`668`o1gbG z-7u3$w|LrjowKsPLoJ(ZbVr-t76v0LyW^XF==s+){aRqD8}8qEc7J`MmeU%O%<3}l z^X)sz0#-?%=A@T-!`j?w7yh6s^-0qyPL?HyHZfjk$>fq1d|fG5IOJcz=CClc)AX}* zi`dw2rqx`0@BIBT!$&py2Z6@*m~cv|i!^ha>lYsCc(VNZl277Gf3aH2^qr=A^_@j5 z@z3p-oS=L(*%M+{LBD)C^(56kaGm_rKJLpU3y;|~X}(ARd}#LUbnxqyd}L->Rx;^P z_m#EhK2q^XbjbWztmiv^tBhw;qT>_4ksqpzzBJZLzY49dNL_I9d!;tpb$$C$%q$Ze zs)z`elkW=UbsPFoc0|Sght?E>(z5cSf^q^3Sb_WMMG5X^(?ezK{sW85*xx&km)A=3 zSt8Dly(~S>dEb#qv$to$FR@qfL!PvFEB~83YIJTE)0sO=ZCZ8$OWLn`hHieosqa_J z&m{kw*}t6=O-FY;*|@PUWaey|J+=60;1~Kcc6?O~j2?$uZfo=?U(w__nX~EOm>o5F zW1jS`mzL5Ihb-uyINr?5%X3=0-^eJ;QB_Yh%J;*-{)cM&&6(x0a_M;A#CvWxcaCUk zXcJD@Nqb)$Mk>`PsOx}?f) zg z00;hWT~M;|D#sz`Miw!QH~`0l-iM{rP*_#9)w<%H(4h+cqow`dBRCgP&+%SRU@0@W z#j@|eA?q;ME>0^5Ik0cuThXAH4XeK)tGtmer^C(e7%(Uoc%vpJx%aOJ9#)?zSuP1KySf2M+x#s|7q3>>oQ;{ z!=_xn5yUNv?+ltIh2yFwsqV(~m3-$3qQKHxUHvRohG)FkA!OBKq zWeMfWb+5BJA>Ee0YhoBWpF92iD9vNzHyCJh~NyFilk zk-H6vVj7%{zj&b^PPDao>*G^=pRsR@iReb56GJ32qz8&JD+-*j=Kvm_>YRZlMP@ZB zAYu;i7$&`vXVCLChpVz*p6@Y;62G5gCYJvn8gln&lkr;DK9>$M^?l85_nF;gd`eKw zy$B8H~V|x4-*e4kxoS4J|xx(%OS#a ztEu<`dX1WgcS;|bm4FV&IeR!i?DJa=sqIK|JLoHlC@gC>`p}XtM#eaZOR{XllL4>V z1X@v}o9xws?{Isxb_q}r^62IyO)Ww}`|3Ah<|CNdx+Udxg7te?I6AuU~cGUdl@ z{&cTym~~u!jjl5)8No`3Cc{pDe_X|#+Kn~i0YVlml!FHTnk^x(2J=; zRpSi%lxp6Ef6~e{B#qRSR?8U3L(wabw(0!R-T# zHTr56Xd5o2J{pHOK>#@LmSO9GUe2Lke!LG+s|t`R5QL6zH46ptKqh62TgoxqjEGwP z<&z{i%pXN+g6`Oj<;(Z}eG>Ckjx<`nLc5Y6e@>xnZ%k2FUILS{_1dc?j;WqRIb)oJ zio8ygPHqCm3ReiJkZ6pvzkOX3M<&)e$qa(k;8e?Kv1uHhOV{o}JBXTA6q5!yjLPXDP6yn3>>L9CbEg=cnZeS zCh^GL`=S4wVpGL+8MJ*C1>>h(+Z^{tWf6qNkm2zAOWwiAMMc@fZ%eN1GdChL{3sA z!eHRV2lM*Y;Dotx{7^v0Qelf=yOI0f_g$@KF5{pLn~L#!WS1X9li(BiMXvZ4qaH7d z@+b~CRXq4Rc;u27neJxWu76kU!x7h2xLebQG3xA=yE9I%1TOh1TRrPhN!`nbEW4KQ znW_e+FKKwAveV-4E{N6y;YWP+Y7;<+0+sEcA?<~w73rUtW(}Bp;hg%VB{ecw@NIR8 z6yI`pGK0m(^-!g`=Z~@}xM~lF%wmVe3@_3j0ZLPgVlF!17jfnMO7#zXeaS9gzSe(K z$}||{q&{uOmX2;5pEsQO?$?XNdd07nHC#>rDFGJp zQ^!v*=&F0=rCVn?A+EHyz$R-L4O;;af8>d@rTjobG=h?B@F3(%{Me3)wjMpayy&BQ z_DE8y4v&kK>|SKjFcifkvGmoCwN8y@jwSj|FsZP8)fp0O;fV(}`Wau`gU&-a(;@yf z%>`vzJA&ccQJiX%cT4rG*ZBII*-{#qTfYo(|O!GSu@%Cl^ zC&kl05B0=8h3jEY^ae1}M8P0lzM;!BJA;Hys9(sRKlz+ z?&1Irb}PZ5p{lj*cKZvkGt!%mepc9c(FX`>5=w&F1E82iZ11@fe|ju{E#qBBCQBLd z^R>P3is;K@^e_UD4;rG)t>qWYXeWxz z9Cz7??Y}gGxVs%HY(dG6*#r9|&IKlIsx~!x*Wm2MXU8TvLKtkkXSbia+ZkweAa%*& z1;zYDczPbR z!LxJZOWhH>wIbGSB4Tvj{eaTnGzPN) z-PS&R0`H5u`vw~lBP^01sC|Q<&RqCE!(bO954%^?mbAvk)=5*+m~(^y9Z}eSVb&{~79<=wxrS^Z#E_0V5ay literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_deny.png b/docs/extend/images/authz_deny.png new file mode 100644 index 0000000000000000000000000000000000000000..fa4a48584abb3db280b8226d18888cb0539de89d GIT binary patch literal 27099 zcmdSBbySsY*Dnf)5~8$(2q@hlA|NS9OM`Tygn}SQH&QM-Rir~Y1VN-Fln`m8B_*Xh z_Po{i+k1awpYz`y`w4>|{)(SmnBm_`vP$p%`-!N8Fx9lYaf;>N>#OjT;s1Lx zpD;Aij7t}k<^H~tPiU0qzn?H4W4u7EAvq-JEXGqk>Gi z80oWdry&CguGZU=9&|Q?1@>}#Eo4UJn`31bFJBtT>(^}eDrlWvy=|))!rnq0)XyxV z#^yfhxo7oI=l#R`) z?Q5OqQKswG{N}J(#MdY~ndle|sWyh7U|Z2E$(U+M4Nq8->25k+U*8!r>iF^gk--?F zax(is{sU)wh4z*t!Dr@;e&^*vC^rS(87+K!>m>T|*}`QjuL*%nbHTg1a*WjRn8Wo= zWjY>LLPL9)(!XCwHuYiDHb(!WgPf^pdgtp~3ew?JEU$PiaG%WQCb|V(zJ}?Y z%kfh!ynnvoSW?pGgpE&M@-1tQn&kaqg+nZS>R1}DeTg8;v8qzxgV~tF{&#|c2Qh#Feh32>)UpQAunIX$j7lRR*`Y)mte)jbLcG{N6trxoib-BB*ddA zZpyEYR5%GJ+i<eGv#=TI>R^!uR?-PY7+XV6U;q3cY^EtGN4WB4W4;CBQ9h_{XoH=eh ztFLn1%Jx`q#B%suyN5d{-It@x^#YfaD6`&m>*3y%!@;J7>)yEYd7IT=M#et;E$=1` z?yB0w1c`ZGG>Pf$zE3SMExNbhRo3>?6_-FIR)da!hNf6b>g@hoALX|;p#k}hj zsJlh>lG>gItLLgQ#3d$LzVKctZF3x6`SCvg@sBc{jcUim^v~N%104=GEp)4%rp-O% z3Ob=Oq3ww0P^aKEe~Kq&(N1#h8yS}J+;_q_HQ)ZB^_tZWqZ$XBb50+}Ix3ADF=EPR zHr9WB&VyS}+0UjB&(3N$Q8RLFLHaVa@H<;rmtK>6K5Eh@Ap6-XU;uLvjtx8giyf;%WV|V4p z3=s!GOO^eM^gyk<^YZnZmfvZ2PWC4ApDlb_SjK^*s)5XVXdyn+3}WE z(iIL}?Slv!u?dH6*imk9_XgT%ybJO)^B;a3{rm>&La&TVK+DSpWwDdYzU|D&a4Ny{ zy5miS1hFIk&2M*pmkSIO8R#v%3|fAa;+Z{Ippy}ZPh%U`Qlq^;J(WwvY@E7Nu@#U$ zQf?JfwD7AZqmWlZ?kL*(bSp(`e`z3}tp>X@Swt_6U8mQflC4#%P}j^VQdW>8+n}Lchp?sL2MlR508JOpRC}c z@J065#>wDF24P@VWQ#Mw>x2Ki54K z#z)_hdc1t&dqhX{>b+`}&aK+C_k41;OAsFCS~q>vboYm`W5_AS0hC-)k{knEQr67R z{=O_Xj~BY$eFza^WL*+!%xo(9al#*-E;20(R$dKc-^8SzQQko%gAt~(vC|rUn`EQZvXD#_9ETz zLZ$O>!)x=0JIh+b;>yu_lcA(+x;my|{p-z#uWmPvOuxGM$T3p4!g|q3pP1UlwsO+z z)5rcc*gV*S>$Jnu{uQiS36YVh- zG3~JM*4a8Cj_rXwlp#!;p8}@iKfcYtiT4x39;AE!sM|Jc`&ULpJG=@PoiysZO+;Fe zv)WO7{fY3$lMal}F_DC+n$};w3o&`@ENMP*wZ7ZH-5S%-Z~>n{Saz(%z=TnTaLUAj zj9Dhw^;fchebZ*3Z_uaVak#&WYpa_+^(}q==cuWS9p>6eS1*1}4WjWHW}x477lo6$ zxz#PEx5mBFK!XZNRg7W%T3CE^!Q*pZZP!@Gqua)QJt3bOroJ+J?KggD!#)iCBoV*y zeyFYLKwY}VQoQuZ=!g4PLz>&#^rX2-Sv3gAtz3y~NIKu%d5~({g~hmD_ChqQ9#DXK zwsHz*p`cn|j@o30Y$|~+ zDzAGaKRsWatr#+V77|W&ky)Pk9!lE;9oI&an?t88^YU`O5+_cz0exlN?sEr)k!1_0 zJnY`ujGr!k*c@ii`|xW*L2Z>!Im=#Q zs-gwDx1Wq;7A9;&Com{nxrXPt`|43L;kJ?cIF!A$XuKwqSj$5f=H<6Zb#r!N%C=_? zzMo%45=;g2&}-K+S*J*^_GAR{%Q;8zn13MUH6%^q5x;>ITlXnB_h5JPChuKwg< z*xJhCaGXEI$&Zs05iXTfpIBB+`RBhUI1*V~u2#GX_4iBWvx<(>I62%YeD=(gE6Dzt zkm)FTw(9j^swa;urjMchm3g0ec|uuNrVw&`w^1gox{FmW;_WQ}*FehzL%5Ouf-qmi zwL94|+Agw^8DvL=g=-mJ-tgj zLc@hV&u|f!qHo?(=J81h+oAE$o$qxP#qVoVevFTItLJ-vOzgbNJT?0UOifAH}tjQ$>^Zkpl^1>I*sRclu~zd%V`C)uJ8Iw zbh}p5=+?7ZQUe9o=-DVacZbk(g-!|Knk9k;8j$W~kJIhKGI? z^_O}R+k_5Zd|CSv%eNj6t1S2?@L&&$l|JZA(64b( zjhH9AY>^~jpB~GiF3o+&$Hkx6dV4jbXXeiDnv5gi!R7T{|L4th(sU8Ro52BP6T21n zy$|DA9;CMQFB$p9-+cHvXreAia2N+g&eK%Zzx?BU&UnY1$G0RwErG;UNhpt*dtMX@ zV$T9t#CEx+M^?Ytj45L-*UGl2?I#bdwB;~7H8h`K__XPH_b>}Pcw zMKJZ->L<;WH!kPet_o()u2s!kCbJQ1>lAS@KE#VpD?w{vK7WBIG1Kbhs>wLSjM{y93*9M|b*!-@mvc3Z3Q{hvqwMo5h68iSuK_GVR& zj~|#`!l#o$;cjGIE59l2n&=kLcyTFN-}ibf53QelOr_UyG8x|Ub&)B?`>$nxHob5X z+P~W)i7%3wqr)T5=XXnJ`*-Dpn}YF}KNmw$(P|~9kpr$~?YULG>=j>XI27k~ zziD4exo~(}i>=!&1`TJQ5$)cS!SFY?Y_j8Ge{17OU_>eS*r1|DpR)_%1fb9cZ<$a% zTN@LsEo?I1-tJR=qfRYHYmM`?nY`kDM`)CIFHXbv*1D9-ECcUUB>^7I^=Pv%3i2@G z1kj)t@dO&ZDUqU&T*55j3=>JyUhwy89tv9@P1g3$h!ig;h8i}am$*ashfiV$qZyrH zG_^GRWrICRKB52nqoiiyRpg0(m@DlFjtY;o`^EQOf1ivFL=+F3>>PX`AfQrr&{OtyoG#~qw_H8&yz8w;C*QI@f82?QG!({7#X!Mbj1EX z*$12cVCdF5+r<70CwWV3JPO`so2J>{q{>n*ht_6BamF<3_kvIBu<0xD59P(wPGXJ= zHIBbCrUhpIINDies`FxuIhOn^nx(dr6XqHB!n1+w`k^}=0ZQ~2$BFE&LcHnr&U&~x zcdpKJ;dPJF9VSKVL0?yQCIZy%MQO?j1-VH}#)i|Gv1)xY$A^+Mf7e7#4>KOf zPb)y2YWn}*2MOXsH-?P7;}(+LmD&@yxBx4~QwciAj#oQbOgD!(Y|Yxdl*X?2ft^Gy$L00sz)O8&4ZX)%fY{2+zuvHDkBtJ* zul3={X#Dh?X<)9*)XyNALC>F2GSoP_*M-M#a_D{NKcEux8ha$_wA5dEn6sFZ9PW>W zKWOE;G=!ZZ>gk5iSYasqyk@O<2_EAqES$*_HL_nHX%f7=Ml3geMX4g7_o&tBpbob!CPBl?IRJIVMv7!yhU{3EsY(v- zslB8@xz8uNuWh*fE(W?(WLV2ear2=gOLnc%=b6^9s}9r6I0uV4Dch4p2A^iTZcjRf ze11vJm>SJ|74D=Ke4qsmlV1enzQ*N7+`Ug(cS=k?J9?7~JWUn7Ov(3!<=y0((q5-ogh=+5 ztfJ!f9S&u5nF#8_K9&bXNkY%&QJ1F+jp!Pp85A@E@hCI^ZV#kM202pgJqpdK@%tIC z*3Ftz7Fc#BFh{|%W^yyG+VkkSE1+Zq)|ELfs^w}GzV29-2hKY?YF9tdmp9WEk?=VP z|I_}Pmq2agLAp8L=My_c(D17U`0wUYesO!tOR^S4_47HS-P)s!zkQiRJBURqo1xgg{nM_R4901+BFT2b5E$c zuT0k0?@USPQTzZ>>-g)tgmQ(&mq-aQ+2w)!oSfSG~dF3tTp@^oAbC+Lbt#!nhU4+_Z;DndM7@6X8@(w?mX}BjxNx7k2`HQUagss z|FG*dV`7z9{xB8?U#=Gse`pUgb-}1BF}bh!S5i*>VLk2+{M*bVS4mi@LqS>U9n!v9 zpqddlm0fmmu%CdsZEwO|dBSa(1f3!D!S~G}uS>E3YjFHMVSu0qircnH zW6qPFeprbH$5!s{`<5bYmV}!T-p7(OX^ynSjEdjeGLN?wk{cDe5%H7z(Dw=YAwg_A zkj!iEn{`c&c2{@shzwmiFXo$J;!^|N=qb7vO0qF^HIev`yrP^l;O1?dj zNAFwJ+gJRP-(~vgsV#W;i-^7DG7_y7eY&$W;1fDqNBtcHwYQfCDXtrQdOk3OhzSbS zTk~CgYpbJ`m1AAHZGFmOPd@(qbBIhnIsdl(c&9Ky8$H^78eh-52~!Qz$cBfp5JV!& zE0Z2uVb865GeZpBk3nixB=9_*e#zx1aHqq#{2uUD$LdRzd@8r?CdxNK`^tH4^wA+} zZm`Dn+3Id17LB&8z`_1ZM1?yPbB;ank%`b)RLtMnQQWk%JfsOHRui^ttWISc<+Ib{ zgB{(ZBcarzlSjRu4=LKQilmd)C_dN0wp$we_>8fN^5#R7QU>g~_%}y#pr|+{y8o_y zU{L4rV_ffGX$SN=~)Yvh^P2Eo<^ebEgHo zR4aRU818PT+9C8a^du)t6fHRU?@Vr*Hvd9XH{i~=9N!>xNxd9l_r{dpImr!-Gq>OZ zEcK*~<7cRSGM2~kV;R&YGh21uUC{}{Q12y|8-vAgT5`Ju7_$;b_kU<$?%~wT)96Wf z+D7sr3RHw8b<~aPiC6SWjD5Xq+BEVElD$t|MK(`1+Guu2(#E+4N2%#Nv@=c1Bcg;b zZ8|}f?R#P=9rOHQqfn40^Fs!dG5>{W(X!j8i55*L^ufE2x|2O$GUVX|%_Q=eZ8MY5 zsUDkP1`*!KcDW9BhN$3xLB?1_%=^@1yplW48q=u>6<1HR?D^eDDGe{7+MW9Gt;8@Qsq<<~&o!ReCg2UPVQO{sLClcnNEH=>o7EqvUA zeJiU^#AIr;ZQL{VxkaqVgn#icgw-~4JK(&~c&F0Hpu=3_xY#?sIUOo~wD-F*meSB` ze6))p?X0$c#OUz`Mlz_`5@#(hiL=jQ*sk(F{(drE7jD@bz>~I^(b~26%P>KE*3G;-&a;p@Zsv-B=+T7D7P+^fU1(Ir^m%&t)^rzDa#C`NL9!fd`O=#Yun8;l{5 zR9#rO#tjRciR0_DYt)yAO2@E7He@eVmZ+k%&5_gN0iviODV@+hv=1W+XsvkU+zo6u zx3XVUjzckazoei%iMdYOgne-^EwQ@Ul3+Ek$2fvoWZO|MOFpjr78l+nV;^JstWgY$ zfGWL|*PDq6*E=;-P9xCbNx47!^;|;5kDcc#;)W0Cv58`@tu)5`wf&!w zbzW|$-h0Va7YUossGjy!&n4=^VN;XlzJ+t+qYVLkA+GzB*U{>2GHveMz*nNX)Ta5E(u{SO zeNVY58>rd$dKjotT=mCgim!NEes?wO9(QUa_l>5BJkCs!q?k70bNr=JJ0IA+D}l1V zFw)7%E$F>kF;pJ&LzJ-nL&?IEZyTmB>^w{h2?NQV9{}OndDQT|du)hx^{43~m9#ooS9CdYF|yLV5$e*$uN`3H)IUxQZl?=>T7 z-|p+lD8`xMQf63_pQG^QI2V7FIZBfDu{26NkZaQT5}{pDVtb>s!5C^^i4M=4+rXbo z(__0w#*4Uhlq=i7xHca2Njo>it+bix`vk3+a;`D=KGJe8>3~r&@rdkF20k<4`)>hJ zSET!0Mqj9skX)w@F4=sM>bV||CwWydw`l}_nVnqKSQqEb@6|Wik^@(&xv5W0NM4c^ zfK{~=aj7Al^aOkIv{{l}@Mu%moUBKQ+gQTgY`%5$1g$0D{+@AA*M7-m=})_heKMaU zb$q!7pN^A0aQIEZo?&brnn1lZJ%f647@%U3aiJ@KVrGFf4}bzg#-FEfU|S;Zgm zjTNOr&@kl2g^BV+xQE|-c>42h?OCoyt{SUZD~c5WT%KXA8}u!?N{@2`o)J+;6SV#K z?!|^~iwv5e=+~0v%SPSE1EJCy8YAbx?W5qaJ!=yhvL#TkJrma9&D8`QW>tpI-Gcf_ zRzIu1|9LQl+c&mxW$$kFZhcjifM%VC>kE;jC8WBWuHd6L5fKr|E^T`8Y}cCy zzpu4Yi*aK;_~cUkC~26^5V0mk1bz<|zEPNzqDVSj*Y`>ET3x)_AmkI|z1B^FiIW+Q zs9rm*@6ji(c+E!1@I8pFu%Btw3SaAHWxA}?O~aN(pPVX8(v8{MAjoof)h!J@d*yG`u7_JWsR2g#+igB!%cNnB_pkl<*)ep&j+@MNcWK2E5GLZDv+}jb<#%R zJzT&eL>48AFnaLr1mL1-u(q1NH5(T31`jG5bs5zqP@!g^j^^89O~n?|CMn&I&a<^NR^rWCT-#XsPZr?+ z^@;!2AEeb0yE9O1B(@N{NbGHJsmNJJEm|4%%)XHF>Kyp}|D=Q+EM6*n)UHf3HwXO*Vu2b#|U7EB!jiC1oh za*F6kEjUJg_*dKy*}!Gr-;}ul{=P#94H+3zh4n!6`HNS^qMhjl`U`Z_`RyjCaPC=9 zAYqAy7Z9Ad7@ldMm?WUO`+KtZ3_vI&zVNz_Q|-ep%8fZYhZE=ug)SWfgWHmmXK z3VKFHd4#{xoryd`V$6KX zeyTcv`px&&#xX1gGv$aybX!KKv%NtId6#2~Pc6(`s9O<@7(^gHv%J0I91k71S3R23 z79g*aB5~*yD{*^jK?kWktwL7Mquqgxh0a$QadWUA=fG{Tmm?+v8M?kD?DC(g!-;8o zF~Kk3nIN~D8&E@o(EYJz-5>R_X3~KlMROVmz3qQ-kt|c(7lYTTm%i~`pB%T-%cvPB zwG5Jhc$=+P?S7?#O&7m5{;31}ZiHVuv|V$??hP_7-v^c>z+m;UJ4I}gr)#3ty`v{X zX6Icw$UwXR4dmVo*6$TWh#tzSh$;%I|G7Y8e=g7sc+2?BxsFc(ozV<0JDj*Vr9mml zq35Xx*{)@giZKF}V|x)^a1Ka1hS_yGP;9=d=OqrZt7Qh;54J(}<&*v>7x#VKjrfPq zuFRMEbKNxV7&o993_4G^$_PDQu~crKIT$R|%X(03h{z|bG7$m4WHIO?Ut=D-w^L7qh8nqr|NaGDv(c$dh2SJV?8HUeNQfd?$!SOS__pclUSw+4Jimi<-d{b8xx3 zHg5;soahZGC2Yr?3tar+b#f>Nno2LpQ3{6P3C$qmte8K{_}8TktQyKn4%i0 zTjmvCzIJxzEG-;0+R5vZfnPu2yeG3`m4_7>s_?^$c5u<*+Rz4Yu=LWzgHVQLp^*F*HI& z(R$uM1-J)lem1l1!+9ki`JT7f#ZU;<-oT$)SBm2;C})v>4(rZ~5hn+s8*ipHWH9}e z(P4=<2B+?FyBh!iDQb}Wzu3GF=CX$f(t#d|a5A(2e$C~Z_@j*+^e)$1%V3dC`Ct5# zi^G<|pyCNM%a?L}^^PQ?##cv#DEKK{IKl}ZcF{jcHu0yho(l;~>VJ)iKab{q_W|!V z#;h&;1>b_oEMX|0B(84=(yYXZ;CM2kzYf{#Rp7$$8hHpby!C6bS=Z}Z76286Rsi8T zQiA!j$5vGDyW&bE@Kk)k{&3sMJHs81ER;{=p_I&@ZGR6IAs67i*vs&W|H*6Nt$v_xy$24n zUyx`;92CKeUWV;@f`y8%cM&V*r<8b$4T?(Y1aoRs&5 z`ynsY50NzX%Iy1Z^5BCmEn}R=eFyc&Fa=!KNuiY(Wzk%vB!M)r0J7oQWq_QU2L@*b z?iKcT6JAG8oFE5LVb#Z|nJG)S11Fu8xENaZ63F(fJUpr(ENXWp^322wBaL`Vz-cM= z8{LBaY%6Tg3K^dS@b|Eq98gl zDICe^H8L08eSzyX3waptug3>l|LHQaAc$u{8lxBTRE@uSu{T?V9!=D^W&sf@1LbdK zZexGkc@igEt5h_Uuhh26VNPDefnC3<CpGp*zNbB(4BL+8pFasQY-s$t~g` z zo)>tnlX4l3g@!G*1K#)|c6OlL(dFf5bhDS z>y&Z{oNqx+?$dV@M*1|MmZfQD5H^-=VhNYC6Q{wDCaCV56~|1a(q`qOm$r* z>pbJtQ|F|lj1F`|g1+Kp=b31{IJE~0p#6c7XI4(u=S&y&3&0UlpvIqITKJiCG=V3? z2sf>Q`~^lm5;iN&m*BaZD_)|M$fC~ZX?t70k@*WvWA%0E$W}j6`w~%{IZLDZ?bvvA zYL(CuwTg?7-?$ zCws)-)IBG>J8Z@a$-G6_-{o3SJ2!maF#!5D?lYA-uW;YF@g2@!hKi}Zz5OPrkjsSh zv7I277w*V5FZX7>tbv#augCt=J#+n{()oS(Fdc||eIA!26470khKji$jPgN2K$%R* z)mZm=Px?Jxn?d$Mv=p}`77_UZ-HJS6=ukrVs4F~v#|H~3JG%7jX4jRhOwEJ&HZXix z`oI&@INd4sUKCTdcu86bRjM5h_x3MAt)))4XmLIQ*>j^AmhXZqg&N4G|I&@$9ONfW z!D!h}GSX~Bk5=BhS(6?au&^l5>K>}JtC`u-HW1Psq2mNZiUWc&2a))Cr>`7Tc1)xN zm4xt$K+d^A_cs8#$DBwC8rvBP_0NgB>?txB+}M6;#4iw=+3!B|wPC4OhF@^RS>`(u zlp!WewONyh4jp0z*bJ(cKjIgqB#Y_Qi*=o5XO`kp4?HdNW-Hz zm?<3>of{8t3{tQ_8H|#ImvtCn%~r5}5DM-eiu_86>er}*Uc_=$W(zFaxrqEp+ zNh%5Qr7?Qsd?w|?zC91y*A+Zz))aUER_tRnuoSHZ^5V9C^+=vI$qOXh9_eVc%L21L zsK^T%Z#aj#5~D8`jmP)p;8O71j=4Q}DQ*hkL^c#+T6#QVBd!j>@jcjDSf-J0wY#VG z2LNZm2hAq2nkVK)e>|1rfJc6n4$uXC4BQ`HbY3@092LR9crJCgnpJUVq%(zzzla`U zzN7kFAfZh~6zl*OSaqVIM@@va&C*J4&>yoY1^#lsl2Bn^4CDD6Hj@XM&!3*t$Ot?A zak?T`CML?E`8N->n3r&-D~~-9t?w{>gD0x=#~Pu54RhN$tjk{@fqJ8wnq3%fuL}_b z8H7h)WNk{IwQu-hQLht*PW(Yn0)R{!UhzNyLVyAZ02tJCtHl8Fp?okgkRXD<&DOtJ z{a{mMnPXK2lz*Q*1qJKeqgF8ck#498N&-*l)dbCdS&1b008^dh{(l1wp~y|MpDDBc z`($4PC?!#tA#VyehzAzIY^$#o_V>vM*73b*hWD3RGb10lC9ESaN<08l7e*BX}Y1ax5;18R%C?Phym9rczE4Amp*IkdUL*fU6>TFMcsl#8=pLQx@vS zjuC`<8KAQaI)OLTl2D(UDxwb+)M9p&t_1O+WXiZ!Ai8(P|u?x2vLDQu_Pvvf!uxt5rh>XJX!`5DsaU+aUSnOxy_Ig_jEJOAbHep7|OuRP-1HA z=a3F(dr(Aj{A=^7UCniH84^H-SAf=M0WcCtIDVW7zk|6bnMw~Vb0#)(P83wpHa^ul z@Zy$K7a^Ey{_A@h2E&!d8^|SkoM0t_$c3cLNkfQ>p?`ZTJw3!(3Dk5fp zEU{ch6Im`gW#;xDFSoWpQhO(sKAS(P?)D4IO;AY_MPX2&m`cb|)|lOV7UB9cBozLXq8}dbUvHw`b2L5q^uPo&mUopg-4#+aR?c4S=jTdatsG!v5m;3dBn@_ z(OGds$VCc!BMJiO!K&qJCNB3e&Va98-El6Wr>Fm=n1zPM_TyHIpAy!zb-69f_?l2HZwaYT0=v_TR3J^%F~5-6Pp`aqRI#R4jB&H z5Vi6plFjH$7fp$!T`Lg#Dn zJARpP!X8CJRaweODXE*Kk@S&5nCqo}vP7R8P?-+z3x3}7QP2H3K+WEDhP_`VPtQ!W z9)y-NbCTA~y@n|ex1WFm=%f9o{LH%D};%Mi^b3%nbpNiuFS_w-LRWA9DgTMW;DZb$x zzA!CIA^h(1hdU+4f4duPWN&*^Xw*sg-*&cwlbPv|X^MBW)_?)m0veu=xI9bSXfOy1 zf7)zQ92)zJ@hk}EY<{7O_%d8-mIdMZwTZ8F*`|v8cNzcmS9$_m3}otTwmoh+KJ(`X z_dQ#kIuAk884kx^|DI5R*&0FfTl;oaEVRSc9x*{`NRM6?Y5hAF7c7Ld@1e{Bg1_w> z!Dfs>)H8pVg!#4s0zca_)BedNDL88XxqhZ^#6h>x7v-e>pQ>;(e_6BBVh=a z%kyyC0Vcv^A!O*d|GVx;iTluFF)IPnOC=o|Yd>Av9|&yXH4}Eb+z1l$S?+slytZQk z63Gx|8KC(j&7|9Q$z%3y*A8=12(D$c2stfP zWjQWes4d&u$g}*u+mg81`3lrI@c^O*4d!f4WcXl+l*jsWY>ELyPO`bu8sy{It@e6{ zK&9`?5K;U(G6$KacmxCf6PFLEhW4+5?Sm*;FeAkT$0-gd>0-}~Y-6#bm78x-x5eB= zcb}Kt$kb$$@Ncq|A56D1NvYK>L>(f@f z*WAVqd06e%?G~hbFO&sL6-}l(p|;TSW0za?Nk3m3%ME*)D!|=N#bX+9`hjV1TPa19 z1LiN&^y|G0u1*^lazh*KO}}?hgc51VwA8T#B$J3zT`6I5?B_^$rByWXZi`^u)uq{m zmsOR*rFFr#)CzLa3qZfFAQ3er;jR$=uO4lnJiEYg-^c=Fr;H-H-J98=@5m$f_g%oI zt0#YFwXLCw+3HS0BrD`AeiT`}Ak~VZ7yv=&+Qo9vdpfV@!T6_+2>|dc>!G5;vPa!T zz!{4`tVxI1TkeVX!5o<;=oPC-Nw@{Xn{5K zavq3TtPpR@gGa5wNfx5+b=rJSbUGQ5QzkSQ!i-4mJD^|ZSofsA0ROz25(Yz*oaNKTc6O4#b=Mjhak7qjREIPRf;7w$1Nzr5=vpmp4F>=5A7l6fbUHK4zx>A%;Xhr4 z9o7HR;0Vt$Hb?!d`CWi^GCn%#-2VGyAB4c5P(ZDb-je?+eAXySm~}uF4vF9WWB;;wiu?D=;C}$F!dj zi6-;khoKw)I3iH`Q%%ij{t`u6^s8X+UByfO+giYJraSu6_n`F+L)RHGUXT6P`e42r z>+f1+|1ZsV)BWjS1t}LH$@~_Yrvl6ma{_aFxe{8s4Yz;=PzujI`kE~Vl+0Lvciu8m(^DtHCLyLc=CT(Rgt^INA8;>p6UnJ~uc03inj2r14XacGzp z{)$ALF7^OJ%7pNNo78_)>D|%yk0JmIsv}b~Na&Y@MNLY`d5s-HUf8_vA*1$6(hpFg-&`7 z-1B1CEM(_yg(b6wPW|}J4je@lPQ4ROB#)W&pX|wPeLPA&*>T80kDNfO$}?{CTLNuc z3uvAuoLDs=MXHltM}>zl*-N^)+Lsei4P(>tii#TGFls^HQ~^Gvg2cDMv=~|hm~r3Y zYjjI>9fSHC|4LzSesi}x=HebWdI7U>`gjVj?N|o{yk~B_#KfPo4|qtN`xr3%VFyhu z&k2vM`>WwXsa{C5bms_Kpa#^30Wd{kD|tB~5)ZSrYS+ZP^bMe#rWM?O&DUe)wso!> zMlIffyLau%5b$g@5EoP-;r9S^M{5xBm%weX2HwvKPtWr{J2mW>R+FG1t2>O0__ZBw zf-#XIs)`X9%Ckw?J1Tieo(GBdK$qeqeWr#&#{ZemqLCDhA2jKxC(1oiTAJqf$7`VH76Rls=!hfvcAX6VaTD5#&5~ z#~~^lZ^~zHfRTGvOpavfYa1@<4;AV?I^Pw?mT(9${2B*eF+*IW6o0TC&$At?GHKbH z4?0$%!6x|)cT_WuRnt)jR9p?vT`PMV<&aFB>2_-egI~l&JI^mN8g6X@zN2?VZde@B z6DVT9%_LR)^j5@OOIh$RX1CO=ZM*z>lWE}wfC9I}Z_Ty#Gl9nP1K8R8VGY3xAV42( z=H?_v!(ClssYj%F@TC>Gi)Fkwn;nY{I(NW-L0qNhT%NUdlcF760TonZO2z@0a&LCe zw3Ny$Iv&38xw^Dah=S7?ldGQ7jwDF?Z9sCFd+R(wsQA)tOCd$HHrX2HM$^B20_(@& z^w?G8lE+j7I*VaVmFDH7SZ39`M(%7c6)+7W#e<v=IB?C5RBx!ld$tX+}my+}I4AqcY(!B|tRvFmAC4TM4(_wAb}9Rn128(luO@ia(P z3BsaTl0laTvGLHo_J3mifN!B79rTJ$5(YJ%?vC>s3He|Hwj~U;@tQVc`5;Wt)6saSA&dhRyKVnUbK6@hg!7&=!>L^&4l2BoMb~Rh6OKwS z1_#}W)O8B{j_m!a)nsW|FnejEGX_ya5&vxUZ9eT-4VI;R-$e<>`*w6K@E2JwZ@j=( z+JrfKy-`v{Nn_ooFlyw+XEo;H$mru_SdcRDtXo8jR`I@Bz*vS1eg}sjQ#L*W)1}t1 z`xr^`VO`H7f_2^1Ls<;aYqU)aHo;#TaKAjh)z*fY?6E25{K)VQ0b&$Ry!mMI`NEUq z_n>y@)Vftn7%dUJ_(mUT3LAP2|shA~$n37lRY(jmrU-hqZyZHW;h0y!l$A#oi z46h)WGdV4d^$3HMO)K{2=O6{>sXLTUK{+}fU-x|3cwu<^y6<-)kc2D`cz5WDumiI! zh~7UeXTCvjj_samzwS?UQl+dwWxW!5TTE(?b%`ejzvdqsn1&JJi~i9gjGai5BgK6H z-<-p2onnzO1`E*bkZ2U+kL@3+1`nu*@@?THqVcb*3tm8lA)qH(4!Zd(GzyV&P}e0p z5^}*M8G{bLp~dKka}z@&++>iJ{W6Y}Afhd)60k*fJsv&qyjqy?uFW?|%+y0<1#0N}u5qNxNs)A!?a)q^VG`{yqm zyIuH?(L&d1Ol;xCa`&&=E*n*oh=cqAB3KvhK78``OJ_U)PHdwG{}JNFF-q^j0zQnq zsrU;U*Cjwb!J9WO2ae3^B%ver7#kQcR=;1JVg6 zil1YWLEmTH?+^Y%p`~mZQu6)d*$MSvge{Z*H+J2dqfC>q4CQ&r^Et5GIH(h{h&qkX z?8~RJn*X|kzu{nAMa(}3O7!gZ7GBg9HD&JtisxStv`Dd8!dQl9`rsN5&O3m_6@dIS=V-ZCqsHRzL zI4+OH1as28p*?y!`ZxCNe@;+)NtwW!ufv-C{K?P>f)8%Bf&(M>{$ZE!4gvIm)YL*SfxF@zLjk!YCmzwWO1hW|dJQqsGvBDJj0Br$J8<*sS<$3VYt z>nxhV+{gBx_5M8_O^M=)Y|?mD>4L3n{;%m4CVEI;rbR;;U8WIb2MpGGKm`svn<(lz9HAna&&)F5qF30fz$O#j zn|Fp7&sP`)<45yD?*^<5o$1!$qvT{zrjXuFtvdZErS5Qv4bjydLu%s~d{}vyk&2Va z>5G~uQq_i&Z7WPp}t)KRT@}nQ!04AMn-mWA4^tH}2zM-c`SSHWl(}(tC zkxzzL&AK$GN5U^E7M`H&K$gmRYclJ1TxmCHd%gpvRe3K+a&IGv%cg2Gs3jUOmq$8? z6E(D4TwiqQkxB~qH2HAHmicN9;CqP-L738=140r9p^qJ~YKH6Oi#99P_?wzy8m62@xQkEkty-vLwEWF)5mR(XhK(am6cY_tXT)% zs3~4<)(&-F>St-3=!D>mFzN)I6^zknTi*`f}h;%@*Y|}qi75q#Xme?XJx7reH zVUm*c$$%-6-!@GY?__QTYwimm(AseFkJ+T6&KO2&O*tIeL`1>2FnHd#27AJ6Ap3Vc zQq#gH9iu<00QwnDGs+Y*8jIS+r@iN_t}*hD^bmt-D$kStBA3dnEaiPK@~f%azKQB6 zTFJRz&S%ErfBK6c22U<|PrF7q&(M&L3G%D;m*SnQ(%;h@w(}#n|7gk0B)*ASC`+ph zDzm%+9i6}LDTBdNOfGSuolvYx>Tj>2^`-ZgO1fGiIqK@+`Pk=JvO|nPZEv8puDwBY z>LtO6C!wq@&32qu<{{H^$L)asQQddHQ~AgLm&|0oDcdm$;n<<z ztYh!&y;s>QTU!d*LK(^4gwONteO=$b;QPbp^MjwxIoEYN_v?PWo{xDLURdqz`QJXt zp2?&xJ{*U)b^d)@Ix&7J#r1sFx2rJh{1~rZ?@2SsV=W94gNDuZ_TzeHtdh>%myNm6 z!xbjH(9K19>kCiDJfxaM#j8YpLB9kz`%)PWksp8_6TqOI@6niS{)@_Eej{ zZcQnmwP{G(ZPBNw_4Y7(7V}TQ{|lc7uwsHN6#JxM^n_kJ4Ie;MDL@1xRP~c)ez)*8 zn*ny);j$D^wz!&b9ab^-mlDt>_YrsatJ|K)UVidtOY>N36E6ELh>mDL#5Rv;qYerv zVbJw~Gw-m+6(~Swz;mT$oVCOq4c4oT3oMm;C-7+gguhteACfG_j>Mz@+_G_Lq(vj8 zEvJAfR1pp!Nunjh9Perj_sc-PGv>hqqef>2#i3|wgf z*b)*XU95KC-3p~+;89oZbMh_lc*jfkglDJB`KlfFQIvK@ z%q+__?P5sCBq7J;=0d*!f;531Xl$vV;nh=XgYGry^6Nf-JS4sLtHNJ5BBUt|j5~s4 z$|U4xai5T~h|0FZ5~SGWgQ%{fmi5&cWejNNvAp`|rSba8wgvDLm03G=R+n*q1xN_| zi|GKz1bpTW(7dbq_fo+G2F*x`h}ul`(kv!`CZxXv<$pn5*gRJnzPBq#&r13YkZM>; zU)=cjNym$`EFY2|?QmwyuK-ueaeA`%IvoxAQ&Fp(qkjqu=_gPMUy=++dtUUg`u;I*$=w}!?d!YT5m+bu5?qx_K>ubudLPXe z=o5_$7Mh-^g7b9BMp^0YwJ=Uy{|NM&3Y@z;gXn^eb`}9GgFqj!+Y<=ruR8x85qb)` zPCjt=0WaA;A{-tVJdFbm$9K^&Ioo(EjZo=l3k8T-r-2`VOp>}vUbxc$sCfAZ&x!41 zLd3xG_gHP|6YVm4-~f#fw4E7`3`Lq!8_#@va6t2|HZ7wwoqnH%uv3=iQ)=I3o7vT= zRwEdfi!v_*;->AMhBF+k4`;f%pp&m&HN|}!tz*u|G`aVr4C=S82;%w2@`5v=9+SxW z(T`3nJ-B7>R0cWCTm zz@|tqjje&?>#al>>21Pco_UXGGYA-c+DUxgF4JR6{3Y->#bBuOf}IC3_ChqE8PSCh z6{!&vCwvOQ=KZg`yXynk$N}~rtf%|IG73v_h_)A{afE%tF!Q7k8tV;Rr9r5J& z+GW+~-mx{%KN$!L@`Pd=;9^WZW44Jqbgpm~+xy?k0nW2HM_?wB=P?KD%!7q=w-f~V z>%hr%JRDi9(<}uoZ`dCQf%}U`q1mjgURr6tO1Z~|mBWpAIC&fhn(@!LD=QxY+{jDs zez-%|@%Tcw_XCGNLhql^pDPRclp%d|QB~(3@;)KU-o<;|N;&&o#N-@`Y8whj_m;ty zYGgas2Uv(E2lJ~##J`@=^ZH&MKFWHXra+H&+mjl3^j!LuV3EHQ(Jq?sX4O2b{>$E7 zm^*_eReMx+50g*(XtHCdFW*Vl_7)ZAWm&C8!}@bEM_OH%mcc9a4xCmMW-L|HR7Q5F zrly3%{ zgkR!PJLS>`6*}vv*_(MFkHk{p>+OituLG^<>S3bvhP*t_5pE?uznem<;Qy}Df2*>7 zEqvheaF`d#a1ClCSRA~#ecwI4z6#eSau?pO?u*!UDNB<3@G;!21)llk*8KT7QiJzm zVSR$CkBlT*IS*bK3Ub`%T2_3e6tSb#-0eI*mv6l~(a1nFd>}<)ibd3oZv#>XTzwx~ zwi^n~52Ky>t_ui0}-pv>G!{?qOzlmV;%-cz6rV7hO3hZnQ z!xQsdy6QK`({ljQaLZLC*LCFh5O(}7hn(#CR`cql4$ZI`?=d81w2^$5e@1n&?52=g z=m2VrEVqM28oRhu2NbcbohcndJ8+nH>IvQvJJrjG{S*?onOvL&ED;|s<~czwBOQMu zsp)<5AJ2+i=wLo~txKU7beb@eze8BQKoGD79VP(-~6OPt;WJ zPr~?y%B-~$iy9`{m`$KQN;od(uxh&72XyX)*5wwRWWzWksU=Ex{LbEeLha(@U+wp4 zciVv=aX?pLw8%ufe@l1kk>J$4ND5{!qJGi_tGTLDbdV2u2=iyybgKW#K}xYyVf{_d z5oOt6;qERQ@Z4cPl4Y(0(#>6{Nio4VpN!d6+H$?}W{p4HY-+iCsg23-h z&@6uS`(La0KkVaH#NRUt%&xnvu{4RU2+z5`&{{obsF|bb= z3nc_bG&Rs~q$|8m~-etE+C*k176>Zd!=5li@NKu;qB$Yq<&QJgRTC!G9E$gtMU`~4t^|k47 zarNL5ephp;KV94x1q`G;cL!M+{ciCdQnek21{*l<{(Sldos*7f57_Pf5K(`0HjE^7 zwSzo$!yFu-h*IVr{1VTsY^I5~;>VPTfl|jC8Ug9~p7$g`7Y*&ihR6A$R{aO51fo5r zzMo_Bi0Ny^Q#Es8ZBGuNK8qjQ^psP_6H_H_58Z9pQz!z?*8%-r7w9@&$m7Ae75C(T zu;hq^W3N@dh;`Sr6$&6q6ho=80(vp*w{(C49s*2aQsS^n)wnj)DT5wFIB@g~GIT(W zCcBF05$vC6pSO+c{|(21qiaE~E50C_m3SR&p7)fH_U70Dkb!7n2us&g4vb3{Aq7hr z+PEQW&vxdh1gX&oi=C(wJQ~(3GceB&jRQ~3vh$qHC9l&>Fy!moxOn}u1xrvL`cHFC zGAg8vv+NzBQscM&wzLR7?AyB)107xCxVXMK;2GR9ZyUIrf*XM5v5$v@@JAj$K0(b^WDx*A6(#xvTZ!}PbI zsvzr@!Nb4-pAKb?&Z;19C-d5fB^wd@$LBAEd*X70W;3Ap(n({8<0wkxq{`6J70v!P z&5aT!#!zt7#imRLi{-?8OZlt*w}U(3fAx$04~|(l+CFKhNaOv;#QyHOY`yF6 zXiJjF@d7&ctP68U@Rt9Y@_{;eex^{1jD+PsDneOK5qu3~|Nngr&uIN23q>kpTm=}a z>Z-oPMJq@)YeAK~Nev>mJ%Iw_+Y%SWa^5NS(B1pk(}@@`{H*a{vyFxEP8E_@(nMLM z)A+GFDiW_<%>Pn6DF0|`KMUXx9hJ9261pWVa0!W4(z%K4Fst(ezxeXkH&C3BIX(4r zN#Pl=;Ud2g;3jSTiHXOkt_=T1T0+OE#vgUo4hrZiMgShdF8~nFcbJ9)_#JH$C%`ki zl_W7cJp@qzlAqdSg&Tg>M6(UM{s17n?fL`2 zQ`z?KuL+LZiESx_$iskC!!8WK7|qb(0NSp)7^PRth=7Z>isHy2!r7XCVJD{Ao zHA%3@c#VWD%Oo`U9Aj-qsU*p1IK{gx9p6P9h?rScNaDPFu+(3Stb?O0WX97 z0NXtQ(|aQf(vdIXokMeA@hgh@1I=C1$Kt8>Z3Z^UtMIm8(6O(LSpS~%%bvDi0Zr;w zbv54H68G*dg>JA%Ukf6}ARGV;U_#T;YO^jdrEamIp`mvab%4tr1kHO8$WmRny*glQ zuSHT^y727}=G+P&got|i;oe#$yr}^3Yah4fd&5AA7z&0U_gf3tOU!;uH2vbN9OBp9 zQfg9qPR?BN-maoG-{GAptf};2*jU{r5rH!oVlXDiPaTv1e*~AMj))pR!I=WlON}E> z)$#b_w_KSzfQR&9f$BpWeAPRI88LC+2T4X_FA(OyQUs)z8@^Hk9YIAJ$xvm!PZB7F z);klo-y`>ch*DS2kTUG)=_y7!*^gLdybKD1uvuilV=arQYwrPuxGIPmDe*&O-3oOWl!pL4ZQqngo|(9JU_>ruvq9PBv5}L2A8(1Y`oSQzeUmoK z3C_aAQ?Oi=3tN3CT9?ZN3JW2<4*_-5$j%~E%+bLeMyHiKG$cqsHqm(_pUykqfu$2n z^!4v#AbO@-`$*S2V!vns91Z-ZaOaIbrb%k^q8yCDcjKu#n!SkAf+lxxS`;B%jY%tZ z=$$n@4<&GfA09%kRf*F+3l) z*5)R(pK7!?9YMpH4j{Q92tt(GaNle?{Wi7JqOcdV!SVWCuZ3_Nd)Q~3jYaI;N3muR ze7Kj@nA<03d3rh{Bh(%lQ1V>M;}*v8$c7Q}Dg{0^#AIJst?&POG*MwYabr!KaV@Q| zVIhOH-}}?T#GaH(#)8Pyj|U}3M$K7@^K=Jk=t^pTvoJY(`*IIY&wmyc7A$;yGZpNi zWrLlP_%1Q+Ovm9NHYPT9Gxs=WChS{eW25AGmxm7@7GGUwlco+0Nlj%m7fvb47)@*_ zE-TBsYbU`}5RFps*{>d7>HO&&IyGe)laYa@ymE_p<;EoDF?$(~60NGMOB-~wZ@Ts< zFVnvN{d?^pR9?uAKgug5E{@}a)vK|o(NUF$&uDiZ_l?+$Gm{f7=_DNU81P*jn4T7B zF|Cxiaf8D+GAb(Qqg5`8hklt}sh>UH(9n>Don5%Y(*3)4v!)oAencj-k`60+W+o;U zJpVv7cKn3-sg&7{BO^UMn{-M_%7h69y)RCAK1k8WM`p-waR2dnZD3+TyS24t;q0u%#Kc6*G1)$|wG}t?MNCXgw>4_#)JTCV zJjSQar1T%lnSHi)?qQM7R(Yv3ipK$M1_2WsX;+v;r-JM1t~oh7D|lqfuH$viMrOop?yq{o*Jp5Kep4{aAv%Brru zx9=t6j6*&AzOcBcTvk~bUQlqw%}RDsQ}<6HWz@;+tcAZe-?f^~PKCp>PWWbGpPvlt z>+6>shKG|nucChaURi;CjfDL(E-H$2yNiUfkeOpr;&QfZMj!PB!bastQgKa9%}2}{ z%zR3Rjnn_?r|qwIiT8{b6H46q{d<;; ztu0$$FZnsH+i5;#W@ba?j%iqnJZ*=^W!GBWaI2NUj2h|5;qD*TJ9s1|Q>MyoOq?c5 zc0fm# zGZtGCpOC<_va*u*+(m(mj7+iUsfWj1Ch}lmQBlPee>uNRSJWpISBMwAGWQ#8h9yGD z;+&j-j3B0;o12@LHp#~150@4fv%L6~g-{|+Ed9!|$BNjL^mJWeVPTyc(VpE> z&CShA$8x?kN%8T~>1u5xK?+i2J<8T)jo$_bxh*X%OUUEz5e*pKuCbfum!4pcAs!`D zljrPAi?JqBA=#{VH^*^KQ@U;@DSqGM@QWZ%y+V5J-@#k&HFsUEr+1@=WH?xk>+9>o z-v4ATd?1_>a{1j468QZMLIQgHNLvi%J88@VEcV4USqa6*0U{zIB_!CUfu+^eNP}$D zmLC&d9q)}xHEX|rbL*8Ryph2~X{f7v+kX4nU+23(m(7a%Ha~CWY%D1$d0ttF4?fAo zM9$mJ#L3(|sev=DGg+qVnC)(ntHT>DDlJttG&0Jy7B-?UGr$o%->*6odqE~k&O%a& zBfWaQVu5ah32wSi{kCIsTiX>i|M1brdhNXgMzov0zwdOmH$*1Bj8l|jM^Wt)QMK>U z=EV-7fbai#n_)Jes=qyv$qFivHO`FbB)MrEGI@!-8@;UeK z-^Abla@yPbeBDE;bH$yD@5)%Hs$R)OGQ&hzP@H%qerHBX_QyUfE*32=FaI+-I=cNW z7>mVHV|IJB Date: Thu, 10 Dec 2015 12:53:27 +0200 Subject: [PATCH 027/534] Fixing documentation comments by @thaJeztah Signed-off-by: Dima Stopel --- docs/extend/authorization.md | 69 ++++++++++++++++++++---------------- 1 file changed, 39 insertions(+), 30 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index d16ba350a7..448497e6a8 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -13,7 +13,7 @@ weight = -1 # Create an authorization plugin Docker’s out-of-the-box authorization model is all or nothing. Any user with -permission to access to the Docker daemon can run any Docker client command. The +permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker's remote API to contact the daemon. If you require greater access control, you can create authorization plugins and add them to your Docker daemon configuration. Using an authorization plugin, a @@ -27,13 +27,13 @@ and methods information available to an authorization plugin developer. ## Basic principles -Docker's [plugin infrastructure](http://docs.docker.com/engine/extend/plugin_api) enables -extending Docker by dynamically loading, removing and communicating with +Docker's [plugin infrastructure](plugin_api.md) enables +extending Docker by loading, removing and communicating with third-party components using a generic API. The access authorization subsystem was built using this mechanism. Using this subsystem, you don't need to rebuild the Docker daemon to add an -authorization plugin. You can add a plugin to a installed Docker daemon. You do +authorization plugin. You can add a plugin to an installed Docker daemon. You do need to restart the Docker daemon to add a new plugin. An authorization plugin approves or denies requests to the Docker daemon based @@ -41,9 +41,9 @@ on both the current authentication context and the command context. The authentication context contains all user details and the authentication method. The command context contains all the relevant request data. -Authorization plugins must follow the rules described in [Docker Plugin API](http://docs.docker.com/engine/extend/plugin_api/). -Each plugin must reside within directories described under the [plugin discovery](http://docs.docker.com/engine/extend/plugin_api/#plugin-discovery) -section. +Authorization plugins must follow the rules described in [Docker Plugin API](plugin_api.md). +Each plugin must reside within directories described under the +[Plugin discovery](plugin_api.md#plugin-discovery) section. ## Basic architecture @@ -58,12 +58,11 @@ authentication plugin(s). The request contains the user (caller) and command context. The plugin is responsible for deciding whether to allow or deny the request. -Below please find tow sequence diagrams describing the *allow* and *deny* -authorization flows: +The sequence diagrams below depict an allow and deny authorization flow: -![Bad Address](images/authz_allow.png) +![Authorization Allow flow](images/authz_allow.png) -![Bad Address](images/authz_deny.png) +![Authorization Deny flow](images/authz_deny.png) Each request sent to the plugin includes the authenticated user, the HTTP headers, and the request/response body. Only the user name and the @@ -72,15 +71,15 @@ credentials or tokens are passed. Finally, not all request/response bodies are sent to the authorization plugin. Only those request/response bodies where the `Content-Type` is either `text/*` or `application/json` are sent. -For commands that can potentially the hijack the HTTP connection (`HTTP -Upgrade`), such as `exec`, the authorization plugin are only called for the +For commands that can potentially hijack the HTTP connection (`HTTP +Upgrade`), such as `exec`, the authorization plugin is only called for the initial HTTP requests. Once the plugin approves the command, authorization is not applied to the rest of the flow. Specifically, the streaming data is not passed to the authorization plugins. For commands that return chunked HTTP -response, such as `logs` and `events`, only the HTTP request are sent to the -authorization plugins as well. +response, such as `logs` and `events`, only the HTTP request is sent to the +authorization plugins. -During request/response processing, some authorization plugins flows might +During request/response processing, some authorization flows might need to do additional queries to the Docker daemon. To complete such flows, plugins can call the daemon API similar to a regular user. To enable these additional queries, the plugin must provide the means for an administrator to @@ -88,7 +87,8 @@ configure proper authentication and security policies. ## Docker client flows -To enable and configure the authorization plugin, the plugin developer must support the Docker client interactions detailed in this section. +To enable and configure the authorization plugin, the plugin developer must +support the Docker client interactions detailed in this section. ### Setting up Docker daemon @@ -96,17 +96,18 @@ Enable the authorization plugin with a dedicated command line flag in the `--authz-plugins=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. This value can be the plugin’s socket or a path to a specification file. -``` -$ docker daemon --authz-plugins=plugin1 --auth-plugins=plugin2,... +```bash +$ docker daemon --authz-plugins=plugin1 --authz-plugins=plugin2,... ``` -Docker's authorization subsystem supports multiple `--authz-plugin` parameters. +Docker's authorization subsystem supports multiple `--authz-plugins` parameters. ### Calling authorized command (allow) -Your plugin must support calling the `allow` command to authorize a command. This call does not impact Docker's command line. +Your plugin must support calling the `allow` command to authorize a command. +This call does not impact Docker's command line. -``` +```bash $ docker pull centos ... f1b10cd84249: Pull complete @@ -115,12 +116,17 @@ f1b10cd84249: Pull complete ### Calling unauthorized command (deny) -Your plugin must support calling the `deny` command to report on the outcome of a plugin interaction. This call returns messages to Docker's command line informing the user of the outcome of each call. +Your plugin must support calling the `deny` command to report on the outcome of +a plugin interaction. This call returns messages to Docker's command line informing +the user of the outcome of each call. -``` +```bash $ docker pull centos … -Authorization failed. Pull command for user 'john_doe' is denied by authorization plugin 'ACME' with message ‘[ACME] User 'john_doe' is not allowed to perform the pull command’ +Authorization failed. Pull command for user 'john_doe' is +denied by authorization plugin 'ACME' with message +‘[ACME] User 'john_doe' is not allowed to perform the pull +command’ ``` Where multiple authorization plugins are installed, multiple messages are expected. @@ -128,7 +134,8 @@ Where multiple authorization plugins are installed, multiple messages are expect ## API schema and implementation -Sample code for a typical plugin can be found here [ADD LINK]. In addition to Docker's standard plugin registration method, each plugin should implement the following two methods: +In addition to Docker's standard plugin registration method, each plugin +should implement the following two methods: * `/AuthzPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. @@ -138,7 +145,7 @@ Sample code for a typical plugin can be found here [ADD LINK]. In addition to Do **Request**: -``` +```json { "User": "The user identification" "UserAuthNMethod": "The authentication method used" @@ -152,7 +159,7 @@ Sample code for a typical plugin can be found here [ADD LINK]. In addition to Do **Response**: -``` +```json { "Allow" : "Determined whether the user is allowed or not" "Msg": "The authorization message" @@ -162,7 +169,8 @@ Sample code for a typical plugin can be found here [ADD LINK]. In addition to Do #### /AuthzPlugin.AuthZRes **Request**: -``` + +```json { "User": "The user identification" "UserAuthNMethod": "The authentication method used" @@ -178,7 +186,8 @@ Sample code for a typical plugin can be found here [ADD LINK]. In addition to Do ``` **Response**: -``` + +```json { "Allow" : "Determined whether the user is allowed or not" "Msg": "The authorization message" From 24e5fce3d9dd7b0522514423a56d5e40c720c7e1 Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Fri, 11 Dec 2015 20:59:15 +0200 Subject: [PATCH 028/534] Change authz plugin argument name Signed-off-by: Liron Levin --- docs/extend/authorization.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 448497e6a8..a48a8fbdf7 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -93,14 +93,14 @@ support the Docker client interactions detailed in this section. ### Setting up Docker daemon Enable the authorization plugin with a dedicated command line flag in the -`--authz-plugins=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. +`--authz-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. This value can be the plugin’s socket or a path to a specification file. ```bash -$ docker daemon --authz-plugins=plugin1 --authz-plugins=plugin2,... +$ docker daemon --authz-plugin=plugin1 --authz-plugin=plugin2,... ``` -Docker's authorization subsystem supports multiple `--authz-plugins` parameters. +Docker's authorization subsystem supports multiple `--authz-plugin` parameters. ### Calling authorized command (allow) From 51e58598c6350b08ea815581f2cf5ec5f854237d Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Wed, 16 Dec 2015 09:52:26 +0100 Subject: [PATCH 029/534] docs: extend: plugins_volume.md: Err default to empty string Signed-off-by: Antonio Murdaca --- docs/extend/plugins_volume.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 98418b735b..a2507f9113 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -65,7 +65,7 @@ Opts is a map of driver specific options passed through from the user request. **Response**: ``` { - "Err": null + "Err": "" } ``` @@ -85,7 +85,7 @@ Delete the specified volume from disk. This request is issued when a user invoke **Response**: ``` { - "Err": null + "Err": "" } ``` @@ -109,7 +109,7 @@ at the first mount request and deprovision at the last corresponding unmount req ``` { "Mountpoint": "/path/to/directory/on/host", - "Err": null + "Err": "" } ``` @@ -131,7 +131,7 @@ Docker needs reminding of the path to the volume on the host. ``` { "Mountpoint": "/path/to/directory/on/host", - "Err": null + "Err": "" } ``` @@ -154,7 +154,7 @@ this point. **Response**: ``` { - "Err": null + "Err": "" } ``` From 6ca77f8dd653397ca2682330d7c84ff5f550d1cf Mon Sep 17 00:00:00 2001 From: Ray Tsang Date: Tue, 15 Dec 2015 11:04:53 -0500 Subject: [PATCH 030/534] Updated REX-Ray plugin platform support. REX-Ray added Google Compute Engine support: https://github.com/emccode/rexray/issues/113 Signed-off-by: Ray Tsang --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 6c17b9072d..f2cf1b7ba4 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -68,7 +68,7 @@ The following plugins exist: * The [REX-Ray plugin](https://github.com/emccode/rexraycli) is a volume plugin which is written in Go and provides advanced storage functionality for many - platforms including EC2, OpenStack, XtremIO, and ScaleIO. + platforms including EC2, Google Compute Engine, OpenStack, XtremIO, and ScaleIO. * The [Contiv Volume Plugin](https://github.com/contiv/volplugin) is an open source volume plugin that provides multi-tenant, persistent, distributed storage From a15a86dbfb8a050dcca71fe4203296a025f62400 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Tue, 15 Dec 2015 09:49:18 +0100 Subject: [PATCH 031/534] pkg: authorization: add Err to tweak response status code Signed-off-by: Antonio Murdaca --- docs/extend/authorization.md | 57 +++++++++++++++++++----------------- 1 file changed, 30 insertions(+), 27 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index a48a8fbdf7..5330e9d245 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -146,13 +146,13 @@ should implement the following two methods: **Request**: ```json -{ - "User": "The user identification" - "UserAuthNMethod": "The authentication method used" - "RequestMethod": "The HTTP method" - "RequestUri": "The HTTP request URI" - "RequestBody": "Byte array containing the raw HTTP request body" - "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string " +{ + "User": "The user identification", + "UserAuthNMethod": "The authentication method used", + "RequestMethod": "The HTTP method", + "RequestUri": "The HTTP request URI", + "RequestBody": "Byte array containing the raw HTTP request body", + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string ", "RequestStatusCode": "Request status code" } ``` @@ -160,27 +160,27 @@ should implement the following two methods: **Response**: ```json -{ - "Allow" : "Determined whether the user is allowed or not" - "Msg": "The authorization message" +{ + "Allow": "Determined whether the user is allowed or not", + "Msg": "The authorization message", + "Err": "The error message if things go wrong" } ``` - #### /AuthzPlugin.AuthZRes **Request**: ```json { - "User": "The user identification" - "UserAuthNMethod": "The authentication method used" - "RequestMethod": "The HTTP method" - "RequestUri": "The HTTP request URI" - "RequestBody": "Byte array containing the raw HTTP request body" - "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string" - "RequestStatusCode": "Request status code" - "ResponseBody": "Byte array containing the raw HTTP response body" - "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string" + "User": "The user identification", + "UserAuthNMethod": "The authentication method used", + "RequestMethod": "The HTTP method", + "RequestUri": "The HTTP request URI", + "RequestBody": "Byte array containing the raw HTTP request body", + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string", + "RequestStatusCode": "Request status code", + "ResponseBody": "Byte array containing the raw HTTP response body", + "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string", "ResponseStatusCode":"Response status code" } ``` @@ -189,11 +189,12 @@ should implement the following two methods: ```json { - "Allow" : "Determined whether the user is allowed or not" - "Msg": "The authorization message" - "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or nil if no changes required)" - "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or nil if no changes required)" - "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" + "Allow": "Determined whether the user is allowed or not", + "Msg": "The authorization message", + "Err": "The error message if things go wrong", + "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or nil if no changes required)", + "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or nil if no changes required)", + "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" } ``` @@ -222,7 +223,8 @@ Request body | []byte | Raw request body Name | Type | Description --------|--------|---------------------------------------------------------------------------------- Allow | bool | Boolean value indicating whether the request is allowed or denied -Message | string | Authorization message (will be returned to the client in case the access is denied) +Msg | string | Authorization message (will be returned to the client in case the access is denied) +Err | string | Error message (will be returned to the client in case the plugin encounter an error) ### Response authorization @@ -249,4 +251,5 @@ Response body | []byte | Raw docker daemon response body Name | Type | Description --------|--------|---------------------------------------------------------------------------------- Allow | bool | Boolean value indicating whether the response is allowed or denied -Message | string | Authorization message (will be returned to the client in case the access is denied) \ No newline at end of file +Msg | string | Authorization message (will be returned to the client in case the access is denied) +Err | string | Error message (will be returned to the client in case the plugin encounter an error) From 751f72e6feb797e89d4fb843c93c3bad40b62251 Mon Sep 17 00:00:00 2001 From: Wen Cheng Ma Date: Thu, 17 Dec 2015 15:40:10 +0800 Subject: [PATCH 032/534] Fix typo error and update index Signed-off-by: Wen Cheng Ma --- docs/extend/index.md | 1 + docs/extend/plugins_network.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 61afb720ca..b533ea1936 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -18,4 +18,5 @@ Currently, you can extend Docker by adding a plugin. This section contains the f * [Understand Docker plugins](plugins.md) * [Write a volume plugin](plugins_volume.md) * [Write a network plugin](plugins_network.md) +* [Write an authorization plugin](authorization.md) * [Docker plugin API](plugin_api.md) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index c33a755056..01e8089f7e 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -51,7 +51,7 @@ documented as part of libnetwork: # Related Information -To interact with the Docker maintainers and other interested users, se the IRC channel `#docker-network`. +To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. - [Docker networks feature overview](../userguide/networking/index.md) - The [LibNetwork](https://github.com/docker/libnetwork) project From e664987429a66a801c68c6a6b92d95288e6ac8f1 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Fri, 18 Dec 2015 12:34:19 +0100 Subject: [PATCH 033/534] authZ: more fixes - fix naming and formatting - provide more context when erroring auth - do not capitalize errors - fix wrong documentation - remove ugly remoteError{} Signed-off-by: Antonio Murdaca --- docs/extend/authorization.md | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 5330e9d245..23a1787ca3 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -104,9 +104,6 @@ Docker's authorization subsystem supports multiple `--authz-plugin` parameters. ### Calling authorized command (allow) -Your plugin must support calling the `allow` command to authorize a command. -This call does not impact Docker's command line. - ```bash $ docker pull centos ... @@ -116,22 +113,20 @@ f1b10cd84249: Pull complete ### Calling unauthorized command (deny) -Your plugin must support calling the `deny` command to report on the outcome of -a plugin interaction. This call returns messages to Docker's command line informing -the user of the outcome of each call. +```bash +$ docker pull centos +... +docker: Error response from daemon: authorization denied by plugin PLUGIN_NAME: volumes are not allowed. +``` + +### Error from plugins ```bash $ docker pull centos -… -Authorization failed. Pull command for user 'john_doe' is -denied by authorization plugin 'ACME' with message -‘[ACME] User 'john_doe' is not allowed to perform the pull -command’ +... +docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZPlugin.AuthZReq: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. ``` -Where multiple authorization plugins are installed, multiple messages are expected. - - ## API schema and implementation In addition to Docker's standard plugin registration method, each plugin From 1f286794cf5e9ff63c25c8af7b4bba4dfb30e316 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Fri, 18 Dec 2015 11:28:28 +0100 Subject: [PATCH 034/534] docs: extend: authorization.md: add a note about confidential stuff in Err Signed-off-by: Antonio Murdaca --- docs/extend/authorization.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 23a1787ca3..80fa6c94b1 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -219,7 +219,7 @@ Name | Type | Description --------|--------|---------------------------------------------------------------------------------- Allow | bool | Boolean value indicating whether the request is allowed or denied Msg | string | Authorization message (will be returned to the client in case the access is denied) -Err | string | Error message (will be returned to the client in case the plugin encounter an error) +Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) ### Response authorization @@ -247,4 +247,4 @@ Name | Type | Description --------|--------|---------------------------------------------------------------------------------- Allow | bool | Boolean value indicating whether the response is allowed or denied Msg | string | Authorization message (will be returned to the client in case the access is denied) -Err | string | Error message (will be returned to the client in case the plugin encounter an error) +Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) From e59096de6965d8cd4346fd3ba6689f7c6fa09436 Mon Sep 17 00:00:00 2001 From: Ilya Dmitrichenko Date: Tue, 22 Dec 2015 16:45:46 +0000 Subject: [PATCH 035/534] Update link for Weave Net plugin docs Signed-off-by: Ilya Dmitrichenko --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index f2cf1b7ba4..feaecd96a2 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -80,7 +80,7 @@ multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. -* The [Weave Network Plugin](https://github.com/weaveworks/docker-plugin) creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. +* The [Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. ## Troubleshooting a plugin From 4e577793b9c99c7adb7b56852863cef5ce72bb29 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 4 Jan 2016 15:42:03 +0100 Subject: [PATCH 036/534] docs: add note to volume-plugins The /var/lib/docker directory is reserved for docker, and should not be touched by plugins. Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins_volume.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index a2507f9113..7a1471e596 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -15,7 +15,7 @@ storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Docker host. See the [plugin documentation](plugins.md) for more information. -# Command-line changes +## Command-line changes A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: @@ -32,13 +32,13 @@ server to another. By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. -# Create a VolumeDriver +## Create a VolumeDriver The container creation endpoint (`/containers/create`) accepts a `VolumeDriver` field of type `string` allowing to specify the name of the driver. It's default value of `"local"` (the default driver for local volumes). -# Volume plugin protocol +## Volume plugin protocol If a plugin registers itself as a `VolumeDriver` when activated, then it is expected to provide writeable paths on the host filesystem for the Docker @@ -47,6 +47,10 @@ daemon to provide to containers to consume. The Docker daemon handles bind-mounting the provided paths into user containers. +> **Note**: Volume plugins should *not* write data to the `/var/lib/docker/` +> directory, including `/var/lib/docker/volumes`. The `/var/lib/docker/` +> directory is reserved for Docker. + ### /VolumeDriver.Create **Request**: From d54175dc963bfa7eec86b9d519d586e34565674c Mon Sep 17 00:00:00 2001 From: Clinton Kitson Date: Mon, 4 Jan 2016 11:28:25 -0800 Subject: [PATCH 037/534] Updated docs for consolidated list of REX-Ray providers This commit includes the correct project link and also includes a consolidated list of drivers that are supported. Signed-off-by: Clinton Kitson --- docs/extend/plugins.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index feaecd96a2..ea483241b4 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -66,10 +66,10 @@ The following plugins exist: is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) repositories at specific commits as volumes within Docker containers. -* The [REX-Ray plugin](https://github.com/emccode/rexraycli) is a volume plugin +* The [REX-Ray plugin](https://github.com/emccode/rexray) is a volume plugin which is written in Go and provides advanced storage functionality for many - platforms including EC2, Google Compute Engine, OpenStack, XtremIO, and ScaleIO. - + platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. + * The [Contiv Volume Plugin](https://github.com/contiv/volplugin) is an open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. From 8ec9e7d272cb9b5755648b3c1c6cd4c59cab248c Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Tue, 5 Jan 2016 21:07:34 -0500 Subject: [PATCH 038/534] Add docs for volume plugin ls/get Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 71 ++++++++++++++++++++++++++++++----- 1 file changed, 61 insertions(+), 10 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 7a1471e596..1814708275 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -54,7 +54,7 @@ containers. ### /VolumeDriver.Create **Request**: -``` +```json { "Name": "volume_name", "Opts": {} @@ -67,7 +67,7 @@ volume on the filesystem yet (until Mount is called). Opts is a map of driver specific options passed through from the user request. **Response**: -``` +```json { "Err": "" } @@ -78,7 +78,7 @@ Respond with a string error if an error occurred. ### /VolumeDriver.Remove **Request**: -``` +```json { "Name": "volume_name" } @@ -87,7 +87,7 @@ Respond with a string error if an error occurred. Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. **Response**: -``` +```json { "Err": "" } @@ -98,7 +98,7 @@ Respond with a string error if an error occurred. ### /VolumeDriver.Mount **Request**: -``` +```json { "Name": "volume_name" } @@ -110,7 +110,7 @@ more than once, the plugin may need to keep track of each new mount request and at the first mount request and deprovision at the last corresponding unmount request. **Response**: -``` +```json { "Mountpoint": "/path/to/directory/on/host", "Err": "" @@ -123,7 +123,7 @@ available, and/or a string error if an error occurred. ### /VolumeDriver.Path **Request**: -``` +```json { "Name": "volume_name" } @@ -132,7 +132,7 @@ available, and/or a string error if an error occurred. Docker needs reminding of the path to the volume on the host. **Response**: -``` +```json { "Mountpoint": "/path/to/directory/on/host", "Err": "" @@ -145,7 +145,7 @@ available, and/or a string error if an error occurred. ### /VolumeDriver.Unmount **Request**: -``` +```json { "Name": "volume_name" } @@ -156,7 +156,7 @@ per container stop. Plugin may deduce that it is safe to deprovision it at this point. **Response**: -``` +```json { "Err": "" } @@ -164,3 +164,54 @@ this point. Respond with a string error if an error occurred. + +### /VolumeDriver.Get + +**Request**: +```json +{ + "Name": "volume_name" +} +``` + +Get the volume info. + + +**Response**: +```json +{ + "Volume": { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host", + }, + "Err": "" +} +``` + +Respond with a string error if an error occurred. + + +### /VolumeDriver.List + +**Request**: +```json +{} +``` + +Get the list of volumes registered with the plugin. + +**Response**: +```json +{ + "Volumes": [ + { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host" + } + ], + "Err": "" +} +``` + +Respond with a string error if an error occurred. + From f93f1adfa3d708d39a241c2271064a3973b35519 Mon Sep 17 00:00:00 2001 From: Tibor Vass Date: Tue, 12 Jan 2016 19:38:18 -0500 Subject: [PATCH 039/534] Rename authz to authorization for greater clarity Signed-off-by: Tibor Vass --- docs/extend/authorization.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 80fa6c94b1..48790833cf 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -12,7 +12,7 @@ weight = -1 # Create an authorization plugin -Docker’s out-of-the-box authorization model is all or nothing. Any user with +Docker's out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker's remote API to contact the daemon. If you require greater access control, you can create authorization plugins and add @@ -45,6 +45,9 @@ Authorization plugins must follow the rules described in [Docker Plugin API](plu Each plugin must reside within directories described under the [Plugin discovery](plugin_api.md#plugin-discovery) section. +**Note**: the abbreviations `AuthZ` and `AuthN` mean authorization and authentication +respectively. + ## Basic architecture You are responsible for registering your plugin as part of the Docker daemon @@ -93,14 +96,14 @@ support the Docker client interactions detailed in this section. ### Setting up Docker daemon Enable the authorization plugin with a dedicated command line flag in the -`--authz-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. -This value can be the plugin’s socket or a path to a specification file. +`--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` +value. This value can be the plugin’s socket or a path to a specification file. ```bash -$ docker daemon --authz-plugin=plugin1 --authz-plugin=plugin2,... +$ docker daemon --authorization-plugin=plugin1 --authorization-plugin=plugin2,... ``` -Docker's authorization subsystem supports multiple `--authz-plugin` parameters. +Docker's authorization subsystem supports multiple `--authorization-plugin` parameters. ### Calling authorized command (allow) From fa565c6b5d66ddfd81938a88c714f5160c52ded8 Mon Sep 17 00:00:00 2001 From: Joe Doliner Date: Mon, 11 Jan 2016 05:34:43 -0800 Subject: [PATCH 040/534] Remove Pachyderm storage driver. Signed-off-by: Joe Doliner --- docs/extend/plugins.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index ea483241b4..09362145a5 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -62,10 +62,6 @@ The following plugins exist: * The [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) is a cluster aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. -* The [Pachyderm PFS plugin](https://github.com/pachyderm/pachyderm/tree/master/src/cmd/pfs-volume-driver) - is a volume plugin written in Go that provides functionality to mount Pachyderm File System (PFS) - repositories at specific commits as volumes within Docker containers. - * The [REX-Ray plugin](https://github.com/emccode/rexray) is a volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. From 4b78ecacfcaff7f0754eb676c12ad675eb4c3a50 Mon Sep 17 00:00:00 2001 From: Mary Anthony Date: Sat, 23 Jan 2016 20:36:40 -0800 Subject: [PATCH 041/534] Creating Engine specific menu Fixing the links Updating with Seb's comments Adding weight Fixing the engine aliases Updating after Arun pushed Removing empty file Signed-off-by: Mary Anthony --- docs/deprecated.md | 150 +++++++++++++++++++++++++++++++++ docs/extend/authorization.md | 12 +-- docs/extend/index.md | 12 +-- docs/extend/plugin_api.md | 2 +- docs/extend/plugins.md | 8 +- docs/extend/plugins_network.md | 16 ++-- docs/extend/plugins_volume.md | 17 ++-- 7 files changed, 183 insertions(+), 34 deletions(-) create mode 100644 docs/deprecated.md diff --git a/docs/deprecated.md b/docs/deprecated.md new file mode 100644 index 0000000000..4ed127f953 --- /dev/null +++ b/docs/deprecated.md @@ -0,0 +1,150 @@ + + +# Deprecated Engine Features + +The following list of features are deprecated in Engine. + +### Ambiguous event fields in API +**Deprecated In Release: v1.10** + +The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. +See the events API documentation for the new format. + +### `-f` flag on `docker tag` +**Deprecated In Release: v1.10** + +**Target For Removal In Release: v1.12** + +To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. + +### HostConfig at API container start +**Deprecated In Release: v1.10** + +**Target For Removal In Release: v1.12** + +Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of +defining it at container creation (`POST /containers/create`). + +### Docker ps 'before' and 'since' options + +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +**Target For Removal In Release: v1.12** + +The `docker ps --before` and `docker ps --since` options are deprecated. +Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. + +### Command line short variant options +**Deprecated In Release: v1.9** + +**Target For Removal In Release: v1.11** + +The following short variant options are deprecated in favor of their long +variants: + + docker run -c (--cpu-shares) + docker build -c (--cpu-shares) + docker create -c (--cpu-shares) + +### Driver Specific Log Tags +**Deprecated In Release: v1.9** + +**Target For Removal In Release: v1.11** + +Log tags are now generated in a standard way across different logging drivers. +Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and +`fluentd-tag` have been deprecated in favor of the generic `tag` option. + + docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" + +### LXC built-in exec driver +**Deprecated In Release: v1.8** + +**Target For Removal In Release: v1.10** + +The built-in LXC execution driver is deprecated for an external implementation. +The lxc-conf flag and API fields will also be removed. + +### Old Command Line Options +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** + +**Target For Removal In Release: v1.10** + +The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: + + docker daemon -H ... + +The following single-dash (`-opt`) variant of certain command line options +are deprecated and replaced with double-dash options (`--opt`): + + docker attach -nostdin + docker attach -sig-proxy + docker build -no-cache + docker build -rm + docker commit -author + docker commit -run + docker events -since + docker history -notrunc + docker images -notrunc + docker inspect -format + docker ps -beforeId + docker ps -notrunc + docker ps -sinceId + docker rm -link + docker run -cidfile + docker run -dns + docker run -entrypoint + docker run -expose + docker run -link + docker run -lxc-conf + docker run -n + docker run -privileged + docker run -volumes-from + docker search -notrunc + docker search -stars + docker search -t + docker search -trusted + docker tag -force + +The following double-dash options are deprecated and have no replacement: + + docker run --cpuset + docker run --networking + docker ps --since-id + docker ps --before-id + docker search --trusted + +### Auto-creating missing host paths for bind mounts +**Deprecated in Release: v1.9** + +**Target for Removal in Release: 1.11** + +When creating a container with a bind-mounted volume-- `docker run -v /host/path:/container/path` -- +docker was automatically creating the `/host/path` if it didn't already exist. + +This auto-creation of the host path is deprecated and docker will error out if +the path does not exist. + +### Interacting with V1 registries + +Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. + +### Docker Content Trust ENV passphrase variables name change +**Deprecated In Release: v1.9** + +**Target For Removal In Release: v1.10** + +As of 1.9, Docker Content Trust Offline key will be renamed to Root key and the Tagging key will be renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables + +- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE will now be named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE +- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE will now be named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE diff --git a/docs/extend/authorization.md b/docs/extend/authorization.md index 48790833cf..3512c56ccd 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/authorization.md @@ -4,7 +4,7 @@ title = "Access authorization plugin" description = "How to create authorization plugins to manage access control to your Docker daemon." keywords = ["security, authorization, authentication, docker, documentation, plugin, extend"] [menu.main] -parent = "mn_extend" +parent = "engine_extend" weight = -1 +++ @@ -41,8 +41,8 @@ on both the current authentication context and the command context. The authentication context contains all user details and the authentication method. The command context contains all the relevant request data. -Authorization plugins must follow the rules described in [Docker Plugin API](plugin_api.md). -Each plugin must reside within directories described under the +Authorization plugins must follow the rules described in [Docker Plugin API](plugin_api.md). +Each plugin must reside within directories described under the [Plugin discovery](plugin_api.md#plugin-discovery) section. **Note**: the abbreviations `AuthZ` and `AuthN` mean authorization and authentication @@ -90,7 +90,7 @@ configure proper authentication and security policies. ## Docker client flows -To enable and configure the authorization plugin, the plugin developer must +To enable and configure the authorization plugin, the plugin developer must support the Docker client interactions detailed in this section. ### Setting up Docker daemon @@ -132,7 +132,7 @@ docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZP ## API schema and implementation -In addition to Docker's standard plugin registration method, each plugin +In addition to Docker's standard plugin registration method, each plugin should implement the following two methods: * `/AuthzPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. @@ -198,7 +198,7 @@ should implement the following two methods: The modified response enables the authorization plugin to manipulate the content of the HTTP response. In case of more than one plugin, each subsequent plugin -receives a response (optionally) modified by a previous plugin. +receives a response (optionally) modified by a previous plugin. ### Request authorization diff --git a/docs/extend/index.md b/docs/extend/index.md index b533ea1936..f491926e9a 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,19 +1,19 @@ -## Extending Docker +## Extending Docker Engine -Currently, you can extend Docker by adding a plugin. This section contains the following topics: +Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: * [Understand Docker plugins](plugins.md) * [Write a volume plugin](plugins_volume.md) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 8e2862f6cb..c6793a3dda 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -4,7 +4,7 @@ title = "Plugins API" description = "How to write Docker plugins extensions " keywords = ["API, Usage, plugins, documentation, developer"] [menu.main] -parent = "mn_extend" +parent = "engine_extend" weight=1 +++ diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 09362145a5..5a91010b1c 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -1,15 +1,15 @@ -# Understand Docker plugins +# Understand Engine plugins You can extend the capabilities of the Docker Engine by loading third-party plugins. This page explains the types of plugins and provides links to several @@ -65,7 +65,7 @@ The following plugins exist: * The [REX-Ray plugin](https://github.com/emccode/rexray) is a volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. - + * The [Contiv Volume Plugin](https://github.com/contiv/volplugin) is an open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 01e8089f7e..ac07273219 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -4,19 +4,19 @@ title = "Docker network driver plugins" description = "Network driver plugins." keywords = ["Examples, Usage, plugins, docker, documentation, user guide"] [menu.main] -parent = "mn_extend" +parent = "engine_extend" +++ -# Docker network driver plugins +# Engine network driver plugins -Docker network plugins enable Docker deployments to be extended to support a -wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or -something completely different. Network driver plugins are supported via the +Docker Engine network plugins enable Engine deployments to be extended to +support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN +or something completely different. Network driver plugins are supported via the LibNetwork project. Each plugin is implemented asa "remote driver" for -LibNetwork, which shares plugin infrastructure with Docker. Effectively, -network driver plugins are activated in the same way as other plugins, and use -the same kind of protocol. +LibNetwork, which shares plugin infrastructure with Engine. Effectively, network +driver plugins are activated in the same way as other plugins, and use the same +kind of protocol. ## Using network driver plugins diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 1814708275..cb1bebf581 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -4,32 +4,32 @@ title = "Volume plugins" description = "How to manage data with external volume plugins" keywords = ["Examples, Usage, volume, docker, data, volumes, plugin, api"] [menu.main] -parent = "mn_extend" +parent = "engine_extend" +++ # Write a volume plugin -Docker volume plugins enable Docker deployments to be integrated with external -storage systems, such as Amazon EBS, and enable data volumes to persist beyond -the lifetime of a single Docker host. See the [plugin documentation](plugins.md) -for more information. +Docker Engine volume plugins enable Engine deployments to be integrated with +external storage systems, such as Amazon EBS, and enable data volumes to persist +beyond the lifetime of a single Engine host. See the [plugin +documentation](plugins.md) for more information. ## Command-line changes -A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: +A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: $ docker run -ti -v volumename:/data --volume-driver=flocker busybox sh This command passes the `volumename` through to the volume plugin as a -user-given name for the volume. The `volumename` must not begin with a `/`. +user-given name for the volume. The `volumename` must not begin with a `/`. By having the user specify a `volumename`, a plugin can associate the volume with an external volume beyond the lifetime of a single container or container host. This can be used, for example, to move a stateful container from one server to another. -By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. +By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. ## Create a VolumeDriver @@ -214,4 +214,3 @@ Get the list of volumes registered with the plugin. ``` Respond with a string error if an error occurred. - From 953a6cf54f33817cf69d9cfca6274f49ccea9cef Mon Sep 17 00:00:00 2001 From: Felix Hupfeld Date: Mon, 1 Feb 2016 11:56:18 +0100 Subject: [PATCH 042/534] Added link to Quobyte's volume plugin. Signed-off-by: Felix Hupfeld --- docs/extend/plugins.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 5a91010b1c..1bc2c2e3f7 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -62,6 +62,8 @@ The following plugins exist: * The [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) is a cluster aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. +* The [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. + * The [REX-Ray plugin](https://github.com/emccode/rexray) is a volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. From b5b91098c322121b1dc74d7b01430b1da8f2b75c Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Mon, 1 Feb 2016 13:30:15 +0100 Subject: [PATCH 043/534] Add docker-volume-ipfs plugin to the list. Signed-off-by: Vincent Demeester --- docs/extend/plugins.md | 23 ++++++++++++++++------- 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 1bc2c2e3f7..8aa0603f78 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -53,6 +53,10 @@ The following plugins exist: another volume plugin that provides multi-host volumes management for Docker using GlusterFS. +* The [IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) + is an open source volume plugin that allows using an + [ipfs](https://ipfs.io/) filesystem as a volume. + * The [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) is a plugin that provides credentials and secret management using Keywhiz as a central repository. @@ -69,16 +73,21 @@ The following plugins exist: platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. * The [Contiv Volume Plugin](https://github.com/contiv/volplugin) is an open -source volume plugin that provides multi-tenant, persistent, distributed storage -with intent based consumption using ceph underneath. + source volume plugin that provides multi-tenant, persistent, distributed storage + with intent based consumption using ceph underneath. * The [Contiv Networking](https://github.com/contiv/netplugin) is an open source -libnetwork plugin to provide infrastructure and security policies for a -multi-tenant micro services deployment, while providing an integration to -physical network for non-container workload. Contiv Networking implements the -remote driver and IPAM APIs available in Docker 1.9 onwards. + libnetwork plugin to provide infrastructure and security policies for a + multi-tenant micro services deployment, while providing an integration to + physical network for non-container workload. Contiv Networking implements the + remote driver and IPAM APIs available in Docker 1.9 onwards. -* The [Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. +* The [Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) + creates a virtual network that connects your Docker containers - + across multiple hosts or clouds and enables automatic discovery of + applications. Weave networks are resilient, partition tolerant, + secure and work in partially connected networks, and other adverse + environments - all configured with delightful simplicity. ## Troubleshooting a plugin From 5a7d72e0a1d55ab9f02ab58feb7945edd6797ddb Mon Sep 17 00:00:00 2001 From: Mohammad Banikazemi Date: Wed, 3 Feb 2016 14:21:54 -0500 Subject: [PATCH 044/534] Add Kuryr Network Plugin to the docs Signed-off-by: Mohammad Banikazemi --- docs/extend/plugins.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 8aa0603f78..6fa14baabe 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -89,6 +89,12 @@ The following plugins exist: secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. +* The [Kuryr Network Plugin](https://github.com/openstack/kuryr) is + developed as part of the OpenStack Kuryr project and implements the + Docker networking (libnetwork) remote driver API by utilizing + Neutron, the OpenStack networking service. It includes an IPAM + driver as well. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 00acb699ada858f06f8efb1e0bd16129fff98cc0 Mon Sep 17 00:00:00 2001 From: Muthukumar R Date: Mon, 8 Feb 2016 12:01:11 -0800 Subject: [PATCH 045/534] Add Horcrux volume plugin to plugins.md Signed-off-by: Muthukumar R --- docs/extend/plugins.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 6fa14baabe..997c9dfbc5 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -53,6 +53,10 @@ The following plugins exist: another volume plugin that provides multi-host volumes management for Docker using GlusterFS. +* The [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) allows on-demand, + version controlled access to your data. Horcrux is an open-source plugin, + written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. + * The [IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) is an open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. From 0a2e0feedec46793b8d9f5eb9b6183a043e1b92c Mon Sep 17 00:00:00 2001 From: Cameron Spear Date: Wed, 24 Feb 2016 23:03:44 -0800 Subject: [PATCH 046/534] Add the Local Persist plugin I wasn't 100% sure if it was appropriate to add plugin here, but @thaJeztah invited me to do so, so here it is! (see https://github.com/CWSpear/local-persist/issues/17#issuecomment-188523784) Signed-off-by: Cameron Spear --- docs/extend/plugins.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 997c9dfbc5..da46a16ed7 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -99,6 +99,11 @@ The following plugins exist: Neutron, the OpenStack networking service. It includes an IPAM driver as well. +* The [Local Persist Plugin](https://github.com/CWSpear/local-persist) + extends the default `local` driver's functionality by allowing you specify + a mountpoint anywhere on the host, which enables the files to *always persist*, + even if the volume is removed via `docker volume rm`. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From efc058c62adb686f00e472ddb0b3e76302dacb86 Mon Sep 17 00:00:00 2001 From: Linus Heckemann Date: Mon, 29 Feb 2016 09:43:44 +0000 Subject: [PATCH 047/534] Document interfaces a plugin can implement Signed-off-by: Linus Heckemann --- docs/extend/plugin_api.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index c6793a3dda..360ab5d71b 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -127,6 +127,13 @@ Plugins are activated via the following "handshake" API call. Responds with a list of Docker subsystems which this plugin implements. After activation, the plugin will then be sent events from this subsystem. +Possible values are: + - [`authz`](authorization.md) + - `GraphDriver` + - [`NetworkDriver`](plugins_network.md) + - [`VolumeDriver`](plugins_volume.md) + + ## Plugin retries Attempts to call a method on a plugin are retried with an exponential backoff From b30ad57b187d4f5d1c5e3079a8e62147574e5bb0 Mon Sep 17 00:00:00 2001 From: Linus Heckemann Date: Mon, 29 Feb 2016 15:59:54 +0000 Subject: [PATCH 048/534] Remove experimental GraphDriver plugin type Signed-off-by: Linus Heckemann --- docs/extend/plugin_api.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 360ab5d71b..e0a91411f3 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -129,7 +129,6 @@ After activation, the plugin will then be sent events from this subsystem. Possible values are: - [`authz`](authorization.md) - - `GraphDriver` - [`NetworkDriver`](plugins_network.md) - [`VolumeDriver`](plugins_volume.md) From cc59d514a3d624f1472cf51ddeaab1af80ae6239 Mon Sep 17 00:00:00 2001 From: Ken Cochrane Date: Mon, 29 Feb 2016 17:51:36 -0800 Subject: [PATCH 049/534] Remove email address field from login This removes the email prompt when you use docker login, and also removes the ability to register via the docker cli. Docker login, will strictly be used for logging into a registry server. Signed-off-by: Ken Cochrane --- docs/deprecated.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 4ed127f953..7a351ac176 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -14,6 +14,13 @@ weight=80 The following list of features are deprecated in Engine. +### `-e` and `--email` flags on `docker login` +**Deprecated In Release: v1.11** + +**Target For Removal In Release: v1.13** + +The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. + ### Ambiguous event fields in API **Deprecated In Release: v1.10** From 776909a369466cb3107ae5756ffcc21cc0d4857f Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Tue, 8 Mar 2016 22:34:35 +0100 Subject: [PATCH 050/534] docs: extend: plugins: mention the sdk + systemd socket activation Signed-off-by: Antonio Murdaca --- docs/extend/index.md | 2 +- docs/extend/plugin_api.md | 44 ++++++++++++++++++- ...horization.md => plugins_authorization.md} | 1 + 3 files changed, 45 insertions(+), 2 deletions(-) rename docs/extend/{authorization.md => plugins_authorization.md} (99%) diff --git a/docs/extend/index.md b/docs/extend/index.md index f491926e9a..8a061e4e29 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -18,5 +18,5 @@ Currently, you can extend Docker Engine by adding a plugin. This section contain * [Understand Docker plugins](plugins.md) * [Write a volume plugin](plugins_volume.md) * [Write a network plugin](plugins_network.md) -* [Write an authorization plugin](authorization.md) +* [Write an authorization plugin](plugins_authorization.md) * [Docker plugin API](plugin_api.md) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index e0a91411f3..1a4237e2c7 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -96,6 +96,43 @@ directory and activates it with a handshake. See Handshake API below. Plugins are *not* activated automatically at Docker daemon startup. Rather, they are activated only lazily, or on-demand, when they are needed. +## Systemd socket activation + +Plugins may also be socket activated by `systemd`. The official [Plugins helpers](https://github.com/docker/go-plugins-helpers) +natively supports socket activation. In order for a plugin to be socket activated it needs +a `service` file and a `socket` file. + +The `service` file (for example `/lib/systemd/system/your-plugin.service`): + +``` +[Unit] +Description=Your plugin +Before=docker.service +After=network.target your-plugin.socket +Requires=your-plugin.socket docker.service + +[Service] +ExecStart=/usr/lib/docker/your-plugin + +[Install] +WantedBy=multi-user.target +``` +The `socket` file (for example `/lib/systemd/system/your-plugin.socket`): +``` +[Unit] +Description=Your plugin + +[Socket] +ListenStream=/run/docker/plugins/your-plugin.sock + +[Install] +WantedBy=sockets.target +``` + +This will allow plugins to be actually started when the Docker daemon connects to +the sockets they're listening on (for instance the first time the daemon uses them +or if one of the plugin goes down accidentally). + ## API design The Plugin API is RPC-style JSON over HTTP, much like webhooks. @@ -128,7 +165,7 @@ Responds with a list of Docker subsystems which this plugin implements. After activation, the plugin will then be sent events from this subsystem. Possible values are: - - [`authz`](authorization.md) + - [`authz`](plugins_authorization.md) - [`NetworkDriver`](plugins_network.md) - [`VolumeDriver`](plugins_volume.md) @@ -139,3 +176,8 @@ Attempts to call a method on a plugin are retried with an exponential backoff for up to 30 seconds. This may help when packaging plugins as containers, since it gives plugin containers a chance to start up before failing any user containers which depend on them. + +## Plugins helpers + +To ease plugins development, we're providing an `sdk` for each kind of plugins +currently supported by Docker at [docker/go-plugins-helpers](https://github.com/docker/go-plugins-helpers). diff --git a/docs/extend/authorization.md b/docs/extend/plugins_authorization.md similarity index 99% rename from docs/extend/authorization.md rename to docs/extend/plugins_authorization.md index 3512c56ccd..7db31b85c4 100644 --- a/docs/extend/authorization.md +++ b/docs/extend/plugins_authorization.md @@ -3,6 +3,7 @@ title = "Access authorization plugin" description = "How to create authorization plugins to manage access control to your Docker daemon." keywords = ["security, authorization, authentication, docker, documentation, plugin, extend"] +aliases = ["/engine/extend/authorization/"] [menu.main] parent = "engine_extend" weight = -1 From 8a60969f3e6d5f9078d62f7f61b5537ed68d9ee5 Mon Sep 17 00:00:00 2001 From: David Calavera Date: Tue, 15 Mar 2016 18:34:29 -0400 Subject: [PATCH 051/534] Consolidate security options to use `=` as separator. All other options we have use `=` as separator, labels, log configurations, graph configurations and so on. We should be consistent and use `=` for the security options too. Signed-off-by: David Calavera --- docs/deprecated.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7a351ac176..d2acf31f67 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -21,6 +21,8 @@ The following list of features are deprecated in Engine. The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. +The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. + ### Ambiguous event fields in API **Deprecated In Release: v1.10** From ea4082b55c030c1f71785a652492665ce3e4a0f4 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Mon, 21 Mar 2016 02:42:10 +0000 Subject: [PATCH 052/534] Update plugin docs for the covering of remote plugins (reference 20188). This fix updates the Plugin API docs to cover the case of remote plugins which could be deployed on a host different from the docker host, through spec or json files. This fix closes reference 20188. Signed-off-by: Yong Tang --- docs/extend/plugin_api.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 1a4237e2c7..1e7ee24361 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -20,8 +20,9 @@ If you just want to learn about or use Docker plugins, look ## What plugins are -A plugin is a process running on the same docker host as the docker daemon, -which registers itself by placing a file in one of the plugin directories described in [Plugin discovery](#plugin-discovery). +A plugin is a process running on the same or a different host as the docker daemon, +which registers itself by placing a file on the same docker host in one of the plugin +directories described in [Plugin discovery](#plugin-discovery). Plugins have human-readable names, which are short, lowercase strings. For example, `flocker` or `weave`. @@ -37,9 +38,12 @@ user or container tries to use one by name. There are three types of files which can be put in the plugin directory. * `.sock` files are UNIX domain sockets. -* `.spec` files are text files containing a URL, such as `unix:///other.sock`. +* `.spec` files are text files containing a URL, such as `unix:///other.sock` or `tcp://localhost:8080`. * `.json` files are text files containing a full json specification for the plugin. +Plugins with UNIX domain socket files must run on the same docker host, whereas +plugins with spec or json files can run on a different host if a remote URL is specified. + UNIX domain socket files must be located under `/run/docker/plugins`, whereas spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker/plugins`. From ab1c5af249a94ce2374d0992e727b21afa14e929 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 30 Mar 2016 15:00:09 -0700 Subject: [PATCH 053/534] Un-deprecate auto-creation of host directories for mounts Auto-creation of host-directories was marked deprecated in Docker 1.9, but was decided to be too much of an backward-incompatible change, so it was decided to keep the feature. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index d2acf31f67..f1c1fb0a77 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -133,17 +133,6 @@ The following double-dash options are deprecated and have no replacement: docker ps --before-id docker search --trusted -### Auto-creating missing host paths for bind mounts -**Deprecated in Release: v1.9** - -**Target for Removal in Release: 1.11** - -When creating a container with a bind-mounted volume-- `docker run -v /host/path:/container/path` -- -docker was automatically creating the `/host/path` if it didn't already exist. - -This auto-creation of the host path is deprecated and docker will error out if -the path does not exist. - ### Interacting with V1 registries Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. From 8b4d516b02b7fa02a4658ab7470c4c2477bbfe91 Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Wed, 30 Mar 2016 21:14:33 +0300 Subject: [PATCH 054/534] Add short description about default authentication method in authorization docs Following the discussion in reference 21556, adding a short description of the default user authentication mechanism (without requiring authentication plugins) Signed-off-by: Liron Levin --- docs/extend/plugins_authorization.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 7db31b85c4..6ebed3542f 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -49,6 +49,11 @@ Each plugin must reside within directories described under the **Note**: the abbreviations `AuthZ` and `AuthN` mean authorization and authentication respectively. +## Default user authorization mechanism + +If TLS is enabled in the [Docker daemon](https://docs.docker.com/engine/security/https/), the default user authorization flow extracts the user details from the certificate subject name. +That is, the `User` field is set to the client certificate subject common name, and the `AuthenticationMethod` field is set to `TLS`. + ## Basic architecture You are responsible for registering your plugin as part of the Docker daemon From 6acad7b0582e3127888fd101d3b9f605a922944b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A1ximo=20Cuadros?= Date: Thu, 31 Mar 2016 15:14:42 +0200 Subject: [PATCH 055/534] documentation: adding gce-docker plugin to plugins.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Máximo Cuadros --- docs/extend/plugins.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index da46a16ed7..01d68eaced 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -68,6 +68,8 @@ The following plugins exist: * The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. +* The [gce-docker plugin](https://github.com/mcuadros/gce-docker) is a volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). + * The [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) is a cluster aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. * The [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. From 77a53f348a0d744e3afedb55fed92f0ac56db484 Mon Sep 17 00:00:00 2001 From: Yi EungJun Date: Wed, 6 Apr 2016 16:10:30 +0900 Subject: [PATCH 056/534] Use 'null' instead of 'nil' for json When describe json response, 'null' is better than 'nil' which is not in json specification. Signed-off-by: Yi EungJun --- docs/extend/plugins_authorization.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 6ebed3542f..a47cd2a55b 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -196,8 +196,8 @@ should implement the following two methods: "Allow": "Determined whether the user is allowed or not", "Msg": "The authorization message", "Err": "The error message if things go wrong", - "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or nil if no changes required)", - "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or nil if no changes required)", + "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or null if no changes required)", + "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or null if no changes required)", "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" } ``` From 090fce081aa291d09fcc0a62a804af0a13002a55 Mon Sep 17 00:00:00 2001 From: Yi EungJun Date: Wed, 6 Apr 2016 15:37:49 +0900 Subject: [PATCH 057/534] Fix incorrect markdown rendering The docker document site [1] rendered the list of plugin implements incorrectly. [1]: https://docs.docker.com/engine/extend/plugin_api Signed-off-by: Yi EungJun --- docs/extend/plugin_api.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 1e7ee24361..a799a13520 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -169,9 +169,10 @@ Responds with a list of Docker subsystems which this plugin implements. After activation, the plugin will then be sent events from this subsystem. Possible values are: - - [`authz`](plugins_authorization.md) - - [`NetworkDriver`](plugins_network.md) - - [`VolumeDriver`](plugins_volume.md) + +* [`authz`](plugins_authorization.md) +* [`NetworkDriver`](plugins_network.md) +* [`VolumeDriver`](plugins_volume.md) ## Plugin retries From 0f922a54ec584a86c272342817a07a3ee2051992 Mon Sep 17 00:00:00 2001 From: Yi EungJun Date: Sat, 9 Apr 2016 19:15:55 +0900 Subject: [PATCH 058/534] Fix errata; s/RequestUri/ReqestURI/ Signed-off-by: Yi EungJun --- docs/extend/plugins_authorization.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index a47cd2a55b..887fd9330e 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -154,7 +154,7 @@ should implement the following two methods: "User": "The user identification", "UserAuthNMethod": "The authentication method used", "RequestMethod": "The HTTP method", - "RequestUri": "The HTTP request URI", + "RequestURI": "The HTTP request URI", "RequestBody": "Byte array containing the raw HTTP request body", "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string ", "RequestStatusCode": "Request status code" @@ -179,7 +179,7 @@ should implement the following two methods: "User": "The user identification", "UserAuthNMethod": "The authentication method used", "RequestMethod": "The HTTP method", - "RequestUri": "The HTTP request URI", + "RequestURI": "The HTTP request URI", "RequestBody": "Byte array containing the raw HTTP request body", "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string", "RequestStatusCode": "Request status code", From facce6f44700af26998597446ba99a0c55b690ae Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Mon, 7 Mar 2016 15:44:43 -0500 Subject: [PATCH 059/534] Allow volume drivers to provide a `Status` field The `Status` field is a `map[string]interface{}` which allows the driver to pass back low-level details about the underlying volume. Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index cb1bebf581..88798497c3 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -15,6 +15,21 @@ external storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Engine host. See the [plugin documentation](plugins.md) for more information. +## Changelog + +### 1.12.0 + +- Add `Status` field to `VolumeDriver.Get` response ([#21006](https://github.com/docker/docker/pull/21006#)) + +### 1.10.0 + +- Add `VolumeDriver.Get` which gets the details about the volume ([#16534](https://github.com/docker/docker/pull/16534)) +- Add `VolumeDriver.List` which lists all volumes owned by the driver ([#16534](https://github.com/docker/docker/pull/16534)) + +### 1.8.0 + +- Initial support for volume driver plugins ([#14659](https://github.com/docker/docker/pull/14659)) + ## Command-line changes A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: @@ -183,6 +198,7 @@ Get the volume info. "Volume": { "Name": "volume_name", "Mountpoint": "/path/to/directory/on/host", + "Status": {} }, "Err": "" } From 55bb46330067f9c5dbdfc02b00ad26c8f74595ff Mon Sep 17 00:00:00 2001 From: Jared Hocutt Date: Tue, 19 Apr 2016 11:05:05 -0400 Subject: [PATCH 060/534] Add the NetApp Docker Volume Plugin to the documentation Signed-off-by: Jared Hocutt --- docs/extend/plugins.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 01d68eaced..9c6854ef84 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -106,6 +106,12 @@ The following plugins exist: a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. +* The [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) provides + direct integration with the Docker ecosystem for the NetApp storage portfolio. + The nDVP package supports the provisioning and management of + storage resources from the storage platform to Docker hosts, with a robust + framework for adding additional platforms in the future. + ## Troubleshooting a plugin If you are having problems with Docker after loading a plugin, ask the authors From 7edcb01ac4498436168abe57a0372bbb517e3415 Mon Sep 17 00:00:00 2001 From: Wen Cheng Ma Date: Wed, 20 Apr 2016 11:14:47 +0800 Subject: [PATCH 061/534] Fix asa Signed-off-by: Wen Cheng Ma --- docs/extend/plugins_network.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index ac07273219..aad01ba29a 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -13,7 +13,7 @@ parent = "engine_extend" Docker Engine network plugins enable Engine deployments to be extended to support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different. Network driver plugins are supported via the -LibNetwork project. Each plugin is implemented asa "remote driver" for +LibNetwork project. Each plugin is implemented as a "remote driver" for LibNetwork, which shares plugin infrastructure with Engine. Effectively, network driver plugins are activated in the same way as other plugins, and use the same kind of protocol. From 895960b9f0d8fa2285c65f651016ea0d0625ad1d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 20 Apr 2016 13:43:33 +0200 Subject: [PATCH 062/534] docs: use tables for available plugins Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins.md | 102 +++++++++++------------------------------ 1 file changed, 28 insertions(+), 74 deletions(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 9c6854ef84..7eec419c97 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -31,86 +31,40 @@ Follow the instructions in the plugin's documentation. ## Finding a plugin -The following plugins exist: +The sections below provide an inexhaustive overview of available plugins. -* The [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) - is a volume plugin that provides access to an extensible set of - container-based persistent storage options. It supports single and multi-host Docker - environments with features that include tenant isolation, automated - provisioning, encryption, secure deletion, snapshots and QoS. + -* The [Convoy plugin](https://github.com/rancher/convoy) is a volume plugin for a - variety of storage back-ends including device mapper and NFS. It's a simple standalone - executable written in Go and provides the framework to support vendor-specific extensions - such as snapshots, backups and restore. +### Network plugins -* The [Flocker plugin](https://clusterhq.com/docker-plugin/) is a volume plugin - which provides multi-host portable volumes for Docker, enabling you to run - databases and other stateful containers and move them around across a cluster - of machines. +Plugin | Description +----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +[Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. +[Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. +[Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. -* The [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) is - another volume plugin that provides multi-host volumes management for Docker - using GlusterFS. +### Volume plugins -* The [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) allows on-demand, - version controlled access to your data. Horcrux is an open-source plugin, - written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. +Plugin | Description +----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +[Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. +[Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. +[Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. +[Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. +[gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). +[GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. +[Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. +[IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. +[Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. +[Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. +[NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. +[Netshare plugin](https://github.com/gondor/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. +[OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. +[Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. +[REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. -* The [IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) - is an open source volume plugin that allows using an - [ipfs](https://ipfs.io/) filesystem as a volume. - -* The [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) is - a plugin that provides credentials and secret management using Keywhiz as - a central repository. - -* The [Netshare plugin](https://github.com/gondor/docker-volume-netshare) is a volume plugin - that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. - -* The [gce-docker plugin](https://github.com/mcuadros/gce-docker) is a volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). - -* The [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) is a cluster aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. - -* The [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. - -* The [REX-Ray plugin](https://github.com/emccode/rexray) is a volume plugin - which is written in Go and provides advanced storage functionality for many - platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. - -* The [Contiv Volume Plugin](https://github.com/contiv/volplugin) is an open - source volume plugin that provides multi-tenant, persistent, distributed storage - with intent based consumption using ceph underneath. - -* The [Contiv Networking](https://github.com/contiv/netplugin) is an open source - libnetwork plugin to provide infrastructure and security policies for a - multi-tenant micro services deployment, while providing an integration to - physical network for non-container workload. Contiv Networking implements the - remote driver and IPAM APIs available in Docker 1.9 onwards. - -* The [Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) - creates a virtual network that connects your Docker containers - - across multiple hosts or clouds and enables automatic discovery of - applications. Weave networks are resilient, partition tolerant, - secure and work in partially connected networks, and other adverse - environments - all configured with delightful simplicity. - -* The [Kuryr Network Plugin](https://github.com/openstack/kuryr) is - developed as part of the OpenStack Kuryr project and implements the - Docker networking (libnetwork) remote driver API by utilizing - Neutron, the OpenStack networking service. It includes an IPAM - driver as well. - -* The [Local Persist Plugin](https://github.com/CWSpear/local-persist) - extends the default `local` driver's functionality by allowing you specify - a mountpoint anywhere on the host, which enables the files to *always persist*, - even if the volume is removed via `docker volume rm`. - -* The [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) provides - direct integration with the Docker ecosystem for the NetApp storage portfolio. - The nDVP package supports the provisioning and management of - storage resources from the storage platform to Docker hosts, with a robust - framework for adding additional platforms in the future. ## Troubleshooting a plugin From d305b9757cdbd5085302eda04cec5ab9e55b24f6 Mon Sep 17 00:00:00 2001 From: Wen Cheng Ma Date: Tue, 19 Apr 2016 13:50:09 +0800 Subject: [PATCH 063/534] Deprecated of docker ps since and before options for v1.12 Deprecated note https://github.com/docker/docker/blob/master/docs/deprecated.md#docker-ps-before-and-since-options Signed-off-by: Wen Cheng Ma --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index f1c1fb0a77..e9b3d73071 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -48,7 +48,7 @@ defining it at container creation (`POST /containers/create`). **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Target For Removal In Release: v1.12** +**Removed In Release: v1.12** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. From fde74829666b562f12b0e981c6ced846d3f15245 Mon Sep 17 00:00:00 2001 From: Lorenzo Fontana Date: Mon, 25 Apr 2016 15:31:23 +0200 Subject: [PATCH 064/534] Mention the fact that authz plugins are available today Signed-off-by: Lorenzo Fontana --- docs/extend/plugins.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 7eec419c97..e40c611282 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -22,7 +22,7 @@ example, a [volume plugin](plugins_volume.md) might enable Docker volumes to persist across multiple Docker hosts and a [network plugin](plugins_network.md) might provide network plumbing. -Currently Docker supports volume and network driver plugins. In the future it +Currently Docker supports authorization, volume and network driver plugins. In the future it will support additional plugin types. ## Installing a plugin @@ -65,6 +65,11 @@ Plugin [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. +### Authorization plugins + + Plugin | Description +------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ + [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). ## Troubleshooting a plugin From 957c5c25770b9259bbd57fd986d1c33eb46d238d Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Mon, 2 May 2016 22:19:23 +0300 Subject: [PATCH 065/534] Remove response modification sections from authorization design doc Signed-off-by: Liron Levin --- docs/extend/plugins_authorization.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 887fd9330e..140db71c35 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -195,17 +195,10 @@ should implement the following two methods: { "Allow": "Determined whether the user is allowed or not", "Msg": "The authorization message", - "Err": "The error message if things go wrong", - "ModifiedBody": "Byte array containing a modified body of the raw HTTP body (or null if no changes required)", - "ModifiedHeader": "Byte array containing a modified header of the HTTP response (or null if no changes required)", - "ModifiedStatusCode": "int containing the modified version of the status code (or 0 if not change is required)" + "Err": "The error message if things go wrong" } ``` -The modified response enables the authorization plugin to manipulate the content -of the HTTP response. In case of more than one plugin, each subsequent plugin -receives a response (optionally) modified by a previous plugin. - ### Request authorization Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. From 4f1ab74b3998bf3a96453645f6a0d35fbf360243 Mon Sep 17 00:00:00 2001 From: Wen Cheng Ma Date: Thu, 28 Apr 2016 14:55:22 +0800 Subject: [PATCH 066/534] Update the `docker daemon` to `dockerd` for document Signed-off-by: Wen Cheng Ma --- docs/reference/commandline/dockerd.md | 959 ++++++++++++++++++++++++++ 1 file changed, 959 insertions(+) create mode 100644 docs/reference/commandline/dockerd.md diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md new file mode 100644 index 0000000000..842e051664 --- /dev/null +++ b/docs/reference/commandline/dockerd.md @@ -0,0 +1,959 @@ + + +# daemon + + Usage: dockerd [OPTIONS] + + A self-sufficient runtime for linux containers. + + Options: + --api-cors-header="" Set CORS headers in the remote API + --authorization-plugin=[] Set authorization plugins to load + -b, --bridge="" Attach containers to a network bridge + --bip="" Specify network bridge IP + --cgroup-parent= Set parent cgroup for all containers + --cluster-store="" URL of the distributed storage backend + --cluster-advertise="" Address of the daemon instance on the cluster + --cluster-store-opt=map[] Set cluster options + --config-file=/etc/docker/daemon.json Daemon configuration file + --containerd Path to containerd socket + -D, --debug Enable debug mode + --default-gateway="" Container default gateway IPv4 address + --default-gateway-v6="" Container default gateway IPv6 address + --dns=[] DNS server to use + --dns-opt=[] DNS options to use + --dns-search=[] DNS search domains to use + --default-ulimit=[] Set default ulimit settings for containers + --exec-opt=[] Set runtime execution options + --exec-root="/var/run/docker" Root directory for execution state files + --fixed-cidr="" IPv4 subnet for fixed IPs + --fixed-cidr-v6="" IPv6 subnet for fixed IPs + -G, --group="docker" Group for the unix socket + -g, --graph="/var/lib/docker" Root of the Docker runtime + -H, --host=[] Daemon socket(s) to connect to + --help Print usage + --icc=true Enable inter-container communication + --insecure-registry=[] Enable insecure registry communication + --ip=0.0.0.0 Default IP when binding container ports + --ip-forward=true Enable net.ipv4.ip_forward + --ip-masq=true Enable IP masquerading + --iptables=true Enable addition of iptables rules + --ipv6 Enable IPv6 networking + -l, --log-level="info" Set the logging level + --label=[] Set key=value labels to the daemon + --log-driver="json-file" Default driver for container logs + --log-opt=[] Log driver specific options + --mtu=0 Set the containers network MTU + --disable-legacy-registry Do not contact legacy registries + -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file + --raw-logs Full timestamps without ANSI coloring + --registry-mirror=[] Preferred Docker registry mirror + -s, --storage-driver="" Storage driver to use + --selinux-enabled Enable selinux support + --storage-opt=[] Set storage driver options + --tls Use TLS; implied by --tlsverify + --tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA + --tlscert="~/.docker/cert.pem" Path to TLS certificate file + --tlskey="~/.docker/key.pem" Path to TLS key file + --tlsverify Use TLS and verify the remote + --userns-remap="default" Enable user namespace remapping + --userland-proxy=true Use userland proxy for loopback traffic + +Options with [] may be specified multiple times. + +dockerd is the persistent process that manages containers. Docker +uses different binaries for the daemon and client. To run the daemon you +type `dockerd`. + +To run the daemon with debug output, use `dockerd -D`. + +## Daemon socket option + +The Docker daemon can listen for [Docker Remote API](../api/docker_remote_api.md) +requests via three different types of Socket: `unix`, `tcp`, and `fd`. + +By default, a `unix` domain socket (or IPC socket) is created at +`/var/run/docker.sock`, requiring either `root` permission, or `docker` group +membership. + +If you need to access the Docker daemon remotely, you need to enable the `tcp` +Socket. Beware that the default setup provides un-encrypted and +un-authenticated direct access to the Docker daemon - and should be secured +either using the [built in HTTPS encrypted socket](../../security/https/), or by +putting a secure web proxy in front of it. You can listen on port `2375` on all +network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network +interface using its IP address: `-H tcp://192.168.59.103:2375`. It is +conventional to use port `2375` for un-encrypted, and port `2376` for encrypted +communication with the daemon. + +> **Note:** +> If you're using an HTTPS encrypted socket, keep in mind that only +> TLS1.0 and greater are supported. Protocols SSLv3 and under are not +> supported anymore for security reasons. + +On Systemd based systems, you can communicate with the daemon via +[Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html), +use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but +you can also specify individual sockets: `dockerd -H fd://3`. If the +specified socket activated files aren't found, then Docker will exit. You can +find examples of using Systemd socket activation with Docker and Systemd in the +[Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/). + +You can configure the Docker daemon to listen to multiple sockets at the same +time using multiple `-H` options: + + # listen using the default unix socket, and on 2 specific IP addresses on this host. + dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 + +The Docker client will honor the `DOCKER_HOST` environment variable to set the +`-H` flag for the client. + + $ docker -H tcp://0.0.0.0:2375 ps + # or + $ export DOCKER_HOST="tcp://0.0.0.0:2375" + $ docker ps + # both are equal + +Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than +the empty string is equivalent to setting the `--tlsverify` flag. The following +are equivalent: + + $ docker --tlsverify ps + # or + $ export DOCKER_TLS_VERIFY=1 + $ docker ps + +The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` +environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes +precedence over `HTTP_PROXY`. + +### Daemon storage-driver option + +The Docker daemon has support for several different image layer storage +drivers: `aufs`, `devicemapper`, `btrfs`, `zfs` and `overlay`. + +The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that +is unlikely to be merged into the main kernel. These are also known to cause +some serious kernel crashes. However, `aufs` is also the only storage driver +that allows containers to share executable and shared library memory, so is a +useful choice when running thousands of containers with the same program or +libraries. + +The `devicemapper` driver uses thin provisioning and Copy on Write (CoW) +snapshots. For each devicemapper graph location – typically +`/var/lib/docker/devicemapper` – a thin pool is created based on two block +devices, one for data and one for metadata. By default, these block devices +are created automatically by using loopback mounts of automatically created +sparse files. Refer to [Storage driver options](#storage-driver-options) below +for a way how to customize this setup. +[~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) +article explains how to tune your existing setup without the use of options. + +The `btrfs` driver is very fast for `docker build` - but like `devicemapper` +does not share executable memory between devices. Use +`dockerd -s btrfs -g /mnt/btrfs_partition`. + +The `zfs` driver is probably not as fast as `btrfs` but has a longer track record +on stability. Thanks to `Single Copy ARC` shared blocks between clones will be +cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem +set `zfs.fsname` option as described in [Storage driver options](#storage-driver-options). + +The `overlay` is a very fast union filesystem. It is now merged in the main +Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). Call +`dockerd -s overlay` to use it. + +> **Note:** +> As promising as `overlay` is, the feature is still quite young and should not +> be used in production. Most notably, using `overlay` can cause excessive +> inode consumption (especially as the number of images grows), as well as +> being incompatible with the use of RPMs. + +> **Note:** +> It is currently unsupported on `btrfs` or any Copy on Write filesystem +> and should only be used over `ext4` partitions. + +### Storage driver options + +Particular storage-driver can be configured with options specified with +`--storage-opt` flags. Options for `devicemapper` are prefixed with `dm` and +options for `zfs` start with `zfs`. + +* `dm.thinpooldev` + + Specifies a custom block storage device to use for the thin pool. + + If using a block device for device mapper storage, it is best to use `lvm` + to create and manage the thin-pool volume. This volume is then handed to Docker + to exclusively create snapshot volumes needed for images and containers. + + Managing the thin-pool outside of Engine makes for the most feature-rich + method of having Docker utilize device mapper thin provisioning as the + backing storage for Docker containers. The highlights of the lvm-based + thin-pool management feature include: automatic or interactive thin-pool + resize support, dynamically changing thin-pool features, automatic thinp + metadata checking when lvm activates the thin-pool, etc. + + As a fallback if no thin pool is provided, loopback files are + created. Loopback is very slow, but can be used without any + pre-configuration of storage. It is strongly recommended that you do + not use loopback in production. Ensure your Engine daemon has a + `--storage-opt dm.thinpooldev` argument provided. + + Example use: + + $ dockerd \ + --storage-opt dm.thinpooldev=/dev/mapper/thin-pool + +* `dm.basesize` + + Specifies the size to use when creating the base device, which limits the + size of images and containers. The default value is 10G. Note, thin devices + are inherently "sparse", so a 10G device which is mostly empty doesn't use + 10 GB of space on the pool. However, the filesystem will use more space for + the empty case the larger the device is. + + The base device size can be increased at daemon restart which will allow + all future images and containers (based on those new images) to be of the + new base device size. + + Example use: + + $ dockerd --storage-opt dm.basesize=50G + + This will increase the base device size to 50G. The Docker daemon will throw an + error if existing base device size is larger than 50G. A user can use + this option to expand the base device size however shrinking is not permitted. + + This value affects the system-wide "base" empty filesystem + that may already be initialized and inherited by pulled images. Typically, + a change to this value requires additional steps to take effect: + + $ sudo service docker stop + $ sudo rm -rf /var/lib/docker + $ sudo service docker start + + Example use: + + $ dockerd --storage-opt dm.basesize=20G + +* `dm.loopdatasize` + + > **Note**: + > This option configures devicemapper loopback, which should not + > be used in production. + + Specifies the size to use when creating the loopback file for the + "data" device which is used for the thin pool. The default size is + 100G. The file is sparse, so it will not initially take up this + much space. + + Example use: + + $ dockerd --storage-opt dm.loopdatasize=200G + +* `dm.loopmetadatasize` + + > **Note**: + > This option configures devicemapper loopback, which should not + > be used in production. + + Specifies the size to use when creating the loopback file for the + "metadata" device which is used for the thin pool. The default size + is 2G. The file is sparse, so it will not initially take up + this much space. + + Example use: + + $ dockerd --storage-opt dm.loopmetadatasize=4G + +* `dm.fs` + + Specifies the filesystem type to use for the base device. The supported + options are "ext4" and "xfs". The default is "xfs" + + Example use: + + $ dockerd --storage-opt dm.fs=ext4 + +* `dm.mkfsarg` + + Specifies extra mkfs arguments to be used when creating the base device. + + Example use: + + $ dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" + +* `dm.mountopt` + + Specifies extra mount options used when mounting the thin devices. + + Example use: + + $ dockerd --storage-opt dm.mountopt=nodiscard + +* `dm.datadev` + + (Deprecated, use `dm.thinpooldev`) + + Specifies a custom blockdevice to use for data for the thin pool. + + If using a block device for device mapper storage, ideally both datadev and + metadatadev should be specified to completely avoid using the loopback + device. + + Example use: + + $ dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 + +* `dm.metadatadev` + + (Deprecated, use `dm.thinpooldev`) + + Specifies a custom blockdevice to use for metadata for the thin pool. + + For best performance the metadata should be on a different spindle than the + data, or even better on an SSD. + + If setting up a new metadata pool it is required to be valid. This can be + achieved by zeroing the first 4k to indicate empty metadata, like this: + + $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 + + Example use: + + $ dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 + +* `dm.blocksize` + + Specifies a custom blocksize to use for the thin pool. The default + blocksize is 64K. + + Example use: + + $ dockerd --storage-opt dm.blocksize=512K + +* `dm.blkdiscard` + + Enables or disables the use of blkdiscard when removing devicemapper + devices. This is enabled by default (only) if using loopback devices and is + required to resparsify the loopback file on image/container removal. + + Disabling this on loopback can lead to *much* faster container removal + times, but will make the space used in `/var/lib/docker` directory not be + returned to the system for other use when containers are removed. + + Example use: + + $ dockerd --storage-opt dm.blkdiscard=false + +* `dm.override_udev_sync_check` + + Overrides the `udev` synchronization checks between `devicemapper` and `udev`. + `udev` is the device manager for the Linux kernel. + + To view the `udev` sync support of a Docker daemon that is using the + `devicemapper` driver, run: + + $ docker info + [...] + Udev Sync Supported: true + [...] + + When `udev` sync support is `true`, then `devicemapper` and udev can + coordinate the activation and deactivation of devices for containers. + + When `udev` sync support is `false`, a race condition occurs between + the`devicemapper` and `udev` during create and cleanup. The race condition + results in errors and failures. (For information on these failures, see + [docker#4036](https://github.com/docker/docker/issues/4036)) + + To allow the `docker` daemon to start, regardless of `udev` sync not being + supported, set `dm.override_udev_sync_check` to true: + + $ dockerd --storage-opt dm.override_udev_sync_check=true + + When this value is `true`, the `devicemapper` continues and simply warns + you the errors are happening. + + > **Note:** + > The ideal is to pursue a `docker` daemon and environment that does + > support synchronizing with `udev`. For further discussion on this + > topic, see [docker#4036](https://github.com/docker/docker/issues/4036). + > Otherwise, set this flag for migrating existing Docker daemons to + > a daemon with a supported environment. + +* `dm.use_deferred_removal` + + Enables use of deferred device removal if `libdm` and the kernel driver + support the mechanism. + + Deferred device removal means that if device is busy when devices are + being removed/deactivated, then a deferred removal is scheduled on + device. And devices automatically go away when last user of the device + exits. + + For example, when a container exits, its associated thin device is removed. + If that device has leaked into some other mount namespace and can't be + removed, the container exit still succeeds and this option causes the + system to schedule the device for deferred removal. It does not wait in a + loop trying to remove a busy device. + + Example use: + + $ dockerd --storage-opt dm.use_deferred_removal=true + +* `dm.use_deferred_deletion` + + Enables use of deferred device deletion for thin pool devices. By default, + thin pool device deletion is synchronous. Before a container is deleted, + the Docker daemon removes any associated devices. If the storage driver + can not remove a device, the container deletion fails and daemon returns. + + Error deleting container: Error response from daemon: Cannot destroy container + + To avoid this failure, enable both deferred device deletion and deferred + device removal on the daemon. + + $ dockerd \ + --storage-opt dm.use_deferred_deletion=true \ + --storage-opt dm.use_deferred_removal=true + + With these two options enabled, if a device is busy when the driver is + deleting a container, the driver marks the device as deleted. Later, when + the device isn't in use, the driver deletes it. + + In general it should be safe to enable this option by default. It will help + when unintentional leaking of mount point happens across multiple mount + namespaces. + +* `dm.min_free_space` + + Specifies the min free space percent in a thin pool require for new device + creation to succeed. This check applies to both free data space as well + as free metadata space. Valid values are from 0% - 99%. Value 0% disables + free space checking logic. If user does not specify a value for this option, + the Engine uses a default value of 10%. + + Whenever a new a thin pool device is created (during `docker pull` or during + container creation), the Engine checks if the minimum free space is + available. If sufficient space is unavailable, then device creation fails + and any relevant `docker` operation fails. + + To recover from this error, you must create more free space in the thin pool + to recover from the error. You can create free space by deleting some images + and containers from the thin pool. You can also add more storage to the thin + pool. + + To add more space to a LVM (logical volume management) thin pool, just add + more storage to the volume group container thin pool; this should automatically + resolve any errors. If your configuration uses loop devices, then stop the + Engine daemon, grow the size of loop files and restart the daemon to resolve + the issue. + + Example use: + + ```bash + $ dockerd --storage-opt dm.min_free_space=10% + ``` + +Currently supported options of `zfs`: + +* `zfs.fsname` + + Set zfs filesystem under which docker will create its own datasets. + By default docker will pick up the zfs filesystem where docker graph + (`/var/lib/docker`) is located. + + Example use: + + $ dockerd -s zfs --storage-opt zfs.fsname=zroot/docker + +## Docker runtime execution options + +The Docker daemon relies on a +[OCI](https://github.com/opencontainers/specs) compliant runtime +(invoked via the `containerd` daemon) as its interface to the Linux +kernel `namespaces`, `cgroups`, and `SELinux`. + +## Options for the runtime + +You can configure the runtime using options specified +with the `--exec-opt` flag. All the flag's options have the `native` prefix. A +single `native.cgroupdriver` option is available. + +The `native.cgroupdriver` option specifies the management of the container's +cgroups. You can specify only specify `cgroupfs` or `systemd`. If you specify +`systemd` and it is not available, the system errors out. If you omit the +`native.cgroupdriver` option,` cgroupfs` is used. + +This example sets the `cgroupdriver` to `systemd`: + + $ sudo dockerd --exec-opt native.cgroupdriver=systemd + +Setting this option applies to all containers the daemon launches. + +Also Windows Container makes use of `--exec-opt` for special purpose. Docker user +can specify default container isolation technology with this, for example: + + $ dockerd --exec-opt isolation=hyperv + +Will make `hyperv` the default isolation technology on Windows, without specifying +isolation value on daemon start, Windows isolation technology will default to `process`. + +## Daemon DNS options + +To set the DNS server for all Docker containers, use +`dockerd --dns 8.8.8.8`. + +To set the DNS search domain for all Docker containers, use +`dockerd --dns-search example.com`. + +## Insecure registries + +Docker considers a private registry either secure or insecure. In the rest of +this section, *registry* is used for *private registry*, and `myregistry:5000` +is a placeholder example for a private registry. + +A secure registry uses TLS and a copy of its CA certificate is placed on the +Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure +registry is either not using TLS (i.e., listening on plain text HTTP), or is +using TLS with a CA certificate not known by the Docker daemon. The latter can +happen when the certificate was not found under +`/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification +failed (i.e., wrong CA). + +By default, Docker assumes all, but local (see local registries below), +registries are secure. Communicating with an insecure registry is not possible +if Docker assumes that registry is secure. In order to communicate with an +insecure registry, the Docker daemon requires `--insecure-registry` in one of +the following two forms: + +* `--insecure-registry myregistry:5000` tells the Docker daemon that + myregistry:5000 should be considered insecure. +* `--insecure-registry 10.1.0.0/16` tells the Docker daemon that all registries + whose domain resolve to an IP address is part of the subnet described by the + CIDR syntax, should be considered insecure. + +The flag can be used multiple times to allow multiple registries to be marked +as insecure. + +If an insecure registry is not marked as insecure, `docker pull`, +`docker push`, and `docker search` will result in an error message prompting +the user to either secure or pass the `--insecure-registry` flag to the Docker +daemon as described above. + +Local registries, whose IP address falls in the 127.0.0.0/8 range, are +automatically marked as insecure as of Docker 1.3.2. It is not recommended to +rely on this, as it may change in the future. + +Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted +communication, can be useful when running a local registry. However, +because its use creates security vulnerabilities it should ONLY be enabled for +testing purposes. For increased security, users should add their CA to their +system's list of trusted CAs instead of enabling `--insecure-registry`. + +## Legacy Registries + +Enabling `--disable-legacy-registry` forces a docker daemon to only interact with registries which support the V2 protocol. Specifically, the daemon will not attempt `push`, `pull` and `login` to v1 registries. The exception to this is `search` which can still be performed on v1 registries. + +## Running a Docker daemon behind a HTTPS_PROXY + +When running inside a LAN that uses a `HTTPS` proxy, the Docker Hub +certificates will be replaced by the proxy's certificates. These certificates +need to be added to your Docker host's configuration: + +1. Install the `ca-certificates` package for your distribution +2. Ask your network admin for the proxy's CA certificate and append them to + `/etc/pki/tls/certs/ca-bundle.crt` +3. Then start your Docker daemon with `HTTPS_PROXY=http://username:password@proxy:port/ dockerd`. + The `username:` and `password@` are optional - and are only needed if your + proxy is set up to require authentication. + +This will only add the proxy and authentication to the Docker daemon's requests - +your `docker build`s and running containers will need extra configuration to +use the proxy + +## Default Ulimits + +`--default-ulimit` allows you to set the default `ulimit` options to use for +all containers. It takes the same options as `--ulimit` for `docker run`. If +these defaults are not set, `ulimit` settings will be inherited, if not set on +`docker run`, from the Docker daemon. Any `--ulimit` options passed to +`docker run` will overwrite these defaults. + +Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to +set the maximum number of processes available to a user, not to a container. For details +please check the [run](run.md) reference. + +## Nodes discovery + +The `--cluster-advertise` option specifies the `host:port` or `interface:port` +combination that this particular daemon instance should use when advertising +itself to the cluster. The daemon is reached by remote hosts through this value. +If you specify an interface, make sure it includes the IP address of the actual +Docker host. For Engine installation created through `docker-machine`, the +interface is typically `eth1`. + +The daemon uses [libkv](https://github.com/docker/libkv/) to advertise +the node within the cluster. Some key-value backends support mutual +TLS. To configure the client TLS settings used by the daemon can be configured +using the `--cluster-store-opt` flag, specifying the paths to PEM encoded +files. For example: + +```bash +dockerd \ + --cluster-advertise 192.168.1.2:2376 \ + --cluster-store etcd://192.168.1.2:2379 \ + --cluster-store-opt kv.cacertfile=/path/to/ca.pem \ + --cluster-store-opt kv.certfile=/path/to/cert.pem \ + --cluster-store-opt kv.keyfile=/path/to/key.pem +``` + +The currently supported cluster store options are: + +* `discovery.heartbeat` + + Specifies the heartbeat timer in seconds which is used by the daemon as a + keepalive mechanism to make sure discovery module treats the node as alive + in the cluster. If not configured, the default value is 20 seconds. + +* `discovery.ttl` + + Specifies the ttl (time-to-live) in seconds which is used by the discovery + module to timeout a node if a valid heartbeat is not received within the + configured ttl value. If not configured, the default value is 60 seconds. + +* `kv.cacertfile` + + Specifies the path to a local file with PEM encoded CA certificates to trust + +* `kv.certfile` + + Specifies the path to a local file with a PEM encoded certificate. This + certificate is used as the client cert for communication with the + Key/Value store. + +* `kv.keyfile` + + Specifies the path to a local file with a PEM encoded private key. This + private key is used as the client key for communication with the + Key/Value store. + +* `kv.path` + + Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. + +## Access authorization + +Docker's access authorization can be extended by authorization plugins that your +organization can purchase or build themselves. You can install one or more +authorization plugins when you start the Docker `daemon` using the +`--authorization-plugin=PLUGIN_ID` option. + +```bash +dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... +``` + +The `PLUGIN_ID` value is either the plugin's name or a path to its specification +file. The plugin's implementation determines whether you can specify a name or +path. Consult with your Docker administrator to get information about the +plugins available to you. + +Once a plugin is installed, requests made to the `daemon` through the command +line or Docker's remote API are allowed or denied by the plugin. If you have +multiple plugins installed, at least one must allow the request for it to +complete. + +For information about how to create an authorization plugin, see [authorization +plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. + + +## Daemon user namespace options + +The Linux kernel [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling +a process, and therefore a container, to have a unique range of user and +group IDs which are outside the traditional user and group range utilized by +the host system. Potentially the most important security improvement is that, +by default, container processes running as the `root` user will have expected +administrative privilege (with some restrictions) inside the container but will +effectively be mapped to an unprivileged `uid` on the host. + +When user namespace support is enabled, Docker creates a single daemon-wide mapping +for all containers running on the same engine instance. The mappings will +utilize the existing subordinate user and group ID feature available on all modern +Linux distributions. +The [`/etc/subuid`](http://man7.org/linux/man-pages/man5/subuid.5.html) and +[`/etc/subgid`](http://man7.org/linux/man-pages/man5/subgid.5.html) files will be +read for the user, and optional group, specified to the `--userns-remap` +parameter. If you do not wish to specify your own user and/or group, you can +provide `default` as the value to this flag, and a user will be created on your behalf +and provided subordinate uid and gid ranges. This default user will be named +`dockremap`, and entries will be created for it in `/etc/passwd` and +`/etc/group` using your distro's standard user and group creation tools. + +> **Note**: The single mapping per-daemon restriction is in place for now +> because Docker shares image layers from its local cache across all +> containers running on the engine instance. Since file ownership must be +> the same for all containers sharing the same layer content, the decision +> was made to map the file ownership on `docker pull` to the daemon's user and +> group mappings so that there is no delay for running containers once the +> content is downloaded. This design preserves the same performance for `docker +> pull`, `docker push`, and container startup as users expect with +> user namespaces disabled. + +### Starting the daemon with user namespaces enabled + +To enable user namespace support, start the daemon with the +`--userns-remap` flag, which accepts values in the following formats: + + - uid + - uid:gid + - username + - username:groupname + +If numeric IDs are provided, translation back to valid user or group names +will occur so that the subordinate uid and gid information can be read, given +these resources are name-based, not id-based. If the numeric ID information +provided does not exist as entries in `/etc/passwd` or `/etc/group`, daemon +startup will fail with an error message. + +> **Note:** On Fedora 22, you have to `touch` the `/etc/subuid` and `/etc/subgid` +> files to have ranges assigned when users are created. This must be done +> *before* the `--userns-remap` option is enabled. Once these files exist, the +> daemon can be (re)started and range assignment on user creation works properly. + +*Example: starting with default Docker user management:* + +```bash +$ dockerd --userns-remap=default +``` + +When `default` is provided, Docker will create - or find the existing - user and group +named `dockremap`. If the user is created, and the Linux distribution has +appropriate support, the `/etc/subuid` and `/etc/subgid` files will be populated +with a contiguous 65536 length range of subordinate user and group IDs, starting +at an offset based on prior entries in those files. For example, Ubuntu will +create the following range, based on an existing user named `user1` already owning +the first 65536 range: + +```bash +$ cat /etc/subuid +user1:100000:65536 +dockremap:165536:65536 +``` + +If you have a preferred/self-managed user with subordinate ID mappings already +configured, you can provide that username or uid to the `--userns-remap` flag. +If you have a group that doesn't match the username, you may provide the `gid` +or group name as well; otherwise the username will be used as the group name +when querying the system for the subordinate group ID range. + +### Detailed information on `subuid`/`subgid` ranges + +Given potential advanced use of the subordinate ID ranges by power users, the +following paragraphs define how the Docker daemon currently uses the range entries +found within the subordinate range files. + +The simplest case is that only one contiguous range is defined for the +provided user or group. In this case, Docker will use that entire contiguous +range for the mapping of host uids and gids to the container process. This +means that the first ID in the range will be the remapped root user, and the +IDs above that initial ID will map host ID 1 through the end of the range. + +From the example `/etc/subuid` content shown above, the remapped root +user would be uid 165536. + +If the system administrator has set up multiple ranges for a single user or +group, the Docker daemon will read all the available ranges and use the +following algorithm to create the mapping ranges: + +1. The range segments found for the particular user will be sorted by *start ID* ascending. +2. Map segments will be created from each range in increasing value with a length matching the length of each segment. Therefore the range segment with the lowest numeric starting value will be equal to the remapped root, and continue up through host uid/gid equal to the range segment length. As an example, if the lowest segment starts at ID 1000 and has a length of 100, then a map of 1000 -> 0 (the remapped root) up through 1100 -> 100 will be created from this segment. If the next segment starts at ID 10000, then the next map will start with mapping 10000 -> 101 up to the length of this second segment. This will continue until no more segments are found in the subordinate files for this user. +3. If more than five range segments exist for a single user, only the first five will be utilized, matching the kernel's limitation of only five entries in `/proc/self/uid_map` and `proc/self/gid_map`. + +### Disable user namespace for a container + +If you enable user namespaces on the daemon, all containers are started +with user namespaces enabled. In some situations you might want to disable +this feature for a container, for example, to start a privileged container (see +[user namespace known restrictions](#user-namespace-known-restrictions)). +To enable those advanced features for a specific container use `--userns=host` +in the `run/exec/create` command. +This option will completely disable user namespace mapping for the container's user. + +### User namespace known restrictions + +The following standard Docker features are currently incompatible when +running a Docker daemon with user namespaces enabled: + + - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`) + - A `--readonly` container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace) + - external (volume or graph) drivers which are unaware/incapable of using daemon user mappings + - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`) + +In general, user namespaces are an advanced feature and will require +coordination with other capabilities. For example, if volumes are mounted from +the host, file ownership will have to be pre-arranged if the user or +administrator wishes the containers to have expected access to the volume +contents. + +Finally, while the `root` user inside a user namespaced container process has +many of the expected admin privileges that go along with being the superuser, the +Linux kernel has restrictions based on internal knowledge that this is a user namespaced +process. The most notable restriction that we are aware of at this time is the +inability to use `mknod`. Permission will be denied for device creation even as +container `root` inside a user namespace. + +## Miscellaneous options + +IP masquerading uses address translation to allow containers without a public +IP to talk to other machines on the Internet. This may interfere with some +network topologies and can be disabled with `--ip-masq=false`. + +Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and +for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be +set like this: + + DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 + # or + export DOCKER_TMPDIR=/mnt/disk2/tmp + /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 + + +## Default cgroup parent + +The `--cgroup-parent` option allows you to set the default cgroup parent +to use for containers. If this option is not set, it defaults to `/docker` for +fs cgroup driver and `system.slice` for systemd cgroup driver. + +If the cgroup has a leading forward slash (`/`), the cgroup is created +under the root cgroup, otherwise the cgroup is created under the daemon +cgroup. + +Assuming the daemon is running in cgroup `daemoncgroup`, +`--cgroup-parent=/foobar` creates a cgroup in +`/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar` +creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar` + +The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd +represents hierarchy by slice and the name of the slice encodes the location in +the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A +name can consist of a dash-separated series of names, which describes the path +to the slice from the root slice. For example, `--cgroup-parent=user-a-b.slice` +means the memory cgroup for the container is created in +`/sys/fs/cgroup/memory/user.slice/user-a.slice/user-a-b.slice/docker-.scope`. + +This setting can also be set per container, using the `--cgroup-parent` +option on `docker create` and `docker run`, and takes precedence over +the `--cgroup-parent` option on the daemon. + +## Daemon configuration file + +The `--config-file` option allows you to set any configuration option +for the daemon in a JSON format. This file uses the same flag names as keys, +except for flags that allow several entries, where it uses the plural +of the flag name, e.g., `labels` for the `label` flag. By default, +docker tries to load a configuration file from `/etc/docker/daemon.json` +on Linux and `%programdata%\docker\config\daemon.json` on Windows. + +The options set in the configuration file must not conflict with options set +via flags. The docker daemon fails to start if an option is duplicated between +the file and the flags, regardless their value. We do this to avoid +silently ignore changes introduced in configuration reloads. +For example, the daemon fails to start if you set daemon labels +in the configuration file and also set daemon labels via the `--label` flag. + +Options that are not present in the file are ignored when the daemon starts. +This is a full example of the allowed configuration options in the file: + +```json +{ + "authorization-plugins": [], + "dns": [], + "dns-opts": [], + "dns-search": [], + "exec-opts": [], + "exec-root": "", + "storage-driver": "", + "storage-opts": [], + "labels": [], + "log-driver": "", + "log-opts": [], + "mtu": 0, + "pidfile": "", + "graph": "", + "cluster-store": "", + "cluster-store-opts": [], + "cluster-advertise": "", + "debug": true, + "hosts": [], + "log-level": "", + "tls": true, + "tlsverify": true, + "tlscacert": "", + "tlscert": "", + "tlskey": "", + "api-cors-headers": "", + "selinux-enabled": false, + "userns-remap": "", + "group": "", + "cgroup-parent": "", + "default-ulimits": {}, + "ipv6": false, + "iptables": false, + "ip-forward": false, + "ip-mask": false, + "userland-proxy": false, + "ip": "0.0.0.0", + "bridge": "", + "bip": "", + "fixed-cidr": "", + "fixed-cidr-v6": "", + "default-gateway": "", + "default-gateway-v6": "", + "icc": false, + "raw-logs": false, + "registry-mirrors": [], + "insecure-registries": [], + "disable-legacy-registry": false +} +``` + +### Configuration reloading + +Some options can be reconfigured when the daemon is running without requiring +to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event +in Windows with the key `Global\docker-daemon-config-$PID`. The options can +be modified in the configuration file but still will check for conflicts with +the provided flags. The daemon fails to reconfigure itself +if there are conflicts, but it won't stop execution. + +The list of currently supported options that can be reconfigured is this: + +- `debug`: it changes the daemon to debug mode when set to true. +- `cluster-store`: it reloads the discovery store with the new address. +- `cluster-store-opts`: it uses the new options to reload the discovery store. +- `cluster-advertise`: it modifies the address advertised after reloading. +- `labels`: it replaces the daemon labels with a new set of labels. + +Updating and reloading the cluster configurations such as `--cluster-store`, +`--cluster-advertise` and `--cluster-store-opts` will take effect only if +these configurations were not previously configured. If `--cluster-store` +has been provided in flags and `cluster-advertise` not, `cluster-advertise` +can be added in the configuration file without accompanied by `--cluster-store` +Configuration reload will log a warning message if it detects a change in +previously configured cluster configurations. From d2d962bd30e8648abeb65ef50ed3d445f29630c1 Mon Sep 17 00:00:00 2001 From: Zhu Guihua Date: Thu, 24 Mar 2016 09:14:38 +0800 Subject: [PATCH 067/534] Add disk quota support for btrfs Signed-off-by: Zhu Guihua --- docs/reference/commandline/dockerd.md | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 842e051664..2ecadbed5c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -185,8 +185,10 @@ Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). Call ### Storage driver options Particular storage-driver can be configured with options specified with -`--storage-opt` flags. Options for `devicemapper` are prefixed with `dm` and -options for `zfs` start with `zfs`. +`--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, +options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. + +#### Devicemapper options * `dm.thinpooldev` @@ -470,7 +472,7 @@ options for `zfs` start with `zfs`. $ dockerd --storage-opt dm.min_free_space=10% ``` -Currently supported options of `zfs`: +#### ZFS options * `zfs.fsname` @@ -482,6 +484,18 @@ Currently supported options of `zfs`: $ dockerd -s zfs --storage-opt zfs.fsname=zroot/docker +#### Btrfs options + +* `btrfs.min_space` + + Specifies the mininum size to use when creating the subvolume which is used + for containers. If user uses disk quota for btrfs when creating or running + a container with **--storage-opt size** option, docker should ensure the + **size** cannot be smaller than **btrfs.min_space**. + + Example use: + $ docker daemon -s btrfs --storage-opt btrfs.min_space=10G + ## Docker runtime execution options The Docker daemon relies on a From 1ac91f58610ff1b50a6303fe70859fd25b8eefdf Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Mon, 7 Mar 2016 21:41:44 -0500 Subject: [PATCH 068/534] When calling volume driver Mount, send opaque ID This generates an ID string for calls to Mount/Unmount, allowing drivers to differentiate between two callers of `Mount` and `Unmount`. Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 88798497c3..5c2ebb49bc 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -115,7 +115,8 @@ Respond with a string error if an error occurred. **Request**: ```json { - "Name": "volume_name" + "Name": "volume_name", + "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" } ``` @@ -124,6 +125,8 @@ name. This is called once per container start. If the same volume_name is reques more than once, the plugin may need to keep track of each new mount request and provision at the first mount request and deprovision at the last corresponding unmount request. +`ID` is a unqiue ID for the caller that is requesting the mount. + **Response**: ```json { @@ -162,7 +165,8 @@ available, and/or a string error if an error occurred. **Request**: ```json { - "Name": "volume_name" + "Name": "volume_name", + "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" } ``` @@ -170,6 +174,8 @@ Indication that Docker no longer is using the named volume. This is called once per container stop. Plugin may deduce that it is safe to deprovision it at this point. +`ID` is a unqiue ID for the caller that is requesting the mount. + **Response**: ```json { From bb8a711260c9a6d9dd9cbe6de9114082788b99cc Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sat, 7 May 2016 08:46:15 -0700 Subject: [PATCH 069/534] Add the missing subtitle in deprecated docs for --security-opt. The colon separator(`:`) of `--security-opt` flag was deprecated in 1.11.0. However, the subtitle in deprecated docs is missing so it is placed under the same subtitle as the deprecated `-e` and `--email` flags. This fix adds the missing subtitle in deprecated docs. Signed-off-by: Yong Tang --- docs/deprecated.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index e9b3d73071..58668d25d8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -21,6 +21,11 @@ The following list of features are deprecated in Engine. The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. +### Separator (`:`) of `--security-opt` flag on `docker run` +**Deprecated In Release: v1.11** + +**Target For Removal In Release: v1.13** + The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. ### Ambiguous event fields in API From 7cd054570535160c2e352593d7700c51328c5e36 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sat, 7 May 2016 09:26:34 -0700 Subject: [PATCH 070/534] Remove deprecated Docker Content Trust ENV passphrase variables Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. The corresponding environment variables `DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE` `DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE` have also been deprecated and renamed to `DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE` `DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE` This fix removed the deprecated ENV passphrase variables for 1.12 and updated the docs. Signed-off-by: Yong Tang --- docs/deprecated.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 58668d25d8..6b438487d8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -145,9 +145,9 @@ Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the d ### Docker Content Trust ENV passphrase variables name change **Deprecated In Release: v1.9** -**Target For Removal In Release: v1.10** +**Removed In Release: v1.12** -As of 1.9, Docker Content Trust Offline key will be renamed to Root key and the Tagging key will be renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables +Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables -- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE will now be named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE -- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE will now be named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE +- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE +- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE From 9f86016de1e839447d23d7dfcb9fe94a5f4568b3 Mon Sep 17 00:00:00 2001 From: cyli Date: Mon, 9 May 2016 10:29:42 -0700 Subject: [PATCH 071/534] Minor tense correction for deprecated docs Signed-off-by: cyli --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 6b438487d8..74c0833207 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -149,5 +149,5 @@ Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the d Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables -- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE -- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE +- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE +- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE From ef29b77883533d2be9d0e6414d01386c0018f273 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Wed, 11 May 2016 06:48:07 -0700 Subject: [PATCH 072/534] Update deprecated docs for cli flags removal. The old command line options have been deprecated in 1.8.0 and eventually removed in 1.10.0 through PR reference 17724, though the deprecated.md still shows `Target For Removal In Release`. This fix updates the deprecated.md and changes `Target For Removal In Release` to `Removed In Release`. Signed-off-by: Yong Tang --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 74c0833207..f61ffd3184 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -92,7 +92,7 @@ The lxc-conf flag and API fields will also be removed. ### Old Command Line Options **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Target For Removal In Release: v1.10** +**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: From b97c9ca7416b9b52eb19ddeb2c661d40ef2750d2 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Mon, 9 May 2016 19:04:09 -0700 Subject: [PATCH 073/534] Remove deprecated driver specific log tags Since 1.9, driver specific log tag options `syslog-tag` `gelf-tag` `fluentd-tag` have been deprecated in favor of the generic tag option which is standard across different logging drivers. This fix removed the deprecated driver specific log tag options of `syslog-tag`, `gelf-tag`, `fluentd-tag` for 1.12 and updated the docs. Signed-off-by: Yong Tang --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index f61ffd3184..5c85c18a18 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -73,7 +73,7 @@ variants: ### Driver Specific Log Tags **Deprecated In Release: v1.9** -**Target For Removal In Release: v1.11** +**Removed In Release: v1.12** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and From 9342eb5917774cb27bd80a239ff999ab87bffbca Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Thu, 5 May 2016 21:45:55 -0700 Subject: [PATCH 074/534] Docker pull/push with max concurrency limits. This fix tries to address issues raised in reference 20936 and reference 22443 where `docker pull` or `docker push` fails because of the concurrent connection failing. Currently, the number of maximum concurrent connections is controlled by `maxDownloadConcurrency` and `maxUploadConcurrency` which are hardcoded to 3 and 5 respectively. Therefore, in situations where network connections don't support multiple downloads/uploads, failures may encounter for `docker push` or `docker pull`. This fix tries changes `maxDownloadConcurrency` and `maxUploadConcurrency` to adjustable by passing `--max-concurrent-uploads` and `--max-concurrent-downloads` to `docker daemon` command. The documentation related to docker daemon has been updated. Additional test case have been added to cover the changes in this fix. This fix fixes reference 20936. This fix fixes reference 22443. Signed-off-by: Yong Tang --- docs/reference/commandline/dockerd.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2ecadbed5c..c3caa10fcd 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -54,6 +54,8 @@ weight = -1 --log-driver="json-file" Default driver for container logs --log-opt=[] Log driver specific options --mtu=0 Set the containers network MTU + --max-concurrent-downloads=3 Set the max concurrent downloads for each pull + --max-concurrent-uploads=5 Set the max concurrent uploads for each push --disable-legacy-registry Do not contact legacy registries -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file --raw-logs Full timestamps without ANSI coloring @@ -913,6 +915,8 @@ This is a full example of the allowed configuration options in the file: "cluster-store": "", "cluster-store-opts": [], "cluster-advertise": "", + "max-concurrent-downloads": 3, + "max-concurrent-uploads": 5, "debug": true, "hosts": [], "log-level": "", @@ -963,6 +967,8 @@ The list of currently supported options that can be reconfigured is this: - `cluster-store-opts`: it uses the new options to reload the discovery store. - `cluster-advertise`: it modifies the address advertised after reloading. - `labels`: it replaces the daemon labels with a new set of labels. +- `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. +- `max-concurrent-uploads`: it updates the max concurrent uploads for each push. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From 80b8ab36940ed41d3580ca4b80223118c17b0a1f Mon Sep 17 00:00:00 2001 From: Yi EungJun Date: Thu, 12 May 2016 14:26:39 +0900 Subject: [PATCH 075/534] docs: Remove RequestStatusCode The requests doesn't include RequestStatusCode field. Signed-off-by: Yi EungJun --- docs/extend/plugins_authorization.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 140db71c35..07840b144d 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -156,8 +156,7 @@ should implement the following two methods: "RequestMethod": "The HTTP method", "RequestURI": "The HTTP request URI", "RequestBody": "Byte array containing the raw HTTP request body", - "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string ", - "RequestStatusCode": "Request status code" + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string " } ``` @@ -182,7 +181,6 @@ should implement the following two methods: "RequestURI": "The HTTP request URI", "RequestBody": "Byte array containing the raw HTTP request body", "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string", - "RequestStatusCode": "Request status code", "ResponseBody": "Byte array containing the raw HTTP response body", "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string", "ResponseStatusCode":"Response status code" From 453b4974bb573e84e714fd8527d35c2acb337a1f Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sat, 14 May 2016 16:55:01 -0700 Subject: [PATCH 076/534] Update deprecated docs for LXC built-in exec driver The LXC built-in exec driver has been deprecated in 1.8 and further removed in 1.10, yet in deprecated.md it still shows: ``` Target For Removal In Release: v1.10 ``` This fix changes the above to `Removed In Release:`. In addition, lxc-conf flag and API fields have already been removed in 1.10 as well so the related description has also been updated in this fix. Signed-off-by: Yong Tang --- docs/deprecated.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 5c85c18a18..df87118478 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -84,10 +84,9 @@ Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` a ### LXC built-in exec driver **Deprecated In Release: v1.8** -**Target For Removal In Release: v1.10** +**Removed In Release: v1.10** -The built-in LXC execution driver is deprecated for an external implementation. -The lxc-conf flag and API fields will also be removed. +The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. ### Old Command Line Options **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** From e70ab70a83eef1e9796bee27d67a690e81d73d66 Mon Sep 17 00:00:00 2001 From: Jeremy Unruh Date: Sun, 15 May 2016 21:20:54 -0700 Subject: [PATCH 077/534] update URL for docker-volume-netshare Signed-off-by: Jeremy Unruh --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index e40c611282..7a3687fced 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -60,7 +60,7 @@ Plugin [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. -[Netshare plugin](https://github.com/gondor/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. +[Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. From 6fe553f22b602d81b8e4f99e5f7eda1890c0a60c Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Tue, 12 Apr 2016 17:21:20 +0200 Subject: [PATCH 078/534] Ignore invalid host header between go1.6 and old docker clients BenchmarkWithHack-4 50000 37082 ns/op 44.50 MB/s 1920 B/op 30 allocs/op BenchmarkNoHack-4 50000 30829 ns/op 53.52 MB/s 0 B/op 0 allocs/op Signed-off-by: Brian Goff Signed-off-by: Antonio Murdaca --- docs/reference/commandline/dockerd.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index c3caa10fcd..9238388880 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -849,6 +849,19 @@ set like this: export DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +Docker clients <= 1.9.2 used an invalid Host header when making request to the +daemon. Docker 1.12 is built using golang 1.6 which is now checking the validity +of the Host header and as such clients <= 1.9.2 can't talk anymore to the daemon. +Docker supports overcoming this issue via a Docker daemon +environment variable. In case you are seeing this error when contacting the +daemon: + + Error response from daemon: 400 Bad Request: malformed Host header + +The `DOCKER_HTTP_HOST_COMPAT` can be set like this: + + DOCKER_HTTP_HOST_COMPAT=1 /usr/local/bin/dockerd ... + ## Default cgroup parent From a8ff0d27f9bcd0f5260c09ebd10787eedbf39bc7 Mon Sep 17 00:00:00 2001 From: Fabrizio Soppelsa Date: Fri, 20 May 2016 13:41:28 +0200 Subject: [PATCH 079/534] Add a --filter option to `docker search` The filtering is made server-side, and the following filters are supported: * is-official (boolean) * is-automated (boolean) * has-stars (integer) Signed-off-by: Fabrizio Soppelsa Signed-off-by: Vincent Demeester --- docs/deprecated.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index df87118478..d80f24e729 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -58,6 +58,15 @@ defining it at container creation (`POST /containers/create`). The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. +### Docker search 'automated' and 'stars' options + +**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +**Removed In Release: v1.14** + +The `docker search --automated` and `docker search --stars` options are deprecated. +Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. + ### Command line short variant options **Deprecated In Release: v1.9** From 58dc5eda0ad86290a21609326de30522721dea1c Mon Sep 17 00:00:00 2001 From: John Howard Date: Thu, 14 Apr 2016 17:12:02 -0700 Subject: [PATCH 080/534] Windows: Default to Hyper-V Containers on client Signed-off-by: John Howard --- docs/reference/commandline/dockerd.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 9238388880..5aece25c06 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -527,8 +527,9 @@ can specify default container isolation technology with this, for example: $ dockerd --exec-opt isolation=hyperv -Will make `hyperv` the default isolation technology on Windows, without specifying -isolation value on daemon start, Windows isolation technology will default to `process`. +Will make `hyperv` the default isolation technology on Windows. If no isolation +value is specified on daemon start, on Windows client, the default is +`hyperv`, and on Windows server, the default is `process`. ## Daemon DNS options From 9804348999c159733a07b73418915b5c41a89ac7 Mon Sep 17 00:00:00 2001 From: root Date: Mon, 16 May 2016 11:18:48 +0800 Subject: [PATCH 081/534] Fix some mistakes in dockerd.md Signed-off-by: Wang Xing --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5aece25c06..2b90df9277 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -939,7 +939,7 @@ This is a full example of the allowed configuration options in the file: "tlscacert": "", "tlscert": "", "tlskey": "", - "api-cors-headers": "", + "api-cors-header": "", "selinux-enabled": false, "userns-remap": "", "group": "", @@ -948,7 +948,7 @@ This is a full example of the allowed configuration options in the file: "ipv6": false, "iptables": false, "ip-forward": false, - "ip-mask": false, + "ip-masq": false, "userland-proxy": false, "ip": "0.0.0.0", "bridge": "", From a7e1ee810c70c2848295a5df497848c7c11453c6 Mon Sep 17 00:00:00 2001 From: Darren Shepherd Date: Sat, 21 May 2016 20:40:15 -0700 Subject: [PATCH 082/534] Remove DOCKER_HTTP_HOST_COMPAT env var Signed-off-by: Darren Shepherd --- docs/reference/commandline/dockerd.md | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2b90df9277..c425f92c63 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -850,20 +850,6 @@ set like this: export DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 -Docker clients <= 1.9.2 used an invalid Host header when making request to the -daemon. Docker 1.12 is built using golang 1.6 which is now checking the validity -of the Host header and as such clients <= 1.9.2 can't talk anymore to the daemon. -Docker supports overcoming this issue via a Docker daemon -environment variable. In case you are seeing this error when contacting the -daemon: - - Error response from daemon: 400 Bad Request: malformed Host header - -The `DOCKER_HTTP_HOST_COMPAT` can be set like this: - - DOCKER_HTTP_HOST_COMPAT=1 /usr/local/bin/dockerd ... - - ## Default cgroup parent The `--cgroup-parent` option allows you to set the default cgroup parent From e05854f9134c0c9f8b1a18b404fc4ef8eaa4387a Mon Sep 17 00:00:00 2001 From: Sven Dowideit Date: Fri, 27 May 2016 00:28:46 +0000 Subject: [PATCH 083/534] Fix up stale links Signed-off-by: Sven Dowideit --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 7a3687fced..7881c73c29 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -43,7 +43,7 @@ Plugin ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. [Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. -[Weave Network Plugin](http://docs.weave.works/weave/latest_release/plugin.html) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. +[Weave Network Plugin](https://www.weave.works/docs/net/latest/introducing-weave/) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. ### Volume plugins From 6db613a9ea0fdae118b58cd4a2f2dfa659660fd7 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Thu, 26 May 2016 20:04:48 -0700 Subject: [PATCH 084/534] Fix error in dockerd.md for incorrect cluster-store-opts example. This fix fixes an error in documentation (dockerd.md). In the example given by dockerd.md, the option `cluster-store-opts` is assigned with an array but this option can only be assigned as a map. Signed-off-by: Yong Tang --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index c425f92c63..1655b132be 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -913,7 +913,7 @@ This is a full example of the allowed configuration options in the file: "pidfile": "", "graph": "", "cluster-store": "", - "cluster-store-opts": [], + "cluster-store-opts": {}, "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, From 6daf6539a37be214abf37f3313f773de3613955f Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Mon, 9 May 2016 19:07:04 -0700 Subject: [PATCH 085/534] Un-deprecated command line short variant options of `-c`. Since 1.9, the following short variant options have been deprecated in favor of their long variants: `docker run -c (--cpu-shares)` `docker build -c (--cpu-shares)` `docker create -c (--cpu-shares)` `docker update -c (--cpu-shares)` However, `-c` is still widely used and is considered as a convenient option for swarm (see reference 16271). This fix undeprecated the command line short variant options of `-c` and updated the deprecated.md. Signed-off-by: Yong Tang --- docs/deprecated.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index d80f24e729..a919a5d54f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -67,18 +67,6 @@ Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. -### Command line short variant options -**Deprecated In Release: v1.9** - -**Target For Removal In Release: v1.11** - -The following short variant options are deprecated in favor of their long -variants: - - docker run -c (--cpu-shares) - docker build -c (--cpu-shares) - docker create -c (--cpu-shares) - ### Driver Specific Log Tags **Deprecated In Release: v1.9** From 467a43b336f1898a2639f175888c1d9628b6a5a1 Mon Sep 17 00:00:00 2001 From: Roland Kammerer Date: Mon, 30 May 2016 17:45:05 +0200 Subject: [PATCH 086/534] Add the DRBD Docker Volume Plugin to the documentation Signed-off-by: Roland Kammerer --- docs/extend/plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 7881c73c29..fe80ac6d98 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -52,6 +52,7 @@ Plugin [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. [Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. +[DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. [Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. [gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. From 693cf57001dedf5717c24fa4b0f560797fe42296 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sun, 29 May 2016 08:16:39 -0700 Subject: [PATCH 087/534] Remove deprecated -f flag on docker tag The -f flag on docker tag has been deprecated in docker 1.10 and is expected to be removed in docker 1.12. This fix removed the -f flag on docker tag and also updated deprecated.md. NOTE: A separate pull request for engine-api has been opened to cover the related changes. Signed-off-by: Yong Tang --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a919a5d54f..9de65b28c2 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -35,9 +35,9 @@ The fields `ID`, `Status` and `From` in the events API have been deprecated in f See the events API documentation for the new format. ### `-f` flag on `docker tag` -**Deprecated In Release: v1.10** +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Target For Removal In Release: v1.12** +**Removed In Release: v1.12.0** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. From dbd9095c1251da9fc5119862869c2bb482b658c9 Mon Sep 17 00:00:00 2001 From: Ritesh H Shukla Date: Wed, 1 Jun 2016 15:29:15 -0700 Subject: [PATCH 088/534] Add VMware Docker Volume Plugin. Add reference to https://github.com/vmware/docker-volume-vsphere to Docker's list of plugins. This is an officially supported plugin from VMware. Signed-off-by: Ritesh H Shukla --- docs/extend/plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index fe80ac6d98..222a2dbf65 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -65,6 +65,7 @@ Plugin [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. +[VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. ### Authorization plugins From b217451c6117e5b3946fb955bdcede8c610a16ba Mon Sep 17 00:00:00 2001 From: Lukasz Zajaczkowski Date: Mon, 16 May 2016 12:41:07 +0200 Subject: [PATCH 089/534] Add documentation for running multiple daemons Signed-off-by: Lukasz Zajaczkowski --- docs/reference/commandline/dockerd.md | 56 +++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1655b132be..2dfbb68002 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -977,3 +977,59 @@ has been provided in flags and `cluster-advertise` not, `cluster-advertise` can be added in the configuration file without accompanied by `--cluster-store` Configuration reload will log a warning message if it detects a change in previously configured cluster configurations. + + +## Running multiple daemons + +> **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of +> unsolved problems. This solution may not work properly in some cases. Solutions are currently under development +> and will be delivered in the near future. + +This section describes how to run multiple Docker daemons on a single host. To +run multiple daemons, you must configure each daemon so that it does not +conflict with other daemons on the same host. You can set these options either +by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file). + +The following daemon options must be configured for each daemon: + +```bash +-b, --bridge= Attach containers to a network bridge +--exec-root=/var/run/docker Root of the Docker execdriver +-g, --graph=/var/lib/docker Root of the Docker runtime +-p, --pidfile=/var/run/docker.pid Path to use for daemon PID file +-H, --host=[] Daemon socket(s) to connect to +--config-file=/etc/docker/daemon.json Daemon configuration file +--tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA +--tlscert="~/.docker/cert.pem" Path to TLS certificate file +--tlskey="~/.docker/key.pem" Path to TLS key file +``` + +When your daemons use different values for these flags, you can run them on the same host without any problems. +It is very important to properly understand the meaning of those options and to use them correctly. + +- The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker. +If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none` +- `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for +your running daemon here. +- `--graph` is the path where images are stored. The default value is `/var/lib/docker`. To avoid any conflict with other daemons +set this parameter separately for each daemon. +- `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your +pid file here. +- `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. +- `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of +daemon flags. Specify the path for each daemon. +- `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections. +The `--tls*` options enable use of specific certificates for individual daemons. + +Example script for a separate “bootstrap” instance of the Docker daemon without network: + +```bash +$ docker daemon \ + -H unix:///var/run/docker-bootstrap.sock \ + -p /var/run/docker-bootstrap.pid \ + --iptables=false \ + --ip-masq=false \ + --bridge=none \ + --graph=/var/lib/docker-bootstrap \ + --exec-root=/var/run/docker-bootstrap +``` From 086994ef0f86dfc3ae7db14daa5eb9888c836225 Mon Sep 17 00:00:00 2001 From: allencloud Date: Sun, 8 May 2016 09:36:10 +0800 Subject: [PATCH 090/534] fix typos Signed-off-by: allencloud --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2dfbb68002..8eab487773 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -587,9 +587,9 @@ system's list of trusted CAs instead of enabling `--insecure-registry`. Enabling `--disable-legacy-registry` forces a docker daemon to only interact with registries which support the V2 protocol. Specifically, the daemon will not attempt `push`, `pull` and `login` to v1 registries. The exception to this is `search` which can still be performed on v1 registries. -## Running a Docker daemon behind a HTTPS_PROXY +## Running a Docker daemon behind an HTTPS_PROXY -When running inside a LAN that uses a `HTTPS` proxy, the Docker Hub +When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub certificates will be replaced by the proxy's certificates. These certificates need to be added to your Docker host's configuration: From a7473b8f019639478e74ab1b6dd1c50d32197561 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Thu, 2 Jun 2016 19:54:35 -0700 Subject: [PATCH 091/534] Add missing links in deprecated.md doc. This fix tries to address several issues in deprecated.md: 1. For deprecated and removal versions, some include link reference to the release tag but some does not point to the release tag. This fix adds the missing links as long as the version is <= 1.12. 2. Technically, 1.12 is not released yet so the link to 1.12 does not exist yet. However, at the time 1.12 is released this deprecated.md doc should have been part of the release as well. There is a circular dependency. This fix adds 1.12 for now. 3. `HostConfig at API container start` has already been removed by reference 22570 so this fix changes `Target For Removal In Release: v1.12` to `Removed In Release: v1.12`. 4. `Docker search 'automated' and 'stars' options` has not been removed yet so this fix changes `Removed In Release: v1.14` to `Target For Removal In Release: v1.14` Signed-off-by: Yong Tang --- docs/deprecated.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9de65b28c2..2710a52b48 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -15,21 +15,21 @@ weight=80 The following list of features are deprecated in Engine. ### `-e` and `--email` flags on `docker login` -**Deprecated In Release: v1.11** +**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Target For Removal In Release: v1.13** The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` -**Deprecated In Release: v1.11** +**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Target For Removal In Release: v1.13** The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. ### Ambiguous event fields in API -**Deprecated In Release: v1.10** +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. See the events API documentation for the new format. @@ -37,14 +37,14 @@ See the events API documentation for the new format. ### `-f` flag on `docker tag` **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: v1.12.0** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start -**Deprecated In Release: v1.10** +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Target For Removal In Release: v1.12** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of defining it at container creation (`POST /containers/create`). @@ -53,7 +53,7 @@ defining it at container creation (`POST /containers/create`). **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: v1.12** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. @@ -62,15 +62,15 @@ Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Removed In Release: v1.14** +**Target For Removal In Release: v1.14** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. ### Driver Specific Log Tags -**Deprecated In Release: v1.9** +**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: v1.12** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and @@ -79,9 +79,9 @@ Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` a docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ### LXC built-in exec driver -**Deprecated In Release: v1.8** +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: v1.10** +**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. @@ -139,9 +139,9 @@ The following double-dash options are deprecated and have no replacement: Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. ### Docker Content Trust ENV passphrase variables name change -**Deprecated In Release: v1.9** +**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: v1.12** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables From 95437ba7264a904decafd5430f28ca329c8feaaf Mon Sep 17 00:00:00 2001 From: Sven Dowideit Date: Fri, 3 Jun 2016 21:43:45 +0000 Subject: [PATCH 092/534] docs validation fixes Signed-off-by: Sven Dowideit --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8eab487773..a170e1c726 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -91,7 +91,7 @@ membership. If you need to access the Docker daemon remotely, you need to enable the `tcp` Socket. Beware that the default setup provides un-encrypted and un-authenticated direct access to the Docker daemon - and should be secured -either using the [built in HTTPS encrypted socket](../../security/https/), or by +either using the [built in HTTPS encrypted socket](../../security/https.md), or by putting a secure web proxy in front of it. You can listen on port `2375` on all network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network interface using its IP address: `-H tcp://192.168.59.103:2375`. It is From 65e0a8af3008a5aec06a396da24be5856f131f1c Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Sun, 5 Jun 2016 18:07:34 +0200 Subject: [PATCH 093/534] Deprecated the old 3-args form of `docker import` It's been deprecated since November 2013 and v0.6.7. Removing the cli side of it. Signed-off-by: Vincent Demeester --- docs/deprecated.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2710a52b48..c054af68ee 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -14,6 +14,13 @@ weight=80 The following list of features are deprecated in Engine. +### Three argument form in `docker import` +**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. + ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** From 9ea56da9906754f05c7f443999d49723006f97b6 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Mon, 11 Apr 2016 11:17:52 -0400 Subject: [PATCH 094/534] Add support for volume scopes This is similar to network scopes where a volume can either be `local` or `global`. A `global` volume is one that exists across the entire cluster where as a `local` volume exists on a single engine. Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 5c2ebb49bc..a70a2c3ee2 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -20,6 +20,7 @@ documentation](plugins.md) for more information. ### 1.12.0 - Add `Status` field to `VolumeDriver.Get` response ([#21006](https://github.com/docker/docker/pull/21006#)) +- Add `VolumeDriver.Capabilities` to get capabilities of the volume driver([#22077](https://github.com/docker/docker/pull/22077)) ### 1.10.0 @@ -236,3 +237,29 @@ Get the list of volumes registered with the plugin. ``` Respond with a string error if an error occurred. + +### /VolumeDriver.Capabilities + +**Request**: +```json +{} +``` + +Get the list of capabilities the driver supports. +The driver is not required to implement this endpoint, however in such cases +the default values will be taken. + +**Response**: +```json +{ + "Capabilities": { + "Scope": "global" + } +} +``` + +Supported scopes are `global` and `local`. Any other value in `Scope` will be +ignored and assumed to be `local`. Scope allows cluster managers to handle the +volume differently, for instance with a scope of `global`, the cluster manager +knows it only needs to create the volume once instead of on every engine. More +capabilities may be added in the future. From 082ebbd823b905f9e8ce0613b486246c65681175 Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Fri, 3 Jun 2016 19:38:03 +0200 Subject: [PATCH 095/534] Deprecate /containers/(id or name)/copy endpoint This endpoint has been deprecated since 1.8. Return an error starting from this API version (1.24) in order to make sure it's not used for the next API version and so that we can remove it some times later. Signed-off-by: Vincent Demeester --- docs/deprecated.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index c054af68ee..79e10c0bbc 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -35,6 +35,14 @@ The docker login command is removing the ability to automatically register for a The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. +### `/containers/(id or name)/copy` endpoint + +**Deprecated In Release: v1.8** + +**Removed In Release: v1.12.0** + +The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. + ### Ambiguous event fields in API **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** From ea142c7eaccb4c7cfd821750a62b092f9b0dfc8c Mon Sep 17 00:00:00 2001 From: Kirill Kolyshkin Date: Wed, 8 Jun 2016 11:24:56 -0700 Subject: [PATCH 096/534] docs/extend/plugins.md: add docker-volume-ploop Added docker-volume-ploop driver to the list of available volume plugins. Signed-off-by: Kir Kolyshkin --- docs/extend/plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 222a2dbf65..5bc29ef23c 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -65,6 +65,7 @@ Plugin [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. +[Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. [VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. ### Authorization plugins From 50792b5dd87fd72150c6879473fbbd4f6668fb27 Mon Sep 17 00:00:00 2001 From: Victoria Bialas Date: Tue, 7 Jun 2016 17:42:00 -0700 Subject: [PATCH 097/534] re-doing Docker Engine overview topics for v.1.12 fixed broken links created from Engine Overview update by adding missing topic to daemon reference page and updating the hrefs in the api pages Signed-off-by: Victoria Bialas --- docs/reference/commandline/dockerd.md | 64 ++++++++++++++++++++++++++- 1 file changed, 63 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a170e1c726..6888c14ef4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -139,6 +139,68 @@ The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes precedence over `HTTP_PROXY`. +### Bind Docker to another host/port or a Unix socket + +> **Warning**: +> Changing the default `docker` daemon binding to a +> TCP port or Unix *docker* user group will increase your security risks +> by allowing non-root users to gain *root* access on the host. Make sure +> you control access to `docker`. If you are binding +> to a TCP port, anyone with access to that port has full Docker access; +> so it is not advisable on an open network. + +With `-H` it is possible to make the Docker daemon to listen on a +specific IP and port. By default, it will listen on +`unix:///var/run/docker.sock` to allow only local connections by the +*root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP +to give access to everybody, but that is **not recommended** because +then it is trivial for someone to gain root access to the host where the +daemon is running. + +Similarly, the Docker client can use `-H` to connect to a custom port. +The Docker client will default to connecting to `unix:///var/run/docker.sock` +on Linux, and `tcp://127.0.0.1:2376` on Windows. + +`-H` accepts host and port assignment in the following format: + + tcp://[host]:[port][path] or unix://path + +For example: + +- `tcp://` -> TCP connection to `127.0.0.1` on either port `2376` when TLS encryption + is on, or port `2375` when communication is in plain text. +- `tcp://host:2375` -> TCP connection on + host:2375 +- `tcp://host:2375/path` -> TCP connection on + host:2375 and prepend path to all requests +- `unix://path/to/socket` -> Unix socket located + at `path/to/socket` + +`-H`, when empty, will default to the same value as +when no `-H` was passed in. + +`-H` also accepts short form for TCP bindings: + + `host:` or `host:port` or `:port` + +Run Docker in daemon mode: + + $ sudo /dockerd -H 0.0.0.0:5555 & + +Download an `ubuntu` image: + + $ docker -H :5555 pull ubuntu + +You can use multiple `-H`, for example, if you want to listen on both +TCP and a Unix socket + + # Run docker in daemon mode + $ sudo /dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & + # Download an ubuntu image, use default Unix socket + $ docker pull ubuntu + # OR use the TCP port + $ docker -H tcp://127.0.0.1:2375 pull ubuntu + ### Daemon storage-driver option The Docker daemon has support for several different image layer storage @@ -529,7 +591,7 @@ can specify default container isolation technology with this, for example: Will make `hyperv` the default isolation technology on Windows. If no isolation value is specified on daemon start, on Windows client, the default is -`hyperv`, and on Windows server, the default is `process`. +`hyperv`, and on Windows server, the default is `process`. ## Daemon DNS options From e9ba18ddacb87be2a9ea8667feba7d26d9276bb4 Mon Sep 17 00:00:00 2001 From: Ahmet Alp Balkan Date: Thu, 9 Jun 2016 13:47:15 -0700 Subject: [PATCH 098/534] docs: Add Azure File Storage Volume Driver plugin Signed-off-by: Ahmet Alp Balkan --- docs/extend/plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 5bc29ef23c..477163fa51 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -49,6 +49,7 @@ Plugin Plugin | Description ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +[Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. [Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. From 24fd2dbbefaac59494a519d81ac568b273b0acd8 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Sun, 12 Jun 2016 16:33:03 -0400 Subject: [PATCH 099/534] Note about mountpoint being optional Signed-off-by: Brian Goff --- docs/extend/plugins_volume.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index a70a2c3ee2..3b84d92f7a 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -159,7 +159,8 @@ Docker needs reminding of the path to the volume on the host. ``` Respond with the path on the host filesystem where the volume has been made -available, and/or a string error if an error occurred. +available, and/or a string error if an error occurred. `Mountpoint` is optional, +however the plugin may be queried again later if one is not provided. ### /VolumeDriver.Unmount @@ -211,7 +212,8 @@ Get the volume info. } ``` -Respond with a string error if an error occurred. +Respond with a string error if an error occurred. `Mountpoint` and `Status` are +optional. ### /VolumeDriver.List @@ -236,7 +238,7 @@ Get the list of volumes registered with the plugin. } ``` -Respond with a string error if an error occurred. +Respond with a string error if an error occurred. `Mountpoint` is optional. ### /VolumeDriver.Capabilities From d35eafe83c7503fa4dec683c9a044893c959863c Mon Sep 17 00:00:00 2001 From: Derek McGowan Date: Mon, 13 Jun 2016 11:06:47 -0700 Subject: [PATCH 100/534] Add documentation for using overlay2 Add mention in dockerd command line and storage driver selection documentation. Signed-off-by: Derek McGowan (github: dmcgowan) --- docs/reference/commandline/dockerd.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6888c14ef4..1d500433d4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -204,7 +204,7 @@ TCP and a Unix socket ### Daemon storage-driver option The Docker daemon has support for several different image layer storage -drivers: `aufs`, `devicemapper`, `btrfs`, `zfs` and `overlay`. +drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that is unlikely to be merged into the main kernel. These are also known to cause @@ -242,9 +242,14 @@ Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). Call > inode consumption (especially as the number of images grows), as well as > being incompatible with the use of RPMs. +The `overlay2` uses the same fast union filesystem but takes advantage of +[additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux +kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2` +to use it. + > **Note:** -> It is currently unsupported on `btrfs` or any Copy on Write filesystem -> and should only be used over `ext4` partitions. +> Both `overlay` and `overlay2` are currently unsupported on `btrfs` or any +> Copy on Write filesystem and should only be used over `ext4` partitions. ### Storage driver options From 052a3599f22c35b4d11fb082ec2d7b22546e3846 Mon Sep 17 00:00:00 2001 From: Kenfe-Mickael Laventure Date: Mon, 23 May 2016 14:49:50 -0700 Subject: [PATCH 101/534] Add support for multiples runtimes Signed-off-by: Kenfe-Mickael Laventure --- docs/reference/commandline/dockerd.md | 45 ++++++++++++++++++++++++++- 1 file changed, 44 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1d500433d4..ec44c0ab90 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -60,6 +60,7 @@ weight = -1 -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file --raw-logs Full timestamps without ANSI coloring --registry-mirror=[] Preferred Docker registry mirror + --add-runtime=[] Register an additional OCI compatible runtime -s, --storage-driver="" Storage driver to use --selinux-enabled Enable selinux support --storage-opt=[] Set storage driver options @@ -572,6 +573,31 @@ The Docker daemon relies on a (invoked via the `containerd` daemon) as its interface to the Linux kernel `namespaces`, `cgroups`, and `SELinux`. +Runtimes can be registered with the daemon either via the +configuration file or using the `--add-runtime` command line argument. + +The following is an example adding 2 runtimes via the configuration: +```json + "default-runtime": "runc", + "runtimes": { + "runc": { + "path": "runc" + }, + "custom": { + "path": "/usr/local/bin/my-runc-replacement", + "runtimeArgs": [ + "--debug" + ] + } + } +``` + +This is the same example via the command line: + + $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement + +**Note**: defining runtime arguments via the command line is not supported. + ## Options for the runtime You can configure the runtime using options specified @@ -1014,7 +1040,19 @@ This is a full example of the allowed configuration options in the file: "raw-logs": false, "registry-mirrors": [], "insecure-registries": [], - "disable-legacy-registry": false + "disable-legacy-registry": false, + "default-runtime": "runc", + "runtimes": { + "runc": { + "path": "runc" + }, + "custom": { + "path": "/usr/local/bin/my-runc-replacement", + "runtimeArgs": [ + "--debug" + ] + } + } } ``` @@ -1036,6 +1074,11 @@ The list of currently supported options that can be reconfigured is this: - `labels`: it replaces the daemon labels with a new set of labels. - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. - `max-concurrent-uploads`: it updates the max concurrent uploads for each push. +- `default-runtime`: it updates the runtime to be used if not is + specified at container creation. It defaults to "default" which is + the runtime shipped with the official docker packages. +- `runtimes`: it updates the list of available OCI runtimes that can + be used to run containers Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From 51792cbff986c670b982b6e6a780f8fa32433af1 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sun, 19 Jun 2016 06:13:57 -0700 Subject: [PATCH 102/534] Add release tag reference links in deprecated.md This fix adds a couple of missed release tag reference links in deprecated.md Signed-off-by: Yong Tang --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 79e10c0bbc..6f575639be 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -37,9 +37,9 @@ The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide ### `/containers/(id or name)/copy` endpoint -**Deprecated In Release: v1.8** +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: v1.12.0** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. From be3384c9f7cdb7435bef2d21e1feb6023cd8aa68 Mon Sep 17 00:00:00 2001 From: Neil Peterson Date: Thu, 16 Jun 2016 21:41:58 -0700 Subject: [PATCH 103/534] Added daemon.json Windows example Signed-off-by: Neil Peterson --- docs/reference/commandline/dockerd.md | 58 ++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 6 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ec44c0ab90..7109e0ee28 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -975,19 +975,23 @@ the `--cgroup-parent` option on the daemon. The `--config-file` option allows you to set any configuration option for the daemon in a JSON format. This file uses the same flag names as keys, except for flags that allow several entries, where it uses the plural -of the flag name, e.g., `labels` for the `label` flag. By default, -docker tries to load a configuration file from `/etc/docker/daemon.json` -on Linux and `%programdata%\docker\config\daemon.json` on Windows. +of the flag name, e.g., `labels` for the `label` flag. The options set in the configuration file must not conflict with options set via flags. The docker daemon fails to start if an option is duplicated between the file and the flags, regardless their value. We do this to avoid silently ignore changes introduced in configuration reloads. For example, the daemon fails to start if you set daemon labels -in the configuration file and also set daemon labels via the `--label` flag. - +in the configuration file and also set daemon labels via the `--label` flag. Options that are not present in the file are ignored when the daemon starts. -This is a full example of the allowed configuration options in the file: + +### Linux configuration file + +The default location of the configuration file on Linux is +`/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a + non-default location. + +This is a full example of the allowed configuration options on Linux: ```json { @@ -1056,6 +1060,48 @@ This is a full example of the allowed configuration options in the file: } ``` +### Windows configuration file + +The default location of the configuration file on Windows is + `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be + used to specify a non-default location. + +This is a full example of the allowed configuration options on Windows: + +```json +{ + "authorization-plugins": [], + "dns": [], + "dns-opts": [], + "dns-search": [], + "exec-opts": [], + "storage-driver": "", + "storage-opts": [], + "labels": [], + "log-driver": "", + "mtu": 0, + "pidfile": "", + "graph": "", + "cluster-store": "", + "cluster-advertise": "", + "debug": true, + "hosts": [], + "log-level": "", + "tlsverify": true, + "tlscacert": "", + "tlscert": "", + "tlskey": "", + "group": "", + "default-ulimits": {}, + "bridge": "", + "fixed-cidr": "", + "raw-logs": false, + "registry-mirrors": [], + "insecure-registries": [], + "disable-legacy-registry": false +} +``` + ### Configuration reloading Some options can be reconfigured when the daemon is running without requiring From 3b2dd80ea9b4bda2cd361b7d6e57612267b4c0fd Mon Sep 17 00:00:00 2001 From: Sven Dowideit Date: Wed, 22 Jun 2016 12:50:51 +0000 Subject: [PATCH 104/534] Removing some url links that can be resolved using src markdown links Signed-off-by: Sven Dowideit --- docs/extend/plugins_authorization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 07840b144d..16e453f86e 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -51,7 +51,7 @@ respectively. ## Default user authorization mechanism -If TLS is enabled in the [Docker daemon](https://docs.docker.com/engine/security/https/), the default user authorization flow extracts the user details from the certificate subject name. +If TLS is enabled in the [Docker daemon](../security/https.md), the default user authorization flow extracts the user details from the certificate subject name. That is, the `User` field is set to the client certificate subject common name, and the `AuthenticationMethod` field is set to `TLS`. ## Basic architecture From 82c007ca49bba21ad66bd7a2e1146dfdbb0643a8 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 25 May 2016 16:44:58 +0200 Subject: [PATCH 105/534] deprecate '-h' shorthand and remove '-help' This deprecates the shorthand '-h', because we don't mention it as an option, and it conflicts with the 'docker create -h/--hostname' option, so cannot be used for all commands consistently. This also removes the (single-dash) '-help' option. The single-dash variant was marked "deprecated" in version 1.5.0 (basically, since it was added in a2b529ead21e6ab9eafcb1b1d2437c725c43a06a), but still kept around to prevent '-help' being treated as '-h -e -l -p', causing confusing warnings. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 6f575639be..fdfb4e86c0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -21,6 +21,17 @@ The following list of features are deprecated in Engine. The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. +### `-h` shorthand for `--help` + +**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +**Target For Removal In Release: [v1.14.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The shorthand (`-h`) is less common than `--help` on Linux and cannot be used +on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on +`docker create`). For this reason, the `-h` shorthand was not printed in the +"usage" output of subcommands, nor docummented, and is now marked "deprecated". + ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** @@ -149,6 +160,16 @@ The following double-dash options are deprecated and have no replacement: docker ps --before-id docker search --trusted +**Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The single-dash (`-help`) was removed, in favor of the double-dash `--help` + + docker -help + docker [COMMAND] -help + + ### Interacting with V1 registries Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. From fd8cc8e8f19c63f5c69b7b19d65a0171b5815efa Mon Sep 17 00:00:00 2001 From: Keith Hudgins Date: Thu, 30 Jun 2016 14:07:03 -0400 Subject: [PATCH 106/534] Added HPE volume plugin listing Signed-off-by: Keith Hudgins --- docs/extend/plugins.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 477163fa51..8d5616f2cb 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -58,6 +58,7 @@ Plugin [gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. +[HPE 3Par Volume Plugin](https://github.com/hpe-storage/python-hpedockerplugin/) | A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage arrays. [IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. @@ -67,7 +68,7 @@ Plugin [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. [Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. -[VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. +[VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. ### Authorization plugins From 6b021376a7c8a2189a54434348a3788b4714c7af Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 1 Jul 2016 17:08:59 -0700 Subject: [PATCH 107/534] fix typo in "readonly" flag in documentation the flag is named '--read-only', not '--readonly' Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 7109e0ee28..9deb06cfec 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -911,7 +911,7 @@ The following standard Docker features are currently incompatible when running a Docker daemon with user namespaces enabled: - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`) - - A `--readonly` container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace) + - A `--read-only` container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace) - external (volume or graph) drivers which are unaware/incapable of using daemon user mappings - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`) From 03558dde5392e8098b989550ceb2996473070459 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Sat, 2 Jul 2016 16:22:23 +0200 Subject: [PATCH 108/534] man: add missing --add-runtime Signed-off-by: Antonio Murdaca --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 9deb06cfec..5eb79c4c0b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -17,6 +17,7 @@ weight = -1 A self-sufficient runtime for linux containers. Options: + --add-runtime=[] Register an additional OCI compatible runtime --api-cors-header="" Set CORS headers in the remote API --authorization-plugin=[] Set authorization plugins to load -b, --bridge="" Attach containers to a network bridge @@ -60,7 +61,6 @@ weight = -1 -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file --raw-logs Full timestamps without ANSI coloring --registry-mirror=[] Preferred Docker registry mirror - --add-runtime=[] Register an additional OCI compatible runtime -s, --storage-driver="" Storage driver to use --selinux-enabled Enable selinux support --storage-opt=[] Set storage driver options From 51d3ac96c7d2d6aa3cbb5b190ab2d5e210109f5d Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Tue, 5 Jul 2016 11:27:42 +0200 Subject: [PATCH 109/534] Merge docker/cli pull request 24325 from Anvil/plugins_volume.md-unqiue Fixed typo: s/unqiue/unique/g (cherry picked from commit 84aa074d18d2f75f54564f21c2b5c0d1bcddd23f) Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins_volume.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 3b84d92f7a..66afa7d439 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -126,7 +126,7 @@ name. This is called once per container start. If the same volume_name is reques more than once, the plugin may need to keep track of each new mount request and provision at the first mount request and deprovision at the last corresponding unmount request. -`ID` is a unqiue ID for the caller that is requesting the mount. +`ID` is a unique ID for the caller that is requesting the mount. **Response**: ```json @@ -176,7 +176,7 @@ Indication that Docker no longer is using the named volume. This is called once per container stop. Plugin may deduce that it is safe to deprovision it at this point. -`ID` is a unqiue ID for the caller that is requesting the mount. +`ID` is a unique ID for the caller that is requesting the mount. **Response**: ```json From 3f7826d6830c05974dc350baa2cd23c32c1ef43e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Jul 2016 14:22:12 -0700 Subject: [PATCH 110/534] add iptables=false to docs for multiple daemons Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5eb79c4c0b..a881b62ac6 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1154,6 +1154,7 @@ The following daemon options must be configured for each daemon: -g, --graph=/var/lib/docker Root of the Docker runtime -p, --pidfile=/var/run/docker.pid Path to use for daemon PID file -H, --host=[] Daemon socket(s) to connect to +--iptables=true Enable addition of iptables rules --config-file=/etc/docker/daemon.json Daemon configuration file --tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA --tlscert="~/.docker/cert.pem" Path to TLS certificate file @@ -1172,6 +1173,10 @@ set this parameter separately for each daemon. - `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your pid file here. - `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. +- `--iptables=false` prevents the Docker daemon from adding iptables rules. If + multiple daemons manage iptables rules, they may overwrite rules set by + another daemon. Be aware that disabling this option requires you to manually + add iptables rules to expose container ports. - `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of daemon flags. Specify the path for each daemon. - `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections. From bfe03650137aaa8d5f8e9d65f73b9b77d3853bbb Mon Sep 17 00:00:00 2001 From: Joao Fernandes Date: Fri, 8 Jul 2016 16:04:03 -0700 Subject: [PATCH 111/534] Fixes broken link in docs. Fixes reference 24428 Signed-off-by: Joao Fernandes --- docs/deprecated.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index fdfb4e86c0..554272ff49 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -17,15 +17,15 @@ The following list of features are deprecated in Engine. ### Three argument form in `docker import` **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. ### `-h` shorthand for `--help` -**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/)** -**Target For Removal In Release: [v1.14.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Target For Removal In Release: [v1.14.0](https://github.com/docker/docker/releases/)** The shorthand (`-h`) is less common than `--help` on Linux and cannot be used on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on @@ -50,7 +50,7 @@ The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. @@ -63,14 +63,14 @@ See the events API documentation for the new format. ### `-f` flag on `docker tag` **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of defining it at container creation (`POST /containers/create`). @@ -79,14 +79,14 @@ defining it at container creation (`POST /containers/create`). **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. ### Docker search 'automated' and 'stars' options -**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/)** **Target For Removal In Release: v1.14** @@ -96,7 +96,7 @@ Use `docker search --filter=is-automated=...` and `docker search --filter=stars= ### Driver Specific Log Tags **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and @@ -162,7 +162,7 @@ The following double-dash options are deprecated and have no replacement: **Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** The single-dash (`-help`) was removed, in favor of the double-dash `--help` @@ -177,7 +177,7 @@ Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the d ### Docker Content Trust ENV passphrase variables name change **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables From 7b884ea18d2477069e0b01b19c069aea3ff022cc Mon Sep 17 00:00:00 2001 From: Derek McGowan Date: Mon, 11 Jul 2016 16:48:08 -0700 Subject: [PATCH 112/534] Allow option to override kernel check in overlay2 Add option to skip kernel check for older kernels which have been patched to support multiple lower directories in overlayfs. Fixes reference 24023 Signed-off-by: Derek McGowan (github: dmcgowan) --- docs/reference/commandline/dockerd.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a881b62ac6..8c53fd02c3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -566,6 +566,17 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: $ docker daemon -s btrfs --storage-opt btrfs.min_space=10G +#### Overlay2 options + +* `overlay2.override_kernel_check` + + Overrides the Linux kernel version check allowing overlay2. Support for + specifying multiple lower directories needed by overlay2 was added to the + Linux kernel in 4.0.0. However some older kernel versions may be patched + to add multiple lower directory support for OverlayFS. This option should + only be used after verifying this support exists in the kernel. Applying + this option on a kernel without this support will cause failures on mount. + ## Docker runtime execution options The Docker daemon relies on a From 6c3cc98ad88f53c96bf775d22f50f187a07bf5c6 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Tue, 12 Jul 2016 10:29:50 -0700 Subject: [PATCH 113/534] add caveat for network plugins in swarm mode Signed-off-by: Charles Smith --- docs/extend/plugins_network.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index aad01ba29a..9199f784ce 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -18,6 +18,17 @@ LibNetwork, which shares plugin infrastructure with Engine. Effectively, network driver plugins are activated in the same way as other plugins, and use the same kind of protocol. +## Network driver plugins and swarm mode + +Docker 1.12 adds support for cluster management and orchestration called +[swarm mode](../swarm/index.md). Docker Engine running in swarm mode currently +only supports the built-in overlay driver for networking. Therefore existing +networking plugins will not work in swarm mode. + +When you run Docker Engine outside of swarm mode, all networking plugins that +worked in Docker 1.11 will continue to function normally. They do not require +any modification. + ## Using network driver plugins The means of installing and running a network driver plugin depend on the From 493be9949bc359e3beab56c39611a2196f3d65a8 Mon Sep 17 00:00:00 2001 From: "Arnaud Porterie (icecrime)" Date: Mon, 6 Jun 2016 16:33:00 -0700 Subject: [PATCH 114/534] Rename `--net` to `--network` Add a `--network` flag which replaces `--net` without deprecating it yet. The `--net` flag remains hidden and supported. Add a `--network-alias` flag which replaces `--net-alias` without deprecating it yet. The `--net-alias` flag remains hidden and supported. Signed-off-by: Arnaud Porterie (icecrime) --- docs/extend/plugins_network.md | 2 +- docs/reference/commandline/dockerd.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 9199f784ce..ec1ccddfea 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -46,7 +46,7 @@ Some network driver plugins are listed in [plugins](plugins.md) The `mynet` network is now owned by `weave`, so subsequent commands referring to that network will be sent to the plugin, - $ docker run --net=mynet busybox top + $ docker run --network=mynet busybox top ## Write a network plugin diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8c53fd02c3..02acdc9468 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -921,7 +921,7 @@ This option will completely disable user namespace mapping for the container's u The following standard Docker features are currently incompatible when running a Docker daemon with user namespaces enabled: - - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`) + - sharing PID or NET namespaces with the host (`--pid=host` or `--network=host`) - A `--read-only` container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace) - external (volume or graph) drivers which are unaware/incapable of using daemon user mappings - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`) From f28a1cbecfec17eac445d2700454f02c38fb80b0 Mon Sep 17 00:00:00 2001 From: Michael Crosby Date: Mon, 11 Jul 2016 15:26:23 -0700 Subject: [PATCH 115/534] Add --oom-score-adjust to daemon This adds an `--oom-score-adjust` flag to the daemon so that the value provided can be set for the docker daemon's process. The default value for the flag is -500. This will allow the docker daemon to have a less chance of being killed before containers do. The default value for processes is 0 with a min/max of -1000/1000. -500 is a good middle ground because it is less than the default for most processes and still not -1000 which basically means never kill this process in an OOM condition on the host machine. The only processes on my machine that have a score less than -500 are dbus at -900 and sshd and xfce( my window manager ) at -1000. I don't think docker should be set lower, by default, than dbus or sshd so that is why I chose -500. Signed-off-by: Michael Crosby --- docs/reference/commandline/dockerd.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 02acdc9468..65a2f5406b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -54,9 +54,10 @@ weight = -1 --label=[] Set key=value labels to the daemon --log-driver="json-file" Default driver for container logs --log-opt=[] Log driver specific options - --mtu=0 Set the containers network MTU --max-concurrent-downloads=3 Set the max concurrent downloads for each pull --max-concurrent-uploads=5 Set the max concurrent uploads for each push + --mtu=0 Set the containers network MTU + --oom-score-adjust=-500 Set the oom_score_adj for the daemon --disable-legacy-registry Do not contact legacy registries -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file --raw-logs Full timestamps without ANSI coloring @@ -1057,6 +1058,7 @@ This is a full example of the allowed configuration options on Linux: "insecure-registries": [], "disable-legacy-registry": false, "default-runtime": "runc", + "oom-score-adjust": -500, "runtimes": { "runc": { "path": "runc" From 37ddab807e812a42fd3334b5c727621f316caac3 Mon Sep 17 00:00:00 2001 From: Qiang Huang Date: Fri, 15 Jul 2016 13:57:38 +0800 Subject: [PATCH 116/534] Fix aufs docs Signed-off-by: Qiang Huang --- docs/reference/commandline/dockerd.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 65a2f5406b..939f3c189c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -210,10 +210,9 @@ drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that is unlikely to be merged into the main kernel. These are also known to cause -some serious kernel crashes. However, `aufs` is also the only storage driver -that allows containers to share executable and shared library memory, so is a -useful choice when running thousands of containers with the same program or -libraries. +some serious kernel crashes. However, `aufs` allows containers to share +executable and shared library memory, so is a useful choice when running +thousands of containers with the same program or libraries. The `devicemapper` driver uses thin provisioning and Copy on Write (CoW) snapshots. For each devicemapper graph location – typically @@ -235,7 +234,10 @@ cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem set `zfs.fsname` option as described in [Storage driver options](#storage-driver-options). The `overlay` is a very fast union filesystem. It is now merged in the main -Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). Call +Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay` +also supports page cache sharing, this means multiple containers accessing +the same file can share a single page cache entry (or entries), it makes +`overlay` as efficient with memory as `aufs` driver. Call `dockerd -s overlay` to use it. > **Note:** From 8365f5514d88b4d16fa43206996206b8302b7a54 Mon Sep 17 00:00:00 2001 From: Yanqiang Miao Date: Thu, 14 Jul 2016 22:49:56 +0800 Subject: [PATCH 117/534] Delete redundant content of 'docs/extend/plugins.md' Signed-off-by: Yanqiang Miao Update plugins.md Update plugins.md --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index 8d5616f2cb..72d50bd150 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -34,7 +34,7 @@ Follow the instructions in the plugin's documentation. The sections below provide an inexhaustive overview of available plugins. ### Network plugins From 42388dd22e62cdaeba861aa22ecb14e4b51da983 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 16 Jul 2016 01:52:59 +0200 Subject: [PATCH 118/534] Improve flag help consistency, and update docs This adds the `--live-restore` option to the documentation. Also synched usage description in the documentation with the actual description, and re-phrased some flag descriptions to be a bit more consistent. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 72 ++++++++++++++++----------- 1 file changed, 42 insertions(+), 30 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 939f3c189c..04e16534c2 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -12,35 +12,45 @@ weight = -1 # daemon +```markdown Usage: dockerd [OPTIONS] - A self-sufficient runtime for linux containers. + A self-sufficient runtime for containers. Options: + +Usage: dockerd [OPTIONS] + +A self-sufficient runtime for containers. + +Options: + --add-runtime=[] Register an additional OCI compatible runtime - --api-cors-header="" Set CORS headers in the remote API - --authorization-plugin=[] Set authorization plugins to load - -b, --bridge="" Attach containers to a network bridge - --bip="" Specify network bridge IP - --cgroup-parent= Set parent cgroup for all containers - --cluster-store="" URL of the distributed storage backend - --cluster-advertise="" Address of the daemon instance on the cluster - --cluster-store-opt=map[] Set cluster options + --api-cors-header Set CORS headers in the remote API + --authorization-plugin=[] Authorization plugins to load + -b, --bridge Attach containers to a network bridge + --bip Specify network bridge IP + --cgroup-parent Set parent cgroup for all containers + --cluster-advertise Address or interface name to advertise + --cluster-store URL of the distributed storage backend + --cluster-store-opt=map[] Set cluster store options --config-file=/etc/docker/daemon.json Daemon configuration file --containerd Path to containerd socket -D, --debug Enable debug mode - --default-gateway="" Container default gateway IPv4 address - --default-gateway-v6="" Container default gateway IPv6 address + --default-gateway Container default gateway IPv4 address + --default-gateway-v6 Container default gateway IPv6 address + --default-runtime=runc Default OCI runtime for containers + --default-ulimit=[] Default ulimits for containers + --disable-legacy-registry Disable contacting legacy registries --dns=[] DNS server to use --dns-opt=[] DNS options to use --dns-search=[] DNS search domains to use - --default-ulimit=[] Set default ulimit settings for containers - --exec-opt=[] Set runtime execution options - --exec-root="/var/run/docker" Root directory for execution state files - --fixed-cidr="" IPv4 subnet for fixed IPs - --fixed-cidr-v6="" IPv6 subnet for fixed IPs - -G, --group="docker" Group for the unix socket - -g, --graph="/var/lib/docker" Root of the Docker runtime + --exec-opt=[] Runtime execution options + --exec-root=/var/run/docker Root directory for execution state files + --fixed-cidr IPv4 subnet for fixed IPs + --fixed-cidr-v6 IPv6 subnet for fixed IPs + -G, --group=docker Group for the unix socket + -g, --graph=/var/lib/docker Root of the Docker runtime -H, --host=[] Daemon socket(s) to connect to --help Print usage --icc=true Enable inter-container communication @@ -50,28 +60,30 @@ weight = -1 --ip-masq=true Enable IP masquerading --iptables=true Enable addition of iptables rules --ipv6 Enable IPv6 networking - -l, --log-level="info" Set the logging level + -l, --log-level=info Set the logging level --label=[] Set key=value labels to the daemon - --log-driver="json-file" Default driver for container logs - --log-opt=[] Log driver specific options + --live-restore Enable live restore of docker when containers are still running + --log-driver=json-file Default driver for container logs + --log-opt=map[] Default log driver options for containers --max-concurrent-downloads=3 Set the max concurrent downloads for each pull --max-concurrent-uploads=5 Set the max concurrent uploads for each push - --mtu=0 Set the containers network MTU + --mtu Set the containers network MTU --oom-score-adjust=-500 Set the oom_score_adj for the daemon - --disable-legacy-registry Do not contact legacy registries - -p, --pidfile="/var/run/docker.pid" Path to use for daemon PID file + -p, --pidfile=/var/run/docker.pid Path to use for daemon PID file --raw-logs Full timestamps without ANSI coloring --registry-mirror=[] Preferred Docker registry mirror - -s, --storage-driver="" Storage driver to use + -s, --storage-driver Storage driver to use --selinux-enabled Enable selinux support - --storage-opt=[] Set storage driver options + --storage-opt=[] Storage driver options --tls Use TLS; implied by --tlsverify - --tlscacert="~/.docker/ca.pem" Trust certs signed only by this CA - --tlscert="~/.docker/cert.pem" Path to TLS certificate file - --tlskey="~/.docker/key.pem" Path to TLS key file + --tlscacert=~/.docker/ca.pem Trust certs signed only by this CA + --tlscert=~/.docker/cert.pem Path to TLS certificate file + --tlskey=~/.docker/key.pem Path to TLS key file --tlsverify Use TLS and verify the remote - --userns-remap="default" Enable user namespace remapping --userland-proxy=true Use userland proxy for loopback traffic + --userns-remap User/Group setting for user namespaces + -v, --version Print version information and quit +``` Options with [] may be specified multiple times. From 3a9a59076d7a9156d136fbfe37eaa177ea380e5f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 20 Jul 2016 15:36:02 +0200 Subject: [PATCH 119/534] fix duplicated usage in docs this removes a copy/pasta whoopsie on my side, introduced in de64324109d2694b1525e62b5c0072267282a36c Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 04e16534c2..8581ec095f 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -13,12 +13,6 @@ weight = -1 # daemon ```markdown - Usage: dockerd [OPTIONS] - - A self-sufficient runtime for containers. - - Options: - Usage: dockerd [OPTIONS] A self-sufficient runtime for containers. From 6f2052f70dbd953fe34d1126c669bbfd752547d9 Mon Sep 17 00:00:00 2001 From: Harry Zhang Date: Tue, 19 Jul 2016 22:42:23 +0800 Subject: [PATCH 120/534] Fix nits in deprecated Signed-off-by: Harry Zhang --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 554272ff49..0a728d97b8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -30,7 +30,7 @@ The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated The shorthand (`-h`) is less common than `--help` on Linux and cannot be used on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on `docker create`). For this reason, the `-h` shorthand was not printed in the -"usage" output of subcommands, nor docummented, and is now marked "deprecated". +"usage" output of subcommands, nor documented, and is now marked "deprecated". ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** @@ -44,7 +44,7 @@ The docker login command is removing the ability to automatically register for a **Target For Removal In Release: v1.13** -The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consinstency with other similar flags, like `--storage-opt`. +The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. ### `/containers/(id or name)/copy` endpoint From f701d249cc1acf57474256b53317aa73df490dce Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Tue, 12 Jul 2016 20:14:11 -0700 Subject: [PATCH 121/534] Extend deprecation cycle to 3 releases by default At the moment docker's deprecation policy is 2 release cycles by default, which is around 5 months. This may not be enough for production environment and there is a need to extend the deprecation cycle to 3 releases (see reference 24494). This fix updates the docs/deprecated.md and extend the deprecation cycle to 3 releases. This fix is related to reference 24494 and reference 24534. This fix fixes reference 24534. Signed-off-by: Yong Tang --- docs/deprecated.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 0a728d97b8..7fc59ad443 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -13,6 +13,9 @@ weight=80 # Deprecated Engine Features The following list of features are deprecated in Engine. +To learn more about Docker Engine's deprecation policy, +see [Feature Deprecation Policy](index.md#feature-deprecation-policy). + ### Three argument form in `docker import` **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** @@ -25,7 +28,7 @@ The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated **Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/)** -**Target For Removal In Release: [v1.14.0](https://github.com/docker/docker/releases/)** +**Target For Removal In Release: v1.15** The shorthand (`-h`) is less common than `--help` on Linux and cannot be used on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on @@ -35,14 +38,14 @@ on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v1.13** +**Target For Removal In Release: v1.14** The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v1.13** +**Target For Removal In Release: v1.14** The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. @@ -88,7 +91,7 @@ Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/)** -**Target For Removal In Release: v1.14** +**Target For Removal In Release: v1.15** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. From e30bfe1df27a7054252a56c5769afb946dba82ca Mon Sep 17 00:00:00 2001 From: Aaron Lehmann Date: Thu, 30 Jun 2016 18:07:35 -0700 Subject: [PATCH 122/534] Split advertised address from listen address There are currently problems with "swarm init" and "swarm join" when an explicit --listen-addr flag is not provided. swarmkit defaults to finding the IP address associated with the default route, and in cloud setups this is often the wrong choice. Introduce a notion of "advertised address", with the client flag --advertise-addr, and the daemon flag --swarm-default-advertise-addr to provide a default. The default listening address is now 0.0.0.0, but a valid advertised address must be detected or specified. If no explicit advertised address is specified, error out if there is more than one usable candidate IP address on the system. This requires a user to explicitly choose instead of letting swarmkit make the wrong choice. For the purposes of this autodetection, we ignore certain interfaces that are unlikely to be relevant (currently docker*). The user is also required to choose a listen address on swarm init if they specify an explicit advertise address that is a hostname or an IP address that's not local to the system. This is a requirement for overlay networking. Also support specifying interface names to --listen-addr, --advertise-addr, and the daemon flag --swarm-default-advertise-addr. This will fail if the interface has multiple IP addresses (unless it has a single IPv4 address and a single IPv6 address - then we resolve the tie in favor of IPv4). This change also exposes the node's externally-reachable address in docker info, as requested by reference 24017. Make corresponding API and CLI docs changes. Signed-off-by: Aaron Lehmann --- docs/reference/commandline/dockerd.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8581ec095f..029d6cbf92 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -69,6 +69,7 @@ Options: -s, --storage-driver Storage driver to use --selinux-enabled Enable selinux support --storage-opt=[] Storage driver options + --swarm-default-advertise-addr Set default address or interface for swarm advertised address --tls Use TLS; implied by --tlsverify --tlscacert=~/.docker/ca.pem Trust certs signed only by this CA --tlscert=~/.docker/cert.pem Path to TLS certificate file @@ -1042,6 +1043,7 @@ This is a full example of the allowed configuration options on Linux: "tlscacert": "", "tlscert": "", "tlskey": "", + "swarm-default-advertise-addr": "", "api-cors-header": "", "selinux-enabled": false, "userns-remap": "", @@ -1112,6 +1114,7 @@ This is a full example of the allowed configuration options on Windows: "tlscacert": "", "tlscert": "", "tlskey": "", + "swarm-default-advertise-addr": "", "group": "", "default-ulimits": {}, "bridge": "", From 40d8d31fcce22334d3f5b82ede5e27cafde6e51b Mon Sep 17 00:00:00 2001 From: Everett Toews Date: Sun, 24 Jul 2016 22:57:48 -0500 Subject: [PATCH 123/534] Fix AuthzPlugin URL fragment in docs These docs have AuthzPlugin with a lower case 'z'. What the plugin api is actually looking for is AuthZPlugin with an upper case 'Z'. See https://github.com/docker/docker/blob/46e3a249a1971f8697ca338c9b02e27d36ddab12/pkg/authorization/api.go#L5-L8 Signed-off-by: Everett Toews --- docs/extend/plugins_authorization.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 16e453f86e..4c11fa26ca 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -141,11 +141,11 @@ docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZP In addition to Docker's standard plugin registration method, each plugin should implement the following two methods: -* `/AuthzPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. +* `/AuthZPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. -* `/AuthzPlugin.AuthZRes` This authorize response method is called before the response is returned from Docker daemon to the client. +* `/AuthZPlugin.AuthZRes` This authorize response method is called before the response is returned from Docker daemon to the client. -#### /AuthzPlugin.AuthZReq +#### /AuthZPlugin.AuthZReq **Request**: @@ -169,7 +169,7 @@ should implement the following two methods: "Err": "The error message if things go wrong" } ``` -#### /AuthzPlugin.AuthZRes +#### /AuthZPlugin.AuthZRes **Request**: From bacac86fe45e01fdf6f4c20827322712b3d8401e Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Mon, 18 Jul 2016 21:40:07 -0700 Subject: [PATCH 124/534] add doc for live-restore setting and daemonless containers, fix admin guide menu Signed-off-by: Charles Smith --- docs/reference/commandline/dockerd.md | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 029d6cbf92..f3ac7e5fb4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -56,7 +56,7 @@ Options: --ipv6 Enable IPv6 networking -l, --log-level=info Set the logging level --label=[] Set key=value labels to the daemon - --live-restore Enable live restore of docker when containers are still running + --live-restore Enables keeping containers alive during daemon downtime --log-driver=json-file Default driver for container logs --log-opt=map[] Default log driver options for containers --max-concurrent-downloads=3 Set the max concurrent downloads for each pull @@ -1003,14 +1003,14 @@ via flags. The docker daemon fails to start if an option is duplicated between the file and the flags, regardless their value. We do this to avoid silently ignore changes introduced in configuration reloads. For example, the daemon fails to start if you set daemon labels -in the configuration file and also set daemon labels via the `--label` flag. +in the configuration file and also set daemon labels via the `--label` flag. Options that are not present in the file are ignored when the daemon starts. ### Linux configuration file -The default location of the configuration file on Linux is +The default location of the configuration file on Linux is `/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a - non-default location. + non-default location. This is a full example of the allowed configuration options on Linux: @@ -1025,6 +1025,7 @@ This is a full example of the allowed configuration options on Linux: "storage-driver": "", "storage-opts": [], "labels": [], + "live-restore": true, "log-driver": "", "log-opts": [], "mtu": 0, @@ -1087,7 +1088,7 @@ This is a full example of the allowed configuration options on Linux: The default location of the configuration file on Windows is `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be - used to specify a non-default location. + used to specify a non-default location. This is a full example of the allowed configuration options on Windows: @@ -1101,7 +1102,8 @@ This is a full example of the allowed configuration options on Windows: "storage-driver": "", "storage-opts": [], "labels": [], - "log-driver": "", + "live-restore": true, + "log-driver": "", "mtu": 0, "pidfile": "", "graph": "", @@ -1142,6 +1144,7 @@ The list of currently supported options that can be reconfigured is this: - `cluster-store-opts`: it uses the new options to reload the discovery store. - `cluster-advertise`: it modifies the address advertised after reloading. - `labels`: it replaces the daemon labels with a new set of labels. +- `live-restore`: Enables [keeping containers alive during daemon downtime](../../admin/live-restore.md). - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. - `max-concurrent-uploads`: it updates the max concurrent uploads for each push. - `default-runtime`: it updates the runtime to be used if not is From 693a61518b2789087d24f5edfe8d9fe8a282a787 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 27 Jul 2016 11:30:42 +0200 Subject: [PATCH 125/534] Add back links to 1.12 release in deprecated.md These links were temporarily changed, because the docs were published before the 1.12.0 tag was available. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7fc59ad443..668d8e89b4 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -20,13 +20,13 @@ see [Feature Deprecation Policy](index.md#feature-deprecation-policy). ### Three argument form in `docker import` **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. ### `-h` shorthand for `--help` -**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** **Target For Removal In Release: v1.15** @@ -53,7 +53,7 @@ The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. @@ -66,14 +66,14 @@ See the events API documentation for the new format. ### `-f` flag on `docker tag` **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of defining it at container creation (`POST /containers/create`). @@ -82,14 +82,14 @@ defining it at container creation (`POST /containers/create`). **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. ### Docker search 'automated' and 'stars' options -**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** **Target For Removal In Release: v1.15** @@ -99,7 +99,7 @@ Use `docker search --filter=is-automated=...` and `docker search --filter=stars= ### Driver Specific Log Tags **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and @@ -165,7 +165,7 @@ The following double-dash options are deprecated and have no replacement: **Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The single-dash (`-help`) was removed, in favor of the double-dash `--help` @@ -180,7 +180,7 @@ Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the d ### Docker Content Trust ENV passphrase variables name change **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/)** +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables From 116215052c2224fb43d4cb8b666c027b3b0cc6ea Mon Sep 17 00:00:00 2001 From: Liron Levin Date: Mon, 16 May 2016 21:12:48 +0300 Subject: [PATCH 126/534] Enable to dynamically reload authorization plugins via daemon.config Following reference 22729, enable to dynamically reload/remove the daemon authorization plugins (via standard reloading mechanism). https://docs.docker.com/engine/reference/commandline/daemon/#daemon- configuration-file Daemon must store a reference to the authorization middleware to refresh the plugin on configuration changes. Signed-off-by: Liron Levin --- docs/extend/plugins_authorization.md | 2 ++ docs/reference/commandline/dockerd.md | 1 + 2 files changed, 3 insertions(+) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 4c11fa26ca..8630bfb761 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -104,6 +104,8 @@ support the Docker client interactions detailed in this section. Enable the authorization plugin with a dedicated command line flag in the `--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. This value can be the plugin’s socket or a path to a specification file. +Authorization plugins can be loaded without restarting the daemon. Refer +to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. ```bash $ docker daemon --authorization-plugin=plugin1 --authorization-plugin=plugin2,... diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index f3ac7e5fb4..9c0d4b71c6 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1152,6 +1152,7 @@ The list of currently supported options that can be reconfigured is this: the runtime shipped with the official docker packages. - `runtimes`: it updates the list of available OCI runtimes that can be used to run containers +- `authorization-plugin`: specifies the authorization plugins to use. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From c723a475583689557ab9f631877c96a3ac59122d Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Wed, 10 Aug 2016 12:47:07 -0700 Subject: [PATCH 127/534] Update docs so that log-opts takes map (`{}`) instead of `[]` This fix updates docs so that log-opts takes map (`{}`) instead of `[]`, as is defined in the impmenetation (`map[string]string`) This fix fixes 22311. Signed-off-by: Yong Tang --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 9c0d4b71c6..6487371360 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1027,7 +1027,7 @@ This is a full example of the allowed configuration options on Linux: "labels": [], "live-restore": true, "log-driver": "", - "log-opts": [], + "log-opts": {}, "mtu": 0, "pidfile": "", "graph": "", From b0f07e226da4d9fa2b7dc3026f5f66a17fc57767 Mon Sep 17 00:00:00 2001 From: Michael Crosby Date: Wed, 17 Aug 2016 09:38:34 -0700 Subject: [PATCH 128/534] Replace old oci specs import with runtime-specs Fixes reference 25804 The upstream repo changed the import paths. Signed-off-by: Michael Crosby --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6487371360..ccf120c3b9 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -590,7 +590,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. ## Docker runtime execution options The Docker daemon relies on a -[OCI](https://github.com/opencontainers/specs) compliant runtime +[OCI](https://github.com/opencontainers/runtime-spec) compliant runtime (invoked via the `containerd` daemon) as its interface to the Linux kernel `namespaces`, `cgroups`, and `SELinux`. From 6c857150607648dc9c25be147544193ef318c157 Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Mon, 15 Aug 2016 11:03:05 -0700 Subject: [PATCH 129/534] adding some documentation about the new plugin system Signed-off-by: Victor Vieux --- docs/extend/index.md | 8 +- docs/extend/legacy/index.md | 22 ++ docs/extend/{ => legacy}/plugin_api.md | 0 docs/extend/{ => legacy}/plugins.md | 0 .../{ => legacy}/plugins_authorization.md | 0 docs/extend/{ => legacy}/plugins_network.md | 0 docs/extend/{ => legacy}/plugins_volume.md | 0 docs/extend/new/index.md | 18 ++ docs/extend/new/plugins.md | 227 ++++++++++++++++++ 9 files changed, 270 insertions(+), 5 deletions(-) create mode 100644 docs/extend/legacy/index.md rename docs/extend/{ => legacy}/plugin_api.md (100%) rename docs/extend/{ => legacy}/plugins.md (100%) rename docs/extend/{ => legacy}/plugins_authorization.md (100%) rename docs/extend/{ => legacy}/plugins_network.md (100%) rename docs/extend/{ => legacy}/plugins_volume.md (100%) create mode 100644 docs/extend/new/index.md create mode 100644 docs/extend/new/plugins.md diff --git a/docs/extend/index.md b/docs/extend/index.md index 8a061e4e29..588dc10a2f 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -15,8 +15,6 @@ weight = 6 Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: -* [Understand Docker plugins](plugins.md) -* [Write a volume plugin](plugins_volume.md) -* [Write a network plugin](plugins_network.md) -* [Write an authorization plugin](plugins_authorization.md) -* [Docker plugin API](plugin_api.md) +* [New Docker Plugin System](new/index.md) +* [Legacy Docker Plugins](legacy/index.md) + diff --git a/docs/extend/legacy/index.md b/docs/extend/legacy/index.md new file mode 100644 index 0000000000..031d7d2394 --- /dev/null +++ b/docs/extend/legacy/index.md @@ -0,0 +1,22 @@ + + + +## Legacy Docker Plugins + +Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: + +* [Understand Docker plugins](plugins.md) +* [Write a volume plugin](plugins_volume.md) +* [Write a network plugin](plugins_network.md) +* [Write an authorization plugin](plugins_authorization.md) +* [Docker plugin API](plugin_api.md) diff --git a/docs/extend/plugin_api.md b/docs/extend/legacy/plugin_api.md similarity index 100% rename from docs/extend/plugin_api.md rename to docs/extend/legacy/plugin_api.md diff --git a/docs/extend/plugins.md b/docs/extend/legacy/plugins.md similarity index 100% rename from docs/extend/plugins.md rename to docs/extend/legacy/plugins.md diff --git a/docs/extend/plugins_authorization.md b/docs/extend/legacy/plugins_authorization.md similarity index 100% rename from docs/extend/plugins_authorization.md rename to docs/extend/legacy/plugins_authorization.md diff --git a/docs/extend/plugins_network.md b/docs/extend/legacy/plugins_network.md similarity index 100% rename from docs/extend/plugins_network.md rename to docs/extend/legacy/plugins_network.md diff --git a/docs/extend/plugins_volume.md b/docs/extend/legacy/plugins_volume.md similarity index 100% rename from docs/extend/plugins_volume.md rename to docs/extend/legacy/plugins_volume.md diff --git a/docs/extend/new/index.md b/docs/extend/new/index.md new file mode 100644 index 0000000000..9c809a03a4 --- /dev/null +++ b/docs/extend/new/index.md @@ -0,0 +1,18 @@ + + + +## New Docker Plugin System + +Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: + +* [Understand Docker plugins](plugins.md) diff --git a/docs/extend/new/plugins.md b/docs/extend/new/plugins.md new file mode 100644 index 0000000000..04f83ea243 --- /dev/null +++ b/docs/extend/new/plugins.md @@ -0,0 +1,227 @@ + + +# New Plugin System + +The goal of this document is to describe the current state of the new plugin system available today in the **experimental build** of Docker 1.12. + +The main difference, compared to legacy plugins, is that plugins are now managed by Docker: plugins are installed, started, stopped and removed by docker. + +Only volume drivers are currently supported but more types will be added in the next release. + +This document is split in two parts, the user perspective, “how to operate a plugin” and the developer perspective “how to create a plugin” + + +## How to operate a plugin + +Plugins are distributed as docker images, so they can be hosted on the Docker Hub or on a private registry. +Installing a plugin is very easy, it’s a simple command: `docker plugin install` +This command is going to pull the plugin from the Docker Hub / Private registry, ask the operator to accept privileges (for example, plugin requires access to a device on the host system), if necessary and enable it. +You can then check the status of the plugin with the docker plugin ls command, the plugin will be marked as ENABLED if it was started without issue. + +Then, the plugin behavior is the same as legacy plugins, here is a full example using a sshfs plugin: + +### install the plugin +```bash +$ docker plugin install vieux/sshfs +Plugin "vieux/sshfs" is requesting the following privileges: + - network: [host] + - capabilities: [CAP_SYS_ADMIN] +Do you grant the above permissions? [y/N] y +vieux/sshfs +``` + +Here the plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able to do mount inside the plugin and `host networking`. + +### verify that the plugin has correctly started +##### by looking at the ENABLED column. (The value should be true) + +```bash +$ docker plugin ls +NAME TAG ENABLED +vieux/sshfs latest true +``` + +### create a volume using the plugin installed above + +```bash +$ docker volume create -d vieux/sshfs --name sshvolume -o sshcmd=user@1.2.3.4:/remote +sshvolume +``` + +### use the volume created above + +```bash +$ docker run -v sshvolume:/data busybox ls /data + +``` + +### verify that the plugin was created successfully + +```bash +$ docker volume ls +DRIVER NAME +vieux/sshfs sshvolume +``` + +It’s also possible to stop a plugin with the `docker plugin disable` command and to remove a plugin with `docker plugin remove`. + +See the [command line reference](../engine/reference/commandline/) for more information. + +## How to create a plugin + +The creation of plugin is currently a manual process, in the future release, a command such as `docker plugin build` will be added to automate the process. So here we are going to describe the format of an existing enabled plugin, to create a plugin you have to manually craft all those files by hand. + +Plugins are stored in `/var/lib/docker/plugins`. See this example: + +```bash +# ls -la /var/lib/docker/plugins +total 20 +drwx------ 4 root root 4096 Aug 8 18:03 . +drwx--x--x 12 root root 4096 Aug 8 17:53 .. +drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 +-rw------- 1 root root 2107 Aug 8 18:03 plugins.json +``` + +The file `plugins.json` is an inventory of all installed plugins, see an example of the content: + +```bash +# cat plugins.json +{ + "cd851ce43a403": { + "plugin": { + "Manifest": { + "Args": { + "Value": null, + "Settable": null, + "Description": "", + "Name": "" + }, + "Env": null, + "Devices": null, + "Mounts": null, + "Capabilities": [ + "CAP_SYS_ADMIN" + ], + "ManifestVersion": "v0.1", + "Description": "sshFS plugin for Docker", + "Documentation": "https://docs.docker.com/engine/extend/plugins/", + "Interface": { + "Socket": "sshfs.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Entrypoint": [ + "/go/bin/docker-volume-sshfs" + ], + "Workdir": "", + "User": {}, + "Network": { + "Type": "host" + } + }, + "Config": { + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] + }, + "Active": true, + "Tag": "latest", + "Name": "vieux/sshfs", + "Id": "cd851ce43a403" + } + } +} +``` + +Each folder represents a plugin, for example: + +```bash +# ls -la /var/lib/docker/plugins/cd851ce43a403 +total 12 +drwx------ 19 root root 4096 Aug 8 17:56 rootfs +-rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json +-rw------- 1 root root 347 Aug 8 17:56 manifest.json +``` + +rootfs represents the root filesystem of the plugin, in this example, it was created from this Dockerfile as follows: + +_Note: `/run/docker/plugins` is mandatory for docker to communicate with the plugin._ + +```bash +$ git clone github.com/vieux/docker-volume-sshfs +$ cd docker-volume-sshfs +$ docker build -t rootfs . +$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created +$ mkdir -p /var/lib/docker/plugins/$id/rootfs +$ docker export "$id" | tar -x -C /var/lib/docker/plugins/$id/rootfs +$ docker rm -vf "$id" +$ docker rmi rootfs +``` + +`manifest.json` describe the plugin and `plugin-config.json` contains some runtime parameters, see for example: + +```bash +# cat manifest.json +{ + "manifestVersion": "v0.1", + "description": "sshFS plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["/go/bin/docker-volume-sshfs"], + "network": { + "type": "host" + }, + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "sshfs.sock" + }, + "capabilities": ["CAP_SYS_ADMIN"] +} +``` + +In this example, you can see the plugin is a volume driver, requires the `CAP_SYS_ADMIN` capability, `host networking`, `/go/bin/docker-volume-sshfs` as entrypoint and is going to use `/run/docker/plugins/sshfs.sock` to communicate with the docker engine. + +```bash +# cat plugin-config.json +{ + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] +} +``` + +No runtime parameters are needed for this plugin. + +Both `manifest.json` and `plugin-config.json` are part of the `plugins.json`. +`manifest.json` is read-only and `plugin-config.json` is read-write. + + + +To sum up, here are the steps required to create a plugin today: + +0. choose the name of the plugins, same format as images, for example `/` +1. create a rootfs as showed above in `/var/lib/docker/plugins/$id/rootfs` +2. create manifest.json file in `/var/lib/docker/plugins/$id/` as shown above +3. create a `plugin-config.json` if needed, as shown above. +4. create or add a section to `/var/lib/docker/plugins/plugins.json` as shown above, use +`/` as “Name” and `$id` as “Id” +5. restart docker +6. `docker plugin ls` + a. if your plugin is listed as `ENABLED=true`, go to 7. + b. if the plugins is not listed or listed as `ENABLED=false` something went wrong, look at the daemon logs. +7. if not logged in already, use `docker login` to authenticate against a registry. +8. push the plugin with `docker plugin push /` + + From 3492c48769e4699d4471945597c85df9d1424f54 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Tue, 16 Aug 2016 11:53:36 -0700 Subject: [PATCH 130/534] edit plugin system doc, fix menu system Signed-off-by: Charles Smith --- docs/extend/index.md | 20 -- docs/extend/legacy/index.md | 22 -- .../{legacy/plugins.md => legacy_plugins.md} | 10 +- docs/extend/{new/index.md => menu.md} | 9 +- docs/extend/new/plugins.md | 227 --------------- docs/extend/{legacy => }/plugin_api.md | 8 +- docs/extend/plugins.md | 272 ++++++++++++++++++ .../{legacy => }/plugins_authorization.md | 6 +- docs/extend/{legacy => }/plugins_network.md | 5 + docs/extend/{legacy => }/plugins_volume.md | 5 + 10 files changed, 305 insertions(+), 279 deletions(-) delete mode 100644 docs/extend/index.md delete mode 100644 docs/extend/legacy/index.md rename docs/extend/{legacy/plugins.md => legacy_plugins.md} (97%) rename docs/extend/{new/index.md => menu.md} (78%) delete mode 100644 docs/extend/new/plugins.md rename docs/extend/{legacy => }/plugin_api.md (96%) create mode 100644 docs/extend/plugins.md rename docs/extend/{legacy => }/plugins_authorization.md (97%) rename docs/extend/{legacy => }/plugins_network.md (91%) rename docs/extend/{legacy => }/plugins_volume.md (96%) diff --git a/docs/extend/index.md b/docs/extend/index.md deleted file mode 100644 index 588dc10a2f..0000000000 --- a/docs/extend/index.md +++ /dev/null @@ -1,20 +0,0 @@ - - - -## Extending Docker Engine - -Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: - -* [New Docker Plugin System](new/index.md) -* [Legacy Docker Plugins](legacy/index.md) - diff --git a/docs/extend/legacy/index.md b/docs/extend/legacy/index.md deleted file mode 100644 index 031d7d2394..0000000000 --- a/docs/extend/legacy/index.md +++ /dev/null @@ -1,22 +0,0 @@ - - - -## Legacy Docker Plugins - -Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: - -* [Understand Docker plugins](plugins.md) -* [Write a volume plugin](plugins_volume.md) -* [Write a network plugin](plugins_network.md) -* [Write an authorization plugin](plugins_authorization.md) -* [Docker plugin API](plugin_api.md) diff --git a/docs/extend/legacy/plugins.md b/docs/extend/legacy_plugins.md similarity index 97% rename from docs/extend/legacy/plugins.md rename to docs/extend/legacy_plugins.md index 72d50bd150..02874f315e 100644 --- a/docs/extend/legacy/plugins.md +++ b/docs/extend/legacy_plugins.md @@ -5,11 +5,15 @@ description = "How to add additional functionality to Docker with plugins extens keywords = ["Examples, Usage, plugins, docker, documentation, user guide"] [menu.main] parent = "engine_extend" -weight=-1 +weight=3 +++ -# Understand Engine plugins +# Understand legacy Docker Engine plugins + +This document describes the Docker Engine plugins generally available in Docker +Engine 1.12 and earlier. To view information on plugins managed by Docker +Engine, refer to [Docker Engine plugin system](plugins.md). You can extend the capabilities of the Docker Engine by loading third-party plugins. This page explains the types of plugins and provides links to several @@ -72,7 +76,7 @@ Plugin ### Authorization plugins - Plugin | Description + Plugin | Description ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). diff --git a/docs/extend/new/index.md b/docs/extend/menu.md similarity index 78% rename from docs/extend/new/index.md rename to docs/extend/menu.md index 9c809a03a4..026e56427f 100644 --- a/docs/extend/new/index.md +++ b/docs/extend/menu.md @@ -1,12 +1,13 @@ diff --git a/docs/extend/new/plugins.md b/docs/extend/new/plugins.md deleted file mode 100644 index 04f83ea243..0000000000 --- a/docs/extend/new/plugins.md +++ /dev/null @@ -1,227 +0,0 @@ - - -# New Plugin System - -The goal of this document is to describe the current state of the new plugin system available today in the **experimental build** of Docker 1.12. - -The main difference, compared to legacy plugins, is that plugins are now managed by Docker: plugins are installed, started, stopped and removed by docker. - -Only volume drivers are currently supported but more types will be added in the next release. - -This document is split in two parts, the user perspective, “how to operate a plugin” and the developer perspective “how to create a plugin” - - -## How to operate a plugin - -Plugins are distributed as docker images, so they can be hosted on the Docker Hub or on a private registry. -Installing a plugin is very easy, it’s a simple command: `docker plugin install` -This command is going to pull the plugin from the Docker Hub / Private registry, ask the operator to accept privileges (for example, plugin requires access to a device on the host system), if necessary and enable it. -You can then check the status of the plugin with the docker plugin ls command, the plugin will be marked as ENABLED if it was started without issue. - -Then, the plugin behavior is the same as legacy plugins, here is a full example using a sshfs plugin: - -### install the plugin -```bash -$ docker plugin install vieux/sshfs -Plugin "vieux/sshfs" is requesting the following privileges: - - network: [host] - - capabilities: [CAP_SYS_ADMIN] -Do you grant the above permissions? [y/N] y -vieux/sshfs -``` - -Here the plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able to do mount inside the plugin and `host networking`. - -### verify that the plugin has correctly started -##### by looking at the ENABLED column. (The value should be true) - -```bash -$ docker plugin ls -NAME TAG ENABLED -vieux/sshfs latest true -``` - -### create a volume using the plugin installed above - -```bash -$ docker volume create -d vieux/sshfs --name sshvolume -o sshcmd=user@1.2.3.4:/remote -sshvolume -``` - -### use the volume created above - -```bash -$ docker run -v sshvolume:/data busybox ls /data - -``` - -### verify that the plugin was created successfully - -```bash -$ docker volume ls -DRIVER NAME -vieux/sshfs sshvolume -``` - -It’s also possible to stop a plugin with the `docker plugin disable` command and to remove a plugin with `docker plugin remove`. - -See the [command line reference](../engine/reference/commandline/) for more information. - -## How to create a plugin - -The creation of plugin is currently a manual process, in the future release, a command such as `docker plugin build` will be added to automate the process. So here we are going to describe the format of an existing enabled plugin, to create a plugin you have to manually craft all those files by hand. - -Plugins are stored in `/var/lib/docker/plugins`. See this example: - -```bash -# ls -la /var/lib/docker/plugins -total 20 -drwx------ 4 root root 4096 Aug 8 18:03 . -drwx--x--x 12 root root 4096 Aug 8 17:53 .. -drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 --rw------- 1 root root 2107 Aug 8 18:03 plugins.json -``` - -The file `plugins.json` is an inventory of all installed plugins, see an example of the content: - -```bash -# cat plugins.json -{ - "cd851ce43a403": { - "plugin": { - "Manifest": { - "Args": { - "Value": null, - "Settable": null, - "Description": "", - "Name": "" - }, - "Env": null, - "Devices": null, - "Mounts": null, - "Capabilities": [ - "CAP_SYS_ADMIN" - ], - "ManifestVersion": "v0.1", - "Description": "sshFS plugin for Docker", - "Documentation": "https://docs.docker.com/engine/extend/plugins/", - "Interface": { - "Socket": "sshfs.sock", - "Types": [ - "docker.volumedriver/1.0" - ] - }, - "Entrypoint": [ - "/go/bin/docker-volume-sshfs" - ], - "Workdir": "", - "User": {}, - "Network": { - "Type": "host" - } - }, - "Config": { - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] - }, - "Active": true, - "Tag": "latest", - "Name": "vieux/sshfs", - "Id": "cd851ce43a403" - } - } -} -``` - -Each folder represents a plugin, for example: - -```bash -# ls -la /var/lib/docker/plugins/cd851ce43a403 -total 12 -drwx------ 19 root root 4096 Aug 8 17:56 rootfs --rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json --rw------- 1 root root 347 Aug 8 17:56 manifest.json -``` - -rootfs represents the root filesystem of the plugin, in this example, it was created from this Dockerfile as follows: - -_Note: `/run/docker/plugins` is mandatory for docker to communicate with the plugin._ - -```bash -$ git clone github.com/vieux/docker-volume-sshfs -$ cd docker-volume-sshfs -$ docker build -t rootfs . -$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created -$ mkdir -p /var/lib/docker/plugins/$id/rootfs -$ docker export "$id" | tar -x -C /var/lib/docker/plugins/$id/rootfs -$ docker rm -vf "$id" -$ docker rmi rootfs -``` - -`manifest.json` describe the plugin and `plugin-config.json` contains some runtime parameters, see for example: - -```bash -# cat manifest.json -{ - "manifestVersion": "v0.1", - "description": "sshFS plugin for Docker", - "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["/go/bin/docker-volume-sshfs"], - "network": { - "type": "host" - }, - "interface" : { - "types": ["docker.volumedriver/1.0"], - "socket": "sshfs.sock" - }, - "capabilities": ["CAP_SYS_ADMIN"] -} -``` - -In this example, you can see the plugin is a volume driver, requires the `CAP_SYS_ADMIN` capability, `host networking`, `/go/bin/docker-volume-sshfs` as entrypoint and is going to use `/run/docker/plugins/sshfs.sock` to communicate with the docker engine. - -```bash -# cat plugin-config.json -{ - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] -} -``` - -No runtime parameters are needed for this plugin. - -Both `manifest.json` and `plugin-config.json` are part of the `plugins.json`. -`manifest.json` is read-only and `plugin-config.json` is read-write. - - - -To sum up, here are the steps required to create a plugin today: - -0. choose the name of the plugins, same format as images, for example `/` -1. create a rootfs as showed above in `/var/lib/docker/plugins/$id/rootfs` -2. create manifest.json file in `/var/lib/docker/plugins/$id/` as shown above -3. create a `plugin-config.json` if needed, as shown above. -4. create or add a section to `/var/lib/docker/plugins/plugins.json` as shown above, use -`/` as “Name” and `$id` as “Id” -5. restart docker -6. `docker plugin ls` - a. if your plugin is listed as `ENABLED=true`, go to 7. - b. if the plugins is not listed or listed as `ENABLED=false` something went wrong, look at the daemon logs. -7. if not logged in already, use `docker login` to authenticate against a registry. -8. push the plugin with `docker plugin push /` - - diff --git a/docs/extend/legacy/plugin_api.md b/docs/extend/plugin_api.md similarity index 96% rename from docs/extend/legacy/plugin_api.md rename to docs/extend/plugin_api.md index a799a13520..1f55b6d528 100644 --- a/docs/extend/legacy/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -5,7 +5,7 @@ description = "How to write Docker plugins extensions " keywords = ["API, Usage, plugins, documentation, developer"] [menu.main] parent = "engine_extend" -weight=1 +weight=7 +++ @@ -14,9 +14,13 @@ weight=1 Docker plugins are out-of-process extensions which add capabilities to the Docker Engine. +This document describes the Docker Engine plugin API generally available in +Docker Engine 1.12 and earlier. To view information on plugins managed by Docker +Engine, refer to [Docker Engine plugin system](plugins.md). + This page is intended for people who want to develop their own Docker plugin. If you just want to learn about or use Docker plugins, look -[here](plugins.md). +[here](legacy_plugins.md). ## What plugins are diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md new file mode 100644 index 0000000000..fb7af836e5 --- /dev/null +++ b/docs/extend/plugins.md @@ -0,0 +1,272 @@ + + +# Docker Engine plugin system + +This document describes the plugin system available today in the **experimental +build** of Docker 1.12: + +* [How to operate an existing plugin](#how-to-operate-a-plugin) +* [How to develop a plugin](#how-to-develop-a-plugin) + +Unlike the legacy plugin system, you now manage plugins using Docker Engine: + +* install plugins +* start plugins +* stop plugins +* remove plugins + +The current Docker Engine plugin system only supports volume drivers. We are +adding more plugin driver types in the future releases. + +For information on Docker Engine plugins generally available in Docker Engine +1.12 and earlier, refer to [Understand legacy Docker Engine plugins](legacy_plugins.md). + +## How to operate a plugin + +Plugins are distributed as Docker images, so develpers can host them on Docker +Hub or on a private registry. + +You install the plugin using a single command: `docker plugin install `. +The `plugin install` command pulls the plugin from the Docker Hub or private +registry. If necessary the CLI prompts you to accept any privilige requriements. +For example the plugin may require access to a device on the host system. +Finally it enables the plugin. + +Run `docker plugin ls` to check the status of installed plugins. The Engine +markes plugins that are started without issues as `ENABLED`. + +After you install a plugin, the plugin behavior is the same as legacy plugins. +The following example demonstrates how to install the `sshfs` plugin and use it +to create a volume. + +1. Install the `sshfs` plugin. + + ```bash + $ docker plugin install vieux/sshfs + + Plugin "vieux/sshfs" is requesting the following privileges: + - network: [host] + - capabilities: [CAP_SYS_ADMIN] + Do you grant the above permissions? [y/N] y + + vieux/sshfs + ``` + + The plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able + to do mount inside the plugin and `host networking`. + +2. Check for a value of `true` the `ENABLED` column to verify the plugin +started without error. + + ```bash + $ docker plugin ls + + NAME TAG ENABLED + vieux/sshfs latest true + ``` + +3. Create a volume using the plugin. + + ```bash + $ docker volume create \ + -d vieux/sshfs \ + --name sshvolume \ + -o sshcmd=user@1.2.3.4:/remote + + sshvolume + ``` + +4. Use the volume `sshvolume`. + + ```bash + $ docker run -v sshvolume:/data busybox ls /data + + + ``` + +5. Verify the plugin successfully crated the volume. + + ```bash + $ docker volume ls + + DRIVER NAME + vieux/sshfs sshvolume + ``` + + You can stop a plugin with the `docker plugin disable` + command or remove a plugin with `docker plugin remove`. + +See the [command line reference](../engine/reference/commandline/) for more +information. + +## How to develop a plugin + +Plugin creation is currently a manual process. We plan to add automation in a +future release with a command such as `docker plugin build`. + +This section describes the format of an existing enabled plugin. You have to +create and format the plugin files by hand. + +Plugins are stored in `/var/lib/docker/plugins`. For instance: + +```bash +# ls -la /var/lib/docker/plugins +total 20 +drwx------ 4 root root 4096 Aug 8 18:03 . +drwx--x--x 12 root root 4096 Aug 8 17:53 .. +drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 +-rw------- 1 root root 2107 Aug 8 18:03 plugins.json +``` + +`plugins.json` is an inventory of all installed plugins. For example: + +```bash +# cat plugins.json +{ + "cd851ce43a403": { + "plugin": { + "Manifest": { + "Args": { + "Value": null, + "Settable": null, + "Description": "", + "Name": "" + }, + "Env": null, + "Devices": null, + "Mounts": null, + "Capabilities": [ + "CAP_SYS_ADMIN" + ], + "ManifestVersion": "v0.1", + "Description": "sshFS plugin for Docker", + "Documentation": "https://docs.docker.com/engine/extend/plugins/", + "Interface": { + "Socket": "sshfs.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Entrypoint": [ + "/go/bin/docker-volume-sshfs" + ], + "Workdir": "", + "User": {}, + "Network": { + "Type": "host" + } + }, + "Config": { + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] + }, + "Active": true, + "Tag": "latest", + "Name": "vieux/sshfs", + "Id": "cd851ce43a403" + } + } +} +``` + +Each folder represents a plugin. For example: + +```bash +# ls -la /var/lib/docker/plugins/cd851ce43a403 +total 12 +drwx------ 19 root root 4096 Aug 8 17:56 rootfs +-rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json +-rw------- 1 root root 347 Aug 8 17:56 manifest.json +``` + +`rootfs` represents the root filesystem of the plugin. In this example, it was +created from a Dockerfile as follows: + +>**Note:** `/run/docker/plugins` is mandatory for docker to communicate with +the plugin._ + +```bash +$ git clone github.com/vieux/docker-volume-sshfs +$ cd docker-volume-sshfs +$ docker build -t rootfs . +$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created +$ mkdir -p /var/lib/docker/plugins/$id/rootfs +$ docker export "$id" | tar -x -C /var/lib/docker/plugins/$id/rootfs +$ docker rm -vf "$id" +$ docker rmi rootfs +``` + +`manifest.json` describes the plugin and `plugin-config.json` contains some +runtime parameters. For example: + +```bash +# cat manifest.json +{ + "manifestVersion": "v0.1", + "description": "sshFS plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["/go/bin/docker-volume-sshfs"], + "network": { + "type": "host" + }, + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "sshfs.sock" + }, + "capabilities": ["CAP_SYS_ADMIN"] +} +``` + +In this example, you can see the plugin is a volume driver, requires the +`CAP_SYS_ADMIN` capability, `host networking`, `/go/bin/docker-volume-sshfs` as +entrypoint and is going to use `/run/docker/plugins/sshfs.sock` to communicate +with the Docker Engine. + +```bash +# cat plugin-config.json +{ + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] +} +``` + +This plugin doesn't require runtime parameters. + +Both `manifest.json` and `plugin-config.json` are part of the `plugins.json`. +`manifest.json` is read-only and `plugin-config.json` is read-write. + +To summarize, follow the steps below to create a plugin: + +0. Choose a name for the plugin. Plugin name uses the same format as images, +for example: `/`. +1. Create a rootfs in `/var/lib/docker/plugins/$id/rootfs`. +2. Create manifest.json file in `/var/lib/docker/plugins/$id/`. +3. Create a `plugin-config.json` if needed. +4. Create or add a section to `/var/lib/docker/plugins/plugins.json`. Use + `/` as “Name” and `$id` as “Id”. +5. Restart the Docker Engine. +6. Run `docker plugin ls`. + * If your plugin is listed as `ENABLED=true`, you can push it to the + registry. + * If the plugin is not listed or if `ENABLED=false`, something went wrong. + Check the daemon logs for errors. +7. If you are not already logged in, use `docker login` to authenticate against + a registry. +8. Run `docker plugin push /` to push the plugin. diff --git a/docs/extend/legacy/plugins_authorization.md b/docs/extend/plugins_authorization.md similarity index 97% rename from docs/extend/legacy/plugins_authorization.md rename to docs/extend/plugins_authorization.md index 8630bfb761..572c6e99be 100644 --- a/docs/extend/legacy/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -6,13 +6,17 @@ keywords = ["security, authorization, authentication, docker, documentation, plu aliases = ["/engine/extend/authorization/"] [menu.main] parent = "engine_extend" -weight = -1 +weight = 4 +++ # Create an authorization plugin +This document describes Docker Engine authorization plugins generally +available in Docker Engine 1.12 and earlier. To view information on plugins +managed by Docker Engine, refer to [Docker Engine plugin system](plugins.md). + Docker's out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The same is true for callers using Docker's remote API to contact the daemon. If you diff --git a/docs/extend/legacy/plugins_network.md b/docs/extend/plugins_network.md similarity index 91% rename from docs/extend/legacy/plugins_network.md rename to docs/extend/plugins_network.md index ec1ccddfea..f1d72ceeae 100644 --- a/docs/extend/legacy/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -5,11 +5,16 @@ description = "Network driver plugins." keywords = ["Examples, Usage, plugins, docker, documentation, user guide"] [menu.main] parent = "engine_extend" +weight=5 +++ # Engine network driver plugins +This document describes Docker Engine network driver plugins generally +available in Docker Engine 1.12 and earlier. To view information on plugins +managed by Docker Engine, refer to [Docker Engine plugin system](plugins.md). + Docker Engine network plugins enable Engine deployments to be extended to support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different. Network driver plugins are supported via the diff --git a/docs/extend/legacy/plugins_volume.md b/docs/extend/plugins_volume.md similarity index 96% rename from docs/extend/legacy/plugins_volume.md rename to docs/extend/plugins_volume.md index 66afa7d439..45eb25eb04 100644 --- a/docs/extend/legacy/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -5,11 +5,16 @@ description = "How to manage data with external volume plugins" keywords = ["Examples, Usage, volume, docker, data, volumes, plugin, api"] [menu.main] parent = "engine_extend" +weight=6 +++ # Write a volume plugin +This document describes Docker Engine volume plugins generally available in +Docker Engine 1.12 and earlier. To view information on plugins managed by Docker +Engine, refer to [Docker Engine plugin system](plugins.md). + Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Engine host. See the [plugin From 388791f1a96671b0ad82b09ef30053c98fa300f6 Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Tue, 16 Aug 2016 14:56:47 -0700 Subject: [PATCH 131/534] fix broken link Signed-off-by: Victor Vieux --- docs/extend/menu.md | 1 + docs/extend/plugins.md | 4 ++-- docs/extend/plugins_volume.md | 4 ---- 3 files changed, 3 insertions(+), 6 deletions(-) diff --git a/docs/extend/menu.md b/docs/extend/menu.md index 026e56427f..1c523e2dfa 100644 --- a/docs/extend/menu.md +++ b/docs/extend/menu.md @@ -17,3 +17,4 @@ weight = 0 Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: * [Understand Docker plugins](plugins.md) +* [Write a volume plugin](plugins_volume.md) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index fb7af836e5..cba09f5cd5 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -97,7 +97,7 @@ started without error. ``` -5. Verify the plugin successfully crated the volume. +5. Verify the plugin successfully created the volume. ```bash $ docker volume ls @@ -109,7 +109,7 @@ started without error. You can stop a plugin with the `docker plugin disable` command or remove a plugin with `docker plugin remove`. -See the [command line reference](../engine/reference/commandline/) for more +See the [command line reference](../reference/commandline/index.md) for more information. ## How to develop a plugin diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 45eb25eb04..b20732febe 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -11,10 +11,6 @@ weight=6 # Write a volume plugin -This document describes Docker Engine volume plugins generally available in -Docker Engine 1.12 and earlier. To view information on plugins managed by Docker -Engine, refer to [Docker Engine plugin system](plugins.md). - Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Engine host. See the [plugin From 60f34524260e4f402fbf86e608cc172d2a6e505d Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Fri, 19 Aug 2016 13:14:03 +0200 Subject: [PATCH 132/534] Remove -run flag from commit command. This flag has been deprecated in version below 1.10 so it's safe to remove now, according to our deprecation policy. Signed-off-by: Vincent Demeester --- docs/deprecated.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 668d8e89b4..907092c6a4 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -172,6 +172,15 @@ The single-dash (`-help`) was removed, in favor of the double-dash `--help` docker -help docker [COMMAND] -help +### `--run` flag on docker commit + +**Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** + +**Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/)** + +The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor +of the `--changes` flag that allows to pass `Dockerfile` commands. + ### Interacting with V1 registries @@ -186,3 +195,4 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the - DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE - DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE + From 7b10177b5ef30f9bd94ccc501e35b2929b88f46d Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Wed, 24 Aug 2016 10:09:33 +0800 Subject: [PATCH 133/534] Modify incorrect repository URL Signed-off-by: yuexiao-wang --- docs/extend/plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins.md b/docs/extend/plugins.md index cba09f5cd5..fd17441714 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/plugins.md @@ -201,7 +201,7 @@ created from a Dockerfile as follows: the plugin._ ```bash -$ git clone github.com/vieux/docker-volume-sshfs +$ git clone https://github.com/vieux/docker-volume-sshfs $ cd docker-volume-sshfs $ docker build -t rootfs . $ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created From 65dd620c041b6d18b8623f44f32b561b15bf9371 Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Thu, 25 Aug 2016 11:47:33 +0800 Subject: [PATCH 134/534] Replace docker command from 'docker daemon' to 'dockerd' Signed-off-by: yuexiao-wang --- docs/extend/plugins_authorization.md | 2 +- docs/reference/commandline/dockerd.md | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 572c6e99be..f5c3a882b9 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -112,7 +112,7 @@ Authorization plugins can be loaded without restarting the daemon. Refer to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. ```bash -$ docker daemon --authorization-plugin=plugin1 --authorization-plugin=plugin2,... +$ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... ``` Docker's authorization subsystem supports multiple `--authorization-plugin` parameters. diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ccf120c3b9..8231060f7d 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -574,7 +574,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. **size** cannot be smaller than **btrfs.min_space**. Example use: - $ docker daemon -s btrfs --storage-opt btrfs.min_space=10G + $ dockerd -s btrfs --storage-opt btrfs.min_space=10G #### Overlay2 options @@ -1213,7 +1213,7 @@ The `--tls*` options enable use of specific certificates for individual daemons. Example script for a separate “bootstrap” instance of the Docker daemon without network: ```bash -$ docker daemon \ +$ dockerd \ -H unix:///var/run/docker-bootstrap.sock \ -p /var/run/docker-bootstrap.pid \ --iptables=false \ From e53a242c93c72d5e1be8b4c5e4cd93b4831e8c27 Mon Sep 17 00:00:00 2001 From: ohmystack Date: Mon, 29 Aug 2016 16:00:52 +0800 Subject: [PATCH 135/534] Add docs for "--containerd" in dockerd Signed-off-by: ohmystack --- docs/reference/commandline/dockerd.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8231060f7d..079e318a63 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -594,6 +594,14 @@ The Docker daemon relies on a (invoked via the `containerd` daemon) as its interface to the Linux kernel `namespaces`, `cgroups`, and `SELinux`. +By default, the Docker daemon automatically starts `containerd`. If you want to +control `containerd` startup, manually start `containerd` and pass the path to +the `containerd` socket using the `--containerd` flag. For example: + +```bash +$ dockerd --containerd /var/run/dev/docker-containerd.sock +``` + Runtimes can be registered with the daemon either via the configuration file or using the `--add-runtime` command line argument. From f4bb9dabe1d05fdf2e85ab3433741922e7c03666 Mon Sep 17 00:00:00 2001 From: Charles Smith Date: Mon, 29 Aug 2016 16:37:01 -0700 Subject: [PATCH 136/534] some fixes to clean up new plugin system docs Signed-off-by: Charles Smith --- docs/extend/{plugins.md => index.md} | 6 +++--- docs/extend/legacy_plugins.md | 11 ++++++----- docs/extend/menu.md | 11 +++-------- docs/extend/plugin_api.md | 6 +++--- docs/extend/plugins_authorization.md | 6 +++--- docs/extend/plugins_network.md | 6 +++--- docs/extend/plugins_volume.md | 4 ++-- 7 files changed, 23 insertions(+), 27 deletions(-) rename docs/extend/{plugins.md => index.md} (98%) diff --git a/docs/extend/plugins.md b/docs/extend/index.md similarity index 98% rename from docs/extend/plugins.md rename to docs/extend/index.md index fd17441714..4e77c698e6 100644 --- a/docs/extend/plugins.md +++ b/docs/extend/index.md @@ -3,8 +3,8 @@ aliases = [ "/engine/extend/" ] -title = "New Plugin System" -description = "How to operate and create a plugin with the new system" +title = "Managed plugin system" +description = "How develop and use a plugin with the managed plugin system" keywords = ["API, Usage, plugins, documentation, developer"] advisory = "experimental" [menu.main] @@ -13,7 +13,7 @@ weight=1 +++ -# Docker Engine plugin system +# Docker Engine managed plugin system This document describes the plugin system available today in the **experimental build** of Docker 1.12: diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 02874f315e..0d3404831c 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -1,6 +1,7 @@ -# Understand legacy Docker Engine plugins +# Use Docker Engine plugins This document describes the Docker Engine plugins generally available in Docker -Engine 1.12 and earlier. To view information on plugins managed by Docker -Engine, refer to [Docker Engine plugin system](plugins.md). +Engine. To view information on plugins managed by Docker Engine currently in +experimental status, refer to [Docker Engine plugin system](index.md). You can extend the capabilities of the Docker Engine by loading third-party plugins. This page explains the types of plugins and provides links to several @@ -21,7 +22,7 @@ volume and network plugins for Docker. ## Types of plugins -Plugins extend Docker's functionality. They come in specific types. For +Plugins extend Docker's functionality. They come in specific types. For example, a [volume plugin](plugins_volume.md) might enable Docker volumes to persist across multiple Docker hosts and a [network plugin](plugins_network.md) might provide network plumbing. diff --git a/docs/extend/menu.md b/docs/extend/menu.md index 1c523e2dfa..4b9955038a 100644 --- a/docs/extend/menu.md +++ b/docs/extend/menu.md @@ -1,7 +1,7 @@ -## New Docker Plugin System - -Currently, you can extend Docker Engine by adding a plugin. This section contains the following topics: - -* [Understand Docker plugins](plugins.md) -* [Write a volume plugin](plugins_volume.md) + diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 1f55b6d528..5801ba9151 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -14,9 +14,9 @@ weight=7 Docker plugins are out-of-process extensions which add capabilities to the Docker Engine. -This document describes the Docker Engine plugin API generally available in -Docker Engine 1.12 and earlier. To view information on plugins managed by Docker -Engine, refer to [Docker Engine plugin system](plugins.md). +This document describes the Docker Engine plugin API. To view information on +plugins managed by Docker Engine currently in experimental status, refer to +[Docker Engine plugin system](index.md). This page is intended for people who want to develop their own Docker plugin. If you just want to learn about or use Docker plugins, look diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index f5c3a882b9..2c3d79e25d 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -13,9 +13,9 @@ weight = 4 # Create an authorization plugin -This document describes Docker Engine authorization plugins generally -available in Docker Engine 1.12 and earlier. To view information on plugins -managed by Docker Engine, refer to [Docker Engine plugin system](plugins.md). +This document describes the Docker Engine plugins generally available in Docker +Engine. To view information on plugins managed by Docker Engine currently in +experimental status, refer to [Docker Engine plugin system](index.md). Docker's out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index f1d72ceeae..7a107e2ce3 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -12,8 +12,8 @@ weight=5 # Engine network driver plugins This document describes Docker Engine network driver plugins generally -available in Docker Engine 1.12 and earlier. To view information on plugins -managed by Docker Engine, refer to [Docker Engine plugin system](plugins.md). +available in Docker Engine. To view information on plugins +managed by Docker Engine, refer to [Docker Engine plugin system](index.md). Docker Engine network plugins enable Engine deployments to be extended to support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN @@ -46,7 +46,7 @@ commands. For example, $ docker network create --driver weave mynet -Some network driver plugins are listed in [plugins](plugins.md) +Some network driver plugins are listed in [plugins](legacy_plugins.md) The `mynet` network is now owned by `weave`, so subsequent commands referring to that network will be sent to the plugin, diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index b20732febe..f9c93a3de1 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -13,8 +13,8 @@ weight=6 Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems, such as Amazon EBS, and enable data volumes to persist -beyond the lifetime of a single Engine host. See the [plugin -documentation](plugins.md) for more information. +beyond the lifetime of a single Engine host. See the +[plugin documentation](legacy_plugins.md) for more information. ## Changelog From 7c48da2c2c54c7b98c1f8f13d120faecb121019e Mon Sep 17 00:00:00 2001 From: unclejack Date: Mon, 5 Sep 2016 12:38:50 +0300 Subject: [PATCH 137/534] docs: mention NFS support for contiv volume plugin Signed-off-by: Cristian Staretu --- docs/extend/legacy_plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 0d3404831c..d9d4e5d192 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -56,7 +56,7 @@ Plugin ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. -[Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption using ceph underneath. +[Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. [DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. [Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. From dd36f78ef21e7c0bf0c0a07806c213dcea7d3fd7 Mon Sep 17 00:00:00 2001 From: Vivek Goyal Date: Wed, 31 Aug 2016 10:13:04 -0400 Subject: [PATCH 138/534] devmapper: Provide a knob dm.xfs_nospace_max_retries When xfs filesystem is being used on top of thin pool, xfs can get ENOSPC errors from thin pool when thin pool is full. As of now xfs retries the IO and keeps on retrying and does not give up. This can result in container application being stuck for a very long time. In fact I have seen instances of unkillable processes. So that means once thin pool is full and process gets stuck, container can't be stopped/killed either and only option left seems to be power recycle of the box. In another instance, writer did not block but failed after a while. But when I tried to exit/stop the container, unmounting xfs hanged and only thing I could do was power cycle the machine. Now upstream kernel has committed patches where it allows user space to customize user space behavior in case of errors. One of the knobs is max_retries, which specifies how many times an IO should be retried when ENOSPC is encountered. This patch sets provides a tunable knob (dm.xfs_nospace_max_retries) so that user can specify value for max_retries and tune xfs behavior. If one sets this value to 0, xfs will not retry IO when ENOSPC error is encountered. It will instead give up and shutdown filesystem. This knob can be useful if one is running into unkillable processes/containers issue on top of xfs. Signed-off-by: Vivek Goyal --- docs/reference/commandline/dockerd.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 079e318a63..3552bf0fd9 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -552,6 +552,22 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ dockerd --storage-opt dm.min_free_space=10% ``` +* `dm.xfs_nospace_max_retries` + + Specifies the maximum number of retries XFS should attempt to complete + IO when ENOSPC (no space) error is returned by underlying storage device. + + By default XFS retries infinitely for IO to finish and this can result + in unkillable process. To change this behavior one can set + xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting + ENOSPC and will shutdown filesystem. + + Example use: + + ```bash + $ dockerd --storage-opt dm.xfs_nospace_max_retries=0 + ``` + #### ZFS options * `zfs.fsname` From 64a3035d08274256f547e81c589ec668cee8ba3d Mon Sep 17 00:00:00 2001 From: Justin Cormack Date: Sat, 6 Aug 2016 14:34:49 +0200 Subject: [PATCH 139/534] Begin process of deprecating MAINTAINER This may take some time, but start by pointing people at LABEL instead. MAINTAINER predates general LABEL and has basically no tooling, only allows a single item to be added, and is has been unofficially deprecated for some time, with many images not including it, but we have never specifically said that it should be replaced by LABEL as a better more generic metadata solution. Signed-off-by: Justin Cormack --- docs/deprecated.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 907092c6a4..c5c0b228ce 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -196,3 +196,7 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the - DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE - DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE +### `MAINTAINER` in Dockerfile +**Deprecated In Release: v1.13.0** + +`MAINTAINER` was an early very limited form of `LABEL` which should be used instead. From 3cb08c6be08a3759363a6ad19e9d798ca321a46e Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Wed, 7 Sep 2016 02:34:57 -0700 Subject: [PATCH 140/534] add some plugin manifest reference doc Signed-off-by: Victor Vieux --- docs/extend/index.md | 16 +-- docs/extend/manifest.md | 222 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 230 insertions(+), 8 deletions(-) create mode 100644 docs/extend/manifest.md diff --git a/docs/extend/index.md b/docs/extend/index.md index 4e77c698e6..4ce1eac1c5 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -151,7 +151,7 @@ drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 "Capabilities": [ "CAP_SYS_ADMIN" ], - "ManifestVersion": "v0.1", + "ManifestVersion": "v0", "Description": "sshFS plugin for Docker", "Documentation": "https://docs.docker.com/engine/extend/plugins/", "Interface": { @@ -212,23 +212,23 @@ $ docker rmi rootfs ``` `manifest.json` describes the plugin and `plugin-config.json` contains some -runtime parameters. For example: +runtime parameters. [See the Plugins Manifest reference](manifest.md). For example: ```bash # cat manifest.json { - "manifestVersion": "v0.1", + "manifestVersion": "v0", "description": "sshFS plugin for Docker", "documentation": "https://docs.docker.com/engine/extend/plugins/", "entrypoint": ["/go/bin/docker-volume-sshfs"], "network": { "type": "host" }, - "interface" : { - "types": ["docker.volumedriver/1.0"], - "socket": "sshfs.sock" - }, - "capabilities": ["CAP_SYS_ADMIN"] + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "sshfs.sock" + }, + "capabilities": ["CAP_SYS_ADMIN"] } ``` diff --git a/docs/extend/manifest.md b/docs/extend/manifest.md new file mode 100644 index 0000000000..d47efce8e8 --- /dev/null +++ b/docs/extend/manifest.md @@ -0,0 +1,222 @@ + + +# Plugin Manifest Version 0 of Plugin V2 + +This document outlines the format of the V0 plugin manifest. The plugin +manifest described herein was introduced in the Docker daemon (experimental version) in the [v1.12.0 +release](https://github.com/docker/docker/commit/f37117045c5398fd3dca8016ea8ca0cb47e7312b). + +Plugin manifests describe the various constituents of a docker plugin. Plugin +manifests can be serialized to JSON format with the following media types: + +Manifest Type | Media Type +------------- | ------------- +manifest | "application/vnd.docker.plugin.v0+json" + + +## *Manifest* Field Descriptions + +Manifest provides the base accessible fields for working with V0 plugin format + in the registry. + +- **`manifestVersion`** *string* + + version of the plugin manifest (This version uses V0) + +- **`description`** *string* + + description of the plugin + +- **`documentation`** *string* + + link to the documentation about the plugin + +- **`interface`** *PluginInterface* + + interface implemented by the plugins, struct consisting of the following fields + + - **`types`** *string array* + + types indicate what interface(s) the plugin currently implements. + + currently supported: + + - **docker.volumedriver/1.0** + + - **`socket`** *string* + + socket is the name of the socket the engine should use to communicate with the plugins. + the socket will be created in `/run/docker/plugins`. + + +- **`entrypoint`** *string array* + + entrypoint of the plugin, see [`ENTRYPOINT`](../reference/builder.md#entrypoint) + +- **`workdir`** *string* + + workdir of the plugin, see [`WORKDIR`](../reference/builder.md#workdir) + +- **`network`** *PluginNetwork* + + network of the plugin, struct consisting of the following fields + + - **`type`** *string* + + network type. + + currently supported: + + - **bridge** + - **host** + - **none** + +- **`capabilities`** *array* + + capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) + +- **`mounts`** *PluginMount array* + + mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) + + - **`name`** *string* + + name of the mount. + + - **`description`** *string* + + description of the mount. + + - **`source`** *string* + + source of the mount. + + - **`destination`** *string* + + destination of the mount. + + - **`type`** *string* + + mount type. + + - **`options`** *string array* + + options of the mount. + +- **`devices`** *PluginDevice array* + + device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) + + - **`name`** *string* + + name of the device. + + - **`description`** *string* + + description of the device. + + - **`path`** *string* + + path of the device. + +- **`env`** *PluginEnv array* + + env of the plugin, struct consisting of the following fields + + - **`name`** *string* + + name of the env. + + - **`description`** *string* + + description of the env. + + - **`value`** *string* + + value of the env. + +- **`args`** *PluginArgs* + + args of the plugin, struct consisting of the following fields + + - **`name`** *string* + + name of the env. + + - **`description`** *string* + + description of the env. + + - **`value`** *string array* + + values of the args. + + +## Example Manifest + +*Example showing the 'tiborvass/no-remove' plugin manifest.* + +``` +{ + "manifestVersion": "v0", + "description": "A test plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["plugin-no-remove", "/data"], + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "plugins.sock" + }, + "network": { + "type": "host" + }, + + "mounts": [ + { + "source": "/data", + "destination": "/data", + "type": "bind", + "options": ["shared", "rbind"] + }, + { + "destination": "/foobar", + "type": "tmpfs" + } + ], + + "args": { + "name": "args", + "description": "command line arguments", + "value": [] + }, + + "env": [ + { + "name": "DEBUG", + "description": "If set, prints debug messages", + "value": "1" + } + ], + + "devices": [ + { + "name": "device", + "description": "a host device to mount", + "path": "/dev/cpu_dma_latency" + } + ] +} + +``` From 43a673c72df7496e964c72618d8dec80ee6ba10f Mon Sep 17 00:00:00 2001 From: Phil Estes Date: Tue, 9 Aug 2016 12:14:38 -0400 Subject: [PATCH 141/534] Remove --read-only restriction when user ns enabled The restriction is no longer necessary given changes at the runc layer related to mount options of the rootfs. Also cleaned up the docs on restrictions left for userns enabled mode. Re-enabled tests related to --read-only when testing a userns-enabled daemon in integration-cli. Docker-DCO-1.1-Signed-off-by: Phil Estes (github: estesp) --- docs/reference/commandline/dockerd.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3552bf0fd9..658a18a48a 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -955,16 +955,16 @@ This option will completely disable user namespace mapping for the container's u The following standard Docker features are currently incompatible when running a Docker daemon with user namespaces enabled: - - sharing PID or NET namespaces with the host (`--pid=host` or `--network=host`) - - A `--read-only` container filesystem (this is a Linux kernel restriction against remounting with modified flags of a currently mounted filesystem when inside a user namespace) - - external (volume or graph) drivers which are unaware/incapable of using daemon user mappings + - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`) - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`) In general, user namespaces are an advanced feature and will require coordination with other capabilities. For example, if volumes are mounted from the host, file ownership will have to be pre-arranged if the user or administrator wishes the containers to have expected access to the volume -contents. +contents. Note that when using external volume or graph driver plugins, those +external software programs must be made aware of user and group mapping ranges +if they are to work seamlessly with user namespace support. Finally, while the `root` user inside a user namespaced container process has many of the expected admin privileges that go along with being the superuser, the From 1bf9161e1c37690ce947247af7df1897f28c69bf Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 16 Sep 2016 17:22:58 +0200 Subject: [PATCH 142/534] Use "sudo" for dockerd examples Because we standardize on using a non-privileged prompt (`$`) instead of `#`, replacing the examples to use `sudo` instead to indicate this has to be run as root. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 241 ++++++++++++++++---------- 1 file changed, 153 insertions(+), 88 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 658a18a48a..3c393b004b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -123,26 +123,32 @@ find examples of using Systemd socket activation with Docker and Systemd in the You can configure the Docker daemon to listen to multiple sockets at the same time using multiple `-H` options: - # listen using the default unix socket, and on 2 specific IP addresses on this host. - dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 +```bash +# listen using the default unix socket, and on 2 specific IP addresses on this host. +$ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 +``` The Docker client will honor the `DOCKER_HOST` environment variable to set the `-H` flag for the client. - $ docker -H tcp://0.0.0.0:2375 ps - # or - $ export DOCKER_HOST="tcp://0.0.0.0:2375" - $ docker ps - # both are equal +```bash +$ docker -H tcp://0.0.0.0:2375 ps +# or +$ export DOCKER_HOST="tcp://0.0.0.0:2375" +$ docker ps +# both are equal +``` Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than the empty string is equivalent to setting the `--tlsverify` flag. The following are equivalent: - $ docker --tlsverify ps - # or - $ export DOCKER_TLS_VERIFY=1 - $ docker ps +```bash +$ docker --tlsverify ps +# or +$ export DOCKER_TLS_VERIFY=1 +$ docker ps +``` The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes @@ -188,27 +194,31 @@ For example: `-H`, when empty, will default to the same value as when no `-H` was passed in. -`-H` also accepts short form for TCP bindings: - - `host:` or `host:port` or `:port` +`-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port` Run Docker in daemon mode: - $ sudo /dockerd -H 0.0.0.0:5555 & +```bash +$ sudo /dockerd -H 0.0.0.0:5555 & +``` Download an `ubuntu` image: - $ docker -H :5555 pull ubuntu +```bash +$ docker -H :5555 pull ubuntu +``` You can use multiple `-H`, for example, if you want to listen on both TCP and a Unix socket - # Run docker in daemon mode - $ sudo /dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & - # Download an ubuntu image, use default Unix socket - $ docker pull ubuntu - # OR use the TCP port - $ docker -H tcp://127.0.0.1:2375 pull ubuntu +```bash +# Run docker in daemon mode +$ sudo /dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & +# Download an ubuntu image, use default Unix socket +$ docker pull ubuntu +# OR use the TCP port +$ docker -H tcp://127.0.0.1:2375 pull ubuntu +``` ### Daemon storage-driver option @@ -272,29 +282,30 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. * `dm.thinpooldev` - Specifies a custom block storage device to use for the thin pool. + Specifies a custom block storage device to use for the thin pool. - If using a block device for device mapper storage, it is best to use `lvm` - to create and manage the thin-pool volume. This volume is then handed to Docker - to exclusively create snapshot volumes needed for images and containers. + If using a block device for device mapper storage, it is best to use `lvm` + to create and manage the thin-pool volume. This volume is then handed to Docker + to exclusively create snapshot volumes needed for images and containers. - Managing the thin-pool outside of Engine makes for the most feature-rich - method of having Docker utilize device mapper thin provisioning as the - backing storage for Docker containers. The highlights of the lvm-based - thin-pool management feature include: automatic or interactive thin-pool - resize support, dynamically changing thin-pool features, automatic thinp - metadata checking when lvm activates the thin-pool, etc. + Managing the thin-pool outside of Engine makes for the most feature-rich + method of having Docker utilize device mapper thin provisioning as the + backing storage for Docker containers. The highlights of the lvm-based + thin-pool management feature include: automatic or interactive thin-pool + resize support, dynamically changing thin-pool features, automatic thinp + metadata checking when lvm activates the thin-pool, etc. - As a fallback if no thin pool is provided, loopback files are - created. Loopback is very slow, but can be used without any - pre-configuration of storage. It is strongly recommended that you do - not use loopback in production. Ensure your Engine daemon has a - `--storage-opt dm.thinpooldev` argument provided. + As a fallback if no thin pool is provided, loopback files are + created. Loopback is very slow, but can be used without any + pre-configuration of storage. It is strongly recommended that you do + not use loopback in production. Ensure your Engine daemon has a + `--storage-opt dm.thinpooldev` argument provided. - Example use: + Example use: - $ dockerd \ - --storage-opt dm.thinpooldev=/dev/mapper/thin-pool + ```bash + $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool + ``` * `dm.basesize` @@ -310,7 +321,10 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.basesize=50G + ```bash + $ sudo dockerd --storage-opt dm.basesize=50G + ``` + This will increase the base device size to 50G. The Docker daemon will throw an error if existing base device size is larger than 50G. A user can use @@ -320,19 +334,23 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. that may already be initialized and inherited by pulled images. Typically, a change to this value requires additional steps to take effect: - $ sudo service docker stop - $ sudo rm -rf /var/lib/docker - $ sudo service docker start + ```bash + $ sudo service docker stop + $ sudo rm -rf /var/lib/docker + $ sudo service docker start + ``` Example use: - $ dockerd --storage-opt dm.basesize=20G + ```bash + $ sudo dockerd --storage-opt dm.basesize=20G + ``` * `dm.loopdatasize` > **Note**: - > This option configures devicemapper loopback, which should not - > be used in production. + > This option configures devicemapper loopback, which should not + > be used in production. Specifies the size to use when creating the loopback file for the "data" device which is used for the thin pool. The default size is @@ -341,7 +359,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.loopdatasize=200G + ```bash + $ sudo dockerd --storage-opt dm.loopdatasize=200G + ``` * `dm.loopmetadatasize` @@ -356,7 +376,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.loopmetadatasize=4G + ```bash + $ sudo dockerd --storage-opt dm.loopmetadatasize=4G + ``` * `dm.fs` @@ -365,7 +387,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.fs=ext4 + ```bash + $ sudo dockerd --storage-opt dm.fs=ext4 + ``` * `dm.mkfsarg` @@ -373,7 +397,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" + ```bash + $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" + ``` * `dm.mountopt` @@ -381,7 +407,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.mountopt=nodiscard + ```bash + $ sudo dockerd --storage-opt dm.mountopt=nodiscard + ``` * `dm.datadev` @@ -395,9 +423,11 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 + ```bash + $ sudo dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 + ``` * `dm.metadatadev` @@ -411,13 +441,17 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. If setting up a new metadata pool it is required to be valid. This can be achieved by zeroing the first 4k to indicate empty metadata, like this: - $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 + ```bash + $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 + ``` Example use: - $ dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 + ```bash + $ sudo dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 + ``` * `dm.blocksize` @@ -426,7 +460,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.blocksize=512K + ```bash + $ sudo dockerd --storage-opt dm.blocksize=512K + ``` * `dm.blkdiscard` @@ -440,7 +476,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.blkdiscard=false + ```bash + $ sudo dockerd --storage-opt dm.blkdiscard=false + ``` * `dm.override_udev_sync_check` @@ -450,10 +488,12 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. To view the `udev` sync support of a Docker daemon that is using the `devicemapper` driver, run: - $ docker info - [...] - Udev Sync Supported: true - [...] + ```bash + $ docker info + [...] + Udev Sync Supported: true + [...] + ``` When `udev` sync support is `true`, then `devicemapper` and udev can coordinate the activation and deactivation of devices for containers. @@ -466,7 +506,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. To allow the `docker` daemon to start, regardless of `udev` sync not being supported, set `dm.override_udev_sync_check` to true: - $ dockerd --storage-opt dm.override_udev_sync_check=true + ```bash + $ sudo dockerd --storage-opt dm.override_udev_sync_check=true + ``` When this value is `true`, the `devicemapper` continues and simply warns you the errors are happening. @@ -496,7 +538,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd --storage-opt dm.use_deferred_removal=true + ```bash + $ sudo dockerd --storage-opt dm.use_deferred_removal=true + ``` * `dm.use_deferred_deletion` @@ -510,9 +554,11 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. To avoid this failure, enable both deferred device deletion and deferred device removal on the daemon. - $ dockerd \ - --storage-opt dm.use_deferred_deletion=true \ - --storage-opt dm.use_deferred_removal=true + ```bash + $ sudo dockerd \ + --storage-opt dm.use_deferred_deletion=true \ + --storage-opt dm.use_deferred_removal=true + ``` With these two options enabled, if a device is busy when the driver is deleting a container, the driver marks the device as deleted. Later, when @@ -549,7 +595,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: ```bash - $ dockerd --storage-opt dm.min_free_space=10% + $ sudo dockerd --storage-opt dm.min_free_space=10% ``` * `dm.xfs_nospace_max_retries` @@ -565,7 +611,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: ```bash - $ dockerd --storage-opt dm.xfs_nospace_max_retries=0 + $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 ``` #### ZFS options @@ -578,7 +624,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Example use: - $ dockerd -s zfs --storage-opt zfs.fsname=zroot/docker + ```bash + $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker + ``` #### Btrfs options @@ -590,7 +638,10 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. **size** cannot be smaller than **btrfs.min_space**. Example use: - $ dockerd -s btrfs --storage-opt btrfs.min_space=10G + + ```bash + $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G + ``` #### Overlay2 options @@ -615,7 +666,7 @@ control `containerd` startup, manually start `containerd` and pass the path to the `containerd` socket using the `--containerd` flag. For example: ```bash -$ dockerd --containerd /var/run/dev/docker-containerd.sock +$ sudo dockerd --containerd /var/run/dev/docker-containerd.sock ``` Runtimes can be registered with the daemon either via the @@ -639,9 +690,11 @@ The following is an example adding 2 runtimes via the configuration: This is the same example via the command line: - $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement +```bash +$ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement +``` -**Note**: defining runtime arguments via the command line is not supported. +> **Note**: defining runtime arguments via the command line is not supported. ## Options for the runtime @@ -656,14 +709,18 @@ cgroups. You can specify only specify `cgroupfs` or `systemd`. If you specify This example sets the `cgroupdriver` to `systemd`: - $ sudo dockerd --exec-opt native.cgroupdriver=systemd +```bash +$ sudo dockerd --exec-opt native.cgroupdriver=systemd +``` Setting this option applies to all containers the daemon launches. Also Windows Container makes use of `--exec-opt` for special purpose. Docker user can specify default container isolation technology with this, for example: - $ dockerd --exec-opt isolation=hyperv +```bash +$ sudo dockerd --exec-opt isolation=hyperv +``` Will make `hyperv` the default isolation technology on Windows. If no isolation value is specified on daemon start, on Windows client, the default is @@ -671,11 +728,19 @@ value is specified on daemon start, on Windows client, the default is ## Daemon DNS options -To set the DNS server for all Docker containers, use -`dockerd --dns 8.8.8.8`. +To set the DNS server for all Docker containers, use: + +```bash +$ sudo dockerd --dns 8.8.8.8 +``` + + +To set the DNS search domain for all Docker containers, use: + +```bash +$ sudo dockerd --dns-search example.com +``` -To set the DNS search domain for all Docker containers, use -`dockerd --dns-search example.com`. ## Insecure registries @@ -770,7 +835,7 @@ using the `--cluster-store-opt` flag, specifying the paths to PEM encoded files. For example: ```bash -dockerd \ +$ sudo dockerd \ --cluster-advertise 192.168.1.2:2376 \ --cluster-store etcd://192.168.1.2:2379 \ --cluster-store-opt kv.cacertfile=/path/to/ca.pem \ @@ -820,7 +885,7 @@ authorization plugins when you start the Docker `daemon` using the `--authorization-plugin=PLUGIN_ID` option. ```bash -dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... +$ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... ``` The `PLUGIN_ID` value is either the plugin's name or a path to its specification @@ -891,10 +956,10 @@ startup will fail with an error message. > *before* the `--userns-remap` option is enabled. Once these files exist, the > daemon can be (re)started and range assignment on user creation works properly. -*Example: starting with default Docker user management:* +**Example: starting with default Docker user management:** ```bash -$ dockerd --userns-remap=default +$ sudo dockerd --userns-remap=default ``` When `default` is provided, Docker will create - or find the existing - user and group @@ -1237,7 +1302,7 @@ The `--tls*` options enable use of specific certificates for individual daemons. Example script for a separate “bootstrap” instance of the Docker daemon without network: ```bash -$ dockerd \ +$ sudo dockerd \ -H unix:///var/run/docker-bootstrap.sock \ -p /var/run/docker-bootstrap.pid \ --iptables=false \ From d2bb042f10ba7338bc4fa795e55720fac01dbf00 Mon Sep 17 00:00:00 2001 From: Michael Crosby Date: Mon, 27 Jun 2016 14:38:47 -0700 Subject: [PATCH 143/534] Add init process for zombie fighting This adds a small C binary for fighting zombies. It is mounted under `/dev/init` and is prepended to the args specified by the user. You enable it via a daemon flag, `dockerd --init`, as it is disable by default for backwards compat. You can also override the daemon option or specify this on a per container basis with `docker run --init=true|false`. You can test this by running a process like this as the pid 1 in a container and see the extra zombie that appears in the container as it is running. ```c int main(int argc, char ** argv) { pid_t pid = fork(); if (pid == 0) { pid = fork(); if (pid == 0) { exit(0); } sleep(3); exit(0); } printf("got pid %d and exited\n", pid); sleep(20); } ``` Signed-off-by: Michael Crosby --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3c393b004b..5fff237b1c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -48,6 +48,7 @@ Options: -H, --host=[] Daemon socket(s) to connect to --help Print usage --icc=true Enable inter-container communication + --init Run an init inside containers to forward signals and reap processes --insecure-registry=[] Enable insecure registry communication --ip=0.0.0.0 Default IP when binding container ports --ip-forward=true Enable net.ipv4.ip_forward @@ -1140,6 +1141,7 @@ This is a full example of the allowed configuration options on Linux: "group": "", "cgroup-parent": "", "default-ulimits": {}, + "init": false, "ipv6": false, "iptables": false, "ip-forward": false, From e9dcfaf3b741339f44231902b4c3b0738e3c8648 Mon Sep 17 00:00:00 2001 From: Jeff Silberman Date: Tue, 20 Sep 2016 20:49:30 -0700 Subject: [PATCH 144/534] Add Portworx Volume Plugin Description Signed-off-by: Jeff Silberman --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index d9d4e5d192..aa4fa4a9b3 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -70,6 +70,7 @@ Plugin [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. [Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. +[Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. [Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. From bedaad92db72bed3ccd81649f568907eaa7679ef Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 Sep 2016 22:38:18 +0200 Subject: [PATCH 145/534] Deprecate "daemon" subcommand The daemon is in a separate (dockerd) binary since docker 1.12, so should no longer be used. This marks the command as deprecated, and adds it to the deprecated features list. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index c5c0b228ce..e7a301aa27 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -17,6 +17,13 @@ To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](index.md#feature-deprecation-policy). +### `docker daemon` subcommand +**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** + +**Target For Removal In Release: v1.16** + +The daemon is moved to a separate binary (`dockerd`), and should be used instead. + ### Three argument form in `docker import` **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** From 038fc8a093887180fefac1a6c349be196bb7cf48 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Tue, 27 Sep 2016 12:51:42 +0200 Subject: [PATCH 146/534] configure docker-init binary path Signed-off-by: Antonio Murdaca --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5fff237b1c..b558895ea5 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -49,6 +49,7 @@ Options: --help Print usage --icc=true Enable inter-container communication --init Run an init inside containers to forward signals and reap processes + --init-path Path to the docker-init binary --insecure-registry=[] Enable insecure registry communication --ip=0.0.0.0 Default IP when binding container ports --ip-forward=true Enable net.ipv4.ip_forward @@ -1142,6 +1143,7 @@ This is a full example of the allowed configuration options on Linux: "cgroup-parent": "", "default-ulimits": {}, "init": false, + "init-path": "/usr/libexec/docker-init", "ipv6": false, "iptables": false, "ip-forward": false, From 34d66cf07426aa932ad811c0ecb8acb3ec4b81f3 Mon Sep 17 00:00:00 2001 From: Antonio Murdaca Date: Sat, 24 Sep 2016 15:44:25 +0200 Subject: [PATCH 147/534] daemon: add --userland-proxy-path flag Signed-off-by: Antonio Murdaca --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b558895ea5..d134f6e102 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -78,6 +78,7 @@ Options: --tlskey=~/.docker/key.pem Path to TLS key file --tlsverify Use TLS and verify the remote --userland-proxy=true Use userland proxy for loopback traffic + --userland-proxy-path="" Path to the userland proxy binary --userns-remap User/Group setting for user namespaces -v, --version Print version information and quit ``` @@ -1149,6 +1150,7 @@ This is a full example of the allowed configuration options on Linux: "ip-forward": false, "ip-masq": false, "userland-proxy": false, + "userland-proxy-path": "/usr/libexec/docker-proxy", "ip": "0.0.0.0", "bridge": "", "bip": "", From 902d32a311426aa37d8bf49bb9e06b07248372c0 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sun, 25 Sep 2016 16:26:46 +0200 Subject: [PATCH 148/534] Improve --log-level help text This information was added in 1efc940e6f547760e5e8f4648acb120ff19fdc58, but removed again in a271eaeba224652e3a12af0287afbae6f82a9333 to make the help-output fit in a 80-chars terminal. This adds the available options again in the help output, and updates the CLI reference documentation to match actual output. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 124 +++++++++++++------------- 1 file changed, 62 insertions(+), 62 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index d134f6e102..0233476ce8 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -19,68 +19,68 @@ A self-sufficient runtime for containers. Options: - --add-runtime=[] Register an additional OCI compatible runtime - --api-cors-header Set CORS headers in the remote API - --authorization-plugin=[] Authorization plugins to load - -b, --bridge Attach containers to a network bridge - --bip Specify network bridge IP - --cgroup-parent Set parent cgroup for all containers - --cluster-advertise Address or interface name to advertise - --cluster-store URL of the distributed storage backend - --cluster-store-opt=map[] Set cluster store options - --config-file=/etc/docker/daemon.json Daemon configuration file - --containerd Path to containerd socket - -D, --debug Enable debug mode - --default-gateway Container default gateway IPv4 address - --default-gateway-v6 Container default gateway IPv6 address - --default-runtime=runc Default OCI runtime for containers - --default-ulimit=[] Default ulimits for containers - --disable-legacy-registry Disable contacting legacy registries - --dns=[] DNS server to use - --dns-opt=[] DNS options to use - --dns-search=[] DNS search domains to use - --exec-opt=[] Runtime execution options - --exec-root=/var/run/docker Root directory for execution state files - --fixed-cidr IPv4 subnet for fixed IPs - --fixed-cidr-v6 IPv6 subnet for fixed IPs - -G, --group=docker Group for the unix socket - -g, --graph=/var/lib/docker Root of the Docker runtime - -H, --host=[] Daemon socket(s) to connect to - --help Print usage - --icc=true Enable inter-container communication - --init Run an init inside containers to forward signals and reap processes - --init-path Path to the docker-init binary - --insecure-registry=[] Enable insecure registry communication - --ip=0.0.0.0 Default IP when binding container ports - --ip-forward=true Enable net.ipv4.ip_forward - --ip-masq=true Enable IP masquerading - --iptables=true Enable addition of iptables rules - --ipv6 Enable IPv6 networking - -l, --log-level=info Set the logging level - --label=[] Set key=value labels to the daemon - --live-restore Enables keeping containers alive during daemon downtime - --log-driver=json-file Default driver for container logs - --log-opt=map[] Default log driver options for containers - --max-concurrent-downloads=3 Set the max concurrent downloads for each pull - --max-concurrent-uploads=5 Set the max concurrent uploads for each push - --mtu Set the containers network MTU - --oom-score-adjust=-500 Set the oom_score_adj for the daemon - -p, --pidfile=/var/run/docker.pid Path to use for daemon PID file - --raw-logs Full timestamps without ANSI coloring - --registry-mirror=[] Preferred Docker registry mirror - -s, --storage-driver Storage driver to use - --selinux-enabled Enable selinux support - --storage-opt=[] Storage driver options - --swarm-default-advertise-addr Set default address or interface for swarm advertised address - --tls Use TLS; implied by --tlsverify - --tlscacert=~/.docker/ca.pem Trust certs signed only by this CA - --tlscert=~/.docker/cert.pem Path to TLS certificate file - --tlskey=~/.docker/key.pem Path to TLS key file - --tlsverify Use TLS and verify the remote - --userland-proxy=true Use userland proxy for loopback traffic - --userland-proxy-path="" Path to the userland proxy binary - --userns-remap User/Group setting for user namespaces - -v, --version Print version information and quit + --add-runtime value Register an additional OCI compatible runtime (default []) + --api-cors-header string Set CORS headers in the remote API + --authorization-plugin value Authorization plugins to load (default []) + --bip string Specify network bridge IP + -b, --bridge string Attach containers to a network bridge + --cgroup-parent string Set parent cgroup for all containers + --cluster-advertise string Address or interface name to advertise + --cluster-store string URL of the distributed storage backend + --cluster-store-opt value Set cluster store options (default map[]) + --config-file string Daemon configuration file (default "/etc/docker/daemon.json") + --containerd string Path to containerd socket + -D, --debug Enable debug mode + --default-gateway value Container default gateway IPv4 address + --default-gateway-v6 value Container default gateway IPv6 address + --default-runtime string Default OCI runtime for containers (default "runc") + --default-ulimit value Default ulimits for containers (default []) + --disable-legacy-registry Disable contacting legacy registries + --dns value DNS server to use (default []) + --dns-opt value DNS options to use (default []) + --dns-search value DNS search domains to use (default []) + --exec-opt value Runtime execution options (default []) + --exec-root string Root directory for execution state files (default "/var/run/docker") + --fixed-cidr string IPv4 subnet for fixed IPs + --fixed-cidr-v6 string IPv6 subnet for fixed IPs + -g, --graph string Root of the Docker runtime (default "/var/lib/docker") + -G, --group string Group for the unix socket (default "docker") + --help Print usage + -H, --host value Daemon socket(s) to connect to (default []) + --icc Enable inter-container communication (default true) + --init Run an init in the container to forward signals and reap processes + --init-path string Path to the docker-init binary + --insecure-registry value Enable insecure registry communication (default []) + --ip value Default IP when binding container ports (default 0.0.0.0) + --ip-forward Enable net.ipv4.ip_forward (default true) + --ip-masq Enable IP masquerading (default true) + --iptables Enable addition of iptables rules (default true) + --ipv6 Enable IPv6 networking + --label value Set key=value labels to the daemon (default []) + --live-restore Enable live restore of docker when containers are still running + --log-driver string Default driver for container logs (default "json-file") + -l, --log-level string Set the logging level (debug, info, warn, error, fatal) (default "info") + --log-opt value Default log driver options for containers (default map[]) + --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) + --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) + --mtu int Set the containers network MTU + --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) + -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") + --raw-logs Full timestamps without ANSI coloring + --registry-mirror value Preferred Docker registry mirror (default []) + --selinux-enabled Enable selinux support + -s, --storage-driver string Storage driver to use + --storage-opt value Storage driver options (default []) + --swarm-default-advertise-addr string Set default address or interface for swarm advertised address + --tls Use TLS; implied by --tlsverify + --tlscacert string Trust certs signed only by this CA (default "/root/.docker/ca.pem") + --tlscert string Path to TLS certificate file (default "/root/.docker/cert.pem") + --tlskey string Path to TLS key file (default "/root/.docker/key.pem") + --tlsverify Use TLS and verify the remote + --userland-proxy Use userland proxy for loopback traffic (default true) + --userland-proxy-path string Path to the userland proxy binary + --userns-remap string User/Group setting for user namespaces + -v, --version Print version information and quit ``` Options with [] may be specified multiple times. From 1e8e7d9f546b1f9c20c01c33f533d93928cb6b12 Mon Sep 17 00:00:00 2001 From: Tonis Tiigi Date: Thu, 6 Oct 2016 17:12:17 -0700 Subject: [PATCH 149/534] Deprecate repo:shortid syntax Signed-off-by: Tonis Tiigi --- docs/deprecated.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index e7a301aa27..a591880ef0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -17,6 +17,13 @@ To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](index.md#feature-deprecation-policy). +### `repository:shortid` image references +**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** + +**Target For Removal In Release: v1.16** + +`repository:shortid` syntax for referencing images is very little used, collides with with tag references can be confused with digest references. + ### `docker daemon` subcommand **Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** From 1f456ceda66a499cc762e63a3a191cb23dac2411 Mon Sep 17 00:00:00 2001 From: John Mulhausen Date: Tue, 11 Oct 2016 12:14:15 -0700 Subject: [PATCH 150/534] Remove old documentation, add README.md with pointer Signed-off-by: John Mulhausen --- docs/deprecated.md | 216 -------------- docs/extend/images/authz_additional_info.png | Bin 45916 -> 0 bytes docs/extend/images/authz_allow.png | Bin 33505 -> 0 bytes docs/extend/images/authz_chunked.png | Bin 33168 -> 0 bytes .../extend/images/authz_connection_hijack.png | Bin 38780 -> 0 bytes docs/extend/images/authz_deny.png | Bin 27099 -> 0 bytes docs/extend/index.md | 272 ------------------ docs/extend/legacy_plugins.md | 93 ------ docs/extend/manifest.md | 222 -------------- docs/extend/menu.md | 15 - docs/extend/plugin_api.md | 192 ------------- docs/extend/plugins_authorization.md | 256 ----------------- docs/extend/plugins_network.md | 73 ----- docs/extend/plugins_volume.md | 268 ----------------- 14 files changed, 1607 deletions(-) delete mode 100644 docs/deprecated.md delete mode 100644 docs/extend/images/authz_additional_info.png delete mode 100644 docs/extend/images/authz_allow.png delete mode 100644 docs/extend/images/authz_chunked.png delete mode 100644 docs/extend/images/authz_connection_hijack.png delete mode 100644 docs/extend/images/authz_deny.png delete mode 100644 docs/extend/index.md delete mode 100644 docs/extend/legacy_plugins.md delete mode 100644 docs/extend/manifest.md delete mode 100644 docs/extend/menu.md delete mode 100644 docs/extend/plugin_api.md delete mode 100644 docs/extend/plugins_authorization.md delete mode 100644 docs/extend/plugins_network.md delete mode 100644 docs/extend/plugins_volume.md diff --git a/docs/deprecated.md b/docs/deprecated.md deleted file mode 100644 index a591880ef0..0000000000 --- a/docs/deprecated.md +++ /dev/null @@ -1,216 +0,0 @@ - - -# Deprecated Engine Features - -The following list of features are deprecated in Engine. -To learn more about Docker Engine's deprecation policy, -see [Feature Deprecation Policy](index.md#feature-deprecation-policy). - - -### `repository:shortid` image references -**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** - -**Target For Removal In Release: v1.16** - -`repository:shortid` syntax for referencing images is very little used, collides with with tag references can be confused with digest references. - -### `docker daemon` subcommand -**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** - -**Target For Removal In Release: v1.16** - -The daemon is moved to a separate binary (`dockerd`), and should be used instead. - -### Three argument form in `docker import` -**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. - -### `-h` shorthand for `--help` - -**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -**Target For Removal In Release: v1.15** - -The shorthand (`-h`) is less common than `--help` on Linux and cannot be used -on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on -`docker create`). For this reason, the `-h` shorthand was not printed in the -"usage" output of subcommands, nor documented, and is now marked "deprecated". - -### `-e` and `--email` flags on `docker login` -**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** - -**Target For Removal In Release: v1.14** - -The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. - -### Separator (`:`) of `--security-opt` flag on `docker run` -**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** - -**Target For Removal In Release: v1.14** - -The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. - -### `/containers/(id or name)/copy` endpoint - -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. - -### Ambiguous event fields in API -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. -See the events API documentation for the new format. - -### `-f` flag on `docker tag` -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. - -### HostConfig at API container start -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of -defining it at container creation (`POST /containers/create`). - -### Docker ps 'before' and 'since' options - -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -The `docker ps --before` and `docker ps --since` options are deprecated. -Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. - -### Docker search 'automated' and 'stars' options - -**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -**Target For Removal In Release: v1.15** - -The `docker search --automated` and `docker search --stars` options are deprecated. -Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. - -### Driver Specific Log Tags -**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -Log tags are now generated in a standard way across different logging drivers. -Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and -`fluentd-tag` have been deprecated in favor of the generic `tag` option. - - docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" - -### LXC built-in exec driver -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** - -**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. - -### Old Command Line Options -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** - -**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** - -The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: - - docker daemon -H ... - -The following single-dash (`-opt`) variant of certain command line options -are deprecated and replaced with double-dash options (`--opt`): - - docker attach -nostdin - docker attach -sig-proxy - docker build -no-cache - docker build -rm - docker commit -author - docker commit -run - docker events -since - docker history -notrunc - docker images -notrunc - docker inspect -format - docker ps -beforeId - docker ps -notrunc - docker ps -sinceId - docker rm -link - docker run -cidfile - docker run -dns - docker run -entrypoint - docker run -expose - docker run -link - docker run -lxc-conf - docker run -n - docker run -privileged - docker run -volumes-from - docker search -notrunc - docker search -stars - docker search -t - docker search -trusted - docker tag -force - -The following double-dash options are deprecated and have no replacement: - - docker run --cpuset - docker run --networking - docker ps --since-id - docker ps --before-id - docker search --trusted - -**Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -The single-dash (`-help`) was removed, in favor of the double-dash `--help` - - docker -help - docker [COMMAND] -help - -### `--run` flag on docker commit - -**Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** - -**Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/)** - -The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor -of the `--changes` flag that allows to pass `Dockerfile` commands. - - -### Interacting with V1 registries - -Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. - -### Docker Content Trust ENV passphrase variables name change -**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables - -- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE -- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE - -### `MAINTAINER` in Dockerfile -**Deprecated In Release: v1.13.0** - -`MAINTAINER` was an early very limited form of `LABEL` which should be used instead. diff --git a/docs/extend/images/authz_additional_info.png b/docs/extend/images/authz_additional_info.png deleted file mode 100644 index 1a6a6d01d2048fcb975b7d2b025cdfabd4c4f5f3..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 45916 zcmd43bx@UE)He)x5RM?JfJle5lt_rQlp>N6($cM@(x9NyQUU^sbT>#NA)(SC4bm-w zNPg=w?&o>uo%!bb^P4&MJ;U}~wXeO`TEAH5*`3>RXU|Zc!N9;cs~|6?8mEj#}{hMznYINBu?&oe6G$A5M8Mr ztp2%?yOZk~NktGSO(E}t`um}Kj_Q>uvLRE#Vv_j#d5(dNzJ`Sk$N2q`=ttiPm#IZt z{P`34KQ{W{zd!!}_0JMA*zHq|{>*G_?Q1M*>9Dh#VM&LRcFg&rSx zqV8j@3jHsWhM6dpm8o)fUw9v_7<@n2sL#1iD7KdpOnpr=U%!@JyZC-ld^Fc=n`(Vf zM)_vzcntLu*&9zJ`cVqiE^F_shw^Vnd;OY=O}q7q&(Hq!q}M^-d*1oauSeN#GtfA# zk7;ij$t%$jUG;sZpjTY`IUxJC_u*EcN>L!KmzLE~zAaVT!R<_nr_a?GDsE&_@Fi0) zP6ZiKT(Jn3mgzU2sByQuS+|h%I9TZ5=Z4@6OjFa=YRMOo=jYgFK8Dl#&*oRJKRBG@ z_I8}AvEDoSIXO@!_2i5&pJ~T8x6MWzY6I`1vQ&e5FZPXG?Gj3b_pTduQ9oglMuoy& zCkj8zwnkoMpS!nhmFGdZ$xQRJ(B*-gYj@1vJ1tQaX2DAceSOW9(*3ISU{|5!cH?dv zt+%?PzS}hR?0d05y!T=k_|2RWuepBSI^6zp?fH3*`3A?ij%ezK^Ig0h-Pc%@lbn96 zj`T%{dwa>naB5Q#iXRr5w1rX=e8P0zWwDv48NsZ6LB_Q#pDb3(sGKAk`rdg(gp5-w zoc&h>qa4-vd*_GWI%B^?2tPE_xU{U2DtlIg`{84{qUgQ#VN{3O#X6@YRZ|m~C5;oz z&gl<<*Df2><)&uZ(j6Yn&FAaieo{^4_T0Wj_Aoi_AbG-jDJ?cpaXVJIGlomUYtLZ3 z(8xd4X{mSFTTob7doH8%1DPk$TFmvwE<2JshbOe6Zrwd;H_7GJ$Ev7d(LNOY{*f`% z`*=I~_U=>zZNAGodH(rc8>fqld3JaPHqKth<O{E%xZw@1v&Si_61$gmMUD=>Y9uD`AKSVZ);YO=~aOVtu2P{+n;3-nzPj= zq^!!r-LRq5JxbRW?`9|`52jQ4bsHROR#xT2suU(rD{t+Dw9#JN`J^&**M3L3o1ONk zTgD);yZhJ9VvqEIL@bVD@X41emc5x_p?8b#E6Dj`VXKRb5&m!w+Q%cmOmXgT&T7cZ zNWHl&&e>YFZ^qct{jSbU-k)lV* zTNPn%C^&YXL{>-d4t5*X^Y{~RrhF+$*>*De@*=JBezPuz6q>?|e%P3k1dY-sr|Py^FNW@{3|+gd_x@e} z!J}b1>N=H074qPY#Tp0x?nch=&9Fk3H65ID0(Di(d&jsWN6G;>XZP$z%RlkjjR`Rl z2ESk2{L!a;alv{ZdrqswEQGy8JxjIK{o!M+kd08K6j68qcEbTZ5%ouF_MvQQ=|Wb6 zxt59cJ#U}n$J5In&e=ansD5+-yLx2noWZ2qVUgodzFz3hN$<(b6*cASb9>)AxzE0_ ziw^U$cx$tX>V&N)a=aDhok?jyQ#tWYfs{G!OQP^><-vOO*^6h$FRu>QKCYN;4Y^2g zNK1cN3^p(y>!OFpj$HLFUMfR6wSn#Qi@bsIUVMLBJhk(|dIn>kJkO<;zhqI?Wc*4& zJhWzeT)+L3#)e|btZms-c6)1}?M~GB1E;#i2Nho@%;t|puZ_{+cV=axNydA|Y*z+y z78*j()BRa$trh76)FMKX-Gq*~q(>oWhX|JDhKbt8A$#>ly9x1pjRM=Jx;t>6o!2Fx z5;8pGf7565aesejr6Xyh{$v8P(U@SiElhC6`+Gaf{7nLHnyW<4B4bq#km9;|J0cNy}-r55^xow;zzj zB^tk6P=EYhjaTzm9BuqY?aw4NeIcI7;7>IWvyRQ6T#jTPRI$fAio%&1dy5V6QJO8T~s7rs(4oNw&7KK${cDNFc4aJ^?&w&$A} zyRzC(u>4)C=$OU%YA)UgN&aZl6f&wI>PDEp@kOIUS;FE6x^? z-S~TE@uMFv@6~x|6VQleex4pjn|8)%9PFvRce0GGcVzIP<|v9pD1^HXtjq1S1ybu8n#%bC+FPfSngyMl&03W^N?PvBi+_( z5*&97vHIivoQ^cO&)c5N(ogZfckzFrU^Z09wcc!@-GaCu@0uBURMexU-S{0-&HjyQ znqsTz_G8Pgs|z0^J_My62;2^MXq&M>p(Y1UV|1#mYa@@C+>{(TpmwMCq z*VVL^o*2&2thSTo{AB(%Yfp>US|?G3GddBmn40vpX~&~qbEGRlP`R-8G&d&9L4wXY zo=TDXwwWbtI-?AIzaUaK;g=cYSEh}ucNJZKPN24>hglm+{pZ@|Q)qMd6VK~9Y&Txh zxFdM9@ z&)XYnrlw-9@T%2^Jh$YqysOeWL}geGVU|=Hzw^FO8RDbcbMkHx822u;FTZf2CbH-i zGVAc*Z25%Qfl*F=#)y|*hC~HSRFUv-Ic_O|2z6cQEC2bl`Y4?$oP=%O4V(u^+{Lys z;2HC`w0C$WLo2ZTvz&H-n55=)nWX<|D|IH5*5HoYH9rlz;&`8a8YwXs+}%pQd&pRQ zAbPwv8_92ed#6>LN$BBxaK!BmNLw-+l)BqW1cYkTc9V5^6ZI)!MPccZ+oEC{LKW79oZM>U8E(-R)DA}A5 zIV7#!Y{DysP}?Z|9s(#k-jRgl<3)$#L-$&18|7p%+f|b`E02t3Nt!rp&N|7e(+?*d zS@J!$Dw;l(t=^$NDDKe}Zv5V!C|uIET04?<5Sh_DJx^&>h1L zIK8w?y%dESx5y$JrRU*tPJv>**U>kBLa{q)bVNdrhM~;VGxiEHj57X`bX_E?XaEb7 zrHgQm*88w0j=eV`Z`iN+((Oc6H?Hq3FUYaWy|!A&t-ikVI@)KyDf({b!7`rYaalX3Equj=e#2 zyz3Re>5s*p@5culoxvCQQO2n0c9vvy-EUjNCSmPwZ@jp_|6^%6dUe^alQ!6R(TBXG zanhva`S}hxspKJD2x4WU6xZWK-HX4AOVmNFs_xuP=!CCq6=6sw3z6|nFs-=$?#5)n z3xaksMm}7_zDh?ENnuEQC0A_kC$+UDFztMr8+V;Ne|^S?j3%yFssUFqS6^ zTLv&bd8wSDQPeSDcf`>B=s?lDqWnxib+rg=vg(@!U$m>fXf2t%DOMg`+8DH3TPqmP zntAd`Z&0bVZbBMM*{tT{t;fA4ujrZ}y=3cuOZ$>gq1EE6!69<5_R+waj!O*==1ts) z!J3Zw;UW_P&N;q?XP%rlTkv`c?Y(&8&+DEhq)B8qcqpuem3wO0uEjwPC_vi)r*fh42NZj=S&#j7%PinPJBl)_@ zTY1AlqP2Oeg&l1*Bq(kUr?ZcqVCsLqPoOoP`YPkyJBrxCbNb0`FP4dNAC&3V^RO?N zwdv~a8C#d-N%fG~Wg8}bZyg_#+o^u#B<1>(m*c15>#&u<4$oFA+7e@f%t~Ebd;W(8 zKQG^PnHDYogo}WRypKNR|HTv+B{qp?mqo3<}UUF>%+W3}v!+a0E z$^ZKHQn}wkCW7Hcl=MX^MWZhZ$=>-k{ktgy_O&}Pl)S$7vQN&OZ_e#E=-heL(}b=HqOsc@BV32x_V&wbFhQ`mPk z(E}?lLnGg1h1_>=^sY#CnFih1^?@EJAk=68m1rJlv9504ugKVTv9?!s*%wJhmCbUf z*4%5`c+lXD^&xOxgJnxq++b%9iqdGhhC<`^NapCmKFdNBxfhdwRXe?IUm8Lm{%u_bi0U2D8)xEB_HOGbkY`631^n91a87 z8ok*X92KA66`tuMj-!0&X{7yw)2#guL-S0kK~s$B8P;{3oo2j+!2$zrH*-7>olOS* zy&S{+(_(^?-Y3^vC=cmOpwG3 zQ3{p&Z33Z!>!3O7P!{k_2L4E6kJ%SS$X$XgRZlr15C_8>7F(Q@9Wu zL6VWDt@ih#gq0_iVr? zg7^pCZpg5WWZ(yS^2_&v(6b{$#&2KQSl{Q|aUPYzFEaW;kLJB&Cijm}GkgtG5kR^Sh!qLnC{QPZ2t{N-GUd=Eb)r^9}DN>495(<;>G?qAQ1$#TmS`Y z3FF^aMpAg&$FEt0KmGR$M5Yjz7xPRc(J51x|Cqf&I7Z97_1u30^p_+gNsLpvLjTc$ z4I-FlNLISy{QeElHN#tx!jBsH`=T+>S}gG1USClAyUsDt0deqVgM9g}{e5qAm@6n7 zHu<`fZ*Et8!H3A_dzTLp0 zXX!4rR2x{JdAwQ_KDV}>)B~GGkCkLYsn~!tMUxzU zID~`?)S3%Iz0%L7>-U<3~m&gnk-Nb>Afk*2&C{j z+PTZDD!#wSs9kCifmD7m-|yci*_6RBuz+1$P)Fo>{A=OCw~xfBC+>=~D;})t7l5`* z9(p`EWZ=yuXg3xPsb1uj$?-|!lhF$MPADYjXixS6_g6xXYYNGFy{GlT+s(Xa^FYHlhMcRn}pjL;ztUac{38b`5xJzDXF`2!QRwD2UHjU!02xKa=l!|beOYEu z;l(P8AE_%%-bxl5pA}aIj`ZD~H&T8^o=^8K`KS~(`r~xMZvEB~&9R(LIVoVa#&E+B zYX8V-AWmG{w5TevdW&vSNzX3oh;G`6^R08&FJM4T2==C%0EKXbOXxBE?NQonauV^O$xeayQs;++Y`Fi{4g(R_qA@UzcjR*bqcc_)IUZh+>ROBY*q zy=N~QF`*xWk5`HH&}Ga1%!sKIT3d&y2FwaV8|7m%irNpJn3lwcRpy4MG}n!Z?2QJw zq|i&y7iLz4uNk zKYMLPiqX9%K#+Fie=W0Kv6y*OU}hlL&9oY7i+biY^@00?t`8|=^9`sIYHsl7Pq$)7 zV$%C39OH4He-b=(V27l|5_2yK4>Q%hlsT9CZ8Wu(6LCXhyTEffbo{e+HbiZ;c6Kn!ckPb`v6 z?b0hD$G1>JOY{5)5mT-A@)UO4Oo@LaWAI^9$U}GYta_kyAb|qRxas$A?W)=8Y)hjR z1wejDrYj{}0tTsm>Zu*U@ehTE9_u)A7^#f7)Yn|pbwyW+d0L1pHW!jGRjMD%Df;qr zc`S6_-Qqdf9ksts_QI0+)?MTG*nXPcD&ON8K0O()cFifd@q*0x;lNk6A9{!O9*0|P zR#6vsTGiPT9rZTK)*qM|xXLm6eip{UXP|h8tM@TkT>Pmh6@-JrsHM~}vFu*G+D95` z)}!TlB4S>Ll_Klo)fFOM5CC3(*DEr9A~Kn!#xm;x#P2{=M$)4Wp}^P37J9t_kK^bC z1oHA2;OLNtZ@!`hM(>QwDT^`6g*xHWhz=)bdbI${9XuoIb?8>G_4vo)o0+_ORjMu4 zW0ggWKp1WT0uW9EO19{^x!vrY9}x(%KQ~%vB;5wwc`#qH_p!UP7jOivRr8+`avmal zk0Ze~&}CE>Xdf-o&(=hXSu&k5X6aue4`35}`Sin}ZiSujne*)5J||*1!Y8am(GCtP6p(s z>}^8PO&_r}huXW=1oJ0KGA^zs0(@v=7OEo_+=p@f(+$#3IQw$2?cv!axE;8SY$?PH876@*NJN5K7F;DadLuIT-8jTRVqJ85cKL~!M6tEJy z5Zpt8y7h_<^;M3C-IxFoBSf0r@ds8MfI-UC3y!BV)bfDUw>-+2Zqqdl!CQmz`b9xI zbv4wy_3BGA%IQ&{zchyp&z)s7#NE_DD$8Y#M?aXT$PAvtrG!11!xxR^i>5q1;?2*`qS_B0Z52 ziQ?2&-N??XQ<$1=frpO0toOhw2XbWg?8Al6?Xc|LH8V}z^-G(@rC655)u8n)z{E-{ z_=L%D5tGB0Xet5Ygyq&+C7#1i_ zE%Z190HY^tiuWess_j>GgE*q4&FK$M;~HWw4;@cO^?29cdR2dC`h`KA$1l_g3sGy_ zXZ)*%a~F>by1W=flsQ$qPw@5=GxDtUhKH_4Jg?s_CuVZU^`nEF8fKVRo8e>6O`+77 zSY3cIiGkYlsR1UNCRKVQ!9T;BN;$N2zH}^%0JHYn-N~-0jIG|A+*z)Ji*qBJ^u2^0 z<1xOw#L90+hH#T<8n;T%NFB&F(za3!vwXVJR6?jy?eXhDwD6(O7}x68u?(Y1*U6=Mo+~~*b7w_W|O1lEpwC6ek*li+0V@pI!Vptc6@7?XQcw8#f8bmLBA8oi# z#+bc#l`z9($cOu;gKDcf9hD&Qj~|INj(wK*u+zk7pL0!wUYJi+SkIA+K7TWikoNLM&35Q_vNnge)(uKj8>bq?aR)yioT0#K z5!A$`zV)fHZv1?X$z~qz@oBBMmduOZsvMRSf}$uM@sOn}xjYgVO*&2)qr9Xj%mi_Q zW9S#{Ur(c#$ z&R4JnkB^TIIzeze^m-OvC3^kJ#?q*d@a3N$q%jy!f(yJRt*pss4p%Y!Tx4Yo_;5Jc zJX#Y(F8=cZ%thbrXRE~*O!RENS3VKg-D;oO5Q_?ZI?wZX*Jg*({)hKngL4l>*8`I< z5i2V7=&Nk7qR9^hDO_2fwRYbK?a6X8AH?0mWa@6GZNBN>Whzo%vz;8*=nP&8}XQ$w6!Siw;ZP<`+ z*gE&&ty{?yC}tZu`mN3$qm7%FYI+21hEwdWQk1ue<41`n_GA5;$Eqfp;;c6NM7bk& z#w$mlFwg^s(EfUTSOj*k&9p9gVhX*)rcZ+vJL<x!Un#tu4m5t{jkG=h0UOt6(V=zo0CZPzLcJa8`e2Qjb(rs8m#xUU2q|WAJ z60PO=_BY*n%y?v+nO@{LsoX>iZiSMK6&5{GXV9UNQ?DBP`ifhyr5f+I)3We%3n)92{igaPb0467W&o`dy%5KLz}{`G!>m+ zpm!o}8jvf!p07LhuKr;CS|gu9|0z!r-CP%+J8XCJ6Rk;T0$*XE(0=--SF(hTT=Z*l z@jY?Xyol)Bo?v)=$GZ8RS0m^kCTY3b9sBW@ZZCGCp?&YhKC%Lt}pFRpOwu z-51QdCIu`gGhWSVqpvrWm6z@n`jMo{QW%|SAw0kA;XhJ*f|BMIwX_wGB1vNByZaj# z`pOeEuoCIFW7HEr^hqX{`tbDZr;I9PtlKD?Iq8}E7YP4=U*qQ(6UEPm_zZh6C&K(9 zNcN%((-@nBex1$d)!?PPNDMHTxUowL`7+%x+K^J_feg?m>RIsRexA|=4%<)dMPFaOdV_kiwr_QLEw^IynLR}Rp++2x?;(Z6%z!8sj! zf^$)S5qtAhK;y%9;yTKIc>qUAIOpZ~i0Hq7-FFxP;KM)?VI9t&wg{N(Gk-(8kY<6a3h)*2o6iSf3DLCE9s=5EDa3?tW2WD7!bMGnA_!VHxc@M(F zqCcaqZ4rhl*PPyEJ}9-R`B`SDP*QOEX801Ll8+cu#HG*Q|06n7iGenH62S7`00T19 zz0KM*{}CPRFp*i|oB!r9T;d&KNQxKg{EzvNzX8XbCrSE``3SlKb1hG5?*1Rqkp)xv zPO6FYzkvv3vMpyg!~P>W&ciY4@^1gbeDuO>Gbkm_{l|Q~V~~)HJcCvq4rxfc8T$>; zl?(ri-9J1?KsYQJT%-LC07UnS0RyV{Y;rV%xOZnkx6F!JK|#Sz94f9<}b z_L_fA)Q)`l(iCdEtd7tpq;2DMI~Wpi#0gGdy-iH=_mZ z_1QFY^KqR+;*b+bq zK*rMPw_j(^0KF;yVBs^@TxV=7zygY`p*&p{jcoM@QTOlizy8q{<8Gs!Euh*q z48W${3r`u-E-^#PYKvmuA~t$=AHYr|z4Yk|eD{OiDJSb^Yv!?o6D!)XFTEY8pm*R< z@;fBv5m5zNN?Dt70sI?tyvoC_-@hpcoB<$DgWxTRcQTAltHUa+if<*8OyNZ`iTFA_ zBcSWu;Z}VY1aw+Y)-cduBGiF1hTvPWT0W)5`f)OWsAujxxBk5~8d0}9q14wdIW!GS z@2-tnEuZdV0fae_F;I@!UE1bP-D;zA>nzc)eZ=Rweox@}_Wnu%hwkK~6)q5fUkK*} zt#TXG-P0+xpx2SO@$L?p)bD^rM@9dv?>2^aaItCUnPxw6V-y9QJ`tS|yGAw>u(5(- zaUG59)VV)?{j5J(lR4WbglpM*m~jpKJAIY;)T*^}Xcqn9_{hk&>AoPgaDu^!KW5*m zI{4OyniA7aGG;{~RT8uAuHPZ}7VNxRJe3EuSQPXPR*7iROSsM0C>#&P)-c+FL%Gk1 z!l|>`nVaOBx&{PNnqi}#D)_Fl4C?C; ztun94*Sif?adZjWp(@x+vQ(vHU_x*3=mltqSr6u_j*uktt zOe&1#`aAEFV?MC6I8!|nKgS%A(#3~FyD4u}B-NwH6Yqr?JNy1jl?T8dT@Ze7e{A3r z!TF|aD$a!gZr2ijCEu@8BG&^{D23Lnl1X;cB-;7(-k}Q%WFY|J#bC(E`!Rp~;i%-p z$m}>?lS_pT_X&YeH=N09@%%bE?gm=(BJbY-l_b%`d2qB|y@ARRq>xfhfhq_~ZpdvO zPo>y#R(8iL$}Qh~@iETB->d!vmnoP|A@=g?M`z!AYf-#FsggSEa4S&zoBxT_Sj^!( zeyGWrt;cNmOj98~89jt+W6xtN_3AuEbjfP|2n$3QnET-Jbt+w+yV8XEzTADo=e?th9G2+s|^GX1gdW8$?xColpkpTwm(f*a%J-OeaB#)8`2 zcuw8&TM$0XD2hPz7GQ8(`zkPd4`jdlz`6CND_s=!+%I)n?pHAToM?G3`KoPv6uX8Z z#Dq(bI${yx)VkVrBL-L(1>jxNJbr!WGik+lreWl$dN3E+lPY%;?jy~j=dG*=I4_vK zR@ieRu297n#IM01+YYo()MY&#voA@u!$n|mj5Ywbs`@d|iNyUM?j{*{?nL+J>E3-X z-^B)Q%a_kcS)PBGCbD;-eDWah&-!72iDmE;E`D+g*hx^q8C}PnV-f!%B9E&sUS@7S;{MS#8`;%pi)&EidcuO z50||#Qs-H{I##7tw>N_)Nu_K`_o?dP!aE4?y>Fy(%IXapv6hAkk`FgO5GsmF_(*)C zn}FE_nd1hSeO3j&K$2v;mh3izHYJ>4B;=|T9m!hx%=KqY+=o>z^Yme}Vx#{fNJUmr zr=HCC9s~2584(~6LcwbsV5t5UOg&cbrV%&Hgqg_wk3~KYtAhko@H*SBf@~z;KR3^A_D=xy^`i zf5PwOWXOWGw0a-k7M+A4D>oIO1J-Ls{#!2$OQ)gozWVuIa#vXTu1F(W7Iq9;E6Tj{T*mWVjC-@Ufm{ET$D(Xvh}Pe;HR_u@3Ue?$nlCO_8x zoXA;T>f8lSUG~zYhe>f>O&^~TXs)e-)D%o5tjbbxM7D-CT@Ju=>1$ce73;xUz;C6< zB(H^J_`dNj? zagBGIsSGoO^M2`wu1n+laRu}P`Ng!!xI9_!Evgyq%z#M}WCZmc@0eSD`s|w!%V2~c z#UY}R+B0OwCVUIEkpjT>=!eU|9Q0rLeE4fSIxO%r9wS|xj&J1_86@X7bW;v)YhoH0FX%P(u4lmK+rdp-Qj<-a&HTXY-CI1*IiJb(GiT(Q@0Dz$3bS zDgkCATZ}^3@Aam+`@(D2cfh@htp&H?I*>i3{RAZ?7dJXgjwY8Ba*8LCQ!=>x%2ABC zo1{RcFgI9H1OYk5j1nm;V=3MZSBS9+a{J(SQn61aQ zg0iXMJ`SOx(5I$ycnPW@&k;vl(xYWIjnkSFh;C)okE*+*wd6m=rW_nVr;N*e>l31y zeII-HB`a(sUFK_>6=QI`l%s#R1Jv_h2?PJn<1eZ`QPk3?lda#$BGmgujqKkuxW$ye(fBSx*=xX6z56JGRdS-_>u%$UcO`uDDKaGXbD6|u`+zE?@- zW=ZDd5b^t7SEXRx|30r8%2*>9338xE(IMRiPzki0dq@dsnP?Ioa;Dl-yPAin!Fs4D z{!$E;mJz?#N+z(a(2*d#bIRXH;sYm=`{)d@5{uCk)TEc#F}>48KtH$f$H}n-e?9{e zIqM@66BBJNj#}U%lWGAE6_2W5nqu*V?cMo(uCq2YoT_4^mPzh{M^o!bnXb zd?Zt3o{=EI7|azf97F%+szyMH=ulcU7FNt#-glyyEM`l-!h9LgLkc6y?YcsY3xiS| zC801Ffz%dky__tU(RZSnO-qVZVNit*2C)}8`Rd@C&37LcUxmRQc;CyzebLWQJY$D|gyL%KwmnJlavmQJWt)S{{JSfBxf3%W?2W2;#*qM1bDh5|OQP(<7)s4LijB-R8w**>n9vlgxW zXt@#qRVt(=c=zE0mESwS$w;!IW;QD~|B@7PLD!NMFcO?XW&<*tA#R{n7D)=dIR zr9dRuqi_EDK4;n*d_nfmd1X)mJkU(gs?nRP&A-(B#vkdE2`7|wy%S16anvY)idws@ z45pC(FJM<$*Z!&-M_YpD`kotm@`3_g4{#bxj|dHqOrA;rZXJj^ldkc@93*6;GFmJn zplGs1anY`rsYD=9#2y{|#FLQ$H{QS^zK0;srz^nt3(P%)CJCh137=X-9rhRtQjY=S zP}$kLGzP5q^?9_fzbKv7?L;B^ftH0TOXtV#=; zi4hsTIfGa*G&H*cQ8ow!Ci^78DBmE68B8`cTK~&`De6ujz)L(?txaD-bjND}>lnkR z7PW?R#IlOzcH`^-fx+APR31vOG^>H^-U@qzS5Pe?t_gPaOuCP>n1l-=&dY8cy+ZX? z1MFe$O?600SBV2V>&G}=6VwcSe$4$bKv{KV!8L*QJZeZ z%-b&}q$E);+Q5o!2pR;aKW??0yuZj2~{ z=k@2e%={}J)Qq_P`fjRGGe29vb=)fbnoaRAt;NlSrGf2Ybp_EHqb~(R_uCtIWt8Sk z0cb2o%tp-Q$gSBnj*|%R8f5kaLt<}-h-(UkY_N0c;z=JWwU2bAehQBihcqKlxN{ngWfFr)Q-rM^E)Dt4D z*L0`8Enjvk`MoM0-n}jKStTmKpM5*{iZ4@O5>tp%J2B_~M$)L8rYc+VKAvo)dCxV3 zwLa$t18N5>-8t~sDthv^ey&V7-Sjbhm(9o^r08^7RTD9Z3l<5TZvZ`p zcuc-tb*rtXuqsJ~QUcNWR62zqGc#rg#}XLa?jO`uMRB+kH}xrCAuuBgpC_&t0FqE-LNHq7`@2h~{*UApL6>4cO8cx>>ohtb0jiuz zG`}~J=5SB>Mn#>Qwuj)arQbMl6HI>&;ioHZjVQFregR^c~IHvSq#vE zggAfb<7|u>xa8!)LubMC`q=?HZE4UqkZs7zOvz=p73Hb8% zAM_^mg`NRL#M&??L&SWJDd`c`mwya>6j~yRQ>Rp3n|R}GfFD!G>yVbe|KuZxwMnah zQjwCJ$s^yS{F^w3eIyuyNY0R4bh;A%FD=&LHdnJw7aJH{_Zj+oO}aBE!~la-(GXRx zMeCQ_XlsMb0_h~7 z{tUWf0C)oh=u|$W!ItC-lGwnbfY%&K@UZ|oa;(~||0dsM60c`&-6@i)l@3PjV7-EL zcgJOQczQej^IiM#k>#vg@3g^127L`Tz^cOt{K89UmHG$)U*k2e4(hZBQkv*}D9RA6 z9Z3COtaDeQu$uYBpSouvEA5Rn!;zyEUS&tpz57eZI zPT!2-R3Y34kE1-|Dgw7$wvGMJ7%|W28DdP`@o4w|mA?(Gp*D#=yZ-%3GF~YSlBJDT z{$}a;lHe6jxMe$i&EKv~shcS<(~(@dS9c%}%c9Lb#WRDqn(xeVUq%=u+eoEjD&oLI zhk(nc^Dm^ovn778%HL?w`GUKSwes}VA5x$g(r}}7Zy^4KxA_`>83lMcH7^b~e>T}$ zAS^IKBa2EZIV;t!+j}#Lh2m7(8zV^_&AKWakkK=*VaduZ_h-#Usx5zzScwqHXcm%| zjHFoW()mrn$>SP!V5>1*b6r>WVx&lG==T#o2YfZP6csQ?9UcnQv?5OO;RI^p^LGI9 zw?}itA`AhLC+WM6!Cydv1SNt|wLt&+t81?Lxq8)_A)$uZ3*dlK9dWBRyWCb&V7sGv zDP8$F8JE@VOQs!Bjv`lS4Mk2j>L&RPTU}6Wi(o*ac#6BnxCsfSwSHJ`p8)w zd-HS!1;!s7?K_($R9j=8Imbh8x@xUUFC-d`P%9bnZyT@NhD^w;6o2ATQ18B-9{*md z^(*}VaK!k#>V(7DYkc@hKGcANBp6GcBb$%tEEa(B=-S6W*%vfL3wg6$E_43+=LPuw z`Pv393gvMXOqkqID#burXT!O>=Xbw7@&-Z($QmF<9{IpP@xyCClyd_ar+GW=Be~~OhouiP^F^on0jxExMvFhDxz-q{zM2Su~` z>ZVwv3>s|Ty}h|CzXNj|hO87%nP1?QfDf=3FN0Ajao>>g^bnZYGd&cNJiaUZn(`;? zzk;t>oNG|856$Up*mz`7V0Xgxvas9RobIi9Xg;U@+yoah(Vvucw^nak!`a@{xP^$y z%!osCx4}FeRxkU_pf}LTnK>Z6`3$J15Z)(qT*?OJGi}@ZKR39c>fJVdGQAhdOS`3rtJ>BA(}xI zyEC{7rH@ED|9Ra>cmVuRuvSX{!(~KVSocgxxxRD8A`1n`XHbuy(3c|ie4V7Oe>AT= z9R>ttObHW`{}5g`f&;mf=l|L2nScO46N!<+i~gHq!X+#q23aOLSRTAO7Vdn4zY~Wg*0a5mpSi-3{XSCPu_;?2F znVA(4zV;}#mumY!W8Hl4pg32*Rzzc@$l)#M-AF1gNC(UI5VB?q92uOqAo@H%0eQ)j zOc=x*$$G@12pTybfaRr`HSda3Ndljx`}7lX(bj2fk|2aX+UkwWmAggS@B!eK z@$ySZ4vqfuw0|YvXu&oJV2MZzAyl_Wt{zkg&hw(!T_4wlT|^JFsj&r{mI;XM7{Cok zhgt9*D4Q=q>yrmQ{vB*fgzdg?%_a9yrty88xkaBI+I!LxBkSLtgsAu|y6M=!_Vw*c zvH_^Ftk55R#}bHLM&OTQz%$Ioo4zezk92TAH5&tbb~|9vbE$$8Zqgv0b0Q%F8UWNL zJb&tC>6B%|4`4{>dmlM7{<>@6$qYC=8fN{BkW!+MieZ?>CE2G8SeOHKo{xV)Op3Zr zfqCKjqf)Sot3e)M2MkOelqr6+r2M_W`{*VT6@h@)&~_`vA*W{@g5DXVrM60F{iVbh zu{^n}fbkP~x(*cRRPSVky37o9-hg#VB%pWgc{VmqOE253&@h$XIfV>JYW!RIeTitr#kVOPt7@NDSJBy(4G!U+5E`I@^ zKo&7JE}3Wz4gpoB7?BvHdQx28AcCL^u0}9r>;|4^`zWD%RRzle{z=h6!e$K++x4!E zx&u4+{iTe7(la)XK)W!7xXA9cJ^u+1KWFy%p(&QiJlDNS2T7$io5?!8XVH&0YVRtI z8^vcCw>($ml(M0MWXt=&{0sjVQAQ9iqQRxf?ieJ5&qP#!1iAvQ&PO+%U^L^_EmfcX zAHYDW+=Z?cqb+Ysz9@|0?724r9Iq)y=$@FF@7s!$!YJ}Tn!5^@n z(v17PaZ&!Yh_k4vWuifuRdk!E(Ik-61O6_cbATOz!k1!;#qiwBsC|7j4vkv_n+VMm zt`6%oMXF#Sbt*j$A&vT53_lD8^iB;tdH)Zz(9#qkDfQ4OnQ4m0<`X6OT@$7A)WF zHWj~43U~sIbE3InP;SU_IK}^wByJ@eer=@ch+;DlbNbr$1LyzX2;y%0=5692lDV4x zI)NdJ&8fQ&8`Mdq;;)?i%74;BOYK0w$O>qCo)?G8l-bG`@67kYiQE+*v1hIdI{=y))L;HhedUl>9VKZ zC7TjT#VA&KXn=Q6e9FdA;?`&SxMY&epN`Jzp+G3E!xR7IB7;geql~uGNaCAtYFQ3PrL)jUeV`^xfSLZOE zviBWe#3dp|T;?1ag&jBqMOxIe9Kb9X&!PV>UEa^trSfzXnwCl^NarjCTD#gM??K!X z+(+QTpQ~C(QllNJso`}Dxwu#)zMGOn6bSp{3!>W>3x-+?TlKU0wjDrTn5~S-)nUSo zJBSqIjl?+7^h>MN!b}Ca0ew8rH)VsDq3tmvJXYx_z5agXvOpR3!d-9ws)GH5r(5!L z@yK>l+ULAxhIS)m>>?JM#)>g{&S@@JxlRb_ox0 zgB5;m`m^#=)^XY7{3FKgmnRGlyib05^O}@sNN+8>?X5MmJ@G}&yvDdY()<{&#EYJ# z5Twau_hk)AF~(LmRBj#8tdgq#0<0l7#-BP+`e?{$o|^C)EiqTlX{Sg=y+y1;R*TvW zxg9)vR~dTM_e1_~-+*j{>&t_ig{fqUqeub)IYJ);da zgx`7uz!V!zP4+R=rghCZ=IQjVL%^#f&!9|TD7(5ssfu-unZBjH3u+LOL>TmQ#n2QN zg2Bcu6Dy>sKJRXhg-YZ)uMYF7YM2WDsnJf5rp(m*LW>>-sKJA4 zcE4H|+l=rDyR60;jYHAVFZc^G;)R7D^3uQ#e*;X}3+*MZgde6mdP2TJ`ecyq3a5<; zJ+R+c-g^vPmUDoH5YGDIg2x)C-g|ZM&agrAR0RBV1_;S|s9i&u(`^W#9zuQ)NqqvO z$26$jA_c591^8wyzsbhZq6wgUiiOm5AsHx5dE^%Y)+#1rAOu;fRsJdDgNS^ch}i>M z_9P7fghCdgty;;i{uUue69L^i24HZT=4Woc2|odF9}ED!nJSFTMB$6jD-;DkkfM)p zI!I~p1dqJqrLtJocYle-%ri1AkkGSaPuHAB_4ro)Kh(W>IMjdpH_SAav5rwuqQTgg z$WmFyzGlr96{2s6P?jOvNTTdZDN9*eM4}``HKG*SktoKRJ(N_BV1fEka{0G!w;_A6|>^ePesiA=o}cnETh1WUiM zx!6w!fTSf7W}~v;&9meRze8{p*%`4pEM@bS{qHJ4AEwqqZC|MAy>r+JNzCX}f@w@n z!$DRZJS&QUu~*+5F+)XafL?HR&j;QgJ!-Sc^j`)q*3sDkBMBXjwbDco3vTm#d|+We zB0iukk$?i~Fsr1En%#B0u?GQ1G2Zw--}D&mAa|1~9JF45ELV)wYCo;E0$InWy|)L= zk~|xvaKD*ju}d)hyPMb_0-~6VK~7e)!Tr0Kc$*0l=7Q@zu=^WgS>_laK5d#jry3)2 zd~~6|eR>bN%9b0hca!!{(hD^}h!*zkjS0&;-CoRz}@JR($N#)Kly=wmjvo%PU`xNflI%Y#~nejihwyS zFzMIaN_`6VB1xTjWtx`wLVEH}vt0*x=1g^d4)HgzQ&yHE(%( ziKV>;A5+8W^+`|NhZ^l;A_Z+f)Q87n2)9pwM=#E5H5*!SOAk$Dok? z%IfQZX75_5XjZ=rt)D+?J!<8_SK*q-kyJj}`=H035G?Yt27KZwC`Q}sCx=Nh_TtHltiqKwBYOO>XOi(dbehjQ< zmxD1JkLVd&f6PX_KQ`iLN0V!$%VtzV*jZpOx3TdOAhzqu3d_2Irk? z*gBnOH|XW|zmEcVeUF48V3m1zUFn?B$n1G3s%0j#F@!myhbU7Kae?sh&V zN@)hrz%-PRw$B&Bi!7X6s|U*lzVK;f7Q zykO|LXR1bQ6>q2UKeS3_oY03D~KH z$S01x^$@&r?XB9u*JQXK5N?$XO8L@KQJA=c$dwR(sej$+rs5hlo?YI6&DX4u{5szaIM9C9O0fJ7o2I{SN)pOqP?SqwKSJe9VOy$Qw}Ko)UDXGRpk zH^7=(fZx_@?FVR>)a;NFxYW6-h_Dg$ThN+VBT2_VG<0#XZ_Wzri?9vg9ze-wJ|$k@ zwnI{PHCgIm_-C`BIc$MD8tV-JkzH~Qe8eW`h{21o_f~9pdOn00L+3wXn-5x}=MY<$ zElsT*e^Y#_#NwP4BKT00i;Tm;5R~V)Fp%^6=;VFg^@1BFV7i7%CZUSY;Y9CaHEN=J zhoyDL)SwcY!HTG$;UPd5)UqnZ_uJt zB7ghNDF(mMqtmn&o4JG7XUL29J*dK7irR*sobzA$bl+4}F2miWj_&N+tW{@rKwIGO zju>#{$=yRq*XzH&)HElfK_W33K$li%y0APIGb~76MN3~>jDoLh#!^JIuQLSph&MP3 zIDBaYpFr`Zoa+q_?;c*>N`2J&N_Ryb5GJG^eAc(J96G$O=-vK}9K2q_5yBIc+3J(2 z_$8{6`N@#|jH9v+^iEY47F}&TNpbW-HL#&dY!ZoH3JjaR_qmv3G!_o&yZT9Gb?rsn-b4g3f zwIC76q*z5TXUf$y2!|BPbqx;`T2JhvNW>8-4 zDfzRT8=nUQD+FdCQt(1B4%OvZ-9*2`-o_Nn@ssa=w*s}@k>C9_buYCof zLGaKsXCwgotj1P@RX{m9zRUY7!!Ni1LePOT_?wyJO@tf(DK_auI70%> znE1j;&&Ybe9`)JaJ>uuJkJqFh0W`)uts0P*l+C&o#Q_YZOcYXCaPo_%lOy>4m!iUr z|8GTwvJV^k(;y%cDZM44mzUnc!1)0np8FvX#R5%JSv>@^A;KGg$m3q4uM1;U+5gi3 zQF#d5U*j9!dVD(wvTM8B2X@NruD-Sky_8lQm*ovB(wUyz#WII{E2iuvoMwt`z%$Yu zu`qATLyHi63&-0FV})iu=P+~JGX}e11uDmn2O_^ux*#*gWZ$EGFbv2?|9W?^1k$vD zf!YYT;l81&W)XWa68JJuhbMt0im@UB13ClypL_2eHNstdk-M!GhCe6-={FH(*U8UJ;;`_7D*ij*2H#r?Bz=+MfYy} zL$W#o(ag!dMn&@A*h(jY&BxT{zy=cD0|T+z4&oew4zD*agvM!9`}gftdK}9eVQMoC zE;($md;tbH3Ol56_z#X^%YXDNyY`|8L~+zgrh_3rVZFFU9}0e&(x-nL@34SW?n_NZ zke-VaLP4Umfz8#c>R^o9gBt+^kPhCv^oFLbH{ej^LsMd_inkyC#xd-lpmh$zlT z%We6=5F##DuT%Z21=w@{+O>ddPp-xD;02%`#wX+ejW+hq=gc4gh#NlRolc?Q3Is>( z?WtGi>OYizP3&ARt{NZ~&x>D!YeHICXMeqJYm6<4X>%&{gS&4r4X`llY=SSJu)T3S z0F0SbY3d1uxt(Tae3q}s@{C3#pQw1{6lzaF?Ok5A(;uf5;SDs^FR}HV;5alXRh2Hiawp^cAWt zwZbbfq1Hh!X1z?#{LR?uRw z(K;l&Wy{;occlBj?JRkIyWWOidUW*8(ZaTl17U!cnbY^^LJDH8i}5kfWTr*CZ?0Ue zp`wPB_gP<7CvN=Ke(v6~px)b}xX2F6_{AzR6r-xw=6`&@OT*;zEVcsWS?R550==R~n zP;d;H;aVI+X2=aP!(aIdzc$@P6bsbqMF>Qc&})!G z0%(E*!@|NyXByp0k`I+`54t=?yuva7W#Hct1=`yOQIk%{9^6)RFt_6 z!HR!hVt@e=)Uve~o^lHooOVG{h&FX$ZjdgRdSUwU;=Oxo&OR&WzcG!v2w)4Xeq%yO z6EESsczeJF&7khbFbm9-D8jjh1U~U0xJ2Go?KimxlaJwXITka+z}tUQ4=cC))fZ+{ z)7OK(@cWcN_1`>0F)OAbFNhbu{?YU_fg>Yfn;Pd|#r<)9kTn`K8R1wGA1q>TtZF@| z*YF9|(uKrCbCugRe+n+4QkM$9t+;q6I{MN{^xEJLo(}`Z5!zFMR$1h?$iWPORMXRs zR-!)KIehNj9UEc5gin*cGI|~8qXeaFpiORAJP6=S{&h+1@Na2Am8*69w8;mF#yYc+ zztR?Et4nZ9#im}xTj}XYIlT=~;dJtoF@0=w_9sIqwuw+1K7So$OoxA;ltE0lsZY+oU5pOHp-`N} z_q{*-lx3Pi!k4v4n`)`cY=Q9*{?rm^{&_HR8SgzlrRGxOdwvtjJZo9y6wKX%Mqlee zLGyL{5yXut0rYhaC;-#{!e!*#6fH*CJmZ}=il=&VS}#L<=I-{|$B#609@ zLNN4a9Tk;*ZjR;7qU&UL*;v8|f4LTP6B9f=cE_l(B(+1g714E2xt-aK0yR_yaO|@t zB4G^P>L6b{2T>YDaovmfd5ey@d+43yoxUC&6J{~s)FFaK zL@@r;iJdu0dfrb?ZpI9PiZ#|l(z>P)brCr1#I2l7 zAc+leJ$*;4J2JU4OqKH!SI58Cyt*S?PB$d}jN5$#C~AZfoqTvhX_b{;#>Olc z#@uhn#QZTIT_2x}ig(mrgEq3a{}5U*kS$OMa1KFOSOFGb$DW5JMGa4Y@q2r{ z7fLnE{|+VaMiJg_!dHpo{T}ZW<8Lx1&a2eU0usnJV2*Z0_I^Lq)7Cd#dx_`U zkkh%@=m?8Z!@0{UD#p(Q(d>(KDW7oz+tcuOsr`J%Gy(_WUPX>9Ime%B-1GR!yE_qF z=Z0S7zTXqltlq_K%@esd31Q~82y28B(SCLFOfq)mIFVrpl5AZrlxD#g(%3WeYqRUe9Acs4r^IvB>3n! zFm&?V8Ur=w_}aPZR~_os6A4cmY}VJCxI?%)L`e%-|2**^KbF)L=c0qoT)6Tnf=Y&i z7w8*kTzqae{lT<+ZOk5JPS(9lZb|4ZDDW$Dul#%!jq*_gx=(IZicpUH6%-r@FZK>_ z$R?3_Tkn@^&9~dzf%I2x2uWXt&y%&-VvRC0@Lp#M3z0q6WefXq(9T}pMjNilWk`g| zme%*>rg#dnHtM0pPH^g&1NNAePigoB8KBsz?Kv`W^Puf3^UL`V=k34f=J&NabKmDzs*#<+-litkb{DXlBTSc~dnKFTKB!ome_B`dmkr(PxpWU9! zoZwc(nN8bp@#-be?AP$&58>=<_!eX7?)fm*l-Ii;Eo|@5H{rHpaUuk%cDL2i%zPY= zRug5LVBe+kE!Bac9VN%kQBKB^gQR7>^H)ng8B-_`!}h?#xf}|T&j?8r>95 zRsT~L(_&sA1b7MS{`IeCU=yO24Ys^qH7pV;cOXFfB9kxUONC#HLq5TPQI_sMjR^pZ zU^?;q${+J$RVPXfAkx^1PyT%lc76=S3CE9%V<4cbubbTflfJy`t&*WtrX_Z}*Z$ZB zqt)hzNPUegk=&u3ep#L7=AXf&-GqzhrWrLi4Oq;b}cGGQ=tM3p+I9W4=By7t2Bh3E&(ZWtM;_Ur<2F|!4p@5wDP@_ zL2eXbnWq)D)yx={NVyCQ4oZM7!44J)AqGP*UKA<`q?=y3wzaYV#%L5;THFRcyJv8l z2~W78GOY+-^Tc;y1zvj$%b;HKN<4P37)S{8_f)hf)E0Ava z31V$wiy9eVh;oi82pK|CAYNra0YC$5^K*&{&mdT0cPRuD&c)JH-ok<+mj~f@=0Ie1 z#~c=UDMH7U4JGwC46Y^68lbkx|H-k<@X;Ewqo+5uk3>9C#8? zllw56l=p;IYbCKCR0i4qywWY%dCLZy7AQwUty>=L8eqm?E+Tx2g7-g5rU7}?B!gCS zNdVa4<7fr?O>a;c`gB@>w6_GvZSy?Xo0AvYl(-T2=$KXkYaaNK|*^BU(S9bU|n@jA6^s(L_hAfocV=Hx((Q!D8VSx}&on zr>Dh!pw;~v8wMhbbjD9uZWno%*uOy!xZ1O)W%P6;;qrVYiWlCo)5F4)7>u(?T1x!^ zm+s7?b>}3wZj8p2)|sNr;UaziQ_s46 zHxHFd^L#n#Ax5U-8y*)e?dSf9MCXNHk3Oip1(#q7>LM>768377G#q1Tro>%X zlnVuFD;|e==xLTDX`(`p(r20K_-So?T3K?El250lyQ0wf#4ZSDUreEwUWgb20w!n% zZ|{L~3LgX?p`J+_hgdeS%O&6CdmxGi&jAk>UV5n^9uE&T1xW~j0{>6U`!tiIejyAoTVde9!uQY}G^z7|spJQnzq$mx!X9-EOH|D}x;j=VpGMu1GdOyaN|=$@)W` z`#C|Q1~=FpBi9e6j2B0O7e{{UA6-FKKDgB$qM$gnZvgs09(x?AYQD*lAEeM8V8 zAsM_2fbry#rsn2W$gOCsYd)BNg6S8mS9y4J^tC9uMfYn8{E^m`o6u@vNzkv`fs+); z$sr@j06gFdU~bAl$>ML$ATkR4lJ5n4W~Xx??XyUzrFa&8oii|9F^6vxTl8`<6Nqp(BF} zS{69y{Pi)2ZP3Zn;%WoVH;LMBQU1ZC4#()8;l%u#A<)`c27w^z%p=k8Fd?Ig zGEPa`(9IU2u)F%T(Y@cxkXz+Ni(x+UB?TPSKhrp z>S+(+K5uvmwD}X11*ZwDGvFN#X3RjK)y{+{{XN3muT0=bMVhu2{@CP|g=XD$9m>9M z8lH|K5i@$!ZCC??+RFYS4v!PNE}O3lzMG;+q3lbm{n4ry7d_Gfd^H&|~*52+U(d_|{lyG$}YiY4>#j!w+au)Voib0J2 z?5^_tnq44<=&L%CeG(=bWXZe?itV`$r+sE{8Qt*Im;%hKl9{>ZXs( z?%D>CQ5$rl>V3N^^>A2<(cH+3GM*EQb!g})zjq`ke^Ua?104XXlnAqO5$ zOOT)z4oEJ^u-WhT*y1)@%Nc+CAzDznUcx4LVSnnjklq&R4ZIhOf4in=uQ?5~A3b-< z1(V;6_)dIK65XY#CjuNFvM(`>Zs`TDqpCN276!>BwCSA8JsgB49(PCX1101nas++g zR2N?fiur|-edW72`?^JZF_rAlcE3S*9&{re^4vowgd(7>Ma|1;i6*$x%6Ygv==yI! z12R8*mDu)FZwF+z-y_&0JctRs3i{BN9zue~QdP5%rvGpZTfL>ABRP+|UJLKKEV<8( z@8Yq+r`*FE^dNY0%^Tcuh*<*51@LI@xzHK{fSz*Hx_fhmE=jcwB1+qM(47e`J4kIkFIBGZ3h*xHLs7Y&Z;@*feUdhxwhsRZ~c zy?ihXk3O-k^U!yMOh?veyJM>b&0eYw$82a|n5$`zp+3uh3{%zCV$XS!;2@$NALID- z-HFB%q#LL=QGw<{^M_7PfhWdst@uIJUbVO%tc({!u0_Mlxq?f=XT{welrgUx;M_z)k#NU)Hc#MYJ1)H|83c!DI=;sw;c{W3*T52s@SQ}VRH87o#Cv%s_IT- zvaZv=^&7-D$jpnN#PAcr&`0ZgvdJ;P2WxnHTIu~Egc*Gz(5ZXBpZ0oH@M&VXy%)zk z*M0b=8FZ-jbv~U8O}!aRV;LmNHbcyc_zVD9tzZ|c=lSsMOW#T~TL1&2H>F82KTTUO zY=RU(OCo?h>j7&5A+`6Wc$JM6=u0Bsg11Hz66UQ|p#t1xo7MFb>yEtf7?@Z|bW@j0 zd!Q5Qx_tEmjJecy$elrmnn@Gkl5}EMg?A%#19qTtG?QKPPjI>kDE{YuWhnr`U=e|(O_v6reEiqb>Y;ER>@|6Y0iPi-MXtx?S{@Szkr7RU z4(&Vyjzk)v&JCPdWi^Azk71ck;Gzr!iY|1g>O;dv4~hW2)@A4ofYHrZ93Nq26jzj5 z?9PASC;9-8=@_B_%B;a)a2bf}lzEU6oxb_IyX2M5Vq4%xs4&^YLlBLM3hqDs+t8<5 z!U0dPtD()@lvs&Hv3+3g+jS8+E`f#@o&dYTx_ygx)J7seVwD9=K`V5S-thbyX9gDm z`a{yZG0=%26*tPz%)SXN9RntJxSnNsLDg({8;PgcaD?1_XU#LX_YyME+zS2Z%d3y$`5X*l6bf#6%g^7(#bRqM~}e$G>PJqqN3D zXwwP&{KeQOkkLB~FgjXQL{$Jpz^>61qK#&p`#@T?>-_YjFYM~<36b;`?;a>k?j1*c zeD>6Z!M^3Q^^1?8>Y)%gil`xu96=RWC>k7>`|mNLdT;gIorM=i4%;E$5XyVr3`bTW{Ha6vPck z(REO{s5Nphd0V@fVjEmO*Il={WTlWJ1f9JA4TE2Nti9{IzZ24qdh&`Pf&IwI3gvHNC4i# za~SbA4U1jsj1Y~lC4Z`=2`KDtPv_-y)`iw}ztG(H+I8zBAhF@=?u*+rgLf=POcY9R zZ42|+Z#Uk)>33;QUMNhVXr^d`;W6SxeY^0*+jO&?1;rsCPeE^**n;Pk@>9iLxVWx*nF z*$)s&S?B4X#T9>h8QMizJ~hCBz4^CZEwQ?rEwNv`sbIatriW&%WD$ZRl&Z(qHNi&Z ze2OXF4W$WyRwN6hO~W%WoPQwh)w5ANKyw!s2=dJZU~G4%Ler!WS2Gd6egLaiKNIJk z!zK`wkOSbl1q6zqjjoSK+JT-wBqAy-&l5|a*#YeWOJOO?x#_>jT$>T*`7=9Mq{3GN}>P3G6$ z!Bkg#V5f#mW zQ!g(Jy|=VBoBJ^PYzCd{BEQLUaHi`UGVz{hOtUAEZcFP0DD&f-6Pf<`;9Pq5;9+O{ z!s;~S;c>&@?TJWUMO7g`-@R)i1*y0Lp3%(GGPy%YfPW!qL zzlN_CIps!lI8Mi3bi*Cgx%K0d-5dk~q8sBy%i-Cp1@&_8!cYnWOO^iDL&b?Sp!d+o zN)Ql`f*?bssj5yfmc%>1&U7-^X&W3}+rY9@%v~@ZL9!@A5u`eI%B4NH67Rg!+|3Y< z!8iPaFD4ZBvKC=Q_~j(wbNar~?{B9dfDx>yx&=~zz%iQ&^rQ%QalkpdWC3_NUtaua zwbIk$V;qo(Aot}EW8y;}$5v)IMBwQz23F@Y!W(Gcu~9#V1vY2|T?75r)3JA;BHIzY zTu<-eNQ`abcwi{B+Og2qP$4dtv)_=j_2BtcOcLU0tsf~ z=}^w&t~OM+*rD&YV~^@t^h5ds zHZcP@L%LZ%cN*!%fti%@fB5rCadJU->LAJ%gl2;QKtWpD5G2WFA=mhR?K^zmkKeMt z`{H`v8%MRgkUw}_+*zhLMy#E|4j|Ff0L^1q2N@Lg9N6B?z+Nx}*|wP|3yt#-fW;C3 z#K1~Jwb*!5ghZ2aA?Dc$~6n2 z?gLLB9`FHlXzT6*FZ~?$rIWw_BGMBGk?E zFr;d%+YZH|7z{X10mDcF=iv6KML_71Ae;G9j!xu#J^_Kx)o9U$F;EuC06qyRD=R;R z1~D1hr;QLX6+HB{=2iOGO%^BDOxMq73vhP?09PjKHb;I-w*?VpqwC^);6i69=xUy7 zeyJb4liWMmK3p5MIPUa*sc`q_wB+DwaRqcLP}BHnX`Jz<&kXvPt z?(6e=6*0LQ$*BGG?OeZe>;Y!Wem}U{O5EXsPJ*JUp`|Fe9%Wu3Z5gZqxE>#Vpe?8H zG=mrpbLqM8QNwn|Dn#K#PR{J?>DXe!;7jy@U9UA-KzA zbJov8CYK|3HaU+ZDS6!^39vu=4=#JRKkIS~AQct{~?k9`l zSdbhWxhF7gux4-p*{Wo|v;w=ZEA2C!Kv{Ib9Y;%uR%@yV(gI=q$+U0#<&XD#KI43J zqJ3D5eI8sh=7ypbSAgUhxbgef_RzlG?*126q24ZsgtSSH;|kpjx^TT1lQnz1vsy^RHBqNoxJ4wCs<($4uC=d>*r3I zS|7$rwaR?0Ptezu3Ff+KDt}KnI07yq(p`2D*m=eDN}_<>Q55cobz%*y=i*{QL{}RI zL)-0=W#8mZ`97YVEZn(z;Jd59k*Yosp#CR$2u{^>rv0XmtDFupfF9J11DO-o=*B!Q z6K;yZW<2>b!oq(^7op}t6Y(#wpUMP$01j`u{;&9plokQc{`{Y2$ZKR&4j91=;fyB} zT^LSv^ixoBe3J4rDmdfDX+dnGUmAy?eo8WhMFL*ECDzNL%8Jg z1elU7A|mP`)ZYQc?1EfoeWOyzDx2WI+W8eOe$WyGXSu+pk@IkX?$76hkGd1t`#(J_ z9`-P&Lr?7C<1+IC1PpoKCO_D72q?;@YRx4;tZnB%mzz_BGvng0Xgf=uZuDMAF%n1B z4!~p!5Mp`3k#{`_(sf!POVAt60-xXT8qr>tLoivs(m?Up#3i_v;-WXd(dUE!?6Iz@ z;*;_`d8J6h3obTO7;zj5(d-LJrJSC|^=S4r1B9OCCrprX$aS|DPchzo_lN>A5%*@m zC}su&2Ez7_j=hyY(msG!U7;hGzCoFWPP397KH4)uYhQ3hgE6MIcpvaa3qnFz&VJ#0 zAyXGXwqyp1uM()dErX~I-Mo_zVSbPjG@ivaF_VYV$BT-Ju%2uMgqa(TH!h$EvsENJ zP$Na|fB5ta$!F4`633@Il>WmJZwDv4C|YMr;Kkz?(CV=#Cn&eEm=TzltX#kxDG%?> z;r7wpHNG-H$2i^&K?DitgzW~r047*VvTCo@$8>_E)^lpPc;16dlQ*u|_eg|w{pE)^ zym(br6=v23-c9pWNO+ioSb-L3QMER90J@BpmeLxJf`14}Ls6s?0014zf;A0E`H1ZT ze=eeE<{*OK9J-Ve({wSB)?psF#}32sP~{l?rdvsztUR}(iLDY7EDGE?6NQXoF!T-I zZ>{37W*^W~a%DBp#uFLmtJmY%94oF@>~_Fb?6SnoA>`ALhdTP~5QnQirqm+Fb11q# zdZGW+%8goikAp^1mYN3)eJ^zuI2`gLDgMkP2^`Lv!$?qClDrs|Y#k1eiK8 zn%O%!bYSTRUF85beA?fU7Ubsgfx~OabSPs!HEh$PVOK?%F3B@iG#SPjsB3YX(yU=E zJMX#C(SuO#Qkun;>*5lz`esb84(kz>U}{3arw-w#L*myl*K56W9ca8?|EB+8JCW%P zz?zO?G?h;C2S%=qu{2=t%dcDs7rHeD44QM7x%i}oJ9sXRLh|k+#=TX+Bu1HZc4F(s zO+FaP(a|EKGbMoZz=zH6bIe&69c_kKIIt2%jn&2O;K0_HQbf8_cpEg;)C8oJe)FmI zDN81>!%AbfP$qaC`uJT_Z$Ds`Alnjb^5j8RO#8D+|*Ihce7U!q>-L zwxCz&jFz^0*QLatl7pP`)sf-==z;{F;5a>eV_w zU4;v6;IJ+Sa$LNd1+gI#E+9e0bXl$tv@_@r(G49CmH$Xu7sx{(h0{gIpf-mHhfNcJ zKp|9c>|*5i^Mkc;%*g2>zwVCU@vN7R{r?t-!%#6qCkr4A47mk6q<{&{0Hs`WI4re& z@;=J%{*HY>OnXDqh~&9wv**i!1)cw9dKiWsXo-M>?}2t#0#=>a{p4;;131nC;!8=8 zI>9jJ0L9gf0u+$C;;%6WC7vQ{LG1pt|Cm6iJ5iX)V!)<)lmBi&_+2n=aQEil@9ZbL_p(j}$j0{a!fiT|#KTtQBNTgRAt7&Z_$_`QL} z^J$<~eFoV7Fnq9R*$3Lyh0;}!`53B$lNd=uV>eh_%i@>}4Iwb4{U2bvuYUx8rCk4Q zk4Dyeps_})RB%ZkjWq-IxE@G#WbX+^yMod1fjj*lEHK6u!rw~b??OOaf7L^f{#@Jo z>B9#Q3*Ma?GXC?8OaOLdXh<{$Kd<1--s3C3e`K35?`8S@{{$DBCp-W#aRyTSfb|$T zZqUMx7L8*gNpm?!CItD>io26WkM@E5=?**fE9eQ(7WwR2X&^WVJEU*J_a=LURecQW zLG9YmWdlo_(7E{j7myJ{Z>;u6KKj$~lv~igix3jhp&|VgTF7m9ZsZR}D8+RQtKqR7 z+bB~@D~|BjH+;VbfrAtXWZe-qSF+Lg&Ejqp&8v$1i4DKK}k$9e>VKq`-%MWnAE2I|F8No!428W8F6 zR%rr_1|ZkU?&@+5mL9Mt1`P}hc+HP@+}}Q*+%LLM|2e|Ag03f^ss(?}lH7Ylg>Hj< zwG+fXqHRw%z=8}8d2LX_5;lP#VQo`9QD;q~Ou|Zx(;5SP=T4m7HHciciDj*lgS+n> zOhTpm@k(cM&I7#n99$DwId7WpEq-?dxbsu?6Td(I3Sc4@ytpUNR;=u}?v=x$XtdgB z!}fqnr{9saUVtQXm*sPCi6i7L_R?F_(*Q*6P|gu+m#(zP+NjJPb8`C|y?VI%GEsbl zp8?qyVRmc=TM7uVG1?uc_A|QeBY(t)l{{3{_=2m4gmZt_fT+JP_Pj7GiMC}fukCrN zlw9^t6RY&nfzM-1mVGDj4U-$O@46+w#dMsA&LV=ce|3|&*zgeG?WSY7%i4xB`unFdx!K!_1T!Qq#&7mKynUuvTvgP@2 z!P0)>m%FBBm6t2J)@Y39N$#}Y^Ma|Zl@bK#8wGrv(D0S$vT1&x?frQYGDm^e_RG zFcBlOOV*-i8FU>yE+gd*J1gDRO33IQa>g*835c%?F;iF^lU)|}m#XLAm56CuF*R#R zWMkWmTl}X?Ry9LpBCg9Zjj6~b3rmn?xOf=d_)Kcd4HlfuA=|e#ey%Ielo`YhdX;1v zJ85w^9AzDV#et$vk^xx#hK?Z;g|#}^lU^W2Bzuw64p=h@E7780`N*+3EkN`LBz^7l`R5N#PdX!R@X zii66%Q{%#m)wzrsfO?9T4-9%%Ybi|O2 z)Ai9Wz1mU*Xn_p0jL)FvbbJTUsB9kXX&>f%+5+S&O3!JKzTmxVL20pJ%MKG zMaw7>JOs_t=$l6%5bPX^&H$)^ku3sBCySt?cS@BvpRSH^6lx~m4&D1==HPL>S3@8VwVIX1+RjKH` zK+bC2`jLO({iEf}^|mvnexRj!xUziw9x!`^o6*ZZucAx^9uck4W%nZz%(^)8&_L0r zoBMRim8_SW%E6*gEwtRr3UX+f8lC|fHH-1n!FkyFw1~D)a;4LywbAPt*&RW-D zZ%_O6zP@nc+kH@EhTr^zbiE-en~13$9}b6`>)N0C9OZSJNSm>-Nb^URt-)@MJ&nsP z`;Mg6AY>cAC<~3<5x&o`-Bd}ELoHQQIQ03a+95=og1z}HkbW{wl5Q(#Km~Yz$2w}2 zTf%vm@r+P}h1$~#u*9qsmi5s-UAKpg9Ycvy1vX^~!3{Z+Xv=-*SIul3%FavK?3rhv z0rRm1piV_@u-m@rrHA$$Lz5y<`!FyCZ5u&BCh;PH6%E=cUF$zm$3YdO zTnahD5uiUtVmIV}(psRO*y`~Z8#Y>a3adLhl9hGrZQjLJ8F8jB>~;WsbZk@nZYt<$ zdc?fNpVe>*SbLU+>V*8pd#BI*gw=Z!RlB+5JZcGz3i#?@#?J?tkA$GS8$Sh5TjxGl z_IrHq%t|tpE150Zx|0icVF&Locixvf|M|YAta|qWl1d`YjIS0vK1XV`&~lca4v;>h zb3LUla4=p28odyo-*CD6Jb<`0pGeTES`A0R;sl@L0GSB)^i&@zuF(doOp2egGu%eq zy}++Ge}@d;Qa_N61^IHe>tFi7+EP)Hr2^{#_k(EC?D>TfEtry>@&LylQ;UYL(%~W4 zWv)lSyJI2uhn~v{r((kCb#PQ2Jk#Lqz zNa!$3#xJ)+SDU$~;*oCnXYk=w{386mnt8R4>r->^c)K!4Kd;{wLW>5xwZL_q;#a|m z)>5|EgEMFlD4>cu?tLq|P-C1{q#!py-CZT)aIV|?V2SDOE@)4@d0RSgey3AYoq9lH zRk}^6;!($i6F;N*&wu*8tifT{ps)1u;@X@>1MWt4Azs|HM!#coP&HY?Fn7Q7wSfM| zn7a>m^jEA4eU1G0z4)&>6puDsg*}ODSm~q`vvwIBpyU=Z*Yp0A8lY0FSQ}kq<3#yO zh6==IPUqNz=D1z9rqSUhZ2B2)7ps>v4LGv}T!o*sJ)b?zTLuS|to1AKjke>dVVhr> zsU*`C8jrQsuTsvlT+&b2S`M^C|&sF~Vo2C^aDi1dA^|r*2%Uhu~ zPUpH5f9*fsmZRJ~SNMsST^I?~56@s8l_u*dL}Tctn5GiJw}kTL!lmWKNG>j>uiVNsv`p%FSK5 zDgGZ(QHrWyDabpmp6DMmG&1@W+8AdEm_TR(?qkrsg3VSezk^l09~M4N-Uk>Sc0~&R zRPd$cp~r38miC<=Gz>bQQ*voV;G4R2Uxi=)z3IeVLw>wMda*QF_6#VZ>?7UB&1;WS zx3TP>en{|KlZIb|mMKu$c)b0s3YG)4Sl(Vp9X@)HeN^VruV${mjawdk%LqS^Tu_iK zC>Gi9KJxs>V@;U{zYq7n6WL^an;7J~cEF=8@pH*}msXDA}UD($H z>!kXRM4Q(fvYpr+uj4cy(j+eWkfm3h5*7L6Q-3YUlENe1Vih$d`hGPWnI0IUs|DTt zv?o&Q{U_V1<{CIq*?Zvw&J3SU*rpd7T@FLJb#K(79~wF9zh^jh*4807_kI1xLwhg_ z9`Pp3xA!O+#@4CkSZ#c-a>sAwZk0LVD{Q~GIw6_btMdGshZQzKAUULO*n+3bwQyU^?Tl%+|yzjMR``uVB_lhL=YYDM1-vofuQSS#;`km_r zGr!{XA8I{<&Jmw}9d7-6MBr|Ej2Iq9rNPT+-5FR=t{n`l<=?{})ikTcv~w4yzvpSMik@;EUO76(hx`uk(x^^oJudl%E-PU#*t+I-M zs$|YIQ5WXstL)$;6?#)8VCr!G`-}csmp9Iez))LhmHb&j;!bjHx}gwM>+pS<#F6T4 za}V=Bk}=VI?75d5xjxWw<$_X708YdU-|tdI&%pxdMvVA?aSB^THd+V22m!y*v+4zY?kG4)Q2#cv5613ZI!Z= zi5SjyYM0dJ-j?$@;ogx4T!qP@#l*8%YW&DIXsk<{DT7O)$wtbb2anQm#8JUVIKjDh zdXp7*=;xK$Psjf-IR0Vgp+X6yZ+%P`{(QbpbRcW_0R2#hwf210>BJM;n%{lBuqhsU zP_)#qDUB=sKHW$^?S@rCN8^kPKSXZe)kO$`r={+Q8Dm3b1I`aH>n`xo;FhOw+?6hn z-wq4g_e~w0Z2rXaF*m_@AE<|ZRZ)9HR+Ma(+3hS|=5tQ!>sL-9XZu%aiqiy5WQN79 zettnmi*v@ezk*ZmHJ(^Yc2{3Vcq#PphG@(>gKaquF|&RjKRmvXbn5K+&6suN$C~)D z6hC4ubk1S(Y3gmS8-6B-UZI<4OT4O^*(G)NxKs?yCeLZ6J$z9+hDd8{d2K>T*l@t1 ze+jwQx_Q!Lq73EJ%G^!I{8{76`V*I8M!CYJU_%o<6{0Tru6ao zfLZARwb|ea-H(=f(Q4sMAm}j=rZ}8*y~*D4GiV{WSF( zG{RCjBWFVBc}?36Xp*mG$-BIr$5*hOXTst#3RovfHGT)})hg%ujE`R;l4sbN>*p}` z_my`g^eJe!7q*&@`NyZ&Q)@d{I13)m>hfJ1vK9)Q_S{b3nu$K$=Ud;!+iJSJ6vEaL zb#}in=Fp7l*`b^>oPk?;cUlG4XgqE41vcvX0log@7nYsk?w6NuXWZ!2pXK=JT!BkJCkk$+sx8VZ! znu2Un)qi9wZaW`)nIJ)`6^0iy#JbC1C@r!gtubb76devNi3FJjOOph?ul#Nb9ZDN> zWno&iV^NVqHd*R3sTdFZ?;=x6D;oc4{g z&H6)sUHz-X=SjJ|7`swDN$h@l@?XyIFjA0&ti1WVJ0yV}25-JM$-ay%CaM|@eV^6yb}|Sc!~cVqyO6Q{^nwcAuFq4kT{?%e`fQ4u{3{2bv{Ja6$yP{|2sy5 zVwQ0D$L1S1sYYzMG~`JEodj=)nv5a<8mIZXA2OBY0ibcsi<3~t4iLnG@>%^G9My3rQ}2vgXj6;DwJ?hbi#iP!Kry(pDXOf4+|yv^0{9 zIQ>`kL}1qeNMe=TA>qGc(H9MZgI9>fNc|AjL3cfQ~JDl zEP(ab`+4e5nerZA`rhO3(}lLUr_8lAigk_Nh|vD13M00RV|mV{6d+lHlN4vbqA~~L z5=xI2gE{IPOc}0U=60)Z1$(R$04@&q7Ic9A7<7IwnhRMKjwlU6u(_#8S`+4-@n?^~ zomc8~vp|;fGv{`+L48UWA}j_#*SpyiELSDa&HB{c-5T5j`(g@zu?C00=m$zDap<^h zLoVr5@00TXeeLV0zyC+Tdn93WJ_q(yY2G&g+HFC*Do;r^E7oI^S2h3E82D6FP#>5A zmZbvdX?VmnG`efSZ`J+%|LW=5t#}G@+>|$LSDvMH*M402Y);0B6m&r6( zOGtyQXz7r0-PW|pamiM0p;;wLE-4c!!Z2j`*RGLDa?_=BZufNRuK9~3Rz2bW8 zC8z^O)}pW?*Kxh3KzMO?Z^%GZJ|F5WHdIPXu*Vsbg-86=etI#yr9ruu0U0`f-=7w2 zP`^wlo6<2Nmn}j7y9qe>UMOffC5TtdKql9{JERjBmbxpW-w)!G)fm8cI$#tUhPDkC zzRkPPanrn>Gvcwf|F#>Zx(&z`3|0{ zZqR#Qvb_>@RhpF|@jf$%XgG~-G=f3I6hKs!69b|aJ|fz74 z`NA{EWP%InF45)F=1ab5LSj zx{P#jvh=3LSNFFviTg1jV(}?stXeK&81e)*EHZ<20Jp}CMN~Dl^R@4OR9Ao|bIy5Y zNkFxr<%Z@#Qan#A>y7W8&?b7C4!x}+adaz$ERA9q8j#WQW%uDKeSKjjT7V5>4+8D~ z(thm1Yv7u*)74ERfLIu_Ej@`xdH_~TkvPE>+gKseg`^n?Uk*zyewgPdDeSTGw_clq zidYf_TYC%a0M8X`pK@Vz#bN||CvsOzG3ws#prLQD>r&CJwl<47{Oa-0RjukHVFG?t zEtq~k-W20mbutk=L-fEw2{37;x=lQpq>7s-GwFmh3Ax%SFVKIx>-V4c_Z~%@T12H(Ar-^1rWx0h24RG zuk%>N1VUWeAqNqf8i5RR(cVLQWJ-#FiO7(IDPec&W32xLYdE#~k0|LzoE%0w<4eBH z>*r@qwN)+Wh#yxR4!>USLZ`GO%Q9bxo2>#k70Y}-6j{bD=j&0|QkmwOL=b|oAh(8W zO-sKmA+&oFxpHaMrsz7TetoTE?%=e5B;mCRuWsI#saWwddqf<35JW`wB)bF*lR(kHd zr{0cl*g^LN2;55T)RwVGffVR7uf#H>u}~i$70ATCZR|+#k$UG|=!z4jFU`rAHG!KU zGraC@f4k2+i^7tKICDe;8DR}X?zI_XByEr9`N6c$nTIulE;th&8>Xr(YkE}AG6)Z? zoH%*qn#B||OTCwK2;Ogv(vVHN`nbX+LU8TjWTmFxt3%_TeM1=ve6c~kzqFLW-B*8# zcaAFwfkFgwXZGW1RGpju9QRbniV`ViW z0?9ptTbPE)Ya@x>JMn=10T_{?+Z$YzOu)Ap#6SCd6sG^1fo!<_ zB2Y%N?(aan&;GDAkVB>QAu3kpzc`no)wZ4u-yp~a*kLP|vkwTi+^?>=rngg%ACD3x zic~IdFfnn@*SwRp2m{T|7;#ZQB#Gi6jmC}x19J$y&;TPCm6ON`vz_hF5=B^DZHT%K z9O@66i*ixgCY?-$mL`V}L&ZTPrcaA(z=ROO7Q-|c+sM)0r8bgQ39r~*htA345(yYnR5`czI#;WUf zbz5I5a~FFHQKnr#|ErraZ()Ct Ol5(_nwX52}O8Ou06wbc@ diff --git a/docs/extend/images/authz_allow.png b/docs/extend/images/authz_allow.png deleted file mode 100644 index f42108040bbbf9facb9fff0c320428323f061e10..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 33505 zcmdSB1yEIg6hBBjN;;%dQfW}S1ZgBh8UaD1rBg{Mr4a;_loDxCN>UIMkWxZGLZm?& zY4%+8xBK7!?9S}Y&d%)4ym98ryZ65PjdRW?&V8hn=q!g+ci8JxciMc^7!KkZ>@vJuSY|M72eJ0*RBf^rxl2wv9?-oiI(lUN6 zFTW}e7?N}zG8p>)$&@ceomwRb_4h-s%3R0l@IfsB4KM!BhYN$k&kUDM4g>j2Tfh;H zC||MCpF=t1ZlOD;t;3&{XH}UwaPy)c z8F$#SHy>}Q*9Dx^o2<#69lQG;Z+3qFK~_~|J8;3M>4C4MCgG9zOY@PkTYBQ(s%(fI zd{4!$4aQdJC#H>uqm_$~U)H-unBT8V;e7}Hxr~4PB?0TMq(oj5kvgBv7d;u021+rs z>~>!YVoC%_c}jH)HAA%CJk(o_6v27QB#zNAf1z_slYPL`xp4U0`{6HzTHRxB9foKK z=|qwivI5PV%p3jmP7nG5JGzTZYVLJ3>{VKIPQ==Lex}Dze6uL=aYXm&+YvM2L8+tF zs^!|6gN^&Ye|^fn`Y=x=;le9-ojNz`nk5t~IjJ?3o zd3weAucUs$)B9aCYpBgreWGMPTv9aX`@GP!&MjX@w()N3Hl3Zv{`_<2uV1gVhG4OK zE`LpARVFIZEsX6+6ISNb={#}WxEtZ>MzTBPz}PQ-v^!!xH)LR*EFX66Vy4Aqs}2J9 z>VO)|qLe8DGO-6as|I>ZH39bS|!itj*|`F#IXAkVzVyWC6cZ?&Caw%Y;ATj zsSmx5OzTOK-oHM|o^BGi9-cRKuXCHbu|}X3VtVv)>6}}T5L?W@{#If z(FotI^;#RVUKO?NXEw`t=e1@}fA+iiY|Tqb{VC6~;fKfP^&WgFxE}MSp5Vdyw<=|Z z^`YVv*M=MUY8}<$Yp<=lC%SwN=2999+QjF&Q@n3A_zb8_y|z|Q7IB~NDbq+3qFL}< z8NQzHyE|kMBj)tw@p!!pg7rBp&2*6+%F(#dTz!igB{D^8XQvk30Ps<&+?rWfmv zT>t*A=CRgOp0pL18l{FB!i?dGGC#VJ?%z{^&gou9drk&1v{#~qJS9#DB|XLb58tdL zC^K$8tJcnuFN3AIlvidk&Yg;kS2-d68-EYK0Lh3TvBFCwkJ5rDLp_!T>&ncV%ZQTs zZjJ4?N_nlOiq~;d=S;oxE-$v}%j}4oJX&vFH?F!Hv(gbmcS-9+A%dxAXX(qkjyR_E z!@Y6mTY(4N=F#rYM4j&?$=VIFvC1gN(w7Au&oY&1$)0@GD>c5tI+<>EHp&v1r)=!u zDB?7hx3$Bew9uR3&TClFvSxLf5V$j-OsILqVff~pJoVkr&*sLO{NFjWN0Nr~*5@XX z;5d7=v{Q$?+sk~j_3KguQ-Hv#5(7!V!5kGU z(c0n87cIJ>i+Zn;B*Ir(&1?O3UvVowrnux3LBwFXSn=YanD5pdDha%gR1D}0lfDZK zxjhn|%e74qIff?d+k#_o+p@{>*Uwj!S#SKE>CpX@w+xo@yLjw_YNv618wXtJ zADDLK7Gvy!BQgZ|1^&^GUD^auTsyILU1xfCI%63+hz()=MPo8O*a#+#W8!wlRa$hN ze5sQg-LQdaM|Af#?Pc3|*Tu=k`+Oc!8<@DahpwK*&no+!ot{XH?ZU{@u^OhkPq(0Z z_8eRNhAl+5v>VPZIM}%vqa(i6&gn2*QkE5X`c3S3HcqzwZeQlw){ev?>8)?Ctz+Jt zitWE<9X)*J8*qBCp!AKHN#?T83#$}u+yb*E$KU6f95Z0$7L2YCO(b2^joy=cey1mO z!1L_XpGqRHKBFi1fwSk8!Z4pQlN!xi3M%jJ|Gq~hfiWM?Z!VK8Dy0w6Q<#X?=ym&0 z#chQVhw{b(DKsiRQx|rvOw)X)siwvRBD(rE{MHd))VNve8+FU_6N~ZA(!_~MY4qL! z$vfvdObHutiCyOv?J^|2MG~(+7a`1$vXV8Wd+RVF^19=NT;n;$np2i^ad#zn0Gqcl zZ&KZd(cNc#h=PbxJ9h}?kEP=d#-=K*B0Sb7Z!u_QO4apbN}H%X<*{Sq*DZ+lSRK`Q zb+hQg_CjAulmAf{)9D6|c))S3nmarqt5XG?YU#RFob+9aN4^f9$=yHWg}DrI3wpkn zAP$Y86FoQV>NxgBb?t03E|B8Qr_awt=iaS1J%Qj#Gh{#6P*=I;gz0`7Cv*5x;*IBu zzFmKIu!TQfQ{~gPp2gK~RUyO}^V`BUy^eRjy->tAg6$JcFMi|m_co%Q6V~aynr5HIj|<1)Ci`vSV_RZO*tFRQxMrAXbU@*JNX{{_qMePtrDJ z#Way@z&_~?K_){`9xh#g?Pqk$xa>eNe)A?r`i={neHO9r+~yuH+c60>z@{sABJ5+0sLQS*uS(xNu;6 z=9PivGHQTP(yL^m)>T2;MHOFq3_EJPqJo^^ozr;r#ch@!=ehO2>TMpz#Pqdf$j+jyrz9lD+@x}N`vdhSbjvXV*gS($mmt7n z?oKl_GOc>czettzzQ?LQadDbPGeg4q+HJ)*HZ5vPa+vK5vRi&^usS7uw?B)1*Hl)< zdrzDYp^VNwQAx|B5k<-SYLMe5XAOb0SE8}GyQJ@(HE+@MT{Vr2<>6B9G$HF7B3A5; zdr!X>8)z+$l$ZYybsS~(YftaDH8Wt$Oml8N(cfDeFZ?wsr!1lnPOvO+?JPgVjLh^4 z)II}+A7Ln>ogA4Nls3m!^48ibYiq^k@Dw_`wlTGWY*b!0CJz3a<32x^sY#UBiyH5v z4^@+^)n_Z_XRq#2ZE0M-^O5Arix0flsJ~4sYn@SF;8kkOS4+N8xc058liL6JwqR5S zW5)A1ocat-Eyj!bC8_k%em7&L1?mTWX?lniXlB^o!E)UoxGO0Y=lg4Rn5%Y;bi;yF zm^;G@M#XLGr|R3IF7EP=6}Q`VSUer*yn19O=eICt;=dl!cA)A=9UDVF$q0Iec>jb{ zq+fHh_t_Qo^AY3IYAJlYB=@COW>(6ZMara|FD%>5<|srY6*K8x?UVkU-mp7tviy*U zu>duFrDZLDQTx3@yr{F;?oDM=a-WXSUn+b?udiMIp6%pXoLT4ieY4*>m*A&Sjy%?G ze1S!lRzi}Op!xkVM&~Km0L}*jy-g5E_nhAOJT~%8cVa zEyPu_oM>uf2&i@Xf3JU-C+TKOCor-}tutG%!ZlIv zsqSkLPt+HmDrEgQiAzVGh)KrLV^SbFZ=5>!h*$#nHh9644EIOCy zD+!cKh)mT)TlvjNG=SE6z~8 zd6tLytLtMjcgjdZr9IO~Fj4CpgNn7pR0k$6_5yzo|9z)GjPzV@Ih*SSt*eg#BQ(aw z{*KXkY{nL$*Y1+ih8cvVp2~DROQzAnj5wE){&)2v4)>rS?fXSs_irwB&y+bg&+U?< z8;I#QS%pa|X&TaFQE_GJhYJaq$C3uHVJIcKeE(r|U=c#K*OCb}0NuR@CAcG63>pLx zwqHd@{jj{?rjOFinuWz6-GN+ubRqO6?xMbOYl zI&#%gzII%3JSlZ=U}0L_}3y`&iFiyC%708ZdKcj z24t-P_X)YrYwYqJESf0blPRFBhjku}E0Fp~hlW@>E|7Mu2FH{*s+^lyQ88K>qC94PBvBWm+m+x+UExs zNYH^wOnQ4Z02fzoagDYsnY*wI-;8e0@oGvpJ493bw9N)a`EVf);d8sY)K5qhL)~{O z$F@pD|6q7I4H&h~Yr<8Df2XZdEl8fJn77aG$sd5gP^=eSbGMmty`@$hu1fFg-g!s~hjUoFIQbmU^(?UwWHX^d8TrMD`~D?OyIfx%}$H zeWAB5AHTB~0?wV;j+mK`&wNi)S~h0d^zOfi+2!=z`kBb0fPGQtMd&f0fh8=yuPD7?G6Eh8&-H6VG!IryE(U%CR4zmYV3DA;Lq+x7=Vo zB$nBiJB51mV{V+xtI(59robeK)5aTd!BGv6>0Ud26I%LG7}r;SJyXi4H&dFs&V6Ao zje`QGzv}l6U+4w%`I(HCsnb2tIq6^=>+Si^Ti?B>J`EKc=oM}NoJ@g<=>mMynAKyb zOeTjzY&lwPF1y<-27s1rM3~}igema2(zW!5-<80zH+H$#nr;@|7|c~B7QO#@M6CA! z2!*5?``zWCLvk&rv!q8L+O}18J3ki-MG@%PTGiNZ(=mwZo@9pFMVcmpW@O@ zm8QdG?zs+xDConN0XUNgB}4H2l-6&k>7hqaQBO`!j=D=tYQ7TZR5|pFrQ!9Y3Mv3Z zF9>D)avzXS-4^QE7ehNbhOfU@!vh&d&ir)}MRxEBBm zOfM5QP$DoParT$c%k)jjH@I>d3bO)?`ycHR0JCT-hw=MxdHVf+?c_1Pszd!Y%4mbGN+^T)5@xJ#^lOgAgs=L>X zV6TR>_hrfQBc5a1romcZ zlJe31w)^vm*7BVn2lu;H2IQ=Y3A=F{DR1=LKf*VBfHlyYCEMJ5r$g{qSmEWs)t0!W z;_{|fdxABm?c`dxr|6&2-J4b7)}d}ZS3oX6HBn+D{0*oEHDJgLfH9033y*Kniy*>K z@W7ppIOlXQ?%Yh{6pE#nx*SZkSX7vKV~t3sV8bNZjb|;)7Tp!uT{gP z;nL?VjgKAj?tQO!JKlG_f)(aK$Y)r>&|<3*fv|?jm-Ox&?{D6CW2>|Dt*S7I@0Kx# zW;$2kej5`?_WeA`$18T4O!(u}pIJQ?`>*;dTqB*f*L1}Fc}B}n4`uy1N_5K;k}}T7 zpUtPc{8=9?{m^yQUEbvi7vkW_Xl=Mnw*+Hes+J4JFpI^e3R}S+SqJEvv_P@j@rH_$ z)bEthuRCodSM(BPoIATvq=hy^eLM^}!N4sbLkTZ^K_-2~e54Vw8*V&tlfhQ~GF=Bl&HIsho`9;QWU#;|J zNM=Ud!n-vKEC1$*8?4A%6*R3?nwdxSB+j0WL$8uw;~XppNeq?UGWj42b*jYJ+`;`< z;;CnW@uTC;c?qgyBJW*K*jnkwGwstzrIl1&%6~jf)W#}t{E*bb8YgwZW&IhZk`eVL z5L0q7lb<`MuL$7OE|5$Q+Ej3$H-;P>?yhvnX?(WuFnDXPC;CvZUO8xmo~3dGS7~!z zyV6qeZIn;#LrcK+yU}6s zL0U?9;}x?-qZ05$H}5YGoSyaKey7xLYi+L?Y#k88!KywCgGF%2QsH6i{-T z*!U_Q0816}1gAF~6>6eSCF2E%IRq9z0lPhItW-#%tEEaki0Bn2_LpArDf!fLi0cKGa@n&@aZYG!W=499G?qbYi&siSU9mD?Dz#Vx`HOS_4 z%C;#p`LiKBW5dy}cLAT4%{crHVWICle~PoFjwV|`jc&CoX*x$;Iqmn}_M|-Uq}Xlg zDZjbdcg5fjN`|3%3VG$Eqmk2f^=9L_k{KH=g6*6^MIt^+0*CD(N_|1W7?g6BW7zn; zBs(`B@BLVGmy*<#kfE5Wtzp971sXD&awq(PR-iA+SowQvyTvCHIpxddO%EoC1baJB z@LN+?XvJKz5f-adBdohic)FAHch`>>9sc=vShe2--pk`)@cH!>F5P}BrG%T>_n}m~ z6?GDKzDHbxju9i(r2Wk~ec{}gnZxjpRt(stb@OQJL?Mj9VcWEVDaAG1H(zF#nMaG= zez@QndcTx(qn|5$AN6LOv6(!zZI58%=d@<{!S+ITs-V@)1+6$47Ry?8O&sSq%{@or z43Si4sXFrQrBrj}LLsug9Y|c%*K%TEea9KFdnCSz;GVpyH2N&ansT<)Gmdlew&wZY z54;GtC%Ls^<1lKl@unK@6B*v z>wG)yB*9;FHsTYq05SdLFjni)^hz3YI19@dJetdAt1e74VjiNe=qt$rLm#F!FrLy3 zVDe{1M~M6EjLkf4Q()hSi?T7aYNa=I_EG!Fne~9o#UXovn2J`v2_bV&g6Aq{DA#hGQFQ3cqgGs0lqk#UzE0=&^SHJ51l(HR zOV(~whS0b_p`L5?d=&95l}U=M3^ok=jJ2HdUb2Ci4)J>HV}H{rA2Z2-$R}?6wdFz? zBym_yw)cI`j!*jJZ8u5iT-w{5TsTohCj~S;EEPXgdxwbXz%f5SNT#hN%dBeQCsmb+w_$8FC zvZ;Bh1P10Y|BUx}{;#bf@&e7}Ij^%0LQ~{~q|liK73HSMeC;hCM-{oR_IQ1aXIU=8 zZ3vEU4M53%56J253%FKlBlt_ZbtfXTpr7(b8K;lu<-LYtU5WLGoXk1iB}+w&N9bRDTo z5ZMMk-g|`}Hfgq|=%5PnH?u0L0((&_@bJkY+eOw)M>?Q)YOd=JX#e9Ha5K@uWPI*Q z{YPy$Dg?tPdFb9M+=)2xvr>yTpYQ+5B#Xv!DZYef+X)j1R+qUV!t2?0<=n;AuSV#fLT>!vFCk zB5-cC-6dMJf4Y1td3HZeU$o6%4L?4}372gtU&PhbwYrFlFoqtzfuVcOO|eYwDTNS* zLIhE=_L=#%AFk1MyQ$JlEu$6Di%2$i_AM$JLW>k2kCXtx#a_NQG0MlG4orlTRq?m} zwegzng6q$FbYjP5_)&SWFXA{%^PmV7RTH zg@peb61hbfLWRC0@qfb~WR0u6$oqxja_B!U3Vwk+lV3qk!oOo4WWLi1vRVF{Z$FsN zeqrm_|K^JZ=BxWM7U8DiG-#A=Gw-p?BWSe#{JiN1SPR7I+Du)~t`zf_)(V(a`xKi^ zHTmm3DAq4!1J2$|7D|wW&{)-mO|WjEAMb;aX?{tsMuz8HaQEAVPq&x#OawU<25q*scI(^@%wuvnwHPk zyMYMp(Eqzc^jTrF45NO2Oi(yS$@5YM)asEcxysRCE?9Y!m1?sPxBu9kd>{RyX*$RJ=;V=w~n zgQXz_7Kqx>-kSCP#!SeA=@>b2i58P!D_pK|lT05yPP6;zpyGW3j`eG~x%u9&@5Q}# z-|OF+Hu&gBc&)ya5SK6y_{&kDfahbPE{xA#{#x|Jt+cJNOcp>op3&)~P{>9sov3Cq zPVgS3fPHV$#V3di!Q2b5`^F@yC&sFNrt$ZrZrK@PfT-$|;l{)!0KikG0!YIe$dk#} z9#QYWqR}Tc;3~(sh?_3yt%=BWCx^Sb3;f1!R34oHe) zDf#A0?W=x!s}}(cCIQEz0*DjqfYsu%d9gmPxe@Iq2)x^AoU4-HJJ_P7m+(*l%)~)* zW*bVxAc6ZqhW4is2VT5fEp8rxK}XMsWbpC9wmJaz38K3n<6E6%DI~2rWA*r4Y6laV zWrIiN&I7;U#3&nJ+M6!UYAE%x4H)7ie5QLl7+B{PZFj*~;9BFqO*FBEd%9JEm@k0a z@MW0v|9#7P1=MUVL>IL=pRfjiNgdc4MQ@D@H@NUwAQA-&kWEL*M7!jsU73HUp2WpM zG9!15vHFE>5E=`=Np*>7quSDFdf?f3slCerE1TA7R`(K!%^j|d@xA7m zh7WK}K;6nN!2I#!2iQpXI(R+D_*lOPI6HoOt@ub16`3MeYsD&(ta=sBoWUHG3>2()v#Un<9W=R&y`o6|)3Y>ZMdL&KSpw0!@o(_7d~_ zeUN;M75yWKnVy1wpm+(hcDphr2sh4j$Su zWUlpCx`E{Jo2ABjh(-r{;%4!dV4#fmFVnXwMBAreE783YDi(OSR2X81S#oGzX&GkB z8~5;Zs~tm;XIX(}kt>^?=Y5-qJVeQ#;H`N=%BuA8MCCYVR5?Vb8`4@ND);1U^CtTt{3+)2o)1YD_hsmm*F_>#8%qpf)a9ROVz>Exf(6F#I-its%2y zghTU9B^`#XE>h#|t2-t-$z%|Jr4koo!86J@cL`naWvG|idB}pUG##g|f!fyEa zGl?GtkP>7({q92Mk+ASrhE7g2%WM|QiIC_@;)Ef{x&y9}n{nQ``LDlee(ZW;P! zT~Iv7{Y>RKlDk?sAu4NKmXgDG!8m3G8j8lB|Gfvn-pfI7j7|_^K0(TCtL=~d%!>_z zmR0eNd}PQ-;PFD`s;G}hvOkU&mfBu~t??dC#}l@EOwHONen9QuqMw4I z*94_t1N-i72y6cmBY=KS>;)gGr-=Y2+*5xpWV2OCkvA#a{F9}RXL13mzqmF5Ioeejwbzv!a zORP*Y^HI2jBvl-V+}9$ZijdD{y#+AoV6(&7k||=Pp4=hP;y^AC1$ux<9*evk_B~Q=FlRMv_e#D7 zQ<}K@e00JI1o^MHhd^>DnF2kIgzi5YI(W_yD?@=snk9XfnClA)UgMiWcYC;Qz4J;2 zXIPOH4iSAk1S+*^$8UMS$O?h4sLO#U$N*73S6I!w54+S1Ola{jkoG(;mF0c!jaOf{A1MQy$DITaNxFbn z{4nSCNLDgNzV0fLU`3!9sPV=06ApYIiidE)zCRO`5#ji$mdruB$!j0iMG9b8)xi*u z+kh_7nkjw%4v;}x0AJMsyz+r2kPIx7dahDTo>`-xy(ZIznXc_6^LxU*i!o1`v7U9dTl1FppX3(w9qN#^$q>dp1vFv?i9Dq_F;vd+ zYE|FYk0Y#hE$<#;6Ou*0fRs?I)B%3NsLh?)nyCO2u!*rlV2_6F{DBz{}?6T?*{=(?Y6CrE0i-r~1*|~45yD4)g zCy@O|h;h*>_gST`dh~pyPnpZid!_st#?Hlkq>v`HnGB*EHxJrhjt6m}F7T|09@5Zt zycWgsze&h^U18c2{Dfg3)@ZN$oj@#2Cu7@gg<_)U)@Ytjf=qhA*~JdjdA^&Il}DVGscHJNl<<5KRg`pJtkbrrQ`?!w|gZurWiIb%~-%S-a&jnC+e7}4@N{5~8a&XXnA{6N%0lmgbGa;brtR?`OY z7R~Ltc`MBcB~Efe4&fk8kBNhr8H_0*850G26+& zRQq)k6C#EZVn;N-De8adG1r?V?q2XrVZ^f|l7Y|5gZ4OV{`2!AzCwDnI<^1l-h}OoKVSC~A=lXSff{!~TLXcYh`#bNG`ELc+ z>GYU)gHD9d07WLHb0N%`k*OJgd% zHDn;x(M;jXcmUbWK9o2`YNF22x?BM=*;iX@xBuWaEIJpN;5%iZ9O4Wu7JxVH&(~(m>oEj< zPtbGedK-G>TSS$-3@W82u zHc?Ir!!SEyXrBNANiPHv?*O{0a8iyS#W>`D*euRN$S;0_T6006M?Qe`$e=@|9|Eb1 z2C_`At@7}D3>5R04>cj?pUV6ZTvo00s)?hzT+}g0Mvgi-J$+W0nZa6$y|(jg*_GKu%Ccw94X#Z9(Bi0B#JU;TKsF%Db0np~c00 zSoV|TZFuy0ZtwQkK7uTQ`W8dHzZbE@yku8dfe^2s7T(Gy|w;FUialCeF^!i^` zC9Ou^^T=Bq0#hKyf+nctcee+|Aqe$e1J1tN(leQcn#L|1m8^0y;wE#Ug<9Z?ankN1 zYV^nc?b=n$P+_t`uHr;z)M7_}EM1NyuVO&9CXHdXS#x;WtGrX!V0&9WgPdCMJS&;_ zJe>GaMTiMb)Q0axJmcDqGQu!kwW37h)`Y24g>hCF!H;>(jUIUd$!gVJW(`uJ3p3mq zlP+2BF&}!Q^xt_5gHgN0dsPytM-cQ*0CRUcsEg-6JbDR^_3iM+{^TE6C%2&zAjZ6T z_pgdBhoO()Q8o2`BKXb72Nx5n3F>Si^?%0|=ir`t#vxuXjmSri8t`WuS~4ZS)}kG!3I1kAZPs)4rW+fW;+P$TxR5b zglx*#rnTX@BSd(Lt(`Ey1elCa_`LFDSUq-YNE-13)%zi|qta}TE zVCc60`jQ0-ia3L6h6|Ywwp7102R1WkKJ z9gKzhEg^?!*pqjAS!8%R|_l@Cr7_i9vp0`Ld!rNuuZQab$e>A442M< z);b6U+R&$ggTuo;n}^MA$KMuW;>x!G*Z()cv8k;6;0>8Gj<|m-5UL$=eIk-c7~7&P z=Ke_xrO$|z3i{=__BycSbUhEMbJY-S_&s^QYd7$4$IchzR_m4^bcns@m&V=}^MDuG z}*0iT@*tzpJo`VH01tbStO5$b2YdQBmg02w+O$lRvDt@yG zr>O=^Z}E_3sgCze= zM_!>MU?U-fVX@V=Nn<0qijlg`r}N^PHv9hySI5=7ai!V=AaQM)?z$|ba2seQa-PLi zZ2(rae}?9}6y^a~&Qf8t9=ro}o-0quPQU@eWe^BPprMAq?DLy{SBUra{B^yCdGbi! zmpC6}NN#A5^I>>=kBH*gW=F>%u?5?%afU3EC&9J~9|dh>?Z0wekqv|z^U7fplt?^g z4Teyy723!2VilY#(8?;K{P{UuTo#vXLL0|Zy202eh61XP1BI4)IIbm;hGt^~io zu$J6Nx(n}@aEpLfq=7Dp|5NnRBqh}yHBC=|UCntT8MMM#Tc z?gbM;k;NEV@u@+HVUB+HY<9C$r5tJ0p*FU}IWC;2jf$B5QWH_%wYMX8pcJrY>Z5P@ zA?4hfMR?k^X_Hmsh7prHilg>;N&Y%>n!1*8n1l`foGAE95o0a^qtvLp{s4s}+4S3= z%|jLcQ0~4~#!l|kgYB5ri8@`gz?0oP8_IC{hPwTkjDGjMhRpY98ActfEu{WZY|#*1 zEAY`v$Pw$qnq}GRC#?5PuB5O==l4s#pfSQDi@c8Evrk+=u}b-7bk+z9mccz(1}@}x zPh(jHpjV(~p|ef8yDyO1BJ`w#&N;p_upBRtO17PGgbWie^f{VyIlk!a`E@Z2{^nz$ zk^5q0dTqKa&h9()Y&}B}%Lix@S^-ajL^0?G{&TwFvZ6OKmxF2JHcyz*la@e;iV%fN75i$znEaVs$zJcDnZb%vPyiRU616bBVK)*X0A z3Pg*HKr#<=5sGU|mHp%P6Nvs<6n;`ENH!p2mP^n=BO#8+fJC?&X@sN`9`gRUcQMx3 zCHFl5VB~gZ<-?7{Qqjz6x`%d&?pZv)8AIGtsj4ub0SW1d6gT4I(S6*w6I}7O9}uCt z%Wl1618d0B5)$Q|r;e1gib#%W!kf;|JvMxZK3TH?T>&(Dr4nnhv9$xWd>9EPk5$e)E*p)0&2r@57@=53QDeeO?!B0Z@7fV|NVx(r( zyc*SATnqHC{O2&FWVX<&EbKR#{*jG`j0qHI%od833u~Uewbh3Be^O~M7YWj#<2coH z3;3f6HGXhz@qowobR$CcBNza_$13_o$TQjf;EDT_IU(#L^o>vTRY0mghLGl`tzi@A z4^eR*s4$tqb|c2W`PT~fjs1|?#^(7F=AHE1-$N47f1iLE_As~*vq_Dx{-yE_MJVc{ z2xk8)4Tyv3MDKlsBzv4!(ZO=A@yb}R{-4=b!OerBoR_E#L-4_WmYoK^6qYwANUon= zmcJy z4nh|?cPcyrTZcA77<3k?R$6{QW#BTzn}fnQRBHV%r4}prRG9()&kllYBA7;s%$u85 zzrPc0wURJDS^KPSZv7)2Y+U$eG|#2)dm(N$1RRGGGnFk9tYcFz_Sd>vc@i8oZS5Gu zA(kj06!YQU2#cLj%HDkOe_LodENHGUhUqQ&#H80+R4Cs2?=0#BA0ZiZYr?(n6Zl~6 zoE{$tc`RzPn<1V6j_c33THC>CSCtQG|Bs;zaL|x1*uZqRzv5HzB|*m3x=t$RLd9d4 zjoJa=3kdrB(nxvMrSU02=PM%P=Ul@}C%dGebqVQs6S8PQ3oZq6Lr=t~`wM-_s2>lI z!Jd}DAh7Z`?j(wN{j&22L@K;_Vsdd`59LSMs34L&- zBNiczbTOS${1)A{sH(I%2Dvo;?D+Rtn>fU;8IND(r{eDPwJKRP?;V z(5txw-Q(}=!>j?C!!bGiCN)mBG{}--dyHnf?vlI?M8@ZCtDA&WRpN;IZB5<-8WQYf z^7gdS#aP6MbGNcJwg_oGt-H4f=*~$XP zVOPhLpu70IlT&fC(2Np45IqN%k-oq2CSRO;zNkQ7jt+fy8vSqcPc}>bt7+x0p+OtX zaD8)-xI^gsF#PU%&y|V9_xW)Hvey{jI}uS=${sG}s@(+cW2JhT-tq`)fw#CMbIzj` zR{>b&Uuy=a*|i23yP10&1g#9onmR5klksybzxC$9?D(T@d8U`&pe!9BoCVjwfcKRG zTWA=6>3HrDR!q|}y2;yH;W?Dwrz0m{5Gqvtsq)EC;Cf;Oa#%lVi#c=y0|Cj*J*ni) z(6<%MBy09-s=1tU{&tzP|KZMHgRlD>FE!B6(^8R_=KGYP9rL-UvwqY}JpZ8OCX?yj z=$H_GItZX~k z(I=wt>X~c)Hfphw{{IQF#>x9F>4wc1hJg^?$4M)?S9F|k zhI6oW7a7JXN+qiS#;-xC}ooJ8~Jcp8R_oHWF(nE4W0_qW&d{U>9I&BcAYW z|6yHWGPwYgD%YVKO8v<&Xn3JdAYZ3@r2X}2LmiL?$6bUiKmKJB=5%|2(%Gh7LSwoEv6^&%exs3-r3b*piU*zhlV0kaOvRKKzx{ z;^oAkHfU2#@%`^O38@t((0DRYt8&@UJQ`w<`)jna?ehfKfM4zd0*mw==Oy)qX?cOT z$n}nS(orQ^)Bh6@UD?$723WV27{JhkILNo9@DNY(V=8{W^@inN4e7IR#I4 z9?-#v_x+;L>%3?NN&OAJT`+=lL8uuLAMFFl1)}B1$6Vm;5WBrq79EVYDhQZbw6dAk zF*b1>Cu#-F(lb~A1~Nm!7?uZsbYc?Y3j7Wdi628m2h zzs&R^xOd}0?mFns?1bS8LC6FQ$|s3IvKS=3g)!FzpEdh#7nGd1Y5wO0px`q#ber$F z2uWU_bq7e?BLx;uy-&!Pk)57=u=e1^s$U(S^9f+kA{=etkg$eo(krgDRdG z5$zv6AvhH~wC!@BFbYCNsfjlYw#$`^qmA{ z?q6Wcvax#qp$zk0f&v$mlw=MJ#G8|^t)nJAj}RjC7;N)Orq3?hwX1$%RQgbR6@!ww zg|rciHke~HF6fKmWB{*x6ztmv&;fIFq0NVE<0=|=XJvgRt2R>*HEdXmivS8uUWwaA zGvKm$IM6HyExZ?wmy<^U6_C%!o31Uv$Qhi6JYN`NUyLdSShf?PVme|NSn*v+MYXHp zXdvvT8!V=vFTyN%Y~g6Pem~0Ta0fa}KV!BPA>88Zh)N*X1W5y~SR2rMGiZqdCQrGt z#_)MRJ;a1{h$oYE0#8w7Y;u)aB&5h%Hw=PZixDG7zN#K%={!}PJI6-CjRe%e!4jKM zT40H-Q7KateBM z^P;SMk$5=`v2T)-mXSHQoiu1v7|#EHMtLFE1gci>;tr7idgY`aim&PBsjvV;{{&g6 z(C%iD?3KfC#*S2>G%^or4H#LKqT@@9s?SnL-vLexzSs$-GuRpgE zSac4MAd+C&cR@{o`e6>1b?D#qDKW#ZE*2fD0Q1;?1@_wNpGRgR!NU;b-Vt~E86Sd$ z--Q@@5f>Q}0MDJj#Kz>mlhEpqL3~xP|B{xHsLp4C3`gjJt607 zko^V&G6#x7J?*_I*e5AS9|7{d39wJNNQN_*dzKkizR<{!(96fMc|$CLFY&d+NcV1U zdXe|RpJY_c89%sp1y=`a`n@xFI>;W}!Mhq#0hm}G2^4RMSu$h;n-G8nKZOmtWNvf@ z9&al9&ZpV10lL^4wTc5@h`~?j0}1JEfi~f!c(7jFf;__suhOWJg2HgCQ1(Q%?pXq{ z1Qi5igC)VRpx^1aEF>3kZJ`DJyM4?SY}OYjXP^RKLlZgDD!puI zk+)JFE$5^d99$3QnqmrsGAIuW{W%R5V`@tn^~ zJg#kK%236I&^3BNE=TmOH*k(gsVtZNDRW5Mk!f?lEj&J@S$OY3GQ3_y6^ObyxVgTK zRo*06?9)JerLAM#f)AJ8S;&?2-sF{;4ob|+`%Dc#5LWQu&r#V!!;BDe~7>jS%CV8^sHR$ylQpr)kEK3G-f#I3pAit z7Z$MeYywYu<6UqPX=JIyy4>KQRpIugKEKBcfecHc%m+Q451fvQg$pcOTtQZ{zjL-+ zTz*LlUkc&4Q(%!Db`3}3j00ytcjWc07PvS-<(%MYy9L7YBPnq5dkJQEL5<25EZ5UJ za9`+Q5Fe0rbq%hLP(*Bfv&s4dJ0^L|Gu#{Xh1J}D zH<@%9uY-p!gINtIB1`>6!39Fs z_C*@}8qlvfscBZij7VI1oGdG8l{QoPJ9tfDH=Xa+h5eL&m2=@J%#1AhC?~MUFUQ~CgolF`neL8y3jZ^(1R~Q&H zJ>uHw0V6ySwwnu3<-EK|$@8@Rg=~q<{_%F-kd#P)WICgE;8&Uv2GCZ5w(;6a`mOF! z7Oz&yGh86niFj#ok8uWk3`r1RU(FgyMOdX`c4A1Dyzj(y0dJK3kjP*wgM`a;OLiGG zBy;{~y-64ASsbVrCYQ-IlngWC9w2x~f+KZ2T=WHK$6b;nkA*@ik?}kT!fCkR!f(ZZ z*+90~j-n&`@pzZG#(7Gg)%R!2`Qyc&;J^}8$y>t;i>oC!Q+C;cQ#s;$!py}YBEUBB zGI6y<*#AMh?a?sKB>GO?RAHjM^0`kq0(>H9^X)b z9QKOJv@EFB3(0aAEX>)8GJ-Xj;n_ZS|L9FJaDD$5GoUI(2O0fb+ajUmX&06)6|8tB zbRI^S0sg!;YX9b&5NefcD=5O7bh;7ae8vLpTsOu10kruHli zsaisyYN4P=Q|=*E$&k46L1x}K<2(oYeZ*5!_1*R@78E$V{kWGwrewYfzX?Suh>o|< zAy@hPbGR~0O377F-D145&x^qS)ws?Mc#uV+!}q9d@8}C2qk1fvP6rJ!5PXM1Fsc}L z0ULTV>dk14{ZN;)1S1=Q=~a+tNEGRIllser(wl4PBXCIus;oUbZex3L43hiEz?1#I zDiZ#ynEr~}a5Ll+EC2;(1p1{al+M>Y> z9?jRFLAoc6K?_eM!g)E(#(0CEw0(%A9X1d8G$2HW*N)-+KdRu^yaST-X4hR5(ATSQ z-%d@Fssc(5clT?rSuupG%d-uj-f`zRk5`wbOW)V$oB=Q`7Y#*sr{Tk@ z`eC+sc%>9#S~-N*%IFN{samZre12{KXFFJH8oa)vN&`K+zy>N4(Tin2p`L=b1|@*u zQw`if?C_o?q1*3K$z`|O!V}@WTgY2{kg_b_yUuj5IWQ1$Yap&7cs~a}APYSxGbA}c z*NPXNdY}tgn>9{)FO`5=K+c8e{r>=SgRdimYo`li0|R2c2L44g7y~O0df(jEC#5|n z417P5?{3~Lywu;v^pz%)Twnk_GZ_jY&Uy`Jde6yY4s>o36+0|eTTDE$47ozhjGqS! zS>DAQs8V>d);-spiQrU_0X6G(AgirkS%H7mp~C$MRu(r9ri#oO>vxui)b&dYS!7T5 zbEUR>gw-ZM!o7%3$)k;d@A?Zk(Zdoj@y^{}8Rlir1WM~ER5dE4#&45cMC}HzAr%`^ z#Fm&hn`KNv>!d2Y?^jg=RAXz>ku#tIg z>L~|WKLxh(TbVeA)@k$R7ci%UmGcsX%6tiZ(sEYz(dFs;236(`JPVMMDh?hVY->jV zM}Q}Vv2PK}I0tW9(VITMab7?#DB%Q%6tBl{S_$!6gEG=i6zz2!p}Swd^=;Gz@x&qUQc2R&y! zg&7ik?GGNN{4dpgbx>9NyEot_50XlVgtQ0>qJVTtsDN}y3W79%g0}m%;Xd40h(E4rL?|o01jPQJ z9HBRTe7Bo#q1o+6|LJeBPyb-4NOlaVU~$0E({7obO4A)<`oc-~UKfm_Ec60}z7q|E zEF@Nc;NeA9R@r z??quvcY{KgTmzft8jEpcC14E!a$dqnYt10)mPWOCY7~;;2!__k8W5*RE!=CGGzOq7 zukc>$mC2(p5{&-V-F5bnCuL=*f&>HTn~OSwFta^A9BIOkgTC8^qYY-!WB zhIssd>D;?leSu-gXfX7|y&7Dhh(mq~=sH>65;BTHaqMu>B4J+`&~Hl(!JPUq8pKRW zyc>~Q_7xxbY9g--84Q0OzOFJ{ zQ)PD-^bnjeIu4704pR68%C{rhNFPi#-A8`Enc0nW;V%3C%MHj=?xo^gr?< z(24M>vo?-k0+Y!(qT91Ma+wY9Oc6DRvS@7pZ!6>!RLm+5Jm|3+UB~2;7={-MJ%k~5 ze@EnT?Sq_8>`P9A^GyAb01=8(x~HGi`;$4@i9vzQo|O2I6_ATUSWDc>&+BFXIynf7 zwrLv)k+)5mp!F4;NDjYXjLDC>=XKn_bY?#Qa>!n*xiQY(zDqA}3{rRsYU-H^54LRS zgt%a^T3p65jlw4w*Tqym!ZB;o2nAJ?il|5`Z6;A-M}RBE<~i(=53A;amE6ogt1+JL5%45iJoOCL8haJFyvZ2o}3h3rr2M96K!yBw0LD?W9 zC4WU7B%V>BOM|{J;9(;?weVqXEMC9@WCLAPY+M%*5+*h?yWsMwTko*mCR5Z}GBA@C zH4{jv`Tza;2#=U1thHsd0y4CD*IXV8)ou`|vy!3zB46yH3%K-e-B3vGrEP_X`O|Ba zQ>cYRe@TR4zE}+JN_W&{{rxUU8}dyE&A16GP%O9Z`EnHywS9vx7qH}j3m4+Ud85N~ zfotKf7{^X%SG2lL$iewPMUoHC<2@1|D8Fq_MkmF5l?iE#UcBvt-~zYFVPG>^YzQBe z{98?wWC#JWEJC1Ko;Xa_+JC()HX%yJ0(W6}E?2ki(L!fR;3o-M=PdX&f2jZaiq7;O zl=H#Yw#ofq%`U##!mJCj@Lb5qbnXVGD4u zA5RTQ-xr(*L6Hgq=oZN#o6tU>`A+rr*W_2Br&mG80!nKPDfB}d7-@q~bZ>oO~?0666%~7@3SLX=CmL6V5*#s<%58O*QkA0vcLNJWe z){42m16Uq3;Ir8PZHNcSq{2XfQ32*T)QebgZx3Xh0Sk|8=mg-_NZs6 z2W5$cytEGhMsaqDu7*==K;YhU>T=0>rg&s`d5XAOaksS>3 zfRBTNmF^`7A%`U<`ySo8j9BGI#*_po`g!wxLLrNwyAwU~gf`|IP&o zpkIrm0`cBJ2Ff__d2wfet;>gbt^yX>S&!v0|Cr8EK9(L|K~dXlHi-xO8#S6KD7)_1 zPl)Au2c6sGingyeL$wQ9BR>0P*Ka!N;ClM+3&?IP?q5<=Q z`BgK%4JpaE=zN>oza^BHy$a=_bg#?wzg|I-P%XK)`*HM%mi;q@7-O zN0Y4&VpUH`hw`Y;JZN-AqkIbGOZ{1h-Pn9lBi{A3yvAd*q)*>)JaRHbeCS@H1X}ZS z60j~WIuzCXY@)lJEo!g~Sg00bc$VW!;J8+N4>o*bA;j(vm8ywiMtAp?d>^Bv8(_P6 zpy7)K<)IREXj9;biiQq>h$`p!IYR?LJ6Mr-4xY*_(MM$L6(f}D>?0(>ej-@BJpPRV zWP2!Uw{hpWgu!yPwDkJ2`LlxE_q0!mDYwVDYl+;=MhMPu0L*2B>uebq0gnZ!GIhJx zfJP-7rP&-Lv9cZ)xI`eT^)<7^q>-%iZuZkL}jWucKdh0|hlxWp>dhQah} z->8*-?XW(aVu2)un1s|t=n-x{ra*R}-$UITj7Rzf*DT5Z6pecz5f;D6nfUj$P86VY z9=6gC{(LUeO>`WYmhivF@qpI@)|7g(e|X1Jz{Zi%Dp~w}J-|}CGyCcjd+Bd+#4ZJz z{ym@IE7pSU*FLAt7M(_+5Vk1e(>iy|`@wurKi@d_5)SVdNI3S&8WdW@kj{xO^S}yW z@#r41fFlZ6^Cn&SbHqBB5C6aT|4f$&5#9>?COaVS4lJ;GkstS+>7?hgf5$dK0>i2X z;#|X?2F3ZU6Ao2->!W4O{d@Lf7+CoR1QVzLaIAm}_HKL(-tEVMOiGH-L3~Sb9f<)= z07!doiN?7NgVpn2bq#bq@!%v1Yx@@a-(pgSsyqe%HbgH&}2pkwa}SrgwwGaV}p=VKaJcnV2?oP#?-c#Ie{uWE&vm9}2bB~=IqZABq5m<^D1yW#9_#1XuT@z62uT%bME zN2|be3dhX{i;DI9|5ZDPJi;Ow=O%F=krPXnKH5*?YJH5 z7XeFx?U37r;S(6nUjvr-dVuoA36|SFtTz@_Q--PmZ^mOh^R>7Ree}l|7D6~R>3kwMwI`tZQ-~s zkgS?OAkm!L0Q375WNjDKx;4pTIU@dF7~7F_7yb#3PrJJDO4{#;}kdGab4IY8~l)&Z$ zL}GmuoNIG+i~<21=dXG3He{J^?MqI9xa@U3mCJ{^ ztq=#U>ApkiJzU};R9B8U>V;VRO z@I!t9BacZ51PRU$%k_vo@b}X=D!h<^0)mL_+eEe6zcIVz-!rfRlem8Zl?TrXL_LP8 zG+pLPkOAqktSVI|*ypu9fA{5^zx#5@`65-aEjy4sj^zXsxPI@Id($05)rSEV6+zc8 z)cFpgNMr$YFdu&FA}q2VVHI#sm}$Rt_0}_esUc+42bzh6tR&yFBZV)|BqXl z@&~@2w{Vus9ya})u^82xzbH+R-bmFb7cLc}Z+ZUW{kvTJ>T&UjRJSdUab+{}a6%na z(`C-W9h5HCxDa2K|IKN7)w9XHYGd7K$@}2bf$!+(>ejOFW9LrK&W$G<-kt89&Ykq> zB#sih%Yj@g@B{oaJ;HZzw8R6Nmyhj{dDtiQ{Z|bvEOPE18>8z6W*a5%HTh>UdQl)x zE?o6K<$wt4AA-HB%-{Gm^eTV5$yt|uar{sAsJAfM@AFn6*cn)FWCHBG z@GVel-iE-{>{{A@j9*_KRsq-leewjkyL3^m=UOb^~EIZcg zM&+?ennm{z%Ck?$dnaAcuf~KObHVJB%r~RpTookNv0haH5H(a664D+Ii1aIFj2{?q*Il8i^H|V`2T@<^=FiF1g+<4r(rk^Fg089hWw+VD z>FR3K`qRtDj)ZK?hf~6wb=cII>C7Vs~?tr(I|v4 zQ*2(^PSX-?JJf5E5#wmA)#cuLbZF^gO2g#N9?|>BEpUj^@e-)^;Af`lDjoj5eiUgL zq?U~(pHY7S{Th>`9AmhL#p%Nd2}j8;cQ+vln`rR}P>Eb?x=dky5RfwIWL1^8tjBfl>ctfW^KkXw4r%E}7jDzU3Z zAazp=ItI_z4xvIwY5|+u2$C!EcbH*Nl=v%^{-QnC*3-`v0PD$uGUe2FO|S(jS<9I; zjPRY=w1o+%2pbWSxf{YPEHY0^0EEfyvI3h`pO$U%(e<4VEK!zv>0gB2Qv(ar8h@Gv znzh(ZhMqiMovTKxYy`bUTiTZLq6RJ07;tdKW;oW5Dea;Nbg(Arg!4V0?fKveK^afI ze)uY~3cx-i#-JzM@ zHFq6D6^~Nx9vk_b1{v&t@;EGe*+6g>LkrOz<9^WyrSi5G!g+3=1$BDu}9o zmhw96*N^>AAequ!K^UdJebq&MpJCgRj^bNWlx14p%XRo_6%Ho;xLBjFE8{D-Q>V=n z12$h~U0+lEfjr<`vyefd0I?G#ADcl=!vg@2Hvc|=LG=Qgz@^*(T9K1?No_?CTgPR4 zW9b~2Itq{wbNf{LD4WIiA&jB-y5^l8(*f_-#wnBD8LM^SWz0~PIsh%oA z8LhlegHm~}?xeH!Kk0?gg{f4}#Tb|e^WG4%k3l@_4QLBvcnI6;)8m$5rfI$S$slvp z4AFTI&Y@x4TjI30{;BQh!R{@wpmW&MeUKwzZNDlQMF{+G?Uq}{QC&1u?tWMCRdYUj zSymMfNT=@ajC4)=O?{WUeN;!aZ@4_n(S5FQ+6r1JWwzS>Fahib6sa4*61!3m%i9W< z-l&csAFyd*J((n$@x9H-sKJ1yFYG09M74xJ?xMbMXY{~uIWhngaLLkK?;TbL3kk-D z&o(=-fphk@&`%XZZnkZv>W1H65~2wK9jx`rJwbI&{7S^ZMIoE|R}_?GAu$GmW} zy$5t}$q`Kt)wdDLMUdc#=b&`e^^?wX(14Ox`q|YDoqIpIQPj*pd6mQXYz6S-NWgJ) z`*F>#oX6U1db27k@5l5&$xD|o%jwcL_Q61-OkbByq%Ky7Q)V};H%G68-6yHqo$$4c z!k$j*8J2C9WlzQmM8hh=YK%LrSsDW}uU2UPT!@HlkFx7*k{DmGfccFFf#pbjIQ7K2 zGY0&gS742pf*9p!SZ@f$-`j;|Uwd1RKTebB{({K4FrDCV zNN$p*i%BU0dwOBsjIYWsk1b)iC$S{(ULm>kkPhs5Ww7S=k>U|y4wc@ewIDS z7)tmRK$*VbNyBCSx>;2tb~mXd9a4E0o`T?$MCOh&M5k0x*uo`mYHM%rcLQB?&nlR= zzHjlA4`s~2GvEFQHPq`K0QjfOL1i1T1~sKq{kj*OcX;*kT z`IXUD7+VTF(#as3X{NZ4!wV_|&CdE@(kPm0&>WsXU*|+V9E6>Rr^5?okiiMKJc{*i z<~~OW^g@EfwT$8Wddl6lHmKh*(Y1tdO@J_mq%b`vx>eIQhs&B zc_&)cy#2$6?p>xEXE@b8Jl-1GLsm_2^6_%*A0*dR^z|$*EdcLtj$tURK^(ybO#M# zk`#*J{REqs*w8l3GT`)GVtra2pTu2%F=@*9U`PK7(mIssHzNNFwWtId*l! z?#;sMul?r{+wg++i+7amiqKIVvoS}~HsY6DWSiD^U|+AfzQ|uu&C5#!hD(J^y8d#c z6J_r|?-XjhucN8PfZJZYu5aXi{StWoJ}O~d!QMBy7eOwXbBqZ@&1CI~GnWek2uV5$ z{p#-64? zJF8o6YCYF_b(?hkQ}t(6qWCFRByzTK`S`~9me>Ihhf~0`3@D!%_-Yo)j+xd7F|x(1 zHjoKLyv0q?CeXr@#8r-<33g9gifBs&76t(91$78C4|TJKz+la6MRCsJFa<_(K)FH z<6J=2UIbG}y-8r^9SK8_0$4lz7G3+{*`p;cjR<8>$naf-f z`vfTR`A>8(oAZ#2i{`&tBHTSX^j=`g_rz>>6{ID31h}uL29OLxCi?T$?)1f(>7<2j z_rAh~7Q`jj8%d;Y2r?@6Oqy?ksF#IQY%N-Jb1aa&J2F%_Bke$5`SJzFWUTpW_s+ z)1Atzw*>nBq&r;_2VOViuy3&loziWY69^EMt{hnxka*ok_z(3rS)7F1fe%SCRfH~h z;2yyhLeaYHt)$T%D<3Q+kT&%N|=nd$J4bhu>3H1 zS1s4Ap<3r>gUROxvf3f$9L=$gaN+yk>693JXn4Q9#T}pzq4#Gij;MHQ9d)i^49B*H z#*vwH{Hi1*(sVeK8*q1?b_mV54doi(1w4^zn^KQYH zG+(;xgYdW5zj1|v+vxolS8m&S0-mJWN+Nvvmj&yQ`#+`jpi-VKs9aHMr_-)@dh1&0 zxJT1sp{O!3Hm9qXD%umrdEm1+ttowJ_oJZzb1Areo*EXn4O0wW9_o7CJX>pO1gX3A zTdh1`885bT3H!B_GUpon&kw$nCw`8YdNc`t@X$3j!VxksEw$qj+)Sp8hJi-7u7^tc z+RV3KRZS$RNE2Vfq{H}rfpFqx!?Qr0HiACU98u$SKj*aLn`ca(KZuneWbFuHa4X_x zm>%n~4%Q_}RvK$;WEgNr2>gEZovawxSouETdF=D1*R8I@W(~chnD%nZsaf25M-GlE zx2b<{IBI~rn)SFH`Q9}DQN`;IX>a^4F6&=c5fIG~eH z(Misw@eu_qZgd2fVG%J99jO`waxO1aFn;TthVY2>vR>nT&dsq}jgxln5rPa|&X=)9 zKKzREG}Z`C`M!L;$%3DTelpM;myydR{zx0C=;C;ne@&uahthI$%vf~r+^nQD#mCDw zK9x0_g=j~+fNys=@xQS8j6>LN?VFj@xrq19p+ehJ&Fg}}b9=meqmi+{j*m5U z>4nkOE#XkM===0oPAdeQ6(=uuv`pe0=Xf|sLVI3TTW;(enVMRZ-{|^`t4E&vg)OJ> z<6H^*g?+~af_2t0XYje1eRlnslI`q#27R1lmfVK2PV_K#QAHij_7qZBTK{mXG*wpW zkImt{nJ;b|dhGs$lz{c)NpJD77G?kyq9xTRYxD2wWDo~iyj*Jjz~KB#Ck4Z{G{Sc- z@1OVWjaOmrw9p{_9mATZf;H{1BOgDu8GeJV=m#N5+j3>sgKvI$Dtu*1l5$V=)*FWq z=pmU;P>Vl%pU$q^_l;6WjQV}7nOXIm=Ly#jF&B!Er!c+lw$uCt2Gs7u>sF29{D0$Z ze^RDpL_MW7L+6J+uaztE%CkA(IbG)<5~p{$}ke(8BA}>YDAGe;CgO6*z4l zi}1hcJ*Q>hwfXZf+P?w2n}n)8&0XPqe;7|DNQdotc2$!v^KklRC6j=OdNYylZ`yAb zvA@CIb<|QlyDF8~Q_1326KNja@(zDRv6>tu@i#1OBrVjH{~$Dw4z-N(59DOFVh=XD zse#4jh*^Zjpn)y5HXhh1ZpinzGDewjCxh^U?H3K?)oU~R%-~f=Gb_7m6D5V4XeP&~ zVa)y4Gnz#KOzAT8@RSLp7TsU{DZ$tumW!+MsrbXDM`^QC7TiB@X0*Sq3Ak!u819 zWIQa2Bqy!5U_B5N8lzVC(N0yV}`& z2v`K{v_KuEHu9D3xaTR{5pM3t777B%TJUNFHs!8{T+ zI{pOzq)5m`Z_EKSQr}=ne?-BiqyS)WHjJi_b601hSPjBZj{-=!LZN_zVDfZ6fM+z$ zx$}~I+R*Y4NM|l>WMo%abetG2u7JRf+rOGdsS2l;&>jOFqp$&vQ?GM!vm>9{_T4Wg z^Ph{4U~!oQq?n)5GiCW4&dATJ=lP^tkkkcx=-2@I~YXqb9;bZoeAbhb(2<>*t>(m zjSQpWb9i>Zk5Ov3+zy*tY9XsAaE5(eWA2pzbw41vs&88Hi>;d&vLi5zJF zt3)Kld5uR!siHfW$%Mzes)KPR#eOk+<(W9u^W!?d(|nD>P;EA3B;~;1n#7(kF?2pk(8!JTcvZ4^ z6s%0j!q{G)JB22Rok0Vj$)Fkr7|8`V?cPew0>fdH3H4lKmw1N9?d6u=gr6GXMxU^z z&Hr

n2L|QNjuH0gY?``98dz!+rP4sz-~>n0h2UJ|yt@5G3@|T*xn29kD-R9|CCI9twHC4;H|uCd8~Rv$ z1i0gpqA{5ZL|$$I!iAh?9>Ym}j=j0pSuQTv4^>>h*3HW|X$1PVY;dwU0R*Q6?qQ6D zdxA}U_R*+2SIvxDYn4!7UZqVJ%iiTHaQnecp?sd#B-4k9_pvqzT46?A8mgI2vR>8V00Sy@-%Q0`L7Ra6 z((r}bZ1IWwl|DS#t(R%aLo=YMzMEson@yh2VC%I*7y1?M$%`p{$|4Mf<<@lJMiO$xC-czcN^$OG@a?3J(6b$buiqzR`N zdk7_ik1UJ>v>A_pU-wg@ZXTCDi@xnES-|_0k!>|O__lavyp=AG0Z>V+%sG2jyXC%R z@%l%&ts_b=x1`;KCv|V{LgiIBfde;k9AJRC8NqQ>`w;jl3jtVX5vsrP-BpNV+zh`c z3&mvnKg4F71ySRygix%BRf7K3OL#IB_t9Y5Npx}p03HqiG_->usp~%Bdy{*SJ(i`V zd|`5KqFt>J5;vc&)A;6A7sJen%RyQh+4l88JB=Y;09dSm#au*cY#HxanWfdI7uqVT z)v!czU%d}wdKlabCsrkIgUd)hA7Gv-GR zoj8S{4z4PdYf3tc4&3Ut`Xi;L!mjMtZojd6Md>lvr{j7vw~)9lQfNXsRd43;>{xPe zn0UWi{7|Pi_*@-;PnNxXeI0PR?EuAE!5%p{aWOIJEuSxa1fuzQ;4pX|uK`xtA#t+e z$tf%>95+Q-DUE|Im8u$`d{bubv(QiX^NkWeyU@NWe}%3y3@Ud2EtAND4#Cm(>G0qk z2b_6GoZA+uRV3+*xXU1_w=J;b5K?j;0?E|D^e85`olqdJ`*Rnr2Tgw;Y65~fr-k={zlyfOERWx(D8w-jvNrt5Zvf@F-*aRyT&H&wM4sOv8lGzq$i7_aN|&5s zMhmxh^_PglegVeNKCSZ$kn_3blSB-R>#&UVv`A-_l3PNn^<$l6MqKDYUGgZ`m7xQ` zOkqiN>E9w9`Pn7DWAl0^rnlUk^x15Txac;BmD0j0prYX^J83azNASWTaKV`5Srt5` zb74o4UCT3q9}BnQ&Rh{ap^7rAz9z+=u$5>3_f$}T34aRks0DIq;19X9bq`AmyVmsx-HtJ33JiH40FuiB z1<_X4y*Fp@surD_qlyljW##kOZj)w-lW9_cS-^rh0SNkI@t)w2L%m|Bu9IkL&dyXW t{`B9d77A&U7VZb{!qJ#((6j~HnNeK-%)Zb~CNyMLr_wfZ6Ib^eZ5npA9(WRnkv0@}@nx)!0 zf;6vg_@mLt5TeTB#2`4yzqU$=OGYp4&il*s(vl-A8W)XAhMxBQR#^XS*Y~7vrzzd4 z(rM_O@zCBorxj;c-pDVr1Wbgqs26`<2xQjoykxnG9fk7u&zDY|u_rRh~JfX_t#|MQt`Kc<G$S()49>g@OyT0UJI2b91^yU{%mO~_LA@D+atkfKeEk+0*|Ngl3r)sM6y?)go{-yGX8YgKBFP2-_ z2zgq|Qs%AtB|4S&di={1H6IH!_J04!E_5skC-eLKima-1 zjFnr*=Weu9K9y%478= z{WzJCo5a_gWBKico}t?o2MZ}rb96qK$ua2^3I!E#iMefviyqDqy)&pOqm7``EYOJI zG8VP|QI`9Ayh@pjtK#v9mRQ5=oAZ;rm>T6bq8^E6WS762tUR_@`AGPrjxp!Ft!n*z z)%aZP+)B`CLF1ggT6_AuSvaZx*Oxqe!Og~u)tRx}U{+c`@6#i9Ny*d3lP&qC#ax-_ zdsKa~%#jYBFs}5$cj}ZQE5bXD3*E`>H4PrSYKyrF$ph*I8mtv-&;P(q2;1&U6`W;N z$q+B9=dp~S3NdyMi4-F&WV2K4K&CERhyIG?1J ztOi@cei30-PQTR~{#2im?6nq!fZgvZj-O<&`5(WHuZ*2a#uT;mic) z$D3bhC>IYuMyyXYl`M>!_?^KUyxu2nvnMZ!m?>5}j_1C{+E8vw7iyOzNWCIuWgqVC z_+>Ire(O=x$Y$Q(nwI=f_jw_toDj@C*~x5^`K}IxGEcihZ^>{?ulFW9Cz??RaH7YiU^o z)!ny81Jf-5maD_>ZM;d8u~Y9j+%x&QWKryNZ~W*JzLWDmqt6k2D z`e%P$ZyhMSz9sttkCKV2>7;Corl8StcSW=R^e8vcNUfgJpqlpkTSfIJw2^lAJ97Gh zRHE>PMRe#s2fwQomqZ)Vrr9@S$_X?1JG}=AX+oMWG1spbqrZXuR^WBKzn-XVN~N>>#O9-C*BSn z&bFJ|KU{o6c}yehHZ7aTnMT3$srA9pcApqt7yd9W$L%x|mq*FJOLGFEF)bH6qVLD! zb{!7n%IjV4YxFu6E~SfB=1y>K^gIl6DluxXnfv(U2d{Df2A=TKhTQ~H-Q;9EDxtwU z`;!gNF+TEIc3wLsvXRB?YJMWyw9$(D;I6omlaS9@J+u4P+=?5_X-}}Q$n!moW7CH0 zGi>X}C6^T`&D+CxEcD7OUM`1x$-XH+xErf1x+(M`TxEAYTRM`0w?%ET+97)}QtV7+ zetoQ>%W1xogiE*abWeZwL?Qr#{L8OT#AKf}jp@z5%XxNj@Ap-o<*6!o9(jL_VI;wH z+t?6$OEhY&e0^-I+3yne%6$jjftEAnQ5WNHyH49Ne&!Na`Zf-57UaB$9+XkzpFKU} zif7Z5+qmm>RL-GOD*kM3f;C0psTL`5Z|kE$V~?GF>F}SWg9I+&hYa1?H%zHWvDA8A z^P_L83Q1JLeMxD#box@aoe_VA}MGLJBnvje{i<_=DefQb{x@S5^~?|Wb?~VwpLO}Uku|b zQu=9LQH7rE-+n8rB^hmT1ls2>m8ZU(QwmG#Jv)h94y-0?3tDpbaHH7pAr3v3GH@wd zg*tW_+NZ9KE@?k3ab3*-@%4}L62 zyh&ix@FEZR2D@7>Pbn>otjNS9Xqx||`3l!noBpgI($DK~wO)K1+Qw5(rD;jLzgEOH zmybQfFDG+{Y$`5zC2|?>;n02ji2qrWxo!=&ao1_J(#Lj2i{kbmrBlD#G8*x%2UQ`% z?ZP+DtdS8`4l5_eVYW=}Vb`zDc=iuUpHtJ*SUF)oe>EpwTxTS|rivNcy}OmB{Uw|- zL4mgG&Q*D*ngtJ&s2m~;ZNIId#iWv<f~O%-RCQ5eeQM~V&2@q8bUU1+j+v=u<~EE-7+j3klMt zanB4J&yH4Cm`=O-`tqd}c0z=Um8fMHgwwOmRX07PCnQ~jJP%3Go z`thjIVLF<=ewi&P&?9-92mf}$P5p<$S>{)$ahQy~A{sZ%$A#@j8=hCqcn>c!J;PqL zZ}vr9QDvIrH*3Q(mqRJ}Flas79`2!elVJLLp*Gjy?~i?hJ1^2~ZK*pDTioJU+wP-R zER99oHZpJ`B*jVnt5weNC~vc9?XC=|D`e<&d&<IV7TaE1K9|P_pjO0PE>Y?Uw)x zs^Xn=@1Fl%N~0@j+y|*V<}W6RxywJ|laph9FdOl@5t3!#fD>Qd)AJ-<`Llk^j-tav zwL?~Xq;5@nm0LAtv`O;*Qvdy*gBcl{d)$Uvp4$rDho#S=RPrE%cP_s1;S8~e>{49T zf7^>mO1$tOGs-l`EF=57Q=dql_H0Gq=eOU*&NlJ+F>&UfM$`^}A*%CS$_ySX$4H-- z^1n-@MMp?Cv@5)WFLsjgR&ts#Nz|)WZ6Tj8%XKqMIK~hzKPSTrN@9BHi0i7BI;Yr{ z#_fe7GnV%(L>JU}xWX5sTBPtDdEzrMpZd*(aeWs*)cJ$&LJpUGt>Z=u?oJz#GS=I1 z?$S@^-H>jwB1~0cj`2lKTmEF&+Uk`h6S0PH@F6QU^-Z!ukaF&!N>ZK>* zx|+f~*S_69y1p)G{Tu&*IsUr92AZb96Q{f^4YJwHRvVC#Ke?fpeM?arq${t-esrr%Wu!y^47DBD{RMa(^f&T<-W0omiB8r*#G6)mA6XIev&zhOw{Kv z+HN1Ep@Nncy7t= zJ}5leU1etIk~9pcZNagbYHYCRN>HrX-PB^Qo^`&e!@aKEenN}(!o)9u(@?Kr7pK^3 z&>D($qo+NgCC!g8=fdl0aktRLBy}(;g^8zld2;=%;-XCyo6}w)3aRa$xk+q16l}Xm zJB_I&@|L@sVCGo`|5hMJshsL0eWdU&5{VMT_BQjN7kQsKJq0!OH8X-4wuyt2!|m{9p8Ds>FA*}$A_zC*(vtVq zvg2O{;sxL9e7LvK?=LR#*uJ5q0k`lq@(`-Ohoq@rZpRHwmZ9}Sa5uf|P!r~KIz4jP zJ-E6+$}<(}sxuozY#+cEje=brjIxfqSHXYN`dEjT`5(|+8jq4)XZ!hUw~Q|W`PVFp zrZKSh!9wXDkZi(^%4V(az{~gV@sTL;WP%%`fNubfapN@D!Qe1urQY)4Oc#HIb6 zKH36o6_~CCQJ-bPV!n$5qv5f7iw@pGlmN zYN)5H^I~o#dEsaoJCRPMze{m-2xdCF)?HWd;%FJ<(O=(7|B(H=@L8`>;+-`FxY(sH zj{XWq)7R{>+`YIA4di94ELhH8Y!kX(fOWEOL>piNrB(*t(SOfV zPQ=8gcH6WY%#S|eK25vpI(SBA?5S$|M&$YMYQ4%6&-2s$GWYET6^kC>)2r9`ZQ^Xk z;oIb0fLY{UTqe(P+x)dv)pV+etj7*&V(RFUn?Lr67LFa)ET(^aPjPnU3dLk`XN}vY zS?WBn1mjBq3ujApqRvYy@rg~1ubiOpV-$3pi$ZRTh2FGP&EoZjp4E(S%9(nrF_y~5 zWpc|oukWEYou3`^J{je132x5J>38=zKXv_4q+L5Zai9Hw+7Ck+}vL3lzY^u z$-9A~sKl^NODSF0y4>p?>gZ;Z{s%LuuIJy7n96m1GX9qJfZI#!wYobSkTg4?D4Ux) zKdU;T67|%y#W(@< zMhQ%L@|hGWrrcKSO%TLHpS_R5hHc#%Wgc7fat_dcVZ1{ z9HsEq&JJert;Z|hUl(#_SqpP}F^g^T4P9!u>Z$SW`E39fl3h=}oUbvz@ig0!*tx%5 zV%e1-Cy~gZtLCw@G{aa@v;ONf0BtRIpW*lV%dfX)hmvxhk%IqV%2zBOHFZvh3u1x5P&lD2D*T3e!(7 z2C}4ZYh2gUx2_{xq7|7p)2)d*u3FAsQ{0Xqj^inVBtLCsZTKjW`WniT8re-1OA>#{m*Q_Nym>#X+ip`za_ zg*>S3Wsr!SX`z>=3f5^`?1(#2Rwl#e{Hz_1@KiRh;rYH?-LpS&g~X(|b_ei%&F{=d z*Bio=eI$(UjU!s?1^=I^DLj{!ho%G?<4CLlR<$rF&`rnQ{XRTbk#eE zNJoJ!Yf!?g^zkqxZlQei(d&3m7SJT(JUIUxNWD|G58~TZ8>Q3dL?d+Q+fBIgbwEuN z{JgrZQmA$FN2%%kREEB^yUey@<*NH1zQ!`k|0uJZt#9{4TT_-<5;#*Ubr~F?7Ic&x zt9n|pW=+obkl5fHur7Oz&0v0}GSSrYx8q02sI8gou&R34X6YkrH|}8J>}Lj3mkIUU zdEukRqRZksae^dsD#B~#R)z|3&pvcxZ>QX$=^~(ifMu|ewC;I!xMY=1+(hBGy3+;iDDwEfE*VWP0wTs7oP_LsEJ)F0xFoQY2qc)gk?bI) zx;U2GcEjRf*AK6j7D{Bs%iZzP}j`B6;Yoyd96>1D6dV_ zh(KW2GpM$2VZS?d>??Y3yxznfVUnkwacR?Nq%Ld+ws2k)-xWz$OePdI`-w?Y$-aea*(?Dk$ zgP&RFR%tOQr$K(v!)+d&)dhqr8R5jVLN9X$y$yu$w}GDrF^Wn6!9K^vFQmX;dzcfr zN2b%appkhGz2fDmeZ|TSgy#ZHoe>%P$HjUTa=JqHt|{Lkn!C;uAi3JOS0kR?c&9{@ zc#_XPeMjX&C;W9q59+@$S^XamIiJ0!uce?SPn*7Ze{0ufX?6pRZkoht@fGRKP?G-l zeV-g#1F<`Wf`cMxIP`UT?LX0nQ{C0#mQGCy%rm`ef*VOV@lz;MX@dnLW#U_(zh!jX zBI6$?g}1L~btS4pukHj`W&e>$I3c$utBP-5A{24!7A+*gZ5ZFwNKp)6=iF`7=Q zs6|*w$l+IDritm*gUWF22R?W^Jpo?V7=d{v-ch|g$c+~}8kk4*Ew5+0S4bH#x8j|@ zkFIw#%t&#pn^Ak$y2(85mOoQQ4nDOoiyH=yO6wAZmcYWFUvf#tlC7y*m^7YOH=M(S zdstPNuMihKY)n{{RTdZX&V9<8wC}Z;<~q2&d-OrDjrkE{{y2Z;DdV>cUe1!J%AHG6 zD67@D-6bjN<@?H)%K(oQ6ok2(`d|kX{TZ(+bl3LzLiAA?zsz&{+o|Nb^b^#Dt{DQq zJkDY_3IlaHYyICvH7XZ2;u+h<HcHo!>o%*%p zDAG>OPn*t}12FLufQHlM9ikTtYqaW1w{_>le}IWP!ENeJ*GaKvsYAS^Um9cVwW~SL zc~iSMHFQE61rP19Gmfi|Uzkjz>VzJ(h+AOdS4C`fl?~sPmHR4bPg}*QCESu)#qEh> z(zU)B5qO2Kqe!n;nZLBEBGA|yR9Ak7qJDR)ypw)ZXhni1j2y1-6F|$uyKP=q7^bf= zhP%JHasNffQh&ClGlPM6BenpaF!RgaP%E5SIeYz<*M3{U`s4dIEkion^QoK?+$5Pq zWzRpgEmEXYdY~maeJV#}J?@{hC%Z2}kAZ5Z=HwrYE~_t-`I~FOan}QsDNW-SQds1k z$HeJ1QmtKH6LIB*l>2DiKaQB(9$dw4GHF307qCmEh_Wuk$H9tJT%KF!|InDaTeGSA z>wR#NSi2`LhR2K>*ZoqLz~}&>wb2jsTJhY?{nXhym^GiG5!Z?CV7LlQGQ4oVFWgl7 zeBX}C#7!7AvSRwjL*X;TW0qos00Wxh#*^HVp>WD+{ih9#d{(`ekAf~OY?ZU~eS0+I z$vJXg86}ilpa+n4fuLT!5veRDRi<$=5jBtd{!iB-&Qlo;mg847PZ}cpSSF>zYpfB( z$={*W%`_rX7N2`c1ByV3~cFgDbmB`5m!wn>I?3xoz5HESR+n>rQ(N#_f8C z%TB%5WIy!;C8AE#5U#IE*^?RQy7Be3y(9{F8`givKrN&Z*qZF4FTp89X=L>3hzte` z3%r`Bt5KC6yEeOFMdKXl8I&yxM48`qQW-f^-0qv)pk2Bu2XijDL!XJ?QYIdCZDKMc z|8pu;5qCBEW){JA7yZ2rD`z8@~u72eSVh?JQ^S-FNW6v9RkE|H$mzm1iE~6;%L-$T!@!atMJo z9-QAc4}M1~Q}LL5_QL0~^mhn2$CT_|ZHy4V^R|O_`kFjN^>lh5I%;n4Bk2j}OdV5f zbB*x*yYYD~i*=<}iy=^@pS#$5S=KIO~Y#n}dteXV79fSQl z3=iQ?`$Bfi=_py|yO`J}E;HfA)tkShXiMcB)CaHy=9of64Q9^G?KWKnJ1kIA2W5%XXmk^#M-$omte z5x4)|Ck^yCUh2ZO$3dHqDBs?(KmTTDmZl9?zFRMV@w}D>LoZ8gEdH0e^bVy$wlC_L zwEcS97m?Bk;y2Ip4Y&B-1QN)+V6Gx^WQ|it<5HAyD6VDF8@)m!Y9RG%w(y}sv^4GL z9`0U!2mRF=!dYxg1@cEDjm6mM8Gf#)=~#C~wX(JqyJJqjyiU*NuzsxS9(Ms6uc2*_ zPV;DX-%lYjyB)PMQu1`N#|4wfelWE>Kv4A-5*&tHLvz!=My)C3doz9Ng;cnpjqoxW zx~(^nZktADuD_aWUHOczEg`Mn5+e#(*_jG1n!{$oU15m;6g^3|;MwR8OSzPjuT^D( z6Xo^pm;dVp5RgY=UcV?Jaf2@QlKL#~8(d-NfpI8{Irl>olP?f%lz1y(TIN)$5@1V^ zs=i+o@CR2|jUHZrSaCFLW^u+e`)Bnr|L_g>fPWCmaZ48bhmT-L1svP@V^i+$*HH{-NIYg(~pIzBbm3@ zf3;GCUE{gue@bI@x2`l`RAc^Nhc6{GEdNQ2JE0pU`lyU7QV^jLdpek}niIkSB*Uv~ zx|gtP&A(hZ+;2LkTC#6jk8JZMF<#=g8I1DSUC}in{qyIE?jh}!+RQfyzm}~ylcQuy z3{_9=mIP-8R)Vnm#YPPjB%xd}M|5YK8?T1jymy;!QvUgqI(pXS9i!vF)|l=zux95o zoc?$6np@z~sV43ahhMz>2`_*CJS!5pU}gefCGJ)`uuxpQtb>?E_qKX`9RS6+XC1VGsk?&pCwQGy5^1kLWep_wKryJOVkhx`x4Qg^gm@l*H zqjph*vT+Wiw1G-neSUEs-US@wcqMt@eqgYfc=l(E)%BLaz)=+(A0>AwSJ1Q-p-CNA z*o1h@4^_M4TnB-lneAw4X4x+uvdd(rAm_%{oq)2x)r!m20gz`NfYCsWla+3@eP$C! z!=7Q?vzsRADn}icJVZ1x0TPpqO%aqKl0ox$-++gSMhE??A3 z3}9?fYxHzSIwb%#OCiacNa3vzS|0|Te(*S4iOc=q1GWLw?Lg1P-Zb^$qm}?n%ejtd zw9jc8%b+F`vaRG~^*O?R6sKk6VZieXruL8lx_e&J4fv<}%0Hvm#JuYrDLw{(M1eZu$ zh1jESw2Z4zTC+geC}2fWxr-FsVwn^r@i|(xxN+%_5v~dATTH}OvG3u}U!Py$5m>4XjB){C zyfcWJh=?B04=;7yfFpVG3sa>2E~olE>&m4kq!MThS6@MF1_RKKI@c1Rhw+Or*BU5e znfHSkypDD{=dY068IHax9Bl4pBxJA*?9V+ZD$X&EV4#u{;YQ{nu%kp=tq3=%1Sf}l zy0CkW<9w%DiIE7nkU~Pw+C4V`H*1QMEz}fhy)P&ZGE2K<9n(A%v9vW*s9l)<hCI^vOPT}HI76rue!V84U;u|7N8M*W!eAm{F1+B-1D_c3+L5&S-Sf2%?! z{l=(yB)0$8tTaC2UEtrc(`O>~YF`vraa+OinqXBwkcCaE)-c9B0I$*XLo_`J z`qsNrp~Q9%T52I&2KvE@UZedOvP5g5_026$@M3&OE;k>W?>wJY9G!}!r$a7Eyo~?| z+-j!m<{2h<&+ZpTBtk|Ug7z>&H3O;`B0u9)e1;Q-TMEZrp|l|-eu?NVK@FEYWaYz- z61PCozrXjgcWy9zt9N&r75RCC=_Bn+%VQOVVUz-#MllAp z&a6g@%ySk_s@f` z&hIRLe{n4spGGTT6)5j8uW$Sifg~6f23nK74y@~=bgiTfVR$XQp0ixt9*x`mMN>AW()?{AG zm@KJRod7#bJ3c(>r;9fRf<*HSLJrv=eX(%W(OT0NIF!Xs%N0`Z@UReMd;_ltBa+e5s&0G)v{c#S z5za#(!Ym-I^7@>4)_`;8){{|{bhlsF{K0tWLT(!-F4sp48$2{Qjp|t-Q^f)M*$J93 zViCc1X~dw>Q#ajfPuKcqZH3;OyROAMD?{?$N6XB)Dw%Pa2M&|zBF`-#o|ORr>=KTo z5i6d1^8Iai4CW#|z6cgJLtgRS#J?+Phg?Y}>h`US`YY7J`9-G0xQzQhtlqG`;>$F< zOeLgY(twl_b#t1|jtV!micv3N$P{S2%YjUZWC8t#-V=4U*Si5xo(llPap~I)ua5OC zz%``RO+0TRgn9bvK?b!{72ElCe+&4}F7VX6!@#y?g= zW}ia`@|0>&txZ?uJB7CCB#~5j;npm=byt)IdH_pVQ zkc8lB9Gw)7`^L22rp*dl?TS^3{b5cAXb34BdR6${uJ`o>2)pw%NPt z&lBWd)_ESXcbi#fq4Z3J%MAgveHQWWP!e{Ar<3*V@1SfkL3ip~KnV7XypEKb6~d4) zdDZe6=st7@Y_LUX-O`G8q4R?f#t0$oS{hEei)+uxsEsCVCf_%)I1 z4Gv0*pwnANKj1?3d&pNOOjtkN#0z>F|N547RuP|$g~|cY%W6(dkmf$rDp28SV4#g= zY=HKmZuy=*r0D5XqrbAWnGez@diVM;-$IY;@6j8e%T#oK1ev2ibc0)5sXH=~T0~FC zWra8|wcia5ZHt?$3F=Tcv2?Odye$K5_1RDT`Cqa;VHt!9^eYsD zah7-jw+%8v7KYzJR`2XQI+%^z|#6<0$XgI&=tks5CQEjG=UBSl`V57BztY+mu)Yjik{WuKM}|o9KPrYLokK!H3ixlkg&=#Qm|w zC1#~=C?N0UsB3U9zs&sdShg7-^?kO;Ro2Gq-C&dX}@@?+>zr7t7eEpEy*`yrOro|$B;D=RgK_pwNbSmVN? z-$vw{h1?3Xe*al`KZL>mU;ut?{~}J& zO>!Pg!VqrwZGvGFVu3Pji+?dEAu_FtdL8ATLJm#@_2lqW3hMMus7B@hupJSbyB}yw7OR=pgt z%*tAe{Exngmsxf(f$ulY`ukh*MdUpMv55J{N2^2t0`h<&Ea58rJ*Uyd1%+CWpp#|X zIGs9D8tLc(A{VJ4%L0M%SI>J5(7XdmryOtt+xTW}_KoflJ1@lbAcw_;{oky_-mm#$Eg+I1aSfGz%C& zA=Mn=GOE{MT?5~NELZ^>w!+q5XbNg=WnSl_xh&^AR`34NXg&t<1PCVEyJ`<*(1sBv znyqrT&Zk2w?^f6h$(g*I4HmSHyKCNl*^i%^rKRD@D-Xb=F+wgj5v#s}pa|_=ZzwyMNpRJ5kJk>#A6@B-%5ihbNY4 zn+_F|p!viO@cb+z0kIHaoIO&^p)1Ht&CG`xrR-|JBXa{OnbCdg+@%-H>dH04(TNuj z)!{EhbzB|3nOl0-WhG}~7W6!=?THqxHaQIB)z>-9zF;%j-i$}*RaZS$o2jINSXdcA zw@vHELjNsCmWTvh^x1D6zt$ka_UpSzJm!O*JJ41VKlJ?k%uBb{`Q4L=YE^>fW|XyE z&`%4dKfvzo0z>r>PoZ!Xlpb*t|ADL{xBi8!6xN?!;2~?fLYFBqt&7WlN`6*Zk@a?U zoQ7&m{=dS})z1j-#nAs}F!qu72PH7FPuoa4AA%66sd4*?-uz1wAw)*zWfu3i0AHRW zdY9z^sY&SQID}GQ=yp1*G;MN|1!DkmM;@|&nG%II2aRej9z=}$j)kI`qr*YPr(pa< zq&K0|^wV`t*3YjP(0*&5kLZP9e96{43gS3-Ww`Zt%KP~J?}-}qN%zIHZ^Tz44NhyC zXHdU64gO}{^!=)CPLRP!g^TTqEI`vx7Czr~UYge7u~xlXX-%iZ&9)r!m={L@jlq}f zV>90RTn-6;`dur!{}$8fctK zuh&|GWr;vH?Df!|E&4n21L;QW5l1>8Y zqV$!oH+sI|ArB4VH;AbG*qq25LyL;oa94cbz>IR83p*6W_#q)NQ~F;w+ZlG?Jqi5= z+RshI6O=Y+04-x_d`?u=@|37nvg)_;82k%XiL2p`abErl%#yG7vu$TB4x zK)44MOtB3J@SYHqb^I}5$c?%3g0K(PnHDve-~brfIb;$I{^Ii=mi7}v_!9?c%ZbY& zfpe>!aHRfo?te*IcCd#}%PT@4js`}y+vyt;0YtX zsi(n>@(ueCnLS0kHU0tZLV;Y)(9@y}+$b}ca$_Jtb^)CICH!`KNZzLmgNZY0g(c|h;JGatPYm!~PBSN`Y**b<^ZD(-|z!KKpFP)#1IDPT>$q*78{9q?#B04KY$yLA0=7fX*$@Shf*y^ zhu`IVdhr?^x!Zlc5CE|mfzH(cUiZ`MR6xmdJt-qBInt5TompqdHa5s_a))mri?V4I zu>iNIy6AJdJAzht(jzcMd_CYE(tuKy4=MD98~Or}z=I!PmgNETSpuPALw)_}FMms~ zjcEzsZgR>TRqxDUJ9l60wUX)zExQ!|vg*0WRtLR*EqzNU_!cXbl|Bd%=|zqT2KnH} zJ0B!}{L3%N+(NA`oH3dNs`GX53x3?HB0&-k9TnSqrlhQDZJsi=un`RGsU{W{c7XGf z9WK(H|A!NL>0Vhhc5Gwlg0wvQO+C+OW#mTg8+);9=!VT~`KWYYNW6*~ggD zWDfcEHkoD7W8KWaOYy+jjOnb|RyTv;i2ZBqc}b5u@X7Og((OV^U2MvTDf9wNiy=pH z{}=8-3C>A=AMYTe3YLH$1cOdUq#7vXao390!EA;b$F|^GVo*c(#{1|UlBz(DHnkIk zfVIw#e)bcByet?6bLq z_$Y5W|K`oo(vS659tJEKNJ4Qwwto zsw$Ij$1kW&SIQ0dMz03r>@Gtzc7 zU5#=glcvx;(#kw<>}6*A54F?7PjF4#&^AD_orz%!Dr;C;Q!3E1toau`VFH@EgK~kq zA+9r!pgwSACUH-Y{~ME|9-+A*I9?OoTz+I{ID8=k(q;jml6Kx!bwjQ=t=}Ugfa`hR zRr^oUx&>zpu@l9)i0W5+khrqOMOWbB=n&-5{XNn%7l|n=8JX0pI9DJ1{k^OGa5R1G zA&c*CvfA&gOjZ)C!B>{XfW#5>3FvlIfSfAtV?;1J&N^nRUn_=V2U;x zLbeZ$`7HKQ@svDIYr}&Dkc2Byx^?fNI9k8#Gyushpn&8cbX$VEqhD8cI23VR3Fcx3 zPjXX+_8k_Ig-4o`-QjeWP^yDqwVWaOqtUAlv@_{4i?4J@SP1X|w=vQHY->oqvh&h7 zl79kJd(C~gKvNzZv9?c~f5ZNBIUZupj3k8_XYsGguFN2dQ3{G z*R1@$MA9U}7L@~Q?;O+>NamKD$1D)3s{#o}2ev+K_aA)2#GBmIAKo02L3}WXjfsOkq}UyqJ;#D&>G* zkAYN$-K!kW7OzpHqpI6gngM5KqZF{y(AtG@L4!8uUjjFtT^kvdk*?jZ2{%(T_kGVU#UDUlG99bkY49KyQHyv1N#zx4UvS3@Q?`^rw@f#e> za?sg^xP+oJSX#ucUCe=Ol|@R;$d37v)oLhC#a#uCK_XTXXI}!J{Eu|kJG_CE9&lDz z20~!FH8x+pe5tzKrNt}$jz+{|a5WB8WElYPnzs%BlW@UVHTr)+g%A(MYcJ9wHW-y`mlhOVrj*1Q#-_GyeO>m@du7W7~d#cG;<%{?`;4hK9wjZN6 zgm-&(mUMf?`a zBkq<7zKQWhY%NE?jTS%?HwK5<>U1X=KZv_MF{jcA9KV)WYT)YTBQlOhIbUde40=>& zg@Jwa8%3zKPkt7GTWgc?>P;MqU4Uh-%`2a7z9*v2zQ<-%U#(W|Ax69oNo)7*2rt;G z9k&-0&5dE}FsS9BhsoIo!4?EQ5^IL{vFp_kCiFiMq~yi=L zV?czlD!-H)0OfQx2KJpmnDP4*{$`T+sB9}s*7C!Fh!~`pr&fj16)A_Nrz&kZ zufqyc`mankcNeEO-DqDfR2G}RPsaFvg;A@dt#Vn_vL48J z2_pJ^Xi;W>PU31X?+;Y}0)OBNUm&teb&v7cJhX7I^})tvxXo@2{U7jr2e$r0Nk@zW zw0^MrmK1OUh(|u`LEGio`s7Vyh8+6>VGd|y9LOvvH$T|V+_MJT0g^U@(w7MyY!L-f zPafo93+Ts3E&gSfN1*fZVYu-Agp(Xck@=9h6BN*S7 zM?YkLP%f$8Y^UV)*m>0OF2O!|ZLC5bq5$UZ*H);8o>T2t`9(HoUyiGAT{nXEfdCG@ z@{T(sodrO&Ii4Ii=+-@Zul81Duyk$n_aFGe&>@snT9&PflhA_UzTU7uh`BKHHW-su z1-$1mjx#jSTA-Wgw-sa-ppDwO`QOYU(SP8%B5{m3mLV-jY>EgV?Fbi!hni%sV6R`o zV1&*h?GcQECQWy(T>-?=V{`@Bbu7TmsRfX*EC6L$=yLhGk^4N!XR1pX;$J72(b6v9 z=2baJ?{xktJRFi~$j1mk&qE?Sx@6TbL0)G9NpnK4K6}?D_}nXmjcI-l6(*Kdnht3d zYAJ#&mV?hKPaR4*%(L(*S_{wznAj8k4^m<-@l0(H<2aObvY_=*m}o$YXh+|a9B4*y zQh1isEK^7!2rBTn&))wc6q%mGWkKT(q!yP9>4ieP`I4W|yIBK_;*#NFI&fMHEC`A17+sf zg6pn}RZp2py@57+9AcIXE^Ybh>Gx`I`wf{Edd{?`^FL0Y;AjrzgbtbRWWFyQ@aU3q z;Cfv%luD~JN4>hf++N9zZ93AAw_?oeRp+|#SV1AobL%T}==Zzp_1PGFCKy3fPwq`+ zWL-%h?Sm4nG{v_tzFg~@{g2n`@Z5mGlU306y6c9HsvTzIG_e@%;!R%Cn|kXpS0@E1 zmWCl8yy90;NZ-KYiuagm{0P%^7t--y{p|&ho@FR=7J776;kgCs$3IVAa)CBjksph$k0<6bXi>yuM*qSiTw2VTToZ#vm} zWfp|m3X>sR{um&f1GLFT%bK1%g(u_0wYD_k ziynxxD!&iB#7|u0fMD-Q6;$u}(Th|k{V;^4T*W5(u#5uAp2O`qWD2^eS$npm%9%=Q z(teP^P5nw@mKNAzZ1U9G<1Kn&T8C!E6w@vjkB;0V% zsy3T7JNkRSSqFB+-I;Hv&y}|p`Ll;^|AlOTz-vgtKGDN#94h@|_B4#Cx{z0rsUNO75_z!(AG#is< zAG8T|wq#tgyb1jK`_(5`ck$3eq|-7=fql%FYw;AqV77k@s*;As{@Q!*BOxF8i{ay5 z!jN>Mzp;e3y*uVK{oIawTw}0B<(DAT5zmNZ7B4(H z%3_*J+^uh5F(M?&kbEx>$=HAXMJkxIPpJOjg^VE+9f#3>7hHNvx9K}_F zXn85B&Nbpw_s8hrjctGV$V{ebHb%T8jls+{wf+<_66c-Dn~xu$DKSn9F^0d$$a-eu z>RfD~@vYaEQtkm-ze01o90i`76jjg9vB_FhiSe%38psL=W!T3QA^G_M>nwI6u{dMz zZ}xe_`to5_>&a$gmuEeD;G}|+NARr##pSg@9MaZBg)k_|>F-tt9dns!4H|%EIT!jVV4kjnLC5QLDX4mYZv0WS9jd-#fywVd`tvnG zM$%y(dnDy{3wAMNZYJoNA^m-hq$a;Bi@Cf$_&aa&069;ceWB;8v#_L*4f&KP2rcsO z)N*s|XIi($9+zNqd<3Q6&5+-+6PV$g5R2|4Riv;5rkhw`|Jh$1K(AbT%~)9wI)q>? zB*gkpU6RT{2YoOqmja;Gxf77W;El_q;3%R6K_(Uoq)zCS3zN@y(g}nQc<-X1*p{co zr74^5PFAmVUFQT^BmvT5p+rcNr>WuI|Cz(oNY2OX^nkX9T)UsOThQjz37R6_D`zlX zAt`W(x3A(@RPX(->b^Uk%f9cMpFNVj_g>keNcP@QRz}F?M~F~<_Ff@Mg~$$tl#!KP zl$}k=3`N7J=X2;hulu~N`?~Juy6@NPc|Ff_{&$|8bo`FraeTk;&wG74li4bh%tvNR z<*8jwZ{S44m=qf-_29=dgrBH+1^1=^^fw8p=0Z5c0ss(cf~JBXmnDDURr0Mm`2TNxCYbt*J{h9)m0ld}8J0*s`VKxFZpJ=hB0qy7L+ z!ltDC$@*)%rGhw)8`BLJPP&h?!+0p6ANe?p5@|Fmha9fJIeu4uX89LE(D3E=6MU_v z+31@pZyLYN1cf#J1*hyH5y<}nr{LmH@T-to1L)$VuTk`TKLNa}sX+jn7(Zh3dVbRI zu%h=#=?jiQ8zI~0-izwsYt3XVM?jhJGatuVAQ`=W9DEr_)DbIg!zu!Lv{RuxYd>NP z+Y@2n6|>-#`Q;}W6u;&=*JsmjccYvD-8*67J{ z3;q~*tX5Qd)1numK6e98{qDu3m&96kPuG&tiz%9i0uy`-_-MhxdYC{Rd+OZA^o<|t z3ef*|5DO9AXTVM)SSVJ)VTw^#u-T3TZvlEv&^biD_Od;u?AzyUdw`;_+)xU}Sj*ha z$ET;)Y&nOO`5{6lwv)*};uH&Dsn!N!-bt-7#%Y7r1A^~M4 zRkligw_{W9*i+c~@LkX0TYflNbQfh<=nf^=MBCUYJ*k`Rt7lBzUTgJYEPbX7% z<4Nx~P&AAi1H7fp6jNxZZIFr{Y*MS~53Qsh8uAV(#5aVFJAnPbOZZg3TzT)4w4oe= z(Hr2P<)?8gl_r`FzfP`- zeaN;GUo_(6zrk=8S1=dX<-}xn*v6CS zcbDAEACj&UPrdHzf_#C5QQx;p6KaQ-XK{5}DNr|>K1M*;ti1S*L^gcVlw;@U-#sR1 zN2n$|)FlfnTIuT;d?S*~2h*6Nb(c7K=-bF3hF1HmZ%8~}*rwSErm_d9ytzj#U`u{%XlG=Q zBWt8~jw^uV&HGmFvS9J9#2CNya3z9Xi^hOzi?KkiRVHHcb9O}BM;Kb9X@>0m@ReXK_3PqWa!6vd*w1QN??kBB+|;>5-Z86GYt7*kk-Isx$IynPt&?BVL%n97bLC0o9gKU&@JdM_M_us+YS_ich!udyRdQH? z)FoAn?!1^>PkQKdVv#dc0n^-ibiG;-OxP3s+{nOZO6TkOY@O;ijPm3FwlFjE@|oYr zg)<4*nGR_Nyx$rtngnQvr!`V*TEF>jZ-noD)Wd(|d`me(L6$W;u=N&d-Q-1TDtAwH zB1{>S0J>Ot?<^bb78U?&Nxx6o;Pd&tR>ns_IFxrXq+EwJK^e?g4(2=+j#%6Oj!ChA zE}?7)ls!xV|FhMZ4{=D-Zo`hAaHZCfXcLC%3uKUt7!SJ50QVIB=vSYBWa*)c*aM9Z5# zu!3Af>Px`qno7x|-3`3~;2nV(Hd?u2)hm=d6)IEtCxGb5$G3F81Gh&D5x!^Qo;VHk zBCQP~n0$R0{*kqK?C6CkIbvH-M{LWbf62B09Lxmvy9a5xQiw$K@XE!XCxLz}?Nu~K zT1Ub8)D8abgmZc25TFr6sCGbif{w&$$JST}Ht#_-#9gTUv^C|WHh>-OebcuYs`tla z&d6h7fYrE_UiJ-?r{3qJ>7vw3jl_mYhVC$7r$G*iabo7VpN<$gMxtX5&(nd_f?eh_ zVze&hYdlV$j&S!EnTnB3ab%W!xspa~8q%c0pbm(c{eZ^DLa;)*vfvPq`lPVo1Awq? z0U*Ip>E0jGwuU8hG$!f7)31grV{aC7`eHp=DUHuS1SpWv2H{&(^>Eij;Ofq2E>^zy z=^EIqFb*vnjvno$73V8%LkWo{n{lQ*wM_XU7+p&ak!F*B68$lb13t_s(;X?+Fpa@h zhr+YOgM$jh!)`uo_yO}-52_k-!2E2!4uv%7km41LAI`ad?@MIP^mD(Pa51IWxbo!1 zTn^B$laju z1I&!(Ri%oNw;w-;dt-lcHoZ&Hrt;P@lgaRHe8g2h&kM^XEX`JRA55SfL;BZg!0KN- zYZW}FWF2XgV=Z~64pPO=R!~{>xAK~GvW4uVw!9ljoyls50tx&uyn&Go(<0u#dwU2t zjQ%^-qc?eU9)2}n3R>tO5&)*3QP>uZdO22dj?W$o=fczf;9*{-5j=(t2boOf4=l+3 z>GB$94XLL|TfAj+-KbnlG4v&=C;CSFV;R03?E01vs;@^jv6Fz5>j~@6YY`X^VkNWZ zL&-&PD1ZG`iesdBF(GWhTNvXbIKywYh%N%<uqiP-4X~aO2EoK)6ET39WxEpyozPY$Y!n0uhfHDWudl@;+ftQ*Y z&XUrzGd&cmaZFaC$H#HD?2c*<#geaIJ>)s^di?>+N(w5dYocbrk+)k+b1yz>B+|*1 zLNOAHpE+JgDHWt`Ej6ijmS%`>LtT%D)vZ95E=4SDx-W%!(LmZ=q=-|0HS5^{Pm{hj#Tu6z46MR6W`o zu$B@uvT|DRAhO|XM!(h@??4f z?!Upoc)GZZ{*m2UK$JJDn?|-rd=B0(qfK8WMgjQAiJ(RbznBY5#TlwH%q502TFQ){WX5 ze+hsM7m0ryxWDN)HQNakuqgw-8@OH`+qL7HgPqJJd)&B;-WPtZZjWzH+g%;QZyo+I}{tPwfUE^ObRsB z8Kp+g{};?c<{LM5{YWXI0nsslm8WoD^13FwxM$Oy!hYR8ML33t%ovN%!QUQ2R8 zvfOkw;?4nFU6Vgom2MFjmAV1Og92~dfZ5gOce)E9#k}6($tqPlkJ~mj*Vf0y!Piye zlhJ-LiEV1LDjzD`x$uao2XB?}SZ$pz0a|pm%Cs46KkyfL0$|DBf#iY&K0h=Q{ZnxW zT49kZ-VN<)oNY{M{tteL5?z}%O8oBIhmA=r$a@=osr;Fbfqvkka%Fz+GF%;IqAx?I zTVFD}hdCa+(O@GgzsRzZ_{`00ZG7}>ec84VA#=M`e@mY~3H^tE>UoE@dlXW$UB;>|f6a z=6M+!SyC*B1BqowiLOW^JFkSJF!d5iQDYmSH<3Pc?Rnc-s2|ErFFN0bVI&!qMfH#u z^6~ zIN~x;rEu)P@L$xRfMG&J3{sW~6zc)(s4aO^_?{I*>%wI&Vt< zqMW$vRU)$P<6mmB23R21c-tm|XL;b|28x=)<5>MzUJj(}J{8ueSU3|(j(xgug<^|- z*-Q7%e#Q^xM>j*WvG!xL@y8Z*T1+#2m4%nTqn3WW(F>*Z^L;Anrh(Gu0^%Z(e?gJ9 zW{5HJv+SMF^TUL}zhiEOl$s2^{dd!4?u{f0{kY=Q>31x-iTsgyZ?F3x8-}VLis^nsWY(Y^RKGGZd) zz*y>1V%_w7Z_mtG+xAf&ckYWB>I)3+AL|LX>uE5*W@r2qHB5cy=LwW`3ZglibT+Y7 z_+!O=l`{xA)hJM%VVsN^BK)RG*Nl`m^E5Y`v^QC}+3#WzX>}ibY={p0=s2DEN@U&h z(;2?NO}%GcAMONRar)FG>E>}(va9{>qS^Gu>xw=K^-v153$B@FgYOsbc?RTP zZE>d_88F4!NEZX>^^F6U=3vp6F{=9{l4nzYU&wKK&~}|PM7AJ6fh1H?>OY0xy&RD zxCj%NPo=%1-Kh8pCb?g+kVTIHjwL2pVNWv$7HCaux>^`u6DMH^jNP7G5gK*g(c05fQ+!)}VPGd|0WHPWnc`>WG7lpLugtI>;pJ$A_mAWA^{#E7vC8G8!@oS#RL5m*WImI4FSwlk?FM)dr$$b z22^neDB3GHS9qDHn}i`dk0&n}x<(LB3XyMs?+jdEJg|XraAeGs!%@Mj zs+t;+mzUQ~mvsuvMBRMp3~O+9v?DI4=z5H#&f-Xb0otO3SrvjcZ#Do>i%&@*_Xf?Y zFF`hPh$~c!4%ZwukCmW1;`ZuseeTI#SHV$g!KQk4_*bpx^le;UP$oOTc$ikuxYryn z4}>23#sCbMkIH8Q`0=w*3ue4uWKkR_M3@!lY>fskxOPcmaeZf-C3F;Pi zf!$XSA9!6Ra%wxXq^|#1x>fVp=CMO;z+$3IxuC19ZLaf3X+}(PGT$qVY4&=0OqZ@4 z_6qw+eXbr(PEJ*2Wt86*s8mc9o{%5Jb(hQXP-gK=NSL_5dgdB8bojd&fk)Q(tkO}t zt2eN7+`_gu!$Mzxi0eG|h-D%U@wm-g-}k_#$2!L`6j@sHs&~uNI5H zIh#Aux<3>0?INb9nBx$*T$E5bI8olX63bVk8X(l3RxtW?fIHjjJGO^EpssCDp$LPa z+ZDe51Y+@XUoD-@wn8{XyKj5Z32lI{>|8_ZZ0p$A1&F}C;Y2rVFkB5!2d&If}! z?U!XL+hA1QEIpUoTJdM==e}CL%Y&FsDFXuo%mIvJ1wLR~)#0IvvcK%Q=EO*@e{E4p z*@8%EJn$7oPB=RnFuRvXHf?k=wK8oROZY4}Z`B8?c=o@4Ck!^b78XLa;(Z9kpm-)* zRae0ZWWs#PCn60A7UqMj<(rIQD-|WY`fw~63!kfI4NP3MU&C&a>+%eCeBa+c$JWwv zT#`6K$o*bi+N2Hp;2Rr>%l-V<_cvtTK)Vq3$nRR$4uBp#5#iSfc6r^G%uWaZ6?(fE zYB9baew$T$ldq-*Z<-o_pqG%nR0SRo58vX9`Ow+jm5V=E%bW*;Hk&jq5Gs-nKdDQT zo)zIW2S$+`Z5D1ox<`aN1(T$1Gp%4?mU(VzC$5g(g5G5@N^fASh(o@-l1c2~6(u~x zKlA}uSdgLGWQPiCzcZiZlyJp$)v0rJ)5PwrIwdOJeBTHvPOGIdy}iAAE-}KF{R7rV zejAuwR!#{D#vx^|`U5ZqYPZQqNTju?C@a@&!31V#27OXBVpHW7wQ+6m=$AxlsHz%{ z!nE1 z#Nq`@oNk&I5b^2yvol|pMX|B5JD$SSCm}9w6y#)PMjr|1zvz&-y{i$3gyw2}lJ-}i zHVRD5&f1U|=>Hm)D4gqj_w~uii#~B_DJkqgatk#zHWuvFEiNt=K7IOhXKO137F?~i z*Jx(LRInVdGh^;@o|}EtHU^KNx`syCdoTrVcW_BbnUF{}5*S8YHSVT%@f%o0g>&&osLo57T|8k?6LbF6o^1K( zF0LOPp(~pAqlRMOu)_6&EMk%&{PQ-*gKiv9UaDj$6YjntYIbG~RypmR7<91(4ZL1S zp4HFsl`5VAy=JkZI9*W$%SDPh;(jk!80jjE`bo5bFW->25}Jv^eGv}+RAadk(>hkV zpy~DoMn(p;!AuNkEu$c-g`#X@n}0L(nEwj!E%U8UVmo{9{719A#p6Pjx--` z`aS2|x|7`N6;aca{*1oPYT#Todg^P+CXDY5AC>*|>C;*bwa#zC?g3LR3V*Bq%@gZI zotX;cCK>EOyL96QN@)LKD&9QyfZT9fuShSv;e}2@Cmco7C_ zYedQf53zHyu?S9R3_>|(M#PB-glPUHnXQozc2zg%K}NOLX|*jcsoI>K=*mYl7E}l| zm2{S!vG({zs6xe4IiY*#Zqq}<$n58gqc2(ZvrC@KovR-rbdl9l)I_4$w#V}Y+SVD% zJjXYLDc0lJ$J4>?>nw5V^1W*kBe8m?%2vL7K3vmK^mt($PP*OH3c-rLLui^TY5K9U;WNJ3;NZ;?FJmEiQ{-=A_cEGymq_|W!2x)BTBR~ zIgXDQn8*f`*osE0j+;oaPgDutSZ^}N2@^f_$##?4+=&|NGYU|-rQA^pH?Y5S@P7zCu zref8lM<<~Ko`k)PE0&LnwVueTZND6Q zLZXLbx0Rdkc#)qxs_I#*F!rWJjm+>%^1&^o(Ay>kgH@r!#678pYQL=gY@n`wwa<(q zpEN3jILK2;c8X6r)!5HoIP}$MN-hJ2?*JsmmmuV5Q%TgWhGVtW38-}@#Y%UW7wWyH zDPnEAP8uY^d#!T83g2xlMvZK>9<;+>CPY-sh2nb#+yt^bDE&oQd_Ng|%J6CO{R>FU zk~`Ec2vt{W>m^xt)=t_XHa9Bb0nyjpRC7*9#0?dl6lAEMaHxL5V^{iN56LiSpoUgXnw`X$HmzkU$q!APcdZ4YYm zm$Nj3NuugJLQY?8h(5>Ctd}L2-u}>zrTRXz{tYu8Aw9y^KL}pkKWn3tbn@io585kF zYV#S2q; z2Na=mvb(G0w%IhHzdj-6p?jtoMwuy$29-m`kuHvIf4~-@5RrkG(~Et@4`@he?u#q%)v~hl=5INUNCu{ zCj2AtuXuG)VFt+8V%|A7wtZ$DqtuP{*BJT7G)|7(qhrp>1iXVP~B-vrs~X zhnTnRfuZZAmqLZJj~)lvKQ7wKD-qgz(8PbrPxS?c0grx;sdYS-?OV$k{!{kFO}zub z?keoZfV)qlY3u1MuWJ-BUGk=L2Yp-r-uLgOlDuOoEyzZCca3B2IZr1UK^AGctIY_m zoOABQP~JAyv*+`_KYZVN*d>2Wzc%fpjY*Q@?-{cSAKRIvN{`0h`An-5&Ty_0PKem7 zoH$C~@uvT!oOXP^EVu68zho{SxrP7pUvZKX4Ivkd${olx*m6Y=-}HclPCj(RnasnJNa+ct zIIS~Svc(VK_TdUYIeNs{YdKS3=%Zj|6#t4e&8i2)wLFQ1O)M*dAJUMF9;5&&7}USU(__WlvsO^ zdI0K8K3HWrL7&fullJn#T(JPj5G^wi%esUG`7>wKfZgnX`k9*V;Dm_j4Z5G{JsI;k9hIotR0*q zcp2k^1Z;I~K(5^g&KO#V{E1=)vO{GXj^^TI2HOn}BvE*km6iFYm<>$7L6paEa_%nD zt_N}od2@5K30dRu-3tznSWFrHq@yRnoS_QJsVdB!nm5U5X?ZW%+Y6MPSfZ3|xD$Ky zI0vy0LacrQ!ZDXJvaqmd2;S<&)D6+4Zph5^A{G+FwF0xW@jVRTL1Kcv$l=4F&J6~^t#O1&#W9+U<5-L%olx;i` z62{FCo%#${PMs_dxVT7Gr!n<{=wNFuL_TllKH)Anv3&t$hrDJJ8t^bdj!k=@JNX(Y z{Fd=HRkoNM8822Ml`Hk&LOeL@7(Hl8McR}CBw8l8V2Tssy4Eb^Pn`Ofb50u3T25$ zUgk?1Q0r9T2UOu@QiY5NuE~VqmD>x)2Aqn2=3Tz7Xw$ASlRf8w@xPf{L%d%3ujR)@KbCwuKV$pg>qTSl@tSNE`xph;OdV5zu%( zJdNq7kU~UNbv$PRiT-L3BX>2&Ad>7h0QV`P_&#S%?9Ykzb`?oC#Y4|h7Srp(DTla%x@h_oP1tQCucwC+7T{Bf{y z!~^w6K|#R*UU^aI>f{pM#wUx7jg32T5y+9sGxlq@I50(=ItQi`yYi_Q4o{1VNqi+M zsP3@LP`ZJ4)*jbV8dsK=WcfIdkaY0sa5ko}1~4w<_7;CE+B?=JylwKs#InTc+^+Ok z5_AR{Ppr6-$$aObZ&n$>3$sYAT~fa0&wQ?U8~N_n!xRk-Q^p@Vk`TGbQqRLW9dZd~ z_tms~>rbO~NFbdqu z#C>g!u6o?j0A!WPS8%ww1@1bIp(4rAw78jr<-rg-$OlI?w?oEOTO6YW2Ik-sOcC{` z9(N2`S=)Pmc=txE>C{veqsGM@k0>oGm{Lo;7(PE#sNmZ~c2!Ts{^x zB6mdPq|#qQ))dh7em}CiM5sOtIIIW<}7H~s&cTNzPdDF7ZhEo1b34M z4bDD=iA$yvJ119Q(@7Z`1*1`qof{OrNdOCNamuz#{R~uTsUG62P}qP(DOlvgECyyJNo~(Ay0)hN+>j&hZ6W3U#qb7q^u$ zCh3x=A@U3*Uom7yh7?KX6R-mzTw_7ezpb7RWmRccrJ3K%jpJ5&^DL9U0BX~`&fVO{ zB%JOmB-he9?kraJ9^1Kl9t#VbPfJzF#8X={{H%Nz69@syHHsJ%CE3L|_)WI~VQATs z5j+8deTq|qV1u=0I}_t>*aLRvCD@qXLuPXMZtoh~%u+WAskk>LoQ_wA-Cp?}rGdSl z3}3-E>y-`XarpH9^KE!-)b+5g=mgZZuQ563JbTOjz*NUmzvB-uW{6vL{2H_-IDGV9 zj?alXwcdyBBqk&r)E76T%SZYS5qap-Q^7RoWnwZ}Sf&%R zWWDp9;V3n#^iR8Vn5G7s8%$`TBUq(bOW3l;WI2z2oK;v^MMVvqsmsGpp^oOEh3*cG zliyR;W6gg9+c|{T=da2c*`tmAfMoR8XHdTgjE9x7LPN4EvEUyqHGS1b%C=$u3#2zz AiU0rr diff --git a/docs/extend/images/authz_connection_hijack.png b/docs/extend/images/authz_connection_hijack.png deleted file mode 100644 index f13a2987b28d70f81bda7096a54ec479f20b2690..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 38780 zcmdRWcQlrN{5QHQWZa>Wd6P|cLS$x-WJmTWTSjDrtVC8;86`53k(E6|W|FMz>~$kq z;dx*6UFY}5^T%_Z^PKaX<9p8c+}-zeeXh@Vzu&L%z8+mylP4ymBE-SLAy!n7y@7*+ zrw0GA;-7#|9MZp9Nu8EL$=dljUBrC8qOcZqwAp>oGp5>Y6`LFzYZtTam6~jq zl(O->Yv}U0x4S0Q8_JNEaN@t7u9aS2%^r1;MNd*kcE>vaFz z5`!y2a9wzpAnfBm*HrLpe6;=dC}A)=ihhkB)vv3_{WAtU3bpE!|9-z-9X{|4FysIC zNnSWjiBAaQ9qex>ynipe^-RF}iRHnK42757b01?{H@w+Mb#iW8>t~h;CXy?v@!A#| zE;Mf?No$QjeRCr6Kf|fhR%So)ZE^ao@hDl4(_ChP5&vrOXw|)at|5biD%SkSd0{MB3haB+ zu?K4qm?#;BPH3G{z`Q2VdZVK%dT%w4(xM~&;%j~3(skto?zsLpx&}HtpZ9wV4(NS% zOkdSl_2f85QNz5RfcZRj{IUXOfuO{3+z6X~jm2MKO~P^O!D_`Eiie6I%FhvxCU>(d zl$|lO9+v)9us$IL8z$#QX3X+XvGUuSr51<3b{*b+Dq^_vIU~C6=+MB(bLI@YUf~Oq zI`7?$jrCUgs?65z-dFN*bIWjPD!8EerNH~Oyl8BNWO6$H|*flr<}TKpI@tT-lk7%*fg_D^WJ#G!72ZQ zhC|$g_&mRCzG1~wyx1q9*nccoaHfEc7Ig9x8!xA zC@7Ur?a=r2^(cw(Q4WjYpvF~pgM|&Q@z0qiFnHWl1j6Bxb-o9NjOj+j z);(o;w>G|i?BD%X;X*xk+2Px~QXEH|^V@Ejx}Im3Iuu5Vt<~eWuZ2%B@`*f#Vnd303ey;0#-cUPNOQTi5y4&vQlKu2>sck;hC7b3B`MJy9J5J@w@I%jnI~@r;Yob1TCDB)r zc{-f$eU;xBbjsjsmPtcE{ss9bly2+gDG^kc-gT;PzbrqLky8BFayzs&*edw2^5}5? z;!^G18Kr3s$E~$*6~)dg#t#lGGf2M+%+VJLfZ~CI1p<%M^Shc4d z?RzOawF{Dwu@7ar-jJo$_o++Rch>d_tYCd%l|3GfdrT8ocy7Ml`wIKk@X`gFFWDM$ zqMSURS4PTb;3@FbyCSGBTm0Htm)j$~x>G+WHwF`;u^&pVUu+e@rTPhIXBvy>}WvM6J z>3f|sjr~43tQgCp*q^K2Uu5jNdo9)Hr+%sZ;k4uPAD0eahlDInG(229q!hdzWj9u> zFZYCoJ<-{72W#qo@7nXa74PlQdvPa~jkkY}ICfOJEoI0)BHN3a@yZ_5J9~I^@@Qw8 zG)dSw-Fttfe53Ic52a$3T8fR`$;#7h*6qeEscwE*d6pgV8w17)%-X%>A{|!P9bi&cq}6dT_zs3pLogx-*SUu0NOc zUmaGtuiTxaz5)kz`srmyqYEoRr_P0Y$MYDk=E5<31LuUh_Utd7+fq!0%3apN-`wuEdD{_mRDM(phX&uM^$Bh4wZ~^a{#+TcoNkF| z=k(XYTIdbcY>(N}1YbT?Jc{lK^dP1z<1u{yqRjS7Hbdx2Z*b)qZo@$~8G7~3Dwq%- z>z-8YYxVnY2CEHwK zULTSo3w*BCk@5o3z(h(HYNt=tjvw^qGRJ)~j~24hzORl}afs}lU$+{bun%}yH+lRR z>4TA+P5kfTU+k*;i4W3=jV}ApH@$jxeREsjOk2%&{c|27rErK99XC=?G|G^sif%Q9 z`Afdhd7EM0`efatK4neBFl6Eek%mc|oI(3j_TxN6Cz+J3#Q9f54yZ_J6VF=BzOY8A zc~J#Y1cwJ$)-Jm2ZdkrzBTDk5TFeev7wbvdE~RYdE8gMH>K(b6ZA$Mym}+&^ef672 zPv6uV)zou4N|AoP_V`vmiEe+LyY47(!Fc7%!0`<_8=r^ak@z~pRWug^nyCDPXQ zBv-@w4eHN!CPnnk^`u=c+iF?p%i6NKKIYTU3R_lAMCX)X-8PS;u*fSBnlC z`0H`oW9vPiJvnN0=5$_waqjLarSX}|O_Svbl1AHZ1q^USxl<*bC_D}0%$;31{24zs zur7I3^O0$I4=akCX#4A6k-3j{#GE1~4{DIcu(Y6)k8$wQx6oZ3Jau`8yH^i)rrY~o z){xiIwTvysz%e+3da}7gKq7X%1maK+KGi^L%pNT%`+=swxX*UV{lR&Dz94E&<9+kn z2_p2xIEJ;}6;!nk=T1s(plvBP0!-ORQt|Dvvt7v|nrlic!=(``8N=__YWKtQ1`3QF z?aYtic8Poae9Er*I(zb~R|gx;4eQED*#+NlS{*B4x`V!0UP zPB0RT?=V~~6<=>Aw;?w}1v7jWbz8&_R%TFh8ZD?h%gn*}o;-G-a=dZS zFa@fEoXH>e#ysU|TMFsF9)A_>XHFn2em`aFu{hqWzT08`bYPMrsTkIe?@E?JG-FF8 z?|mPZPRS~t{W3h8)QG?&8LPfxEH-~->ceC1PBrQI*xjJ>R`=z!3$6NhHR(Gu`j*BL zmmY3C#x@q^=@&09dim7d-^!)F>|!)XCw{-g`Jxp~ne9OL+k@F3aU9xrO?}wy{rO1* z59lBKnru%J$g=d|eT8kMfAB@y`S@owLu-d~WqcjzS$mzEH;VbFrix6hURz5nGU;p= zmG2pJ<(~Q>a0cIeGfz}p`3w=KR`!d*X@RN-N4}#JwXuG$@IOs|*Hi2WJ1&jW#7Sus za1y`iMCgGOhPRsJf?W7@eTM8k*o;I>1{W@x*$CMWGmTcdX%;UQ-rcznDVDBtIp3f> zKbl*M`u0ftMS~2P5R#ZnwmO%X9%<+Eg}*-bLPOp-81t>5c25awUk8c7X3M^L`GqP3 zkGlrfsAW&*x0#sxILYo3y?`z5d?S#8iBL!TZ_#pL(<~;cpl5f+t8GvYwI2 z7bJ6H^cQXuX4MIIW5gxHs^O%cXvCM%*?B}PHc@KoiE0)xedI5Bw&}~jrua{ML7OiM zSNE3%`klNNc-19C#i{K02nNX~?xG;q#iM8xO@7Sxwm5p0I*i_Y`!Z1P&GA!bDH|qd zjjQijT+quYB(LXKhYBmcN8-q^|G4tK!~{otujBf-WKI~|`^hwe?0IAaJB)=;uiW{X zmo!lCN|aM2=Iv5Cpzsj+@~?`*zB zZKIc^(4*AoL>s)qTeSeYAmjGk`>dibd=nWS318}b(G;%I$(RB4_aOO;){m*AJu&~F z3_mG9y5D8>-@rzpJ{2b@R-8VS`Cnx75I*q_K#K6c{!G*%G=7w9ApRGL6tYTA zTJ(fiEB5p*2*%^|Zmy~LKUYS-aN@u2^zdJ#wG1$6Xme8k-z)RtXYiIu zzt?yrN1UVmMh9Ox{B3$LG4)vkVs{VQb6k2Dm)XwfyEE+~)vxnXFXOo_eu+LoOyzNf z_LK7%Dek11*(tn+YWj=s<{0Qvjf}~)2R{w&4433Y=B~R+cQUJCdN=TbL;kZUvG6n2)2siv|2aUFM)g?Uf7i)_41UJc=<}cZOT!L0MM=l< z?;sH{nkxHV{onf=;a?Z_#BFN&&ma*<8;igFpZlMJr{yv6Nc?w@Vg#}&PJ4jHW_IV& z5}M!Th(41mp;EDz?7$K8q%p=t0KES4-Uy^C9*{hFP zQ#~3DV*DZv4(e>+AacxgCJ9{K^xByWiSW`MFVr>F1T!v-q5W1mQcO{dp`JmPI78C`ifRUv&~k7mC=;-Hzdt)7JD~JRE&`Am`y#!hUPBQ!l=*tKhco;27k;@Ct49EcIw0J? ztYzv9u&QT_j@{TtdxGm-OL}Wqa&PsP#1NqPq#Ab<{KLIp{7x_ zFFo(q1w`{~hF-+usreDJakJ5-;!YpSE>dg$+TC32Xaby+0oncCy|uBy0J7<~K_2t`ajnHaT@T~x02n@SBfK>B$%OO=R6 zRG$lVEuFlm9M2WQs!rEXcrpUT7>!e%gU}|v2YVlZh3J3COg7RW${Jn|@stYyoOUCD zDIstYFE{F0xym!@2m_Ajk9I;fqI@-Tj~TTez^RvMlcb?V-x+@Whdgvb~u9P0`Y+bKGqbN4x6mI(dnA zFPPuXfJ*=F-j6%wT)bnXy<2l>50st@*?$rC2H=z_7eR%mTVD2r_DVaT)VJ=Z9|t+_ zkNfVcrp;a@H?#+n<{mT;xV|K-X3i*>ik# z8aD46F(`-FA|(#rQ;9ZzB3yOzxT`T{9x2YU^KSYM;6E}CO2RW78Px=B-B&Pg zZ5PkFYY;8JaJ;uN%)GVOPyTBSH|y9*5oOA zgQixjt(@?G%uR-&a_4Kk;%jJgrghq!nHYW$~Rzs2a4C4 zvyUfbiI~k3iKp})B>lMW|KSmEb8lV&cT_G+J;?CQ`27O_D+O{~pB#u>E@Y2x9#w01 zSJS`EmR;x5rBp$wrA_DqWj85(ceL75;Drjr6KA(kh&*51E*g|+?YBnK>+U1f-2Phi z<`aWjh&j5gBc-<1G>(uVMSpe*^iZ}I7TNT3xK+zX(zUDuK7V80_gYop$#L5YgLV(K z3-r3J=Rdv7KRnn!N!$>6`ofn3*j|()Vn80Ie}1c1B%S#NjtM6>clTBksl@wR<+pno z@K@OaoyDrD`<3W7LPEA*Rr*iv+Q>YIl#%m90$YD;8Q@849Osa>lkKD8#~VGoh96 z;8flCu;5h)G>y#>WGbMA>e%=qw^9qWP7*b-7MhIihZg&r-VGkYGM@}idcfN}gIPt( z+}8!^-ca=l`{Ui>=fbZ}kg2?oDO^*r+s%yLHIJIjtr%TrQM#XL7W zj5(s`v6D@~^9|=Td#<{6%pcV`J*#t_>TrHSBZAuImlXu za@$18%m_(5hTa}yMCK}IP7xBNm z@Zp@XmW=RBM}ls##OYsQFZAqR)^2OaC(bR4+@e32L8V&W^v1(K?{|-8>@;f7i>mfI zBZj`I1m7=s;icw^<(C1EFijsZ$~8R$LjUQ=t9#kcgvrCjRyMHZ5jP8`l}qEK2Isn- z4=hqdeqck=Ii)7Om`7nao%qeJPkx*OeOUGLw$CP-0=Fc*&wuOLc}*l0%plU!-@->t zGX4X7Ie`7Ef8wF!$-BgTF_IV*`nf*?XQxhM&EuEcp)AdFrt>X!*xbgF5Dfd8k4we< zD(N7?MHhld{Kbjg-TIpnyj%{-kJTkf`zDX*&HTiOQ{_6f3`=fDBh1C4J1eyQZ1rS# z!+E~Bt;G_q!;)j`kBLL&rM<~y@3Vbv-z<;Vc^4t%Fj98WC>A4HdQxaikuFppPvbPr zlb51-_)i9j^AtBXdnBsd*j~RiyD9k9sPP8Q&g+#9-etdfrM!nz3n`5f8&`ey+JFrB zWmMQ@m`YcP{Yl)K96!Q@#>Wk17JDHVPWjTz7b_skOUoPH6FS5jv9~o~tQ+!U14q$w z?Rm%7k)8C1DgT#*TxxfW6nJx1eRxzgjCj2#J$Gjk z`puFPqeFTw`Z^MmmNn;H`~J4`w(t`i+aKc9tL{gSF@}X~%)(|xN}aB<^xG>Vy0jsQ zxVFu)o`lbv59_2uQDO$@wwoDV=EiStUOJ;Tx#Ep!J$C0pt=+@}@v}TF33lENyqX#h zSkTLA7jRWE9Aav>iGnX+S16=W0^C^O4*gs4$D6UwRTEDvUG&pF##aX-g2BNmZM8mH z#pC*v*VRSO@TSN%W z*~q}{6QnryJFSsHJd@5J!bO{RC(CJyb}+suYTPN|Cp=Vz8I{{^K4RnRFZL!w=usV{ zFKBo2CTRt-Fxg7+TnnBA+tv_2k ztrr=%Q*_Oaj2DDC@ZErg@@n;=<~o%T@xG2%i|E7-*QBREJ!Oj!SJa$b;bM<>vNDF` zjc!ooeKe6No6wgoB4tkt>a)vbK_t{dDqXWQqORHYqN%Pd{wCYTg_|xcrER%qtX`PC z*CbV595yc#s&>K-9-lrL>sI^7bM7FP@_s{wkd2sHLqmh%AEDDM0;k$W@AV01RV$f1 zOdiyevrXTgIOA!MPg6`Ew9Wnju1rxlItMK`E+q;FZe>n*N_Uc5Y0C&E=~AehoUY~y z10Z@$!}=ZjA0!HR37c@JaYu!>G52TU+lWU}iTdZlX2|ghE*0RP!zIjeingx!^VKI9 zzrjz|X+AnNG#U&#jlBq3pINx30W3dRrPF!xH|(uf1hAg`g*xxl->}>76)+znm0o%x|G;k6(=VLv-f%4B z{s&P%VF4cC|IUC+_oK(5YP&*3#3>K?$= zjQ8}f(3&wq?A>RAk7nn06p8@QwZoy#gai_O!*T(b@Rj|lDVEfu^f%Zc7O zPf#KNLJc>Rn-k;<#TE{~g{n`3^jfPw|p^VrtPp zOhWRjohI=LdlXFZs&hEe+kSqsM!7TH?}L){crtvp%vsd50$U|11xU z3M@}UN>lDX4glfDusp?k4@(*TrY@X#VR=s8BYE?W&_2Nh%hSC``Zf5UkoS4Z@SO4vv@EKi?58;M{{`bbdg}RZ_UrreEm^y+nSjZ09~83Hzn{AhLNL?#|6X zQCq5d?jq|!#Ys@MxnOIZ3+)Do(gqAbCNRpME8T4MAs}jj&4S}Uy{{93AAjBVz_&-=eC8@ zf9J=3tSW!At`msq%s0a z)@b{1<0KI!MkjCL8c-RzQ~p2DY|k1%K+ZI*xSPx6RO!9@PJ_o$Eb0Xe_jBEY2N&LY zHhnAC*{%lz!38yQ^z*#Xuf0vmj(y+frNf1sjDQ{kq0+;2%%` zMb&b$F-T6$N(U5iP3b#3*hw4VQvr@3PPe~#=#BYxxK;4rBk06WaUO1;v}!#zCm%t@ zoFeMR-%ey6Y8$IAgBdb4U_#d0lTi)5R-@-_r+l4L4x?8+2zg z-d=wh0Q!K;qcC8S43!eN-v)a-kr+Ki(5yjx%Q}z^_q)jPX1;%nU2Gs)>juosq|bj; za%K4E!S;9ua2O^}&$P8XcdX5ckr7O%2g-&O!BBLi09&(N)Wx@+ssNvL(V&Zvt`d|w zg-2w}PbmXF!LSmj$PPfwj$>7BHIiymNfy?aF>gI)R1N#3D(+kxiJ0p|$o(Dj!*84u zSMGdfDqpTY9zeED7=c7RvlKA&kd5N*Zp>)Pu+A>?Sy0$fOZZk(Z8Uy5zfn%EQn?&( z{8Vn&!;_@#Mdq!jCbi+!ZxyeCm$BEo(>3Y`##4A2(~-BO`!MfJ&~gw0<2L7d zsCkGcO?Bu{&A^M!$l*2J<6U8vE-VwG9kfqirEAOP41Ul1D`b$4NOOT1-?hdyb@tvZ zmgZQY&@(+AH29AY60T%I>L!u$0)JuT4s&wr(x;a)qF1(7*{NJpX1h~3HD9ZoCfc5) z3dQfDwwL3I8xrKaLO1(^U`WoJ!t4WY^}-z}3}YcCaF}NH?79^ArCGf#XsSxBux8fk zT6nLDq=8;Fzee_9uff+oSjKt-e)nZPfBeQKIzDytmT--ARmvyQonL^`?Jqe(D(!Dr z9J#^`fD%vHq(1!w)~jEnN{m62SFH5jnvr4xcM27EGa2DvFoPuBTw!W#Rs(s!Nl`KN zVP3&tN(PT2%nLNp0Wd$=49d-miNsP1l+T(vWl9K|Tf~ zpAKTHJTI+9Fg(QhHr=oR6T@SUb-dT3%uF52t*h$Vne3lf}QwUp>ZF#*b} zUQ>@)zjR(j1zdoo|H(&wdc_Y1ru4NX7b)uej)~5=`1ZF~va)MqHF5L)a|A|}U;&ck zx9-j}YesvJB9D##M81OH5|K+fv0ga=9g7N~B+k_*y8@_1eeNA;F@y6TV$YnRgQ79? z5}q?RZ8Dkd#qHvn#IF%aH#~$}E2W4`l5lH6t+PE%SZIRVup0vt{{AKb=af$g`dqBm z#n%a$`1EuBRU`)~+3VfbXANG#Geo>k!!n6W7ABQyNJ%GwZL@0@TNgOBRnP1NF8JmFX}y85bhh5YZ+AP+@e_EIR;e5eEwmFYNqobjSv zOc6s!a9!}4Y1TVMxDYPXfrOOjc`*Ki7vRfYKHXu$)au?iXG}<{4tC@OhNUXzweBP+*y*fD6E0uQ`J4kM~@%Jf}X?F7{aWmBw%v{bxbWZ^8Q+e;l0(nfFv+9lScPzSpYssM+P<`+XuNQr_ zQ_0~03t{!{?DKc73pYgEmlOR>#%q1rn_QrLo7RIe*y4Lz^k=Y$h=6Ud#CAZ4vQm)*K2U0l4rRv^V{Qa#(#jkn#ncxuN_t|rGqe;yLk6s%m8*O67hcgp39hjVnL~AE{H4rE zsVm+y72w{>fw*ANovaV2n|-=9@*K#&?puO{Gry(pW`G+I#eV78>_N)U^x9s%@!rYQ zbGfMXJHnE@*+uJ%xXeC<$>em5tDT2RA|C27joPC_Fa4VPrCfT2l2AvnL10e+yGwQ= zTj(=DT{)ZYSHT|%#pZcLU0nWFp;`NDLs^Q-(Q@LM%Uri2R79Ded)at_)$uc!IT1DP zX`BD`07{ukC{5cyLAU5lzgFf+C+e!uUksr=_o_mqSPAbB9raOnB>YB6a={i3n;DCY zE|a50xbTYo1nQVJ1*Yi)&m0ycG?fXs3#!Sf}Els7@DChpHnMYMC~~P$Ek$ zs}GLZ?>p<$@tzA0(YBWDPc>+wtRVwr0`TE?nN>-VO*xg33Oa=5O5>$ytWq>%5Cz+H zNsQQK$1%H-=hVpQo_H)tXiVUdiegN`saD0G<<`oM>ZKqGY4F+vU6ZP{lbW*Zoj~Y$ zd+-RxkCZ!hG(0@nkH!RlFBt`;lz3>t2cO0MV=QX|Y@Wqi9oyurYS|~NcfUiCR^YB7 z-)!HY`4kl93OFTNSA6;pU~z2{dy`p%X(u0jK6_gnv9+EDI;8g7`r7o#bwWrv|HPRFt}De250CQ@3?J_qdae5W={%y? z!k}6e7B8yfD>U;b5pM|^+g?2vF@k<;a;jk)D0cj*Z^#*pP_=vxC13!6XZAM9q?BTO zNkfyJ#gD9MTBJ&1T7vtreqo=^LWaeapJUq*)Uy=Lb~tQXpa}Z@mvzBApR-?zc_g7j zN`W;%jJEcvh$yw<&q}f-Tip*1;7kApSC`I55TIQng$sa3I8m*2O>n~m8SGy^ks*9m zoJgW6JuzaVbDh2q+Z3M*y;XT8sOb5E7Zox|-I?C^HBA%F*} zEhaGa-2lBO=T=o25uMl|f6eZTa(>?BkdvYTJsRa%X}&i-lQ^_)F!#R!#tRTAh zOpr7eh1Ji7>KTF_vW-B=<|xBh7S-SWZOeR3E_-u#hUzGEERwOv3NN7NHqM|;@3|2r zgC?E$jQEV=IyK*?i+1S`@nt)#QG7;v>9|EUCD_xU@%!xFDE+%Pz^}ueLm=>g4C5b) z@uR#%RPY%=xAjke!=R;2MT&*zz;>5n@WXij{!sQ9#GBjm z*UJ1+G%7ghh$2gp&+v!rID`Yek#b88$OxF>c-^B)9*kg~LQ0r^$#y{bx8h#uP9cz)oEArGIyvX@U}uLjB1C zp-v!t{0Qcq`Lv#uGG39A-xRx!SLzCj{~-9LD5wuQbz&qH2(F6-QCvO+H^*my;h`I? zG?>uJD7I9yobn+ZvQS!NO8N%u*a-rY{m!a0@FX5b%wCdYurrhx3p?T76Lly|s*v5F zBn=W3(Py9*NKdC8!Y6YQ!VLw_84UYBTL#%~NFl-mUy}|<3i^xv`v2}blAqCkmpXU& zt@Gj0*I}+d$TS{<|DD7{F)N?de`ldsolY%y8{@P8Q@i!~C0iC}XXp3=#%p1r1)FV5 zV7GA?$Wnc-1IB*_sDhs8x--#a8`V@U50`S1<+h8)NNRz6~C|b z?6M;_xJsTPF@4LnGM)jDHDG7qt0_CxQk!5x^7ogj(=6SNW>^9-;)@VhN)>4TN}Dfg|L zQ4&OqQ=qu~jL&rTo@tLW+LS#|OBKthz6nsODx8u}8gnpzn=bUNi9^vP!UqcBk+EVO+L=P>my@ z6-_Kw!j|Pfyd~nk)py+X>izLmTA4)r7Wr^0us}rsZmZXDOdjOSBZr7@ zP9W&;O|Pq#`igs=Bua{sVfGH3B#znz@J+Q}aRm#NewkfTo^hQ~>2Wh+nTJ4o-@u(h zJfBeB=eoFiIt-U6J_?8CHYKF4aAFAVFC>=n6YDk=XB|v1_GS0_xvVjg`e1p{1lb;9 z9oT!uk6B4Zc9p{yk`Y@YyLiKWBnYB#o3Ve02872}f9aC15Z~@J@_LBD#1FFvTmYrpb(3srZTVQ6lYrSZNV_!d9U?unK+7Ql~!; zmB(=FGhO_%<@{BmgK_4%5^wJQ5|>;5&cpuNh0I?#fo{V6##3FSBox0>jHf$_aU-aLRh}cQ!D@)9WOc35~Y8 zDQ5bX3tGgIpwfu-3URyOkI%qsKP*^2{-DW;$oR+-hx8L-SKHV*@@ia-5B6xRfx>_l zF-5Db(td^A6j%>CrAN3Pf8mXjZXZXB;s8OBTEopWrMZvy$nb@bVW~dd{iDmER3uM^ zp5xTvny~f61lzXt&IDW$3PPguWW$k{E_)MG~eCG-p)g=*2~zTQ4H z!mRlPG%)<#N^u>$)+rC>LQq2v4c zbJC3sA?5PL4>WPJx>7x#ym1yiO|Sh%m*#lJkV7)E?1^;E-WLiV|fR1s5; z8rc~gWOQQw5~dhKaGLu7_h|>cQWet>zSiy2_Utcrw`4O~(pyt7jFm7ewBoqF#_!35 zS>wX%WlIJ8zeHzY|Mxx8@nxTTOx;DDcH1v{_vd+-ChYJKs~!MBMd*qc<+zzHOMpZ? zc%W+_U(C33$GF2~a?15||GKo3us`I4Q(wQfL*G&LGxvqh*Ty`iBRpC!EWm;?ZJk|G z`aSECcqFU{(b3TH{92`xS$J%D>WuSW*MP~vLcQ}HH?ZojaUjqQB2(;TelfMbe$IqO z`@?Da2uKaDlI1g*rtr0!kwvwCet>&gA(kbAP)Dww-w5nOaZu8=E%v`T)*kZ6haA%N zec7JG%~9#ogsczP_=tbYd6Kf^J3yU%2kgHXl!Ha9LG+9P;4bf4m8`Q1)_u{pN>T-+ zL7JbU`DOkLH0yL5NMZ=ctU*vdID?~?1`0ZWg|-AzBIbfT$->vS&yluE7WGsvARXdh zPL!aX==cJd?0CQod}*m_tyZ&<6DZpod3t>QY< zz|IWqs5ipEIaJ>#j}k#0}m;Q`DY3!YE_JN5_(lG}x?_4h9W z*L$|h;C6i0ttt^yhZIl8;n^%UDj-`v+7)hnIeT1Bd!%QDB)l zSS6JxKuj>MbYHRQ=Fcr-PfpLhGDP`j1vv5E8kEOFRg`_sEP28WkzanJ+p!w2do!MU zV5>rssS-L9;U9GJ*pQ-b0ihcFw{yXF9N>c1Hz>2yNX-HPhS1l@qTUa8(eOXn{futV znbJxBJ;3*t&7r5;sRJ7|=pE<6+db{(E7R>9EzROqL7qJ|O!BaN0m=St?1v80Bc~;@ z=EbBZDN+vFb^gq4BECsBv5YxnYb|G0?4F6jM>{atXjP&45YaE~#(oG1Etz2mgGBKt zSQBHQ5*6{lzM#3{u3~RaUw|Zm=nL{0I&W_x9m*YW!s5yaWW1p$(jFR25lAo}`|4K8 zN#0}yjL~IdmcOci`Iib+V&pe;UC#FwcEvTldU#GiRHzzemIbk&+@RkXC#L-_?T$7?WVLUHI&+>S9qRGom&8izm-eTV(2Bnxt@B60EPduN&c zu0=y&W;(B(AF0eo%Z;@o{W;UbU8Xzm>D?Hvp$7*n;8VLH6W^wH(tTv+5ssJRisUV~ zn(#2W$rIvg8iP4@WPeUzBS8rW1wU7-Nrd11?FATp6VS}SHIWI*T#4j7=Jh<3cKps$ zvK8R-#9XH-Qi2T<3r2rVN2F`+_<&Rf=I}Pu3hzKKJd+&Z9euW8o-)0uY*fVfZ6PQ5 zx@u^zwAi`DO6XOSK}#!`&EJJw%^~Rxj@aB3wjOSEV>eZ>$D4(0C8zfTDk`-e;{H8n z{;1P_)&w#(XmYeLpnzxyB01-R$DJII8sO^!&@=JB+v6^G8p26W$pe?`-MQ|Iu(8hm znplB~EfY2$hcNX%noBs?#KvwUWwRfril0vg?LU{zX|=)d`VGKZ4&~4?#M?R)5IS@d zI)?dN?<8auR8Glz@Bdi10CoC?_ExAk{oLn=&rXj7X1o<{$%0N%3xxSw|AoO4PxO85B1>yKrzVQuF;PP5d7t0fPOU5gs_dL(lZj z&H$Fd>GVWi<-hh9CbSt|Xiqxx6C821N7Jou{>m3nt%gEGCvp>d{&z%YiV_Cs`Ty=a zhByMwzdU|}>?pru_!HYbAR!RK5MI9MPr-`;iv1d8{>>;R z*Pze~BaHlWLXjH554ik4ZB_q#O9!SWFB?nwdxyeTfWrrT_wN6r3;B`aK^+_qZ8E#7 zRcqgFKo77wE%f|XCYFzqIfnqQH$#XByMdx9cXJHNGel*g!{TV(%Qb=~S5?1iiozJpg3yce@)sc1vSEzBIvEu@`?h}BI zM4UEEU}AJ1+63}SG8BM(5We7J27f&!oL&n1R5Fj}+{U$BNZ5rg9R&#dPfvnL!n5p_ z_r`Zc_M0Wy22{&Ub^n2zZV2B|}%6By) zj(SV*OxVO5Ay5%%a$sD9f<4RW?yDABGJGCnRW_^_n9y;kbPd6muQ1O5?=hoAfQ#TY z0xmxadHcX82gk4PER~0IL~DigQm6s!;Xy3E-q1>g z2rKM2GD6GkJdlt3{j0D+NP>-@)=Vk1s}8g=mHU9 zx%4q_#%p3_!g{5rY%iN5uafYUcy3zNEd5LsvcC!K0b$NukkH6fo08?4-~|w;)%@tP z@Yf`D(oo6LnBi-&n!?3&$k|{963*z5D+QGvfMXSG`jbYSNuxK#4h4~e7-+zu`bmT} zJi_0378YrUt#SZU>@^IOZHrlU-kH;$Wd9lq(brJJ<&He?+LPif;f42UY&JsAV^Ao_ zuPnfWDg-1aF%zId01*c?MOQ>gs*u;`@pFAi1h+@xY<&v|F|rb{IP+`fWl<<;=@-yH zO`*=f&?OBI-U$75?8!g$g$QKNlFSeEgW$59V9gJsHlKmie`h0NIv2dhwa9MP+NgUX0(bJX@cA96fgU& zOMeT6|G9YLA7&BV$(3{t373HgqcHfX*aV?k(CWX*;m{Khq20}|kblN#xntS}&CA+h zDj-b3FsNl|^)cpJFeyDZ5F&9fk;*{X&3t`pfY)&BwC7Cw8MmZc&!LqhJ-;XZ;&Y4y z{`IE=|IunOR^po_=96!YDv6OMXCyI5_)|<4NPizNLEe?oH2po^ZoKvu68V2SwrpzQ zEi>!D?Mb3l5_zSFC*ydG$Gny7)2`f8gLY0fggAtk0NnKbwJz@ojYV-gP)3Jr+&w`0 z9FXR7f&$hw?;DUTSdk7kL(n2`xzRL*qV*8&6sV+jP+b|ezqv$;?JpP0i5$L;&{gd7 z2MwO>tO=f}3nVszKy>?8s{xi!X8cT0^L~q!&6HwyvBMW{Rcc|hpnvq$aL7whdx-Pz zEaBPErjsQK@9UaAhFHQuYd#Ac2-(oMM>bHam!m6~4rZ_cOv>>pp3cA6iBr#cD8esizq>b{ESll$AN7h$ypgYhOk12F#g-awz=qTyd!C``BV|Ps3oH&rO(q&enJ(`yE}W4500e*pjU3Qc9K@JjV(7nBDc}|pjk6tt-QU)!-R;8$5Mg!XpndCrtrSN}ca^ zNKnE3g=*=8C;Q&8c2BnRf}fr008J!R9I>+$_LSZR&f5mh1%Jg7-ck~sEaK7*8*=(g zcp{@mKquaD={NA+n;XzuYtS3mi8ipLi)kzZE`cz34C!%1b|)yc*(>KF7}R!vlx~ zNkK$(k#v-Mc3YZUGedy^kbX40I_DO&ZZJGJ*h(N16b_zPMtW?CXP9w9Fjy+h9ad-+ zEO#6~i_)$`oRLp*z+7NMX$P-n;=rLUQUkb&$ zi6v*0g)n|}W2~VXsDTSO^uhJsL2k12rj^*h4i{xo9>V;Pl2 z3109!za-Sq6e#JaQrF6ttekBsGzdV`67Y?Z3I>OkE(z?U?CM0_Qq?Auxd2i=BWYOa zwr6d=jLKIq8h=?952Z#qpQJLeA9!t@>M6&eo1bSz%IHCxda3!Ispm;Oh}vLmy~=O% zg{?f-?7hGju5TRZK-#b@jHJa`NU*0>G2Jugf1U2_BJiEz)21&@fu`x)V0KD;4;D z^p>2<5`dfh)5kCIlRfp9Uo(@2yk9!|V&zj};2RZiApN5xo0uuo2~e#aj4=p7>Q2oG zPQoY@lwV}H+K;njEgF;+foq&k(|r`${w@X?M^qr{*WTMo@VlxYZZLKb$jEd)BLU@c$D>noQ&uk7Vw4s90q%+(^ zqp{nqvy{Gbwn?nw#S3GKYn_(zZIk%c6OF$m0?MA=?c|?}`Ezl|Rg_~(=~n=Ihhx5& zoZuP<9+Y%!AciF|24QsjA1fWI<)yO+x-7IlO8jAC+(|IWV-QN*9RXj<{KgDl;;fIKt>NOqS zGjt`~DxPDF|7kf0TMwsPK>)p=fvD7{+;?kmOWmok+25);&tEr7Aa0N}8?3`b5+`H) z9MwiAM6`n*H3hFHYuj;xjb9vxQ^!#z|_4I0|0bD}h+FH7Kk=ge6 zW?|vWL%Ls}C>X(;G2e+59eg%3xp`(d!p8vb3Gslus!ID@#%Rx=6Ik^XRydps8n^1^ zeSWi9?x`3y*F{tH37w+JK3TUKU|CHMhxWa^X87y-2jVQ?G!O@4zSA#{0bQLFBp`|K zM8#zi$qpw45Az3yw(<5~hR@8wwg*`UE)e-L$%o=)kvdI%+dXUUY}Hu+`M%b2*b4TD zYwi-Ud3yDsSBDnt7NXvxfwe0%SWJzjdg)tmqwde5UG`W$)%vf-1ust*3SQDpmH?fFY!pO|ABVyN%j%oJ}zeScC65P8OOFVZKp&JO%9On?F4Tm@X zm6qY3O$*~0TWL)=mN8f>Ugy0I>JX{@EZemkJ8Z|%rh9RPzA#))8iIIhxo*a?Rte>w zsMkYteiUfFZ-2B8oSYH7O6KMXLhk|w46cmM(-YhOu(EVGT_^Ue@Sp8ElYHXw$-$2U zPR-fzvPj*s0o6Z7+0!Wi8rC=P!4EgLLr)-6C+|@C2;~)`m+Tc;WW(8$W8a>fndn&} zI)Gh~-tbkm!y8p?<%H|uUeyVUU|{WOCly^rZ8_l#=!Rp#QeJyZIYsqSy|87wWOhEveI8W;W>VvZDC~zN-_K*Y-CdH37JJj&sa}s~ zG#h=~0z^^Aw#(prJOO7etyVzUH8BIQu%2K^$(xzUZ*o>X(PQAwXXGP0`c4RC&S=m- zC>)sliY|dnJGlJs=|5J_o|C-ciiGj`U8dEhN*&v?cdA$o;s_`p+r&*SaTYySp211< zqtm8Px_={LnBmedtUc(Uto^vQS2S*cv+P|w!8PE_7f@U#ur#|6Tr4~~XBV*iS~#*C zqF1Z8YKzM#L$eEJ-vH#2p_{zgGwA*0ia@D`a-kRDb~S1oNvq{BgT^Isk2KNZbLsG-11dZj^xd_db)r$_|3(ZlHzTE!HsUzwcZw!26ExZh9qT?jCxsSRijP!h=sjf5>0FyzR_}O_=?u-1=^R%z-|zB7<2Ij2{`t$hO9m^F$ED=-%jE-$+62ISly*D^&*!*i%(C? z)-FcL?s#=?Px16zmu9_}Ymkp-gMF9b0Y1`WN?q$#O{vq zT~GL#G>IH39B55E%Uz_X+1n+!@%iLY$c0ps^USgj{2#`V5 zksRMr68o`>nX)Hvo;Hnr);KwVeXB~n31HSIz&{Pv8qm3dP$9xQ`BVEmfdK#sot=hVV9w71n>8xz`Cr7gg|HbiO^ zEyxMW0>1@|(ru!F+S#5H$~bE}Frp>FVT}$;G-^Z3zPDMj<4?8AZA5GrPKPPW$ZCg0 zTo}xAPO`m5_ciN9_NvK;A@zq{PTFgy5Dy@Sa{sTV1wH(7)?rOVd}gNdLE%L!EsH5G zh~uupnx%q?;l<7_o=fv=JC(})M2Z;i?kQQA;JnJUY}dNWbD7~ek&bqY)(9>fSO=3q zn+VJcXLVP!8$bE|Aj1l!>o>Nz0gL>x=kRHcVis*RVU%oHMNOiKe15MP*xr^B7Z$p@ za-1toO)XV}LmjQada5C+iB4G{*?E*Sha1a&1Hj;zihF@>LeuCE{x$pDqgOwe5S9;w zX&`-V1{lXodQ}@}^7;ovm~uE93Q--Wn|y5T-E7x-|2EQ(O=25|60bQ8cpXPR1Z4TwGAP$W{J4ecD(duP> z91%gqXFx1my{Ssm z<88_61n_hVMo^tnNTof!5EQ=9S6PI+OOH#U(jMc85YIHw(g4K$9=Ak zt}cA|5^~kOcS;7#I2aTw89y>6e2&m84lu9F_md)2-r6NvnX&vqY9Bqntx8(CDvpCE zLr9Y_@7cwZy@^f9J=NZa_&QfdCkc({`BPoS%XGIIYxLfCZK#?sq057xwDjWBAD$^$ zC{I+KOY_$kHtXcMcT=@|H}qo0gwq5j%tgFGn%yo6t%KA8zYo?Lts%nZE5m{lhltWD z>r=p(L++)!O5OC@EQ^Bl%}kO77SohR3fKqE2{t%>1udBqi>HlaZb1|e>Nt=sM%#W8%qGW5*EDL~rg^TyYhz*I&g3QLVTZ*&?nrUC zrmXfQp0n-`D+Le%H-mvPdg&{#ZL@j6>X$h_y@66>e|w+L8Hbk&qRZz+93k#>w2YD& zZm}zF_(m**0%#vgg?*cMV&}oXg-VDD9Sd8;-&{&8}~13Mg&D+lamIKjO!yn3sg(Db~S`2Hzjb zhh{b7|3uDoJF5X<$EO0wRHQo_VJoKsi))k8Q)HQkt^-MEPmLuc{pkOZm|2kh`Gol9 z%h9O4s?l3_L;<6EZn)@3yE*|NPkei8jGmGhN%g;?oh}LTd<>q&8u$}gc|s^MJvkPI zu)hgF=WT@AeAu(QeFrLg2|1F-^jPz4gw$j37_Y_e=$nfl9R45!>@gat2y>qznr?1v z%_U2Tk#-vN!QEg6!mN5FX~KVb8WXJcdU=LtI-Sc*;>)3azjI3NGjPYd+M$r8+*gPr z<06q2Hk^(elk{T{uP^dGy*4_iQ*jy!m|d41EbOu?4P;C&L|wb%vKtJUi4vy2{n>6| z;lAdaoWfnO(lReQ4;_xVw)bwY;j=4k3vC!LlUfJ$U-$iH+chJFFg4{8MZUr1UQP78 zfbnCHk)^4D87qri{vz@N>6;XGfqvATeV+b7p8vQ2<0vd0fhZG^VfwLw+Bp)arvUG! zVm

#p8!y=e%VSEz5A1!{}AOuQ0US)*&9cEQaDFTK4dXv*Ss2JHEZmqF*js$nh!g z7(PsTXuHQ67S_@q_Z*)uCN6R;d{j;IJ4VvHGM16B>XM=jEF?5eHM4mdr{asBpRXTD z-^GxJxM7R1ouXq#q|^xLGIbCUq(55SXt?xM33H$-@wfl^{X<6I_pCysXP#LpuhxhR zo^%;aF?uNv;)&ky(2Ff<{K}6w{cymn$bDnuNKiS@#<&?6elVBqC{bbw(29dI_9F@R zU|=cxcNhVVYnu2gxNRKb52_ym(j7}$7px`TQ^QsHCkM9oxDa?B7J<#xOKcVI+W9hU ze8-_dm?jN-7W^L4IjFt;pNggR^8T%+@utFqo(7y{VaOFsuU<>q{|-IOTka!)rPCoI zkt^(-YD4E3coMQ|@J;GGd{0n*! zxwy=q?<3)U7VsR^;NT)nPdpFb`Wx2=(GRHE83by29O-%2^d?LAeEi;;ncvg=M=d!{ zQ(5koaBcsL5*Z9)d1= z(}}O&GCvN~-p!C4*%;RdGE-*0RmlKfW!zfl%Uq|4>6hEauJY)E+hV^aE;e2EV2xJi zJ{{5w3ps2rj>EQ8i$>dp!De@!{aNJxI{Z@OrL8uoO&``;SD2tcH+@vtZA#ZMWV6vq=&r;om~9#2cM?*N|YfX zDc%r#TYK`?;9&FJ8_QG7Mwy<^0bNwCaWH)7Kdw#@*kEch!2kAnD2~KGvQ=R}sTIDu z8uE?3d*5_1`i;P37kXVu$zN>W^bmIKfTkXP#u(fgYjfsn&%)d@7AueEIz9QYigh^f z^f{PvOBDAo#Jyjixs*|dfp+EhYQh}Nq;BQ5ue~nvHVvo5M-!0CmHn{=F7z(6_9qk% z`|AW#-FWN&_$(au@|r+XR|oMgh!>omy%MMBsa_9KU_{730g!3FWtoprh>XqV0Wjb z+_-do-4VSP;rg9C^1CYL9#N9YjB*%&H$b48wNjVLxzAdRz7IIW_%CbWp$$m^p*USt zKh*t}YiR0@u$aF#zkao%LWmIO@vfFZQPknl?14^%J_3Ei;*MWhrhCgbq&qH&PNs83EiZkbbD&k2@2Mq`!nf$? zJPl=p{Y%9enL`+)m<)Jy{JAJ%Wn8~~iVEINZ74IU>pwhr*tBkI-QD#cc3GvEJhj^3 zUC?J~oZaU15h7N|?|NOD<6ZrNqPVyFss2mC0y%rZQ}2*7b4~Bm5lYcqRr&4j{gbpo z+czkRo#V!zZ(?o%gy|IH=q`4;{Y||{W+0+jRcd& z)BjldMpEhEUV`3sXzDW<*wqrB2z7N^NHRoz{~EkPat0(sB&Sz^m%}@3y2hN{yNMXY zKx0oSMV>{Qw*bUAM|UDuC1xhn+7rHoNT1Tgf0CjP4X|{nZb5~;Av`jc28i;K*lCx+ zv0NDb=Ga(@#x8D)&p6bO+N>nDA%?_v2E_xBUU>BJrOs4eS+n1SU*%Xb=_$`qcfj}n zE8S6|Nk#T08FFB?AcHKOUH{sqX(QRAgwT(Xl8A23VyaoWt zqeTKIZ!VwdV^11FfP)K<^gBTLUDn{{T-ER?!dbuGZUfkV zkvQW>=vf;+k*wiW71WZJIkq1%bP?a2Av|b{1;KU89V7mnM+s*pYm|?F4wH9&yE4fp z&))VZKoctVwq53~CvU3(PIE3@>I@0+n@3PlO2%8oNxp7hR8%1G;03X#XqrH-d1J*# z?_ax;pd1dens!t7>Z?z)ko@#=UHiSvq?jAqFAVO_JvnBa&~kiKC7QSI?hOmKSN9L5 z{h4OuYfPaw1*PSI8ZQ0p4tUa=kzW=jF9GHo|0zWieCV&d!9Rd%F^}x2ty@Nfl54%) zK;KgF98FcNOy!!fcW`K8SbV1p3T+X@<0vG;8PY{)CdkTLAN_pnMb-A^^2?0d_>PuI zU1W>R)|t-iJ7SU%@g~_&l%%p2XZ-pO+)la7y*eQ-o%zb;TQ~kZP3I(N+E*Ce-bSDK zNLvb75G-2!`v!dd=cM<2ZC|`7`59bVaID0^cu7a${C%Z9>1N*6}@Z^bc^$>cIFmHJ}_FfpH0dc+h*```U|`HCJ$eu;mC(9Wn~(-F@dP9la#bewn5O|XFu6qM>@Ky>DS7_F5)u;qCsInqBDpB;5y_OEYH3mVX-0Sr zdZ~-w-fFjlme0t>c+sbq=R|I|@_MtiDd_0y2ajfcv5ej+IMa?1-HMFXj@15r%r#&_bpC;s2G- zhB&ELl*ligFQ2$DLxXY zcuzaMCQUV1k<#?%*2&$g<5ON8fjQFr-E-idE^FYhoHx@io}e61R(TelD)R4^e6?}g z#|*U1{xk0X!kyUJJ*9LjX#R59NN^Kj)mr;E*85W8Oc4`*w1o)F2Sw#T8dk~w`xnF} zHiTVkU}+hBsP$gW<|k;MW_%kS9=2Eixv88RFm$dsmjBT83!(Umk{A2QZq2SnWqcdN z1HxlUzAUg3y>ek*-1R2<-jz_%2p2NWdHiEfdmtMojE7^W3L<6hfkH~?rhx3Vp?;G^ zw~wQYnha=^xJ8IB_(C2V^tVbx@1jgsz)@VM!j2U(kHD5_bf~*zC~VIFguK$qEgsS^ zG7XI|PKbE25Jn|h-%i0CjTmI9jZ}-0=yDar&9G%E=IG~0z`N%PibffJbV9C>cm#w) zbru{@GAykDf{a8*DlO8z5qw$EQ-sf|qe#nVjDUTTH2@*I7BKv6FmlsPh-?uLH?!BZ z%u;Y*Zex^*|0vxU-~iFX7sh9?IzN;v~?`7_Jc z5T*B7vTh>vD}#t#dYHt+@Mybxq>oGVD>c{Gc)c&ULhVXy)F)tnGoX}5CM>%Bn*xXZ%S=;q5C z8+H`I=iRZpxV~ifi4*rmw&tfxv1FkR_7qovUmDL!8?&rZlz8z3amZ?LMyrrCNcp$G zzB;9|TQ^)%EV2Mw|MqI1%O-T$VjX%(jI8uY(t!HuM9I9C8)!^T*5F)UnR+U*9<9iC zAh+3-_n|GCkLDLYl!&8!O`9D{P6%4}lDS6&hXTSK15%#>Np*0lFo<1l#@S&Wi?5$& z;j1o#;sMHgC&il|I-UonskHmPPDZLl@JsI`wJ&_z{GV}ISM3R94{De@kGVuN9;g%> zaBoC55>v>FhHhx(r=yK?PlQSO zHLk$3cE9@WPA*qYKky@-1G8DBA9t@k;(Yqcbg>0~EiE=Gn3SeqfWjg-m?ovKztcck zjN>&WH=Ska+I>K{ZmHys;GFp`i4$hputQshr^{JbwtS=27HnO0LT{EOw+E zP-SQ>=lc~s{^?lJn?H+VO;;R?dMc4X~uHZS6alTo6a#>JCv{y4{Mh)u(!I3GgOTEkS?>DsTIB;n} z)_Vo#4^eX*nJGLaF9>h<&(C{J9IH#W%QL!FLvQ^eL1R{lktosTy6A%_QFL^5C#Gf5 zUZR{2js&UIv#0KFc`^Oqoy*xDbbrrv_g&(ULW+prZat9kwM6H5;oprV-ix5bCxlwg~>ze4X) zs-(+p)r+s>SVZO-IX>=ANC_rg6YmMX9G-`hKL=z9@s`M0P7q!$XcUGZlRPw6{8-xt^0G?LsobyzoIsguZ^Jj75n?Ocx%Vi^b z6c6KQCL=+l(eYQC>`fvzj=VytYSlX4Q(x1Vx;5q&-!E4pQYrKPsgMz3E}TSV+ag)w zMAQYIKl{JNurJ&QaI!G_od;dO!LIUiPE0wVxu4NZfW952g~TGZWs&f)BXhC-=ei<* zzVa1m%eCW*qu=AukzL>dPQWoRjL7^*g+O8!w!_r+Pa}Xq2s`sqi#7Ecwe+Sw^^(+TtNdrv zKii1uQBFRgpX3~0ZvwTP45kxoxdpUKj6^9gjG6Va=;)1D?Yj>GV4TbB(0aFQfS-v0 z&OS18WCXy@bExzh87v9G4M+L0D#4EdX{6a-yGWbG?K)4TC1u?(Sj-zjk3X0lT>qD{#j<5%lo3DcpsIK7gfHg_xF9If?6(ggMnofwW@HW}_-mn#lqW-i7N@Aoui|@=cvq%wlWC5>`yZQM zj?0w7FKbHknWS?R!|z1zzaXugSwtKZkJZoM;fuxwey{8|$fY+!4PQ7b7RI&b$%19y zX!d_c(jryHTx3O*J183RI+Mij(@(i*oQL$)ErY!dd-*C19<^snB5Vs{xjL~1vQ`k03UusA{>r^ z9Zi`jn>QK=sIO(;@&*TfNnd(wEMK0&wG|hQLl`X9@;~79EeUKoK~3iCm$WO=@4sSl z?xLRWv~wK4jX{M^cnTk>?m5w22>oPqg z+zhOA`#XSsZNH%Wx=NZ7Y1Fleu2r%kCdT15dBb87puBv?*H@g;0K_r8{o{8z!5q>b zocnon9j7%%+e19UiZ?cbut_#Hfrm(f&Q+hUc!LX9_P2N_K1()y?v{gU;v9TYf2#%G z^CWxHacFSgL{Z%LPwTJ*aUr@|FL^lH!1cwyrsEzXTrZb^=~7cej*`s3pQK5*M~D1+ zDRZR-%xk!AQ`i4X<29A>E_=0QOWf`MD+n}D5PU!4@D|;wKLMs#0+)J-&QN~-4gZU~ z|8BDd9}#6NEd8JH@iYl;&%(F;)ji`(y^j0HXB4aWvoQGoqXdMTUk~M-=W7o;ZhSaI z0)M5t_(#9E4X#JZ=;`ZAianNXIQ;(678KYr7i~^$RTP%J*pJFjY<*XV3^vBCYi*n{ zx(V9Fch#dg9QlGcS9S`XdM_d(#w-yuJb&q89R_rSiUvD0;nc$-;mYCsYxF>VTZL`S zCR6Xi2x+G%DuV{J@E%%)kBQLkU>I7SM=yGI%Gh`M+%A#Zj2-;XKzs)mz|ufBc(3vX z(ELRhk2|agy9f5a2^c(Gm3?_8Vv&343iU6AMba5jS`v(??!5+dC)iHyy=yk-5;V8o zl3WPx<<>V%<}dGp7y{_UmmBA{*4*;P8DM3(cbrw7-%`pb9wqgam}CAcVOL<;Ob+q( z&3P4)H^W_`hab~l+-X%U(_-q}>o(DFe@TQ(-`1x49v|D6(t`NRhE=y0oa zY)OB28F`ejeWR`}jF>N_4T}A7tIWz_^LHvty9T$qPe5(HOcthP#;vaGpw8c^FXaXH z-qnLc|J!V^m1OVvmP!5h@l@n|AW{6kyP!v}TwC)U&#PeyrFoH;2jPa2nh`xwIWYdB zklJ|MJkXRLgXGQ7%d4>9BOqS4GYDHbCu)uosWLJv<6|LrjIMCIgVbiFq1)c%bL#!k zV31sN5tv11r@m+qJ<{rw;y(Hn_3)`_l0!3vTguQs=lAa-;;|z01sJnd4|pkcZ??K6 zjQvuvFJ};czFSCt5(-5~2|b8WHH1FPa*laFt9!b_Ma{70OJKN@MRYi&p;b!OO1DJH z6ZNmSgoGYTnOP9`;cUoGs?p(eL_<0sGkOT(1I<)HlP%m=hz1kB2AM@P=Df?8*>}*9 zU;<=XYWnpoTpFdGO^uxQG>nCkloZzuH?@OvvxIm}m1zq{vvxicT5beS>9K#%}ZBI^SO|~JwB#zOx9tg5E z#>HzzO%Tj5(dT!&`OS^=j#2}d^-Ct1Fda5{zmv4#h|=eV$LFiQMrcnHscIC$_aY+) zeagprFa2lzk(>E1>UKcEZJl(P@9|Ekx2HhJCGgzXi0fFN6>)LsZ+j@0pQz*RvWHss zh!2F!$qQG}jlXoRW!QTRiI3-`%w)zy4x}Nq2hjeDBelNmv}$7@ugy641jCZ49pRS& ztIrK7Zw>IHlwPFKvq(xxnzJp($lNhI#wt&MC2WVO?pf^k@OS%EoH&7Q3M+wOv7NXHWVD zDql?Sa!-SoW&7cLft@MJ*Z1}&y5l1eK2zH9rAwH9{KQ4PevOQ52Lp?sYU2-T>C>BK zNN$oLz;c>gc%sx1o-fW#Zm_Xkmb? zo_2>#{=R?Fg^N4qv}7gEFB};MS(09q1uJfm#29&wGX#nmPjS{U+aI+&w2}wffUInW zV-Tf%L&u4s4y|WFy23ufVK!as89g{RP85}fSfYty^}Dn#8>r3*D2a>2&%mnO0}9#3 zQ@@~eA$C(*KVrE%*7MU&Ki|!8|AOiu*oedvs_(QKY7GA=;`HE2yc|@oj;l+y2yHm9 z;xoSLC?%tgb$fX`6q!YP0Y=^1^!Z%b6$Ta|zMbP-v(3+DZolfh4%Wb`lJFyJL-Jal zUZ7Q*Q@t4y$e8UK1C*uHW$l0y&_*?c9QXvr53Da_?&m6T+_n8PWR`I!PbO%1% zBpJp~0P zP?5Ko9x7dZ)92>w8K@qdD(S{571zbJdO!OzZMe^8RKKF4In0t_1>b0%9O^`NZG%cY z&|G3*Yh0FT2Rde1M6%-_uro^g0ptGkAINq+YIb2NK*2`{>gvYOVo!G_0`e~cY{Z2m z``5STy5rvHA@#K)C+>Z`ui5Xzp@9b3OFe(vH4jAi;5E^wfg<^CG*rz;V!q>kEWg5u z>@d!6e$t@bTJ-q%m-G{#i=6e_$yBsUZJ3Yn3180*1#sMwvwd)g-*863XwghsEOl#_ z5&e3fCyXK0nz@hVcqoW>7JvVFcgOPdlhDj1!PQF-oL`n{?~^ECMFC3bA$JW&W5Sbj zX!`q`i=e7(DJ-!G)@A#72_ML(tk%4dpx zZ`gRrHFDXsBx9e>UhA!4%AGDt-muxEJ8dmyVmG428t$YH>|t}?QF;A0A=vj`QejKl z{@HhrGsUP=K2mcNSG{aBT4Fh&6+CP7gdnf&yuDB_frDxl6xmV~u|?u|Do9DH6>dhe zy=I^qyM=6#q8QyLZZS{4I;6i^cEhtzdhM|X5bv!W*n8pA)1R}7)a@a>ZhR``sX}y9 zA5hw+w^~H#P5ZF@zL1;FDjKJ1Cni{OD%z~dA&7?AqvJe=2u!tEH+(%Y9*05p0u`kO zi)W~Gft(UXA}hj#G;BMLM@Rd{G}|Z#a&9+j`s%B?=YdFL@eH3#+K}m$f^Q=(UBTWP zsNKx_ZTdg#)%Rtu#71j04wqin?)x;MW4*tM>w8yKGC@=*-E{h_e=z3)#$+*~3Qg@g zA%yxYX%43CCbZH@wV$U;tpyok)wOan2FvOe(w<-Qf>ut2!Db?u=Gn4wLHkX0yNlHi z#qJAakC>QI5E4zu-Y@pixs_TvdPSv}qj*&EXI=pJ$zy>q3ToTDlI0dq&G$de&*?yr zGt9TaU4G0(TTaUd^U8Ls4j+qyPoLKApo`&YpOY76>gl$x`z?R)s?CAq-x)=Vd*3`< zeeu1}0{LUl-L~?XxN-${-h=sj$r%wpx{uXVRr>aGcC_lt8HToaB4jO)sA>yUp%m26 zsU%JI@>;b@ za?Wi3Tzq=KN6}6g_9SNU#dX!YefIL4HFz6H(t_ZoLD!1*{eh?n#y2{;!k4O309oLtUKF zl;C>itVJf9^oL(JMg0f+RI(z@sFYMIlnqci6ntMF4ZBcW@^M|qSuVDT*gIzzDEi+} zZAuDLjmSBgp3^pb?&<9g<&XCcTHJT7WV>puurVu1sb$!FK(07f4uIA(DckdAblKE6 zYKl5Icmz4#e$(&f&URlp$Nq!<*BO8NP?-77R9;tLQ822R_^x!J$<_T$Ks&=_8%eR= zrK%zxv6m@a8)=I8WAjU$#7B1-xs@F7_gQ}D@C(|cbOz5rVU^!Sk9T_S#P~?J-S#>Z z)$xx)E#DT=UT;%)Z=z}Xrt>$ux@HlrXt+(9GO$+YC$5l2q< z-ea67ms2VhVw^FgHdxr_6ql7U`gi_ zzYPb81~6w~MZVR_^OYYrw=mHhX)U3j}izPk*X~ww&N_5|rl+mNp5iE!nnf(RiM{({(?J z7vm%Qa>kg_SDiz9O52tMW#nBID54JSFm~;zj8r^%-FtTWmp4B}t!nAApv!r5djWa% zoo9R_@aXZ7(t_aU<%Qu|aunrwL|)5*(`668`o1gbG z-7u3$w|LrjowKsPLoJ(ZbVr-t76v0LyW^XF==s+){aRqD8}8qEc7J`MmeU%O%<3}l z^X)sz0#-?%=A@T-!`j?w7yh6s^-0qyPL?HyHZfjk$>fq1d|fG5IOJcz=CClc)AX}* zi`dw2rqx`0@BIBT!$&py2Z6@*m~cv|i!^ha>lYsCc(VNZl277Gf3aH2^qr=A^_@j5 z@z3p-oS=L(*%M+{LBD)C^(56kaGm_rKJLpU3y;|~X}(ARd}#LUbnxqyd}L->Rx;^P z_m#EhK2q^XbjbWztmiv^tBhw;qT>_4ksqpzzBJZLzY49dNL_I9d!;tpb$$C$%q$Ze zs)z`elkW=UbsPFoc0|Sght?E>(z5cSf^q^3Sb_WMMG5X^(?ezK{sW85*xx&km)A=3 zSt8Dly(~S>dEb#qv$to$FR@qfL!PvFEB~83YIJTE)0sO=ZCZ8$OWLn`hHieosqa_J z&m{kw*}t6=O-FY;*|@PUWaey|J+=60;1~Kcc6?O~j2?$uZfo=?U(w__nX~EOm>o5F zW1jS`mzL5Ihb-uyINr?5%X3=0-^eJ;QB_Yh%J;*-{)cM&&6(x0a_M;A#CvWxcaCUk zXcJD@Nqb)$Mk>`PsOx}?f) zg z00;hWT~M;|D#sz`Miw!QH~`0l-iM{rP*_#9)w<%H(4h+cqow`dBRCgP&+%SRU@0@W z#j@|eA?q;ME>0^5Ik0cuThXAH4XeK)tGtmer^C(e7%(Uoc%vpJx%aOJ9#)?zSuP1KySf2M+x#s|7q3>>oQ;{ z!=_xn5yUNv?+ltIh2yFwsqV(~m3-$3qQKHxUHvRohG)FkA!OBKq zWeMfWb+5BJA>Ee0YhoBWpF92iD9vNzHyCJh~NyFilk zk-H6vVj7%{zj&b^PPDao>*G^=pRsR@iReb56GJ32qz8&JD+-*j=Kvm_>YRZlMP@ZB zAYu;i7$&`vXVCLChpVz*p6@Y;62G5gCYJvn8gln&lkr;DK9>$M^?l85_nF;gd`eKw zy$B8H~V|x4-*e4kxoS4J|xx(%OS#a ztEu<`dX1WgcS;|bm4FV&IeR!i?DJa=sqIK|JLoHlC@gC>`p}XtM#eaZOR{XllL4>V z1X@v}o9xws?{Isxb_q}r^62IyO)Ww}`|3Ah<|CNdx+Udxg7te?I6AuU~cGUdl@ z{&cTym~~u!jjl5)8No`3Cc{pDe_X|#+Kn~i0YVlml!FHTnk^x(2J=; zRpSi%lxp6Ef6~e{B#qRSR?8U3L(wabw(0!R-T# zHTr56Xd5o2J{pHOK>#@LmSO9GUe2Lke!LG+s|t`R5QL6zH46ptKqh62TgoxqjEGwP z<&z{i%pXN+g6`Oj<;(Z}eG>Ckjx<`nLc5Y6e@>xnZ%k2FUILS{_1dc?j;WqRIb)oJ zio8ygPHqCm3ReiJkZ6pvzkOX3M<&)e$qa(k;8e?Kv1uHhOV{o}JBXTA6q5!yjLPXDP6yn3>>L9CbEg=cnZeS zCh^GL`=S4wVpGL+8MJ*C1>>h(+Z^{tWf6qNkm2zAOWwiAMMc@fZ%eN1GdChL{3sA z!eHRV2lM*Y;Dotx{7^v0Qelf=yOI0f_g$@KF5{pLn~L#!WS1X9li(BiMXvZ4qaH7d z@+b~CRXq4Rc;u27neJxWu76kU!x7h2xLebQG3xA=yE9I%1TOh1TRrPhN!`nbEW4KQ znW_e+FKKwAveV-4E{N6y;YWP+Y7;<+0+sEcA?<~w73rUtW(}Bp;hg%VB{ecw@NIR8 z6yI`pGK0m(^-!g`=Z~@}xM~lF%wmVe3@_3j0ZLPgVlF!17jfnMO7#zXeaS9gzSe(K z$}||{q&{uOmX2;5pEsQO?$?XNdd07nHC#>rDFGJp zQ^!v*=&F0=rCVn?A+EHyz$R-L4O;;af8>d@rTjobG=h?B@F3(%{Me3)wjMpayy&BQ z_DE8y4v&kK>|SKjFcifkvGmoCwN8y@jwSj|FsZP8)fp0O;fV(}`Wau`gU&-a(;@yf z%>`vzJA&ccQJiX%cT4rG*ZBII*-{#qTfYo(|O!GSu@%Cl^ zC&kl05B0=8h3jEY^ae1}M8P0lzM;!BJA;Hys9(sRKlz+ z?&1Irb}PZ5p{lj*cKZvkGt!%mepc9c(FX`>5=w&F1E82iZ11@fe|ju{E#qBBCQBLd z^R>P3is;K@^e_UD4;rG)t>qWYXeWxz z9Cz7??Y}gGxVs%HY(dG6*#r9|&IKlIsx~!x*Wm2MXU8TvLKtkkXSbia+ZkweAa%*& z1;zYDczPbR z!LxJZOWhH>wIbGSB4Tvj{eaTnGzPN) z-PS&R0`H5u`vw~lBP^01sC|Q<&RqCE!(bO954%^?mbAvk)=5*+m~(^y9Z}eSVb&{~79<=wxrS^Z#E_0V5ay diff --git a/docs/extend/images/authz_deny.png b/docs/extend/images/authz_deny.png deleted file mode 100644 index fa4a48584abb3db280b8226d18888cb0539de89d..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 27099 zcmdSBbySsY*Dnf)5~8$(2q@hlA|NS9OM`Tygn}SQH&QM-Rir~Y1VN-Fln`m8B_*Xh z_Po{i+k1awpYz`y`w4>|{)(SmnBm_`vP$p%`-!N8Fx9lYaf;>N>#OjT;s1Lx zpD;Aij7t}k<^H~tPiU0qzn?H4W4u7EAvq-JEXGqk>Gi z80oWdry&CguGZU=9&|Q?1@>}#Eo4UJn`31bFJBtT>(^}eDrlWvy=|))!rnq0)XyxV z#^yfhxo7oI=l#R`) z?Q5OqQKswG{N}J(#MdY~ndle|sWyh7U|Z2E$(U+M4Nq8->25k+U*8!r>iF^gk--?F zax(is{sU)wh4z*t!Dr@;e&^*vC^rS(87+K!>m>T|*}`QjuL*%nbHTg1a*WjRn8Wo= zWjY>LLPL9)(!XCwHuYiDHb(!WgPf^pdgtp~3ew?JEU$PiaG%WQCb|V(zJ}?Y z%kfh!ynnvoSW?pGgpE&M@-1tQn&kaqg+nZS>R1}DeTg8;v8qzxgV~tF{&#|c2Qh#Feh32>)UpQAunIX$j7lRR*`Y)mte)jbLcG{N6trxoib-BB*ddA zZpyEYR5%GJ+i<eGv#=TI>R^!uR?-PY7+XV6U;q3cY^EtGN4WB4W4;CBQ9h_{XoH=eh ztFLn1%Jx`q#B%suyN5d{-It@x^#YfaD6`&m>*3y%!@;J7>)yEYd7IT=M#et;E$=1` z?yB0w1c`ZGG>Pf$zE3SMExNbhRo3>?6_-FIR)da!hNf6b>g@hoALX|;p#k}hj zsJlh>lG>gItLLgQ#3d$LzVKctZF3x6`SCvg@sBc{jcUim^v~N%104=GEp)4%rp-O% z3Ob=Oq3ww0P^aKEe~Kq&(N1#h8yS}J+;_q_HQ)ZB^_tZWqZ$XBb50+}Ix3ADF=EPR zHr9WB&VyS}+0UjB&(3N$Q8RLFLHaVa@H<;rmtK>6K5Eh@Ap6-XU;uLvjtx8giyf;%WV|V4p z3=s!GOO^eM^gyk<^YZnZmfvZ2PWC4ApDlb_SjK^*s)5XVXdyn+3}WE z(iIL}?Slv!u?dH6*imk9_XgT%ybJO)^B;a3{rm>&La&TVK+DSpWwDdYzU|D&a4Ny{ zy5miS1hFIk&2M*pmkSIO8R#v%3|fAa;+Z{Ippy}ZPh%U`Qlq^;J(WwvY@E7Nu@#U$ zQf?JfwD7AZqmWlZ?kL*(bSp(`e`z3}tp>X@Swt_6U8mQflC4#%P}j^VQdW>8+n}Lchp?sL2MlR508JOpRC}c z@J065#>wDF24P@VWQ#Mw>x2Ki54K z#z)_hdc1t&dqhX{>b+`}&aK+C_k41;OAsFCS~q>vboYm`W5_AS0hC-)k{knEQr67R z{=O_Xj~BY$eFza^WL*+!%xo(9al#*-E;20(R$dKc-^8SzQQko%gAt~(vC|rUn`EQZvXD#_9ETz zLZ$O>!)x=0JIh+b;>yu_lcA(+x;my|{p-z#uWmPvOuxGM$T3p4!g|q3pP1UlwsO+z z)5rcc*gV*S>$Jnu{uQiS36YVh- zG3~JM*4a8Cj_rXwlp#!;p8}@iKfcYtiT4x39;AE!sM|Jc`&ULpJG=@PoiysZO+;Fe zv)WO7{fY3$lMal}F_DC+n$};w3o&`@ENMP*wZ7ZH-5S%-Z~>n{Saz(%z=TnTaLUAj zj9Dhw^;fchebZ*3Z_uaVak#&WYpa_+^(}q==cuWS9p>6eS1*1}4WjWHW}x477lo6$ zxz#PEx5mBFK!XZNRg7W%T3CE^!Q*pZZP!@Gqua)QJt3bOroJ+J?KggD!#)iCBoV*y zeyFYLKwY}VQoQuZ=!g4PLz>&#^rX2-Sv3gAtz3y~NIKu%d5~({g~hmD_ChqQ9#DXK zwsHz*p`cn|j@o30Y$|~+ zDzAGaKRsWatr#+V77|W&ky)Pk9!lE;9oI&an?t88^YU`O5+_cz0exlN?sEr)k!1_0 zJnY`ujGr!k*c@ii`|xW*L2Z>!Im=#Q zs-gwDx1Wq;7A9;&Com{nxrXPt`|43L;kJ?cIF!A$XuKwqSj$5f=H<6Zb#r!N%C=_? zzMo%45=;g2&}-K+S*J*^_GAR{%Q;8zn13MUH6%^q5x;>ITlXnB_h5JPChuKwg< z*xJhCaGXEI$&Zs05iXTfpIBB+`RBhUI1*V~u2#GX_4iBWvx<(>I62%YeD=(gE6Dzt zkm)FTw(9j^swa;urjMchm3g0ec|uuNrVw&`w^1gox{FmW;_WQ}*FehzL%5Ouf-qmi zwL94|+Agw^8DvL=g=-mJ-tgj zLc@hV&u|f!qHo?(=J81h+oAE$o$qxP#qVoVevFTItLJ-vOzgbNJT?0UOifAH}tjQ$>^Zkpl^1>I*sRclu~zd%V`C)uJ8Iw zbh}p5=+?7ZQUe9o=-DVacZbk(g-!|Knk9k;8j$W~kJIhKGI? z^_O}R+k_5Zd|CSv%eNj6t1S2?@L&&$l|JZA(64b( zjhH9AY>^~jpB~GiF3o+&$Hkx6dV4jbXXeiDnv5gi!R7T{|L4th(sU8Ro52BP6T21n zy$|DA9;CMQFB$p9-+cHvXreAia2N+g&eK%Zzx?BU&UnY1$G0RwErG;UNhpt*dtMX@ zV$T9t#CEx+M^?Ytj45L-*UGl2?I#bdwB;~7H8h`K__XPH_b>}Pcw zMKJZ->L<;WH!kPet_o()u2s!kCbJQ1>lAS@KE#VpD?w{vK7WBIG1Kbhs>wLSjM{y93*9M|b*!-@mvc3Z3Q{hvqwMo5h68iSuK_GVR& zj~|#`!l#o$;cjGIE59l2n&=kLcyTFN-}ibf53QelOr_UyG8x|Ub&)B?`>$nxHob5X z+P~W)i7%3wqr)T5=XXnJ`*-Dpn}YF}KNmw$(P|~9kpr$~?YULG>=j>XI27k~ zziD4exo~(}i>=!&1`TJQ5$)cS!SFY?Y_j8Ge{17OU_>eS*r1|DpR)_%1fb9cZ<$a% zTN@LsEo?I1-tJR=qfRYHYmM`?nY`kDM`)CIFHXbv*1D9-ECcUUB>^7I^=Pv%3i2@G z1kj)t@dO&ZDUqU&T*55j3=>JyUhwy89tv9@P1g3$h!ig;h8i}am$*ashfiV$qZyrH zG_^GRWrICRKB52nqoiiyRpg0(m@DlFjtY;o`^EQOf1ivFL=+F3>>PX`AfQrr&{OtyoG#~qw_H8&yz8w;C*QI@f82?QG!({7#X!Mbj1EX z*$12cVCdF5+r<70CwWV3JPO`so2J>{q{>n*ht_6BamF<3_kvIBu<0xD59P(wPGXJ= zHIBbCrUhpIINDies`FxuIhOn^nx(dr6XqHB!n1+w`k^}=0ZQ~2$BFE&LcHnr&U&~x zcdpKJ;dPJF9VSKVL0?yQCIZy%MQO?j1-VH}#)i|Gv1)xY$A^+Mf7e7#4>KOf zPb)y2YWn}*2MOXsH-?P7;}(+LmD&@yxBx4~QwciAj#oQbOgD!(Y|Yxdl*X?2ft^Gy$L00sz)O8&4ZX)%fY{2+zuvHDkBtJ* zul3={X#Dh?X<)9*)XyNALC>F2GSoP_*M-M#a_D{NKcEux8ha$_wA5dEn6sFZ9PW>W zKWOE;G=!ZZ>gk5iSYasqyk@O<2_EAqES$*_HL_nHX%f7=Ml3geMX4g7_o&tBpbob!CPBl?IRJIVMv7!yhU{3EsY(v- zslB8@xz8uNuWh*fE(W?(WLV2ear2=gOLnc%=b6^9s}9r6I0uV4Dch4p2A^iTZcjRf ze11vJm>SJ|74D=Ke4qsmlV1enzQ*N7+`Ug(cS=k?J9?7~JWUn7Ov(3!<=y0((q5-ogh=+5 ztfJ!f9S&u5nF#8_K9&bXNkY%&QJ1F+jp!Pp85A@E@hCI^ZV#kM202pgJqpdK@%tIC z*3Ftz7Fc#BFh{|%W^yyG+VkkSE1+Zq)|ELfs^w}GzV29-2hKY?YF9tdmp9WEk?=VP z|I_}Pmq2agLAp8L=My_c(D17U`0wUYesO!tOR^S4_47HS-P)s!zkQiRJBURqo1xgg{nM_R4901+BFT2b5E$c zuT0k0?@USPQTzZ>>-g)tgmQ(&mq-aQ+2w)!oSfSG~dF3tTp@^oAbC+Lbt#!nhU4+_Z;DndM7@6X8@(w?mX}BjxNx7k2`HQUagss z|FG*dV`7z9{xB8?U#=Gse`pUgb-}1BF}bh!S5i*>VLk2+{M*bVS4mi@LqS>U9n!v9 zpqddlm0fmmu%CdsZEwO|dBSa(1f3!D!S~G}uS>E3YjFHMVSu0qircnH zW6qPFeprbH$5!s{`<5bYmV}!T-p7(OX^ynSjEdjeGLN?wk{cDe5%H7z(Dw=YAwg_A zkj!iEn{`c&c2{@shzwmiFXo$J;!^|N=qb7vO0qF^HIev`yrP^l;O1?dj zNAFwJ+gJRP-(~vgsV#W;i-^7DG7_y7eY&$W;1fDqNBtcHwYQfCDXtrQdOk3OhzSbS zTk~CgYpbJ`m1AAHZGFmOPd@(qbBIhnIsdl(c&9Ky8$H^78eh-52~!Qz$cBfp5JV!& zE0Z2uVb865GeZpBk3nixB=9_*e#zx1aHqq#{2uUD$LdRzd@8r?CdxNK`^tH4^wA+} zZm`Dn+3Id17LB&8z`_1ZM1?yPbB;ank%`b)RLtMnQQWk%JfsOHRui^ttWISc<+Ib{ zgB{(ZBcarzlSjRu4=LKQilmd)C_dN0wp$we_>8fN^5#R7QU>g~_%}y#pr|+{y8o_y zU{L4rV_ffGX$SN=~)Yvh^P2Eo<^ebEgHo zR4aRU818PT+9C8a^du)t6fHRU?@Vr*Hvd9XH{i~=9N!>xNxd9l_r{dpImr!-Gq>OZ zEcK*~<7cRSGM2~kV;R&YGh21uUC{}{Q12y|8-vAgT5`Ju7_$;b_kU<$?%~wT)96Wf z+D7sr3RHw8b<~aPiC6SWjD5Xq+BEVElD$t|MK(`1+Guu2(#E+4N2%#Nv@=c1Bcg;b zZ8|}f?R#P=9rOHQqfn40^Fs!dG5>{W(X!j8i55*L^ufE2x|2O$GUVX|%_Q=eZ8MY5 zsUDkP1`*!KcDW9BhN$3xLB?1_%=^@1yplW48q=u>6<1HR?D^eDDGe{7+MW9Gt;8@Qsq<<~&o!ReCg2UPVQO{sLClcnNEH=>o7EqvUA zeJiU^#AIr;ZQL{VxkaqVgn#icgw-~4JK(&~c&F0Hpu=3_xY#?sIUOo~wD-F*meSB` ze6))p?X0$c#OUz`Mlz_`5@#(hiL=jQ*sk(F{(drE7jD@bz>~I^(b~26%P>KE*3G;-&a;p@Zsv-B=+T7D7P+^fU1(Ir^m%&t)^rzDa#C`NL9!fd`O=#Yun8;l{5 zR9#rO#tjRciR0_DYt)yAO2@E7He@eVmZ+k%&5_gN0iviODV@+hv=1W+XsvkU+zo6u zx3XVUjzckazoei%iMdYOgne-^EwQ@Ul3+Ek$2fvoWZO|MOFpjr78l+nV;^JstWgY$ zfGWL|*PDq6*E=;-P9xCbNx47!^;|;5kDcc#;)W0Cv58`@tu)5`wf&!w zbzW|$-h0Va7YUossGjy!&n4=^VN;XlzJ+t+qYVLkA+GzB*U{>2GHveMz*nNX)Ta5E(u{SO zeNVY58>rd$dKjotT=mCgim!NEes?wO9(QUa_l>5BJkCs!q?k70bNr=JJ0IA+D}l1V zFw)7%E$F>kF;pJ&LzJ-nL&?IEZyTmB>^w{h2?NQV9{}OndDQT|du)hx^{43~m9#ooS9CdYF|yLV5$e*$uN`3H)IUxQZl?=>T7 z-|p+lD8`xMQf63_pQG^QI2V7FIZBfDu{26NkZaQT5}{pDVtb>s!5C^^i4M=4+rXbo z(__0w#*4Uhlq=i7xHca2Njo>it+bix`vk3+a;`D=KGJe8>3~r&@rdkF20k<4`)>hJ zSET!0Mqj9skX)w@F4=sM>bV||CwWydw`l}_nVnqKSQqEb@6|Wik^@(&xv5W0NM4c^ zfK{~=aj7Al^aOkIv{{l}@Mu%moUBKQ+gQTgY`%5$1g$0D{+@AA*M7-m=})_heKMaU zb$q!7pN^A0aQIEZo?&brnn1lZJ%f647@%U3aiJ@KVrGFf4}bzg#-FEfU|S;Zgm zjTNOr&@kl2g^BV+xQE|-c>42h?OCoyt{SUZD~c5WT%KXA8}u!?N{@2`o)J+;6SV#K z?!|^~iwv5e=+~0v%SPSE1EJCy8YAbx?W5qaJ!=yhvL#TkJrma9&D8`QW>tpI-Gcf_ zRzIu1|9LQl+c&mxW$$kFZhcjifM%VC>kE;jC8WBWuHd6L5fKr|E^T`8Y}cCy zzpu4Yi*aK;_~cUkC~26^5V0mk1bz<|zEPNzqDVSj*Y`>ET3x)_AmkI|z1B^FiIW+Q zs9rm*@6ji(c+E!1@I8pFu%Btw3SaAHWxA}?O~aN(pPVX8(v8{MAjoof)h!J@d*yG`u7_JWsR2g#+igB!%cNnB_pkl<*)ep&j+@MNcWK2E5GLZDv+}jb<#%R zJzT&eL>48AFnaLr1mL1-u(q1NH5(T31`jG5bs5zqP@!g^j^^89O~n?|CMn&I&a<^NR^rWCT-#XsPZr?+ z^@;!2AEeb0yE9O1B(@N{NbGHJsmNJJEm|4%%)XHF>Kyp}|D=Q+EM6*n)UHf3HwXO*Vu2b#|U7EB!jiC1oh za*F6kEjUJg_*dKy*}!Gr-;}ul{=P#94H+3zh4n!6`HNS^qMhjl`U`Z_`RyjCaPC=9 zAYqAy7Z9Ad7@ldMm?WUO`+KtZ3_vI&zVNz_Q|-ep%8fZYhZE=ug)SWfgWHmmXK z3VKFHd4#{xoryd`V$6KX zeyTcv`px&&#xX1gGv$aybX!KKv%NtId6#2~Pc6(`s9O<@7(^gHv%J0I91k71S3R23 z79g*aB5~*yD{*^jK?kWktwL7Mquqgxh0a$QadWUA=fG{Tmm?+v8M?kD?DC(g!-;8o zF~Kk3nIN~D8&E@o(EYJz-5>R_X3~KlMROVmz3qQ-kt|c(7lYTTm%i~`pB%T-%cvPB zwG5Jhc$=+P?S7?#O&7m5{;31}ZiHVuv|V$??hP_7-v^c>z+m;UJ4I}gr)#3ty`v{X zX6Icw$UwXR4dmVo*6$TWh#tzSh$;%I|G7Y8e=g7sc+2?BxsFc(ozV<0JDj*Vr9mml zq35Xx*{)@giZKF}V|x)^a1Ka1hS_yGP;9=d=OqrZt7Qh;54J(}<&*v>7x#VKjrfPq zuFRMEbKNxV7&o993_4G^$_PDQu~crKIT$R|%X(03h{z|bG7$m4WHIO?Ut=D-w^L7qh8nqr|NaGDv(c$dh2SJV?8HUeNQfd?$!SOS__pclUSw+4Jimi<-d{b8xx3 zHg5;soahZGC2Yr?3tar+b#f>Nno2LpQ3{6P3C$qmte8K{_}8TktQyKn4%i0 zTjmvCzIJxzEG-;0+R5vZfnPu2yeG3`m4_7>s_?^$c5u<*+Rz4Yu=LWzgHVQLp^*F*HI& z(R$uM1-J)lem1l1!+9ki`JT7f#ZU;<-oT$)SBm2;C})v>4(rZ~5hn+s8*ipHWH9}e z(P4=<2B+?FyBh!iDQb}Wzu3GF=CX$f(t#d|a5A(2e$C~Z_@j*+^e)$1%V3dC`Ct5# zi^G<|pyCNM%a?L}^^PQ?##cv#DEKK{IKl}ZcF{jcHu0yho(l;~>VJ)iKab{q_W|!V z#;h&;1>b_oEMX|0B(84=(yYXZ;CM2kzYf{#Rp7$$8hHpby!C6bS=Z}Z76286Rsi8T zQiA!j$5vGDyW&bE@Kk)k{&3sMJHs81ER;{=p_I&@ZGR6IAs67i*vs&W|H*6Nt$v_xy$24n zUyx`;92CKeUWV;@f`y8%cM&V*r<8b$4T?(Y1aoRs&5 z`ynsY50NzX%Iy1Z^5BCmEn}R=eFyc&Fa=!KNuiY(Wzk%vB!M)r0J7oQWq_QU2L@*b z?iKcT6JAG8oFE5LVb#Z|nJG)S11Fu8xENaZ63F(fJUpr(ENXWp^322wBaL`Vz-cM= z8{LBaY%6Tg3K^dS@b|Eq98gl zDICe^H8L08eSzyX3waptug3>l|LHQaAc$u{8lxBTRE@uSu{T?V9!=D^W&sf@1LbdK zZexGkc@igEt5h_Uuhh26VNPDefnC3<CpGp*zNbB(4BL+8pFasQY-s$t~g` z zo)>tnlX4l3g@!G*1K#)|c6OlL(dFf5bhDS z>y&Z{oNqx+?$dV@M*1|MmZfQD5H^-=VhNYC6Q{wDCaCV56~|1a(q`qOm$r* z>pbJtQ|F|lj1F`|g1+Kp=b31{IJE~0p#6c7XI4(u=S&y&3&0UlpvIqITKJiCG=V3? z2sf>Q`~^lm5;iN&m*BaZD_)|M$fC~ZX?t70k@*WvWA%0E$W}j6`w~%{IZLDZ?bvvA zYL(CuwTg?7-?$ zCws)-)IBG>J8Z@a$-G6_-{o3SJ2!maF#!5D?lYA-uW;YF@g2@!hKi}Zz5OPrkjsSh zv7I277w*V5FZX7>tbv#augCt=J#+n{()oS(Fdc||eIA!26470khKji$jPgN2K$%R* z)mZm=Px?Jxn?d$Mv=p}`77_UZ-HJS6=ukrVs4F~v#|H~3JG%7jX4jRhOwEJ&HZXix z`oI&@INd4sUKCTdcu86bRjM5h_x3MAt)))4XmLIQ*>j^AmhXZqg&N4G|I&@$9ONfW z!D!h}GSX~Bk5=BhS(6?au&^l5>K>}JtC`u-HW1Psq2mNZiUWc&2a))Cr>`7Tc1)xN zm4xt$K+d^A_cs8#$DBwC8rvBP_0NgB>?txB+}M6;#4iw=+3!B|wPC4OhF@^RS>`(u zlp!WewONyh4jp0z*bJ(cKjIgqB#Y_Qi*=o5XO`kp4?HdNW-Hz zm?<3>of{8t3{tQ_8H|#ImvtCn%~r5}5DM-eiu_86>er}*Uc_=$W(zFaxrqEp+ zNh%5Qr7?Qsd?w|?zC91y*A+Zz))aUER_tRnuoSHZ^5V9C^+=vI$qOXh9_eVc%L21L zsK^T%Z#aj#5~D8`jmP)p;8O71j=4Q}DQ*hkL^c#+T6#QVBd!j>@jcjDSf-J0wY#VG z2LNZm2hAq2nkVK)e>|1rfJc6n4$uXC4BQ`HbY3@092LR9crJCgnpJUVq%(zzzla`U zzN7kFAfZh~6zl*OSaqVIM@@va&C*J4&>yoY1^#lsl2Bn^4CDD6Hj@XM&!3*t$Ot?A zak?T`CML?E`8N->n3r&-D~~-9t?w{>gD0x=#~Pu54RhN$tjk{@fqJ8wnq3%fuL}_b z8H7h)WNk{IwQu-hQLht*PW(Yn0)R{!UhzNyLVyAZ02tJCtHl8Fp?okgkRXD<&DOtJ z{a{mMnPXK2lz*Q*1qJKeqgF8ck#498N&-*l)dbCdS&1b008^dh{(l1wp~y|MpDDBc z`($4PC?!#tA#VyehzAzIY^$#o_V>vM*73b*hWD3RGb10lC9ESaN<08l7e*BX}Y1ax5;18R%C?Phym9rczE4Amp*IkdUL*fU6>TFMcsl#8=pLQx@vS zjuC`<8KAQaI)OLTl2D(UDxwb+)M9p&t_1O+WXiZ!Ai8(P|u?x2vLDQu_Pvvf!uxt5rh>XJX!`5DsaU+aUSnOxy_Ig_jEJOAbHep7|OuRP-1HA z=a3F(dr(Aj{A=^7UCniH84^H-SAf=M0WcCtIDVW7zk|6bnMw~Vb0#)(P83wpHa^ul z@Zy$K7a^Ey{_A@h2E&!d8^|SkoM0t_$c3cLNkfQ>p?`ZTJw3!(3Dk5fp zEU{ch6Im`gW#;xDFSoWpQhO(sKAS(P?)D4IO;AY_MPX2&m`cb|)|lOV7UB9cBozLXq8}dbUvHw`b2L5q^uPo&mUopg-4#+aR?c4S=jTdatsG!v5m;3dBn@_ z(OGds$VCc!BMJiO!K&qJCNB3e&Va98-El6Wr>Fm=n1zPM_TyHIpAy!zb-69f_?l2HZwaYT0=v_TR3J^%F~5-6Pp`aqRI#R4jB&H z5Vi6plFjH$7fp$!T`Lg#Dn zJARpP!X8CJRaweODXE*Kk@S&5nCqo}vP7R8P?-+z3x3}7QP2H3K+WEDhP_`VPtQ!W z9)y-NbCTA~y@n|ex1WFm=%f9o{LH%D};%Mi^b3%nbpNiuFS_w-LRWA9DgTMW;DZb$x zzA!CIA^h(1hdU+4f4duPWN&*^Xw*sg-*&cwlbPv|X^MBW)_?)m0veu=xI9bSXfOy1 zf7)zQ92)zJ@hk}EY<{7O_%d8-mIdMZwTZ8F*`|v8cNzcmS9$_m3}otTwmoh+KJ(`X z_dQ#kIuAk884kx^|DI5R*&0FfTl;oaEVRSc9x*{`NRM6?Y5hAF7c7Ld@1e{Bg1_w> z!Dfs>)H8pVg!#4s0zca_)BedNDL88XxqhZ^#6h>x7v-e>pQ>;(e_6BBVh=a z%kyyC0Vcv^A!O*d|GVx;iTluFF)IPnOC=o|Yd>Av9|&yXH4}Eb+z1l$S?+slytZQk z63Gx|8KC(j&7|9Q$z%3y*A8=12(D$c2stfP zWjQWes4d&u$g}*u+mg81`3lrI@c^O*4d!f4WcXl+l*jsWY>ELyPO`bu8sy{It@e6{ zK&9`?5K;U(G6$KacmxCf6PFLEhW4+5?Sm*;FeAkT$0-gd>0-}~Y-6#bm78x-x5eB= zcb}Kt$kb$$@Ncq|A56D1NvYK>L>(f@f z*WAVqd06e%?G~hbFO&sL6-}l(p|;TSW0za?Nk3m3%ME*)D!|=N#bX+9`hjV1TPa19 z1LiN&^y|G0u1*^lazh*KO}}?hgc51VwA8T#B$J3zT`6I5?B_^$rByWXZi`^u)uq{m zmsOR*rFFr#)CzLa3qZfFAQ3er;jR$=uO4lnJiEYg-^c=Fr;H-H-J98=@5m$f_g%oI zt0#YFwXLCw+3HS0BrD`AeiT`}Ak~VZ7yv=&+Qo9vdpfV@!T6_+2>|dc>!G5;vPa!T zz!{4`tVxI1TkeVX!5o<;=oPC-Nw@{Xn{5K zavq3TtPpR@gGa5wNfx5+b=rJSbUGQ5QzkSQ!i-4mJD^|ZSofsA0ROz25(Yz*oaNKTc6O4#b=Mjhak7qjREIPRf;7w$1Nzr5=vpmp4F>=5A7l6fbUHK4zx>A%;Xhr4 z9o7HR;0Vt$Hb?!d`CWi^GCn%#-2VGyAB4c5P(ZDb-je?+eAXySm~}uF4vF9WWB;;wiu?D=;C}$F!dj zi6-;khoKw)I3iH`Q%%ij{t`u6^s8X+UByfO+giYJraSu6_n`F+L)RHGUXT6P`e42r z>+f1+|1ZsV)BWjS1t}LH$@~_Yrvl6ma{_aFxe{8s4Yz;=PzujI`kE~Vl+0Lvciu8m(^DtHCLyLc=CT(Rgt^INA8;>p6UnJ~uc03inj2r14XacGzp z{)$ALF7^OJ%7pNNo78_)>D|%yk0JmIsv}b~Na&Y@MNLY`d5s-HUf8_vA*1$6(hpFg-&`7 z-1B1CEM(_yg(b6wPW|}J4je@lPQ4ROB#)W&pX|wPeLPA&*>T80kDNfO$}?{CTLNuc z3uvAuoLDs=MXHltM}>zl*-N^)+Lsei4P(>tii#TGFls^HQ~^Gvg2cDMv=~|hm~r3Y zYjjI>9fSHC|4LzSesi}x=HebWdI7U>`gjVj?N|o{yk~B_#KfPo4|qtN`xr3%VFyhu z&k2vM`>WwXsa{C5bms_Kpa#^30Wd{kD|tB~5)ZSrYS+ZP^bMe#rWM?O&DUe)wso!> zMlIffyLau%5b$g@5EoP-;r9S^M{5xBm%weX2HwvKPtWr{J2mW>R+FG1t2>O0__ZBw zf-#XIs)`X9%Ckw?J1Tieo(GBdK$qeqeWr#&#{ZemqLCDhA2jKxC(1oiTAJqf$7`VH76Rls=!hfvcAX6VaTD5#&5~ z#~~^lZ^~zHfRTGvOpavfYa1@<4;AV?I^Pw?mT(9${2B*eF+*IW6o0TC&$At?GHKbH z4?0$%!6x|)cT_WuRnt)jR9p?vT`PMV<&aFB>2_-egI~l&JI^mN8g6X@zN2?VZde@B z6DVT9%_LR)^j5@OOIh$RX1CO=ZM*z>lWE}wfC9I}Z_Ty#Gl9nP1K8R8VGY3xAV42( z=H?_v!(ClssYj%F@TC>Gi)Fkwn;nY{I(NW-L0qNhT%NUdlcF760TonZO2z@0a&LCe zw3Ny$Iv&38xw^Dah=S7?ldGQ7jwDF?Z9sCFd+R(wsQA)tOCd$HHrX2HM$^B20_(@& z^w?G8lE+j7I*VaVmFDH7SZ39`M(%7c6)+7W#e<v=IB?C5RBx!ld$tX+}my+}I4AqcY(!B|tRvFmAC4TM4(_wAb}9Rn128(luO@ia(P z3BsaTl0laTvGLHo_J3mifN!B79rTJ$5(YJ%?vC>s3He|Hwj~U;@tQVc`5;Wt)6saSA&dhRyKVnUbK6@hg!7&=!>L^&4l2BoMb~Rh6OKwS z1_#}W)O8B{j_m!a)nsW|FnejEGX_ya5&vxUZ9eT-4VI;R-$e<>`*w6K@E2JwZ@j=( z+JrfKy-`v{Nn_ooFlyw+XEo;H$mru_SdcRDtXo8jR`I@Bz*vS1eg}sjQ#L*W)1}t1 z`xr^`VO`H7f_2^1Ls<;aYqU)aHo;#TaKAjh)z*fY?6E25{K)VQ0b&$Ry!mMI`NEUq z_n>y@)Vftn7%dUJ_(mUT3LAP2|shA~$n37lRY(jmrU-hqZyZHW;h0y!l$A#oi z46h)WGdV4d^$3HMO)K{2=O6{>sXLTUK{+}fU-x|3cwu<^y6<-)kc2D`cz5WDumiI! zh~7UeXTCvjj_samzwS?UQl+dwWxW!5TTE(?b%`ejzvdqsn1&JJi~i9gjGai5BgK6H z-<-p2onnzO1`E*bkZ2U+kL@3+1`nu*@@?THqVcb*3tm8lA)qH(4!Zd(GzyV&P}e0p z5^}*M8G{bLp~dKka}z@&++>iJ{W6Y}Afhd)60k*fJsv&qyjqy?uFW?|%+y0<1#0N}u5qNxNs)A!?a)q^VG`{yqm zyIuH?(L&d1Ol;xCa`&&=E*n*oh=cqAB3KvhK78``OJ_U)PHdwG{}JNFF-q^j0zQnq zsrU;U*Cjwb!J9WO2ae3^B%ver7#kQcR=;1JVg6 zil1YWLEmTH?+^Y%p`~mZQu6)d*$MSvge{Z*H+J2dqfC>q4CQ&r^Et5GIH(h{h&qkX z?8~RJn*X|kzu{nAMa(}3O7!gZ7GBg9HD&JtisxStv`Dd8!dQl9`rsN5&O3m_6@dIS=V-ZCqsHRzL zI4+OH1as28p*?y!`ZxCNe@;+)NtwW!ufv-C{K?P>f)8%Bf&(M>{$ZE!4gvIm)YL*SfxF@zLjk!YCmzwWO1hW|dJQqsGvBDJj0Br$J8<*sS<$3VYt z>nxhV+{gBx_5M8_O^M=)Y|?mD>4L3n{;%m4CVEI;rbR;;U8WIb2MpGGKm`svn<(lz9HAna&&)F5qF30fz$O#j zn|Fp7&sP`)<45yD?*^<5o$1!$qvT{zrjXuFtvdZErS5Qv4bjydLu%s~d{}vyk&2Va z>5G~uQq_i&Z7WPp}t)KRT@}nQ!04AMn-mWA4^tH}2zM-c`SSHWl(}(tC zkxzzL&AK$GN5U^E7M`H&K$gmRYclJ1TxmCHd%gpvRe3K+a&IGv%cg2Gs3jUOmq$8? z6E(D4TwiqQkxB~qH2HAHmicN9;CqP-L738=140r9p^qJ~YKH6Oi#99P_?wzy8m62@xQkEkty-vLwEWF)5mR(XhK(am6cY_tXT)% zs3~4<)(&-F>St-3=!D>mFzN)I6^zknTi*`f}h;%@*Y|}qi75q#Xme?XJx7reH zVUm*c$$%-6-!@GY?__QTYwimm(AseFkJ+T6&KO2&O*tIeL`1>2FnHd#27AJ6Ap3Vc zQq#gH9iu<00QwnDGs+Y*8jIS+r@iN_t}*hD^bmt-D$kStBA3dnEaiPK@~f%azKQB6 zTFJRz&S%ErfBK6c22U<|PrF7q&(M&L3G%D;m*SnQ(%;h@w(}#n|7gk0B)*ASC`+ph zDzm%+9i6}LDTBdNOfGSuolvYx>Tj>2^`-ZgO1fGiIqK@+`Pk=JvO|nPZEv8puDwBY z>LtO6C!wq@&32qu<{{H^$L)asQQddHQ~AgLm&|0oDcdm$;n<<z ztYh!&y;s>QTU!d*LK(^4gwONteO=$b;QPbp^MjwxIoEYN_v?PWo{xDLURdqz`QJXt zp2?&xJ{*U)b^d)@Ix&7J#r1sFx2rJh{1~rZ?@2SsV=W94gNDuZ_TzeHtdh>%myNm6 z!xbjH(9K19>kCiDJfxaM#j8YpLB9kz`%)PWksp8_6TqOI@6niS{)@_Eej{ zZcQnmwP{G(ZPBNw_4Y7(7V}TQ{|lc7uwsHN6#JxM^n_kJ4Ie;MDL@1xRP~c)ez)*8 zn*ny);j$D^wz!&b9ab^-mlDt>_YrsatJ|K)UVidtOY>N36E6ELh>mDL#5Rv;qYerv zVbJw~Gw-m+6(~Swz;mT$oVCOq4c4oT3oMm;C-7+gguhteACfG_j>Mz@+_G_Lq(vj8 zEvJAfR1pp!Nunjh9Perj_sc-PGv>hqqef>2#i3|wgf z*b)*XU95KC-3p~+;89oZbMh_lc*jfkglDJB`KlfFQIvK@ z%q+__?P5sCBq7J;=0d*!f;531Xl$vV;nh=XgYGry^6Nf-JS4sLtHNJ5BBUt|j5~s4 z$|U4xai5T~h|0FZ5~SGWgQ%{fmi5&cWejNNvAp`|rSba8wgvDLm03G=R+n*q1xN_| zi|GKz1bpTW(7dbq_fo+G2F*x`h}ul`(kv!`CZxXv<$pn5*gRJnzPBq#&r13YkZM>; zU)=cjNym$`EFY2|?QmwyuK-ueaeA`%IvoxAQ&Fp(qkjqu=_gPMUy=++dtUUg`u;I*$=w}!?d!YT5m+bu5?qx_K>ubudLPXe z=o5_$7Mh-^g7b9BMp^0YwJ=Uy{|NM&3Y@z;gXn^eb`}9GgFqj!+Y<=ruR8x85qb)` zPCjt=0WaA;A{-tVJdFbm$9K^&Ioo(EjZo=l3k8T-r-2`VOp>}vUbxc$sCfAZ&x!41 zLd3xG_gHP|6YVm4-~f#fw4E7`3`Lq!8_#@va6t2|HZ7wwoqnH%uv3=iQ)=I3o7vT= zRwEdfi!v_*;->AMhBF+k4`;f%pp&m&HN|}!tz*u|G`aVr4C=S82;%w2@`5v=9+SxW z(T`3nJ-B7>R0cWCTm zz@|tqjje&?>#al>>21Pco_UXGGYA-c+DUxgF4JR6{3Y->#bBuOf}IC3_ChqE8PSCh z6{!&vCwvOQ=KZg`yXynk$N}~rtf%|IG73v_h_)A{afE%tF!Q7k8tV;Rr9r5J& z+GW+~-mx{%KN$!L@`Pd=;9^WZW44Jqbgpm~+xy?k0nW2HM_?wB=P?KD%!7q=w-f~V z>%hr%JRDi9(<}uoZ`dCQf%}U`q1mjgURr6tO1Z~|mBWpAIC&fhn(@!LD=QxY+{jDs zez-%|@%Tcw_XCGNLhql^pDPRclp%d|QB~(3@;)KU-o<;|N;&&o#N-@`Y8whj_m;ty zYGgas2Uv(E2lJ~##J`@=^ZH&MKFWHXra+H&+mjl3^j!LuV3EHQ(Jq?sX4O2b{>$E7 zm^*_eReMx+50g*(XtHCdFW*Vl_7)ZAWm&C8!}@bEM_OH%mcc9a4xCmMW-L|HR7Q5F zrly3%{ zgkR!PJLS>`6*}vv*_(MFkHk{p>+OituLG^<>S3bvhP*t_5pE?uznem<;Qy}Df2*>7 zEqvheaF`d#a1ClCSRA~#ecwI4z6#eSau?pO?u*!UDNB<3@G;!21)llk*8KT7QiJzm zVSR$CkBlT*IS*bK3Ub`%T2_3e6tSb#-0eI*mv6l~(a1nFd>}<)ibd3oZv#>XTzwx~ zwi^n~52Ky>t_ui0}-pv>G!{?qOzlmV;%-cz6rV7hO3hZnQ z!xQsdy6QK`({ljQaLZLC*LCFh5O(}7hn(#CR`cql4$ZI`?=d81w2^$5e@1n&?52=g z=m2VrEVqM28oRhu2NbcbohcndJ8+nH>IvQvJJrjG{S*?onOvL&ED;|s<~czwBOQMu zsp)<5AJ2+i=wLo~txKU7beb@eze8BQKoGD79VP(-~6OPt;WJ zPr~?y%B-~$iy9`{m`$KQN;od(uxh&72XyX)*5wwRWWzWksU=Ex{LbEeLha(@U+wp4 zciVv=aX?pLw8%ufe@l1kk>J$4ND5{!qJGi_tGTLDbdV2u2=iyybgKW#K}xYyVf{_d z5oOt6;qERQ@Z4cPl4Y(0(#>6{Nio4VpN!d6+H$?}W{p4HY-+iCsg23-h z&@6uS`(La0KkVaH#NRUt%&xnvu{4RU2+z5`&{{obsF|bb= z3nc_bG&Rs~q$|8m~-etE+C*k176>Zd!=5li@NKu;qB$Yq<&QJgRTC!G9E$gtMU`~4t^|k47 zarNL5ephp;KV94x1q`G;cL!M+{ciCdQnek21{*l<{(Sldos*7f57_Pf5K(`0HjE^7 zwSzo$!yFu-h*IVr{1VTsY^I5~;>VPTfl|jC8Ug9~p7$g`7Y*&ihR6A$R{aO51fo5r zzMo_Bi0Ny^Q#Es8ZBGuNK8qjQ^psP_6H_H_58Z9pQz!z?*8%-r7w9@&$m7Ae75C(T zu;hq^W3N@dh;`Sr6$&6q6ho=80(vp*w{(C49s*2aQsS^n)wnj)DT5wFIB@g~GIT(W zCcBF05$vC6pSO+c{|(21qiaE~E50C_m3SR&p7)fH_U70Dkb!7n2us&g4vb3{Aq7hr z+PEQW&vxdh1gX&oi=C(wJQ~(3GceB&jRQ~3vh$qHC9l&>Fy!moxOn}u1xrvL`cHFC zGAg8vv+NzBQscM&wzLR7?AyB)107xCxVXMK;2GR9ZyUIrf*XM5v5$v@@JAj$K0(b^WDx*A6(#xvTZ!}PbI zsvzr@!Nb4-pAKb?&Z;19C-d5fB^wd@$LBAEd*X70W;3Ap(n({8<0wkxq{`6J70v!P z&5aT!#!zt7#imRLi{-?8OZlt*w}U(3fAx$04~|(l+CFKhNaOv;#QyHOY`yF6 zXiJjF@d7&ctP68U@Rt9Y@_{;eex^{1jD+PsDneOK5qu3~|Nngr&uIN23q>kpTm=}a z>Z-oPMJq@)YeAK~Nev>mJ%Iw_+Y%SWa^5NS(B1pk(}@@`{H*a{vyFxEP8E_@(nMLM z)A+GFDiW_<%>Pn6DF0|`KMUXx9hJ9261pWVa0!W4(z%K4Fst(ezxeXkH&C3BIX(4r zN#Pl=;Ud2g;3jSTiHXOkt_=T1T0+OE#vgUo4hrZiMgShdF8~nFcbJ9)_#JH$C%`ki zl_W7cJp@qzlAqdSg&Tg>M6(UM{s17n?fL`2 zQ`z?KuL+LZiESx_$iskC!!8WK7|qb(0NSp)7^PRth=7Z>isHy2!r7XCVJD{Ao zHA%3@c#VWD%Oo`U9Aj-qsU*p1IK{gx9p6P9h?rScNaDPFu+(3Stb?O0WX97 z0NXtQ(|aQf(vdIXokMeA@hgh@1I=C1$Kt8>Z3Z^UtMIm8(6O(LSpS~%%bvDi0Zr;w zbv54H68G*dg>JA%Ukf6}ARGV;U_#T;YO^jdrEamIp`mvab%4tr1kHO8$WmRny*glQ zuSHT^y727}=G+P&got|i;oe#$yr}^3Yah4fd&5AA7z&0U_gf3tOU!;uH2vbN9OBp9 zQfg9qPR?BN-maoG-{GAptf};2*jU{r5rH!oVlXDiPaTv1e*~AMj))pR!I=WlON}E> z)$#b_w_KSzfQR&9f$BpWeAPRI88LC+2T4X_FA(OyQUs)z8@^Hk9YIAJ$xvm!PZB7F z);klo-y`>ch*DS2kTUG)=_y7!*^gLdybKD1uvuilV=arQYwrPuxGIPmDe*&O-3oOWl!pL4ZQqngo|(9JU_>ruvq9PBv5}L2A8(1Y`oSQzeUmoK z3C_aAQ?Oi=3tN3CT9?ZN3JW2<4*_-5$j%~E%+bLeMyHiKG$cqsHqm(_pUykqfu$2n z^!4v#AbO@-`$*S2V!vns91Z-ZaOaIbrb%k^q8yCDcjKu#n!SkAf+lxxS`;B%jY%tZ z=$$n@4<&GfA09%kRf*F+3l) z*5)R(pK7!?9YMpH4j{Q92tt(GaNle?{Wi7JqOcdV!SVWCuZ3_Nd)Q~3jYaI;N3muR ze7Kj@nA<03d3rh{Bh(%lQ1V>M;}*v8$c7Q}Dg{0^#AIJst?&POG*MwYabr!KaV@Q| zVIhOH-}}?T#GaH(#)8Pyj|U}3M$K7@^K=Jk=t^pTvoJY(`*IIY&wmyc7A$;yGZpNi zWrLlP_%1Q+Ovm9NHYPT9Gxs=WChS{eW25AGmxm7@7GGUwlco+0Nlj%m7fvb47)@*_ zE-TBsYbU`}5RFps*{>d7>HO&&IyGe)laYa@ymE_p<;EoDF?$(~60NGMOB-~wZ@Ts< zFVnvN{d?^pR9?uAKgug5E{@}a)vK|o(NUF$&uDiZ_l?+$Gm{f7=_DNU81P*jn4T7B zF|Cxiaf8D+GAb(Qqg5`8hklt}sh>UH(9n>Don5%Y(*3)4v!)oAencj-k`60+W+o;U zJpVv7cKn3-sg&7{BO^UMn{-M_%7h69y)RCAK1k8WM`p-waR2dnZD3+TyS24t;q0u%#Kc6*G1)$|wG}t?MNCXgw>4_#)JTCV zJjSQar1T%lnSHi)?qQM7R(Yv3ipK$M1_2WsX;+v;r-JM1t~oh7D|lqfuH$viMrOop?yq{o*Jp5Kep4{aAv%Brru zx9=t6j6*&AzOcBcTvk~bUQlqw%}RDsQ}<6HWz@;+tcAZe-?f^~PKCp>PWWbGpPvlt z>+6>shKG|nucChaURi;CjfDL(E-H$2yNiUfkeOpr;&QfZMj!PB!bastQgKa9%}2}{ z%zR3Rjnn_?r|qwIiT8{b6H46q{d<;; ztu0$$FZnsH+i5;#W@ba?j%iqnJZ*=^W!GBWaI2NUj2h|5;qD*TJ9s1|Q>MyoOq?c5 zc0fm# zGZtGCpOC<_va*u*+(m(mj7+iUsfWj1Ch}lmQBlPee>uNRSJWpISBMwAGWQ#8h9yGD z;+&j-j3B0;o12@LHp#~150@4fv%L6~g-{|+Ed9!|$BNjL^mJWeVPTyc(VpE> z&CShA$8x?kN%8T~>1u5xK?+i2J<8T)jo$_bxh*X%OUUEz5e*pKuCbfum!4pcAs!`D zljrPAi?JqBA=#{VH^*^KQ@U;@DSqGM@QWZ%y+V5J-@#k&HFsUEr+1@=WH?xk>+9>o z-v4ATd?1_>a{1j468QZMLIQgHNLvi%J88@VEcV4USqa6*0U{zIB_!CUfu+^eNP}$D zmLC&d9q)}xHEX|rbL*8Ryph2~X{f7v+kX4nU+23(m(7a%Ha~CWY%D1$d0ttF4?fAo zM9$mJ#L3(|sev=DGg+qVnC)(ntHT>DDlJttG&0Jy7B-?UGr$o%->*6odqE~k&O%a& zBfWaQVu5ah32wSi{kCIsTiX>i|M1brdhNXgMzov0zwdOmH$*1Bj8l|jM^Wt)QMK>U z=EV-7fbai#n_)Jes=qyv$qFivHO`FbB)MrEGI@!-8@;UeK z-^Abla@yPbeBDE;bH$yD@5)%Hs$R)OGQ&hzP@H%qerHBX_QyUfE*32=FaI+-I=cNW z7>mVHV|IJB -+++ -aliases = [ -"/engine/extend/" -] -title = "Managed plugin system" -description = "How develop and use a plugin with the managed plugin system" -keywords = ["API, Usage, plugins, documentation, developer"] -advisory = "experimental" -[menu.main] -parent = "engine_extend" -weight=1 -+++ - - -# Docker Engine managed plugin system - -This document describes the plugin system available today in the **experimental -build** of Docker 1.12: - -* [How to operate an existing plugin](#how-to-operate-a-plugin) -* [How to develop a plugin](#how-to-develop-a-plugin) - -Unlike the legacy plugin system, you now manage plugins using Docker Engine: - -* install plugins -* start plugins -* stop plugins -* remove plugins - -The current Docker Engine plugin system only supports volume drivers. We are -adding more plugin driver types in the future releases. - -For information on Docker Engine plugins generally available in Docker Engine -1.12 and earlier, refer to [Understand legacy Docker Engine plugins](legacy_plugins.md). - -## How to operate a plugin - -Plugins are distributed as Docker images, so develpers can host them on Docker -Hub or on a private registry. - -You install the plugin using a single command: `docker plugin install `. -The `plugin install` command pulls the plugin from the Docker Hub or private -registry. If necessary the CLI prompts you to accept any privilige requriements. -For example the plugin may require access to a device on the host system. -Finally it enables the plugin. - -Run `docker plugin ls` to check the status of installed plugins. The Engine -markes plugins that are started without issues as `ENABLED`. - -After you install a plugin, the plugin behavior is the same as legacy plugins. -The following example demonstrates how to install the `sshfs` plugin and use it -to create a volume. - -1. Install the `sshfs` plugin. - - ```bash - $ docker plugin install vieux/sshfs - - Plugin "vieux/sshfs" is requesting the following privileges: - - network: [host] - - capabilities: [CAP_SYS_ADMIN] - Do you grant the above permissions? [y/N] y - - vieux/sshfs - ``` - - The plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able - to do mount inside the plugin and `host networking`. - -2. Check for a value of `true` the `ENABLED` column to verify the plugin -started without error. - - ```bash - $ docker plugin ls - - NAME TAG ENABLED - vieux/sshfs latest true - ``` - -3. Create a volume using the plugin. - - ```bash - $ docker volume create \ - -d vieux/sshfs \ - --name sshvolume \ - -o sshcmd=user@1.2.3.4:/remote - - sshvolume - ``` - -4. Use the volume `sshvolume`. - - ```bash - $ docker run -v sshvolume:/data busybox ls /data - - - ``` - -5. Verify the plugin successfully created the volume. - - ```bash - $ docker volume ls - - DRIVER NAME - vieux/sshfs sshvolume - ``` - - You can stop a plugin with the `docker plugin disable` - command or remove a plugin with `docker plugin remove`. - -See the [command line reference](../reference/commandline/index.md) for more -information. - -## How to develop a plugin - -Plugin creation is currently a manual process. We plan to add automation in a -future release with a command such as `docker plugin build`. - -This section describes the format of an existing enabled plugin. You have to -create and format the plugin files by hand. - -Plugins are stored in `/var/lib/docker/plugins`. For instance: - -```bash -# ls -la /var/lib/docker/plugins -total 20 -drwx------ 4 root root 4096 Aug 8 18:03 . -drwx--x--x 12 root root 4096 Aug 8 17:53 .. -drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 --rw------- 1 root root 2107 Aug 8 18:03 plugins.json -``` - -`plugins.json` is an inventory of all installed plugins. For example: - -```bash -# cat plugins.json -{ - "cd851ce43a403": { - "plugin": { - "Manifest": { - "Args": { - "Value": null, - "Settable": null, - "Description": "", - "Name": "" - }, - "Env": null, - "Devices": null, - "Mounts": null, - "Capabilities": [ - "CAP_SYS_ADMIN" - ], - "ManifestVersion": "v0", - "Description": "sshFS plugin for Docker", - "Documentation": "https://docs.docker.com/engine/extend/plugins/", - "Interface": { - "Socket": "sshfs.sock", - "Types": [ - "docker.volumedriver/1.0" - ] - }, - "Entrypoint": [ - "/go/bin/docker-volume-sshfs" - ], - "Workdir": "", - "User": {}, - "Network": { - "Type": "host" - } - }, - "Config": { - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] - }, - "Active": true, - "Tag": "latest", - "Name": "vieux/sshfs", - "Id": "cd851ce43a403" - } - } -} -``` - -Each folder represents a plugin. For example: - -```bash -# ls -la /var/lib/docker/plugins/cd851ce43a403 -total 12 -drwx------ 19 root root 4096 Aug 8 17:56 rootfs --rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json --rw------- 1 root root 347 Aug 8 17:56 manifest.json -``` - -`rootfs` represents the root filesystem of the plugin. In this example, it was -created from a Dockerfile as follows: - ->**Note:** `/run/docker/plugins` is mandatory for docker to communicate with -the plugin._ - -```bash -$ git clone https://github.com/vieux/docker-volume-sshfs -$ cd docker-volume-sshfs -$ docker build -t rootfs . -$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created -$ mkdir -p /var/lib/docker/plugins/$id/rootfs -$ docker export "$id" | tar -x -C /var/lib/docker/plugins/$id/rootfs -$ docker rm -vf "$id" -$ docker rmi rootfs -``` - -`manifest.json` describes the plugin and `plugin-config.json` contains some -runtime parameters. [See the Plugins Manifest reference](manifest.md). For example: - -```bash -# cat manifest.json -{ - "manifestVersion": "v0", - "description": "sshFS plugin for Docker", - "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["/go/bin/docker-volume-sshfs"], - "network": { - "type": "host" - }, - "interface" : { - "types": ["docker.volumedriver/1.0"], - "socket": "sshfs.sock" - }, - "capabilities": ["CAP_SYS_ADMIN"] -} -``` - -In this example, you can see the plugin is a volume driver, requires the -`CAP_SYS_ADMIN` capability, `host networking`, `/go/bin/docker-volume-sshfs` as -entrypoint and is going to use `/run/docker/plugins/sshfs.sock` to communicate -with the Docker Engine. - -```bash -# cat plugin-config.json -{ - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] -} -``` - -This plugin doesn't require runtime parameters. - -Both `manifest.json` and `plugin-config.json` are part of the `plugins.json`. -`manifest.json` is read-only and `plugin-config.json` is read-write. - -To summarize, follow the steps below to create a plugin: - -0. Choose a name for the plugin. Plugin name uses the same format as images, -for example: `/`. -1. Create a rootfs in `/var/lib/docker/plugins/$id/rootfs`. -2. Create manifest.json file in `/var/lib/docker/plugins/$id/`. -3. Create a `plugin-config.json` if needed. -4. Create or add a section to `/var/lib/docker/plugins/plugins.json`. Use - `/` as “Name” and `$id` as “Id”. -5. Restart the Docker Engine. -6. Run `docker plugin ls`. - * If your plugin is listed as `ENABLED=true`, you can push it to the - registry. - * If the plugin is not listed or if `ENABLED=false`, something went wrong. - Check the daemon logs for errors. -7. If you are not already logged in, use `docker login` to authenticate against - a registry. -8. Run `docker plugin push /` to push the plugin. diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md deleted file mode 100644 index aa4fa4a9b3..0000000000 --- a/docs/extend/legacy_plugins.md +++ /dev/null @@ -1,93 +0,0 @@ - - -# Use Docker Engine plugins - -This document describes the Docker Engine plugins generally available in Docker -Engine. To view information on plugins managed by Docker Engine currently in -experimental status, refer to [Docker Engine plugin system](index.md). - -You can extend the capabilities of the Docker Engine by loading third-party -plugins. This page explains the types of plugins and provides links to several -volume and network plugins for Docker. - -## Types of plugins - -Plugins extend Docker's functionality. They come in specific types. For -example, a [volume plugin](plugins_volume.md) might enable Docker -volumes to persist across multiple Docker hosts and a -[network plugin](plugins_network.md) might provide network plumbing. - -Currently Docker supports authorization, volume and network driver plugins. In the future it -will support additional plugin types. - -## Installing a plugin - -Follow the instructions in the plugin's documentation. - -## Finding a plugin - -The sections below provide an inexhaustive overview of available plugins. - - - -### Network plugins - -Plugin | Description ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -[Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. -[Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. -[Weave Network Plugin](https://www.weave.works/docs/net/latest/introducing-weave/) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. - -### Volume plugins - -Plugin | Description ------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- -[Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). -[Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. -[Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. -[Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. -[DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. -[Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. -[gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). -[GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. -[Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. -[HPE 3Par Volume Plugin](https://github.com/hpe-storage/python-hpedockerplugin/) | A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage arrays. -[IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. -[Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. -[Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. -[NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. -[Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. -[OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. -[Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. -[Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. -[REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. -[Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. -[VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. - -### Authorization plugins - - Plugin | Description -------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ - [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). - -## Troubleshooting a plugin - -If you are having problems with Docker after loading a plugin, ask the authors -of the plugin for help. The Docker team may not be able to assist you. - -## Writing a plugin - -If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](plugin_api.md). diff --git a/docs/extend/manifest.md b/docs/extend/manifest.md deleted file mode 100644 index d47efce8e8..0000000000 --- a/docs/extend/manifest.md +++ /dev/null @@ -1,222 +0,0 @@ - - -# Plugin Manifest Version 0 of Plugin V2 - -This document outlines the format of the V0 plugin manifest. The plugin -manifest described herein was introduced in the Docker daemon (experimental version) in the [v1.12.0 -release](https://github.com/docker/docker/commit/f37117045c5398fd3dca8016ea8ca0cb47e7312b). - -Plugin manifests describe the various constituents of a docker plugin. Plugin -manifests can be serialized to JSON format with the following media types: - -Manifest Type | Media Type -------------- | ------------- -manifest | "application/vnd.docker.plugin.v0+json" - - -## *Manifest* Field Descriptions - -Manifest provides the base accessible fields for working with V0 plugin format - in the registry. - -- **`manifestVersion`** *string* - - version of the plugin manifest (This version uses V0) - -- **`description`** *string* - - description of the plugin - -- **`documentation`** *string* - - link to the documentation about the plugin - -- **`interface`** *PluginInterface* - - interface implemented by the plugins, struct consisting of the following fields - - - **`types`** *string array* - - types indicate what interface(s) the plugin currently implements. - - currently supported: - - - **docker.volumedriver/1.0** - - - **`socket`** *string* - - socket is the name of the socket the engine should use to communicate with the plugins. - the socket will be created in `/run/docker/plugins`. - - -- **`entrypoint`** *string array* - - entrypoint of the plugin, see [`ENTRYPOINT`](../reference/builder.md#entrypoint) - -- **`workdir`** *string* - - workdir of the plugin, see [`WORKDIR`](../reference/builder.md#workdir) - -- **`network`** *PluginNetwork* - - network of the plugin, struct consisting of the following fields - - - **`type`** *string* - - network type. - - currently supported: - - - **bridge** - - **host** - - **none** - -- **`capabilities`** *array* - - capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) - -- **`mounts`** *PluginMount array* - - mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) - - - **`name`** *string* - - name of the mount. - - - **`description`** *string* - - description of the mount. - - - **`source`** *string* - - source of the mount. - - - **`destination`** *string* - - destination of the mount. - - - **`type`** *string* - - mount type. - - - **`options`** *string array* - - options of the mount. - -- **`devices`** *PluginDevice array* - - device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) - - - **`name`** *string* - - name of the device. - - - **`description`** *string* - - description of the device. - - - **`path`** *string* - - path of the device. - -- **`env`** *PluginEnv array* - - env of the plugin, struct consisting of the following fields - - - **`name`** *string* - - name of the env. - - - **`description`** *string* - - description of the env. - - - **`value`** *string* - - value of the env. - -- **`args`** *PluginArgs* - - args of the plugin, struct consisting of the following fields - - - **`name`** *string* - - name of the env. - - - **`description`** *string* - - description of the env. - - - **`value`** *string array* - - values of the args. - - -## Example Manifest - -*Example showing the 'tiborvass/no-remove' plugin manifest.* - -``` -{ - "manifestVersion": "v0", - "description": "A test plugin for Docker", - "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["plugin-no-remove", "/data"], - "interface" : { - "types": ["docker.volumedriver/1.0"], - "socket": "plugins.sock" - }, - "network": { - "type": "host" - }, - - "mounts": [ - { - "source": "/data", - "destination": "/data", - "type": "bind", - "options": ["shared", "rbind"] - }, - { - "destination": "/foobar", - "type": "tmpfs" - } - ], - - "args": { - "name": "args", - "description": "command line arguments", - "value": [] - }, - - "env": [ - { - "name": "DEBUG", - "description": "If set, prints debug messages", - "value": "1" - } - ], - - "devices": [ - { - "name": "device", - "description": "a host device to mount", - "path": "/dev/cpu_dma_latency" - } - ] -} - -``` diff --git a/docs/extend/menu.md b/docs/extend/menu.md deleted file mode 100644 index 4b9955038a..0000000000 --- a/docs/extend/menu.md +++ /dev/null @@ -1,15 +0,0 @@ - - - - diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md deleted file mode 100644 index 5801ba9151..0000000000 --- a/docs/extend/plugin_api.md +++ /dev/null @@ -1,192 +0,0 @@ - - -# Docker Plugin API - -Docker plugins are out-of-process extensions which add capabilities to the -Docker Engine. - -This document describes the Docker Engine plugin API. To view information on -plugins managed by Docker Engine currently in experimental status, refer to -[Docker Engine plugin system](index.md). - -This page is intended for people who want to develop their own Docker plugin. -If you just want to learn about or use Docker plugins, look -[here](legacy_plugins.md). - -## What plugins are - -A plugin is a process running on the same or a different host as the docker daemon, -which registers itself by placing a file on the same docker host in one of the plugin -directories described in [Plugin discovery](#plugin-discovery). - -Plugins have human-readable names, which are short, lowercase strings. For -example, `flocker` or `weave`. - -Plugins can run inside or outside containers. Currently running them outside -containers is recommended. - -## Plugin discovery - -Docker discovers plugins by looking for them in the plugin directory whenever a -user or container tries to use one by name. - -There are three types of files which can be put in the plugin directory. - -* `.sock` files are UNIX domain sockets. -* `.spec` files are text files containing a URL, such as `unix:///other.sock` or `tcp://localhost:8080`. -* `.json` files are text files containing a full json specification for the plugin. - -Plugins with UNIX domain socket files must run on the same docker host, whereas -plugins with spec or json files can run on a different host if a remote URL is specified. - -UNIX domain socket files must be located under `/run/docker/plugins`, whereas -spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker/plugins`. - -The name of the file (excluding the extension) determines the plugin name. - -For example, the `flocker` plugin might create a UNIX socket at -`/run/docker/plugins/flocker.sock`. - -You can define each plugin into a separated subdirectory if you want to isolate definitions from each other. -For example, you can create the `flocker` socket under `/run/docker/plugins/flocker/flocker.sock` and only -mount `/run/docker/plugins/flocker` inside the `flocker` container. - -Docker always searches for unix sockets in `/run/docker/plugins` first. It checks for spec or json files under -`/etc/docker/plugins` and `/usr/lib/docker/plugins` if the socket doesn't exist. The directory scan stops as -soon as it finds the first plugin definition with the given name. - -### JSON specification - -This is the JSON format for a plugin: - -```json -{ - "Name": "plugin-example", - "Addr": "https://example.com/docker/plugin", - "TLSConfig": { - "InsecureSkipVerify": false, - "CAFile": "/usr/shared/docker/certs/example-ca.pem", - "CertFile": "/usr/shared/docker/certs/example-cert.pem", - "KeyFile": "/usr/shared/docker/certs/example-key.pem", - } -} -``` - -The `TLSConfig` field is optional and TLS will only be verified if this configuration is present. - -## Plugin lifecycle - -Plugins should be started before Docker, and stopped after Docker. For -example, when packaging a plugin for a platform which supports `systemd`, you -might use [`systemd` dependencies]( -http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to -manage startup and shutdown order. - -When upgrading a plugin, you should first stop the Docker daemon, upgrade the -plugin, then start Docker again. - -## Plugin activation - -When a plugin is first referred to -- either by a user referring to it by name -(e.g. `docker run --volume-driver=foo`) or a container already configured to -use a plugin being started -- Docker looks for the named plugin in the plugin -directory and activates it with a handshake. See Handshake API below. - -Plugins are *not* activated automatically at Docker daemon startup. Rather, -they are activated only lazily, or on-demand, when they are needed. - -## Systemd socket activation - -Plugins may also be socket activated by `systemd`. The official [Plugins helpers](https://github.com/docker/go-plugins-helpers) -natively supports socket activation. In order for a plugin to be socket activated it needs -a `service` file and a `socket` file. - -The `service` file (for example `/lib/systemd/system/your-plugin.service`): - -``` -[Unit] -Description=Your plugin -Before=docker.service -After=network.target your-plugin.socket -Requires=your-plugin.socket docker.service - -[Service] -ExecStart=/usr/lib/docker/your-plugin - -[Install] -WantedBy=multi-user.target -``` -The `socket` file (for example `/lib/systemd/system/your-plugin.socket`): -``` -[Unit] -Description=Your plugin - -[Socket] -ListenStream=/run/docker/plugins/your-plugin.sock - -[Install] -WantedBy=sockets.target -``` - -This will allow plugins to be actually started when the Docker daemon connects to -the sockets they're listening on (for instance the first time the daemon uses them -or if one of the plugin goes down accidentally). - -## API design - -The Plugin API is RPC-style JSON over HTTP, much like webhooks. - -Requests flow *from* the Docker daemon *to* the plugin. So the plugin needs to -implement an HTTP server and bind this to the UNIX socket mentioned in the -"plugin discovery" section. - -All requests are HTTP `POST` requests. - -The API is versioned via an Accept header, which currently is always set to -`application/vnd.docker.plugins.v1+json`. - -## Handshake API - -Plugins are activated via the following "handshake" API call. - -### /Plugin.Activate - -**Request:** empty body - -**Response:** -``` -{ - "Implements": ["VolumeDriver"] -} -``` - -Responds with a list of Docker subsystems which this plugin implements. -After activation, the plugin will then be sent events from this subsystem. - -Possible values are: - -* [`authz`](plugins_authorization.md) -* [`NetworkDriver`](plugins_network.md) -* [`VolumeDriver`](plugins_volume.md) - - -## Plugin retries - -Attempts to call a method on a plugin are retried with an exponential backoff -for up to 30 seconds. This may help when packaging plugins as containers, since -it gives plugin containers a chance to start up before failing any user -containers which depend on them. - -## Plugins helpers - -To ease plugins development, we're providing an `sdk` for each kind of plugins -currently supported by Docker at [docker/go-plugins-helpers](https://github.com/docker/go-plugins-helpers). diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md deleted file mode 100644 index 2c3d79e25d..0000000000 --- a/docs/extend/plugins_authorization.md +++ /dev/null @@ -1,256 +0,0 @@ - - - -# Create an authorization plugin - -This document describes the Docker Engine plugins generally available in Docker -Engine. To view information on plugins managed by Docker Engine currently in -experimental status, refer to [Docker Engine plugin system](index.md). - -Docker's out-of-the-box authorization model is all or nothing. Any user with -permission to access the Docker daemon can run any Docker client command. The -same is true for callers using Docker's remote API to contact the daemon. If you -require greater access control, you can create authorization plugins and add -them to your Docker daemon configuration. Using an authorization plugin, a -Docker administrator can configure granular access policies for managing access -to Docker daemon. - -Anyone with the appropriate skills can develop an authorization plugin. These -skills, at their most basic, are knowledge of Docker, understanding of REST, and -sound programming knowledge. This document describes the architecture, state, -and methods information available to an authorization plugin developer. - -## Basic principles - -Docker's [plugin infrastructure](plugin_api.md) enables -extending Docker by loading, removing and communicating with -third-party components using a generic API. The access authorization subsystem -was built using this mechanism. - -Using this subsystem, you don't need to rebuild the Docker daemon to add an -authorization plugin. You can add a plugin to an installed Docker daemon. You do -need to restart the Docker daemon to add a new plugin. - -An authorization plugin approves or denies requests to the Docker daemon based -on both the current authentication context and the command context. The -authentication context contains all user details and the authentication method. -The command context contains all the relevant request data. - -Authorization plugins must follow the rules described in [Docker Plugin API](plugin_api.md). -Each plugin must reside within directories described under the -[Plugin discovery](plugin_api.md#plugin-discovery) section. - -**Note**: the abbreviations `AuthZ` and `AuthN` mean authorization and authentication -respectively. - -## Default user authorization mechanism - -If TLS is enabled in the [Docker daemon](../security/https.md), the default user authorization flow extracts the user details from the certificate subject name. -That is, the `User` field is set to the client certificate subject common name, and the `AuthenticationMethod` field is set to `TLS`. - -## Basic architecture - -You are responsible for registering your plugin as part of the Docker daemon -startup. You can install multiple plugins and chain them together. This chain -can be ordered. Each request to the daemon passes in order through the chain. -Only when all the plugins grant access to the resource, is the access granted. - -When an HTTP request is made to the Docker daemon through the CLI or via the -remote API, the authentication subsystem passes the request to the installed -authentication plugin(s). The request contains the user (caller) and command -context. The plugin is responsible for deciding whether to allow or deny the -request. - -The sequence diagrams below depict an allow and deny authorization flow: - -![Authorization Allow flow](images/authz_allow.png) - -![Authorization Deny flow](images/authz_deny.png) - -Each request sent to the plugin includes the authenticated user, the HTTP -headers, and the request/response body. Only the user name and the -authentication method used are passed to the plugin. Most importantly, no user -credentials or tokens are passed. Finally, not all request/response bodies -are sent to the authorization plugin. Only those request/response bodies where -the `Content-Type` is either `text/*` or `application/json` are sent. - -For commands that can potentially hijack the HTTP connection (`HTTP -Upgrade`), such as `exec`, the authorization plugin is only called for the -initial HTTP requests. Once the plugin approves the command, authorization is -not applied to the rest of the flow. Specifically, the streaming data is not -passed to the authorization plugins. For commands that return chunked HTTP -response, such as `logs` and `events`, only the HTTP request is sent to the -authorization plugins. - -During request/response processing, some authorization flows might -need to do additional queries to the Docker daemon. To complete such flows, -plugins can call the daemon API similar to a regular user. To enable these -additional queries, the plugin must provide the means for an administrator to -configure proper authentication and security policies. - -## Docker client flows - -To enable and configure the authorization plugin, the plugin developer must -support the Docker client interactions detailed in this section. - -### Setting up Docker daemon - -Enable the authorization plugin with a dedicated command line flag in the -`--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` -value. This value can be the plugin’s socket or a path to a specification file. -Authorization plugins can be loaded without restarting the daemon. Refer -to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. - -```bash -$ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... -``` - -Docker's authorization subsystem supports multiple `--authorization-plugin` parameters. - -### Calling authorized command (allow) - -```bash -$ docker pull centos -... -f1b10cd84249: Pull complete -... -``` - -### Calling unauthorized command (deny) - -```bash -$ docker pull centos -... -docker: Error response from daemon: authorization denied by plugin PLUGIN_NAME: volumes are not allowed. -``` - -### Error from plugins - -```bash -$ docker pull centos -... -docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZPlugin.AuthZReq: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. -``` - -## API schema and implementation - -In addition to Docker's standard plugin registration method, each plugin -should implement the following two methods: - -* `/AuthZPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. - -* `/AuthZPlugin.AuthZRes` This authorize response method is called before the response is returned from Docker daemon to the client. - -#### /AuthZPlugin.AuthZReq - -**Request**: - -```json -{ - "User": "The user identification", - "UserAuthNMethod": "The authentication method used", - "RequestMethod": "The HTTP method", - "RequestURI": "The HTTP request URI", - "RequestBody": "Byte array containing the raw HTTP request body", - "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string " -} -``` - -**Response**: - -```json -{ - "Allow": "Determined whether the user is allowed or not", - "Msg": "The authorization message", - "Err": "The error message if things go wrong" -} -``` -#### /AuthZPlugin.AuthZRes - -**Request**: - -```json -{ - "User": "The user identification", - "UserAuthNMethod": "The authentication method used", - "RequestMethod": "The HTTP method", - "RequestURI": "The HTTP request URI", - "RequestBody": "Byte array containing the raw HTTP request body", - "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string", - "ResponseBody": "Byte array containing the raw HTTP response body", - "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string", - "ResponseStatusCode":"Response status code" -} -``` - -**Response**: - -```json -{ - "Allow": "Determined whether the user is allowed or not", - "Msg": "The authorization message", - "Err": "The error message if things go wrong" -} -``` - -### Request authorization - -Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. - -#### Daemon -> Plugin - -Name | Type | Description ------------------------|-------------------|------------------------------------------------------- -User | string | The user identification -Authentication method | string | The authentication method used -Request method | enum | The HTTP method (GET/DELETE/POST) -Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) -Request headers | map[string]string | Request headers as key value pairs (without the authorization header) -Request body | []byte | Raw request body - - -#### Plugin -> Daemon - -Name | Type | Description ---------|--------|---------------------------------------------------------------------------------- -Allow | bool | Boolean value indicating whether the request is allowed or denied -Msg | string | Authorization message (will be returned to the client in case the access is denied) -Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) - -### Response authorization - -The plugin must support two authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. - -#### Daemon -> Plugin - - -Name | Type | Description ------------------------ |------------------ |---------------------------------------------------- -User | string | The user identification -Authentication method | string | The authentication method used -Request method | string | The HTTP method (GET/DELETE/POST) -Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) -Request headers | map[string]string | Request headers as key value pairs (without the authorization header) -Request body | []byte | Raw request body -Response status code | int | Status code from the docker daemon -Response headers | map[string]string | Response headers as key value pairs -Response body | []byte | Raw docker daemon response body - - -#### Plugin -> Daemon - -Name | Type | Description ---------|--------|---------------------------------------------------------------------------------- -Allow | bool | Boolean value indicating whether the response is allowed or denied -Msg | string | Authorization message (will be returned to the client in case the access is denied) -Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md deleted file mode 100644 index 7a107e2ce3..0000000000 --- a/docs/extend/plugins_network.md +++ /dev/null @@ -1,73 +0,0 @@ - - -# Engine network driver plugins - -This document describes Docker Engine network driver plugins generally -available in Docker Engine. To view information on plugins -managed by Docker Engine, refer to [Docker Engine plugin system](index.md). - -Docker Engine network plugins enable Engine deployments to be extended to -support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN -or something completely different. Network driver plugins are supported via the -LibNetwork project. Each plugin is implemented as a "remote driver" for -LibNetwork, which shares plugin infrastructure with Engine. Effectively, network -driver plugins are activated in the same way as other plugins, and use the same -kind of protocol. - -## Network driver plugins and swarm mode - -Docker 1.12 adds support for cluster management and orchestration called -[swarm mode](../swarm/index.md). Docker Engine running in swarm mode currently -only supports the built-in overlay driver for networking. Therefore existing -networking plugins will not work in swarm mode. - -When you run Docker Engine outside of swarm mode, all networking plugins that -worked in Docker 1.11 will continue to function normally. They do not require -any modification. - -## Using network driver plugins - -The means of installing and running a network driver plugin depend on the -particular plugin. So, be sure to install your plugin according to the -instructions obtained from the plugin developer. - -Once running however, network driver plugins are used just like the built-in -network drivers: by being mentioned as a driver in network-oriented Docker -commands. For example, - - $ docker network create --driver weave mynet - -Some network driver plugins are listed in [plugins](legacy_plugins.md) - -The `mynet` network is now owned by `weave`, so subsequent commands -referring to that network will be sent to the plugin, - - $ docker run --network=mynet busybox top - - -## Write a network plugin - -Network plugins implement the [Docker plugin -API](https://docs.docker.com/extend/plugin_api/) and the network plugin protocol - -## Network plugin protocol - -The network driver protocol, in addition to the plugin activation call, is -documented as part of libnetwork: -[https://github.com/docker/libnetwork/blob/master/docs/remote.md](https://github.com/docker/libnetwork/blob/master/docs/remote.md). - -# Related Information - -To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. - -- [Docker networks feature overview](../userguide/networking/index.md) -- The [LibNetwork](https://github.com/docker/libnetwork) project diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md deleted file mode 100644 index f9c93a3de1..0000000000 --- a/docs/extend/plugins_volume.md +++ /dev/null @@ -1,268 +0,0 @@ - - -# Write a volume plugin - -Docker Engine volume plugins enable Engine deployments to be integrated with -external storage systems, such as Amazon EBS, and enable data volumes to persist -beyond the lifetime of a single Engine host. See the -[plugin documentation](legacy_plugins.md) for more information. - -## Changelog - -### 1.12.0 - -- Add `Status` field to `VolumeDriver.Get` response ([#21006](https://github.com/docker/docker/pull/21006#)) -- Add `VolumeDriver.Capabilities` to get capabilities of the volume driver([#22077](https://github.com/docker/docker/pull/22077)) - -### 1.10.0 - -- Add `VolumeDriver.Get` which gets the details about the volume ([#16534](https://github.com/docker/docker/pull/16534)) -- Add `VolumeDriver.List` which lists all volumes owned by the driver ([#16534](https://github.com/docker/docker/pull/16534)) - -### 1.8.0 - -- Initial support for volume driver plugins ([#14659](https://github.com/docker/docker/pull/14659)) - -## Command-line changes - -A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: - - $ docker run -ti -v volumename:/data --volume-driver=flocker busybox sh - -This command passes the `volumename` through to the volume plugin as a -user-given name for the volume. The `volumename` must not begin with a `/`. - -By having the user specify a `volumename`, a plugin can associate the volume -with an external volume beyond the lifetime of a single container or container -host. This can be used, for example, to move a stateful container from one -server to another. - -By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. - - -## Create a VolumeDriver - -The container creation endpoint (`/containers/create`) accepts a `VolumeDriver` -field of type `string` allowing to specify the name of the driver. It's default -value of `"local"` (the default driver for local volumes). - -## Volume plugin protocol - -If a plugin registers itself as a `VolumeDriver` when activated, then it is -expected to provide writeable paths on the host filesystem for the Docker -daemon to provide to containers to consume. - -The Docker daemon handles bind-mounting the provided paths into user -containers. - -> **Note**: Volume plugins should *not* write data to the `/var/lib/docker/` -> directory, including `/var/lib/docker/volumes`. The `/var/lib/docker/` -> directory is reserved for Docker. - -### /VolumeDriver.Create - -**Request**: -```json -{ - "Name": "volume_name", - "Opts": {} -} -``` - -Instruct the plugin that the user wants to create a volume, given a user -specified volume name. The plugin does not need to actually manifest the -volume on the filesystem yet (until Mount is called). -Opts is a map of driver specific options passed through from the user request. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a string error if an error occurred. - -### /VolumeDriver.Remove - -**Request**: -```json -{ - "Name": "volume_name" -} -``` - -Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a string error if an error occurred. - -### /VolumeDriver.Mount - -**Request**: -```json -{ - "Name": "volume_name", - "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" -} -``` - -Docker requires the plugin to provide a volume, given a user specified volume -name. This is called once per container start. If the same volume_name is requested -more than once, the plugin may need to keep track of each new mount request and provision -at the first mount request and deprovision at the last corresponding unmount request. - -`ID` is a unique ID for the caller that is requesting the mount. - -**Response**: -```json -{ - "Mountpoint": "/path/to/directory/on/host", - "Err": "" -} -``` - -Respond with the path on the host filesystem where the volume has been made -available, and/or a string error if an error occurred. - -### /VolumeDriver.Path - -**Request**: -```json -{ - "Name": "volume_name" -} -``` - -Docker needs reminding of the path to the volume on the host. - -**Response**: -```json -{ - "Mountpoint": "/path/to/directory/on/host", - "Err": "" -} -``` - -Respond with the path on the host filesystem where the volume has been made -available, and/or a string error if an error occurred. `Mountpoint` is optional, -however the plugin may be queried again later if one is not provided. - -### /VolumeDriver.Unmount - -**Request**: -```json -{ - "Name": "volume_name", - "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" -} -``` - -Indication that Docker no longer is using the named volume. This is called once -per container stop. Plugin may deduce that it is safe to deprovision it at -this point. - -`ID` is a unique ID for the caller that is requesting the mount. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a string error if an error occurred. - - -### /VolumeDriver.Get - -**Request**: -```json -{ - "Name": "volume_name" -} -``` - -Get the volume info. - - -**Response**: -```json -{ - "Volume": { - "Name": "volume_name", - "Mountpoint": "/path/to/directory/on/host", - "Status": {} - }, - "Err": "" -} -``` - -Respond with a string error if an error occurred. `Mountpoint` and `Status` are -optional. - - -### /VolumeDriver.List - -**Request**: -```json -{} -``` - -Get the list of volumes registered with the plugin. - -**Response**: -```json -{ - "Volumes": [ - { - "Name": "volume_name", - "Mountpoint": "/path/to/directory/on/host" - } - ], - "Err": "" -} -``` - -Respond with a string error if an error occurred. `Mountpoint` is optional. - -### /VolumeDriver.Capabilities - -**Request**: -```json -{} -``` - -Get the list of capabilities the driver supports. -The driver is not required to implement this endpoint, however in such cases -the default values will be taken. - -**Response**: -```json -{ - "Capabilities": { - "Scope": "global" - } -} -``` - -Supported scopes are `global` and `local`. Any other value in `Scope` will be -ignored and assumed to be `local`. Scope allows cluster managers to handle the -volume differently, for instance with a scope of `global`, the cluster manager -knows it only needs to create the volume once instead of on every engine. More -capabilities may be added in the future. From 43098d3b479d0ee6798b14fb4e4ef111581b8261 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Wed, 12 Oct 2016 16:23:01 -0700 Subject: [PATCH 151/534] Add a link to the new build instructions Signed-off-by: Misty Stanley-Jones --- docs/deprecated.md | 216 ++++++++++++++ docs/extend/images/authz_additional_info.png | Bin 0 -> 45916 bytes docs/extend/images/authz_allow.png | Bin 0 -> 33505 bytes docs/extend/images/authz_chunked.png | Bin 0 -> 33168 bytes .../extend/images/authz_connection_hijack.png | Bin 0 -> 38780 bytes docs/extend/images/authz_deny.png | Bin 0 -> 27099 bytes docs/extend/index.md | 272 ++++++++++++++++++ docs/extend/legacy_plugins.md | 93 ++++++ docs/extend/manifest.md | 222 ++++++++++++++ docs/extend/menu.md | 15 + docs/extend/plugin_api.md | 192 +++++++++++++ docs/extend/plugins_authorization.md | 256 +++++++++++++++++ docs/extend/plugins_network.md | 73 +++++ docs/extend/plugins_volume.md | 268 +++++++++++++++++ 14 files changed, 1607 insertions(+) create mode 100644 docs/deprecated.md create mode 100644 docs/extend/images/authz_additional_info.png create mode 100644 docs/extend/images/authz_allow.png create mode 100644 docs/extend/images/authz_chunked.png create mode 100644 docs/extend/images/authz_connection_hijack.png create mode 100644 docs/extend/images/authz_deny.png create mode 100644 docs/extend/index.md create mode 100644 docs/extend/legacy_plugins.md create mode 100644 docs/extend/manifest.md create mode 100644 docs/extend/menu.md create mode 100644 docs/extend/plugin_api.md create mode 100644 docs/extend/plugins_authorization.md create mode 100644 docs/extend/plugins_network.md create mode 100644 docs/extend/plugins_volume.md diff --git a/docs/deprecated.md b/docs/deprecated.md new file mode 100644 index 0000000000..a591880ef0 --- /dev/null +++ b/docs/deprecated.md @@ -0,0 +1,216 @@ + + +# Deprecated Engine Features + +The following list of features are deprecated in Engine. +To learn more about Docker Engine's deprecation policy, +see [Feature Deprecation Policy](index.md#feature-deprecation-policy). + + +### `repository:shortid` image references +**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** + +**Target For Removal In Release: v1.16** + +`repository:shortid` syntax for referencing images is very little used, collides with with tag references can be confused with digest references. + +### `docker daemon` subcommand +**Deprecated In Release: [v1.13](https://github.com/docker/docker/releases/)** + +**Target For Removal In Release: v1.16** + +The daemon is moved to a separate binary (`dockerd`), and should be used instead. + +### Three argument form in `docker import` +**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The `docker import` command format 'file|URL|- [REPOSITORY [TAG]]' is deprecated since November 2013. It's no more supported. + +### `-h` shorthand for `--help` + +**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +**Target For Removal In Release: v1.15** + +The shorthand (`-h`) is less common than `--help` on Linux and cannot be used +on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on +`docker create`). For this reason, the `-h` shorthand was not printed in the +"usage" output of subcommands, nor documented, and is now marked "deprecated". + +### `-e` and `--email` flags on `docker login` +**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** + +**Target For Removal In Release: v1.14** + +The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. + +### Separator (`:`) of `--security-opt` flag on `docker run` +**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** + +**Target For Removal In Release: v1.14** + +The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. + +### `/containers/(id or name)/copy` endpoint + +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. + +### Ambiguous event fields in API +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. +See the events API documentation for the new format. + +### `-f` flag on `docker tag` +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. + +### HostConfig at API container start +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of +defining it at container creation (`POST /containers/create`). + +### Docker ps 'before' and 'since' options + +**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The `docker ps --before` and `docker ps --since` options are deprecated. +Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. + +### Docker search 'automated' and 'stars' options + +**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +**Target For Removal In Release: v1.15** + +The `docker search --automated` and `docker search --stars` options are deprecated. +Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. + +### Driver Specific Log Tags +**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +Log tags are now generated in a standard way across different logging drivers. +Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and +`fluentd-tag` have been deprecated in favor of the generic `tag` option. + + docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" + +### LXC built-in exec driver +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** + +**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. + +### Old Command Line Options +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** + +**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** + +The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: + + docker daemon -H ... + +The following single-dash (`-opt`) variant of certain command line options +are deprecated and replaced with double-dash options (`--opt`): + + docker attach -nostdin + docker attach -sig-proxy + docker build -no-cache + docker build -rm + docker commit -author + docker commit -run + docker events -since + docker history -notrunc + docker images -notrunc + docker inspect -format + docker ps -beforeId + docker ps -notrunc + docker ps -sinceId + docker rm -link + docker run -cidfile + docker run -dns + docker run -entrypoint + docker run -expose + docker run -link + docker run -lxc-conf + docker run -n + docker run -privileged + docker run -volumes-from + docker search -notrunc + docker search -stars + docker search -t + docker search -trusted + docker tag -force + +The following double-dash options are deprecated and have no replacement: + + docker run --cpuset + docker run --networking + docker ps --since-id + docker ps --before-id + docker search --trusted + +**Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The single-dash (`-help`) was removed, in favor of the double-dash `--help` + + docker -help + docker [COMMAND] -help + +### `--run` flag on docker commit + +**Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** + +**Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/)** + +The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor +of the `--changes` flag that allows to pass `Dockerfile` commands. + + +### Interacting with V1 registries + +Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though disabled by default, this signals the intent to deprecate the v1 protocol. + +### Docker Content Trust ENV passphrase variables name change +**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables + +- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE +- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE + +### `MAINTAINER` in Dockerfile +**Deprecated In Release: v1.13.0** + +`MAINTAINER` was an early very limited form of `LABEL` which should be used instead. diff --git a/docs/extend/images/authz_additional_info.png b/docs/extend/images/authz_additional_info.png new file mode 100644 index 0000000000000000000000000000000000000000..1a6a6d01d2048fcb975b7d2b025cdfabd4c4f5f3 GIT binary patch literal 45916 zcmd43bx@UE)He)x5RM?JfJle5lt_rQlp>N6($cM@(x9NyQUU^sbT>#NA)(SC4bm-w zNPg=w?&o>uo%!bb^P4&MJ;U}~wXeO`TEAH5*`3>RXU|Zc!N9;cs~|6?8mEj#}{hMznYINBu?&oe6G$A5M8Mr ztp2%?yOZk~NktGSO(E}t`um}Kj_Q>uvLRE#Vv_j#d5(dNzJ`Sk$N2q`=ttiPm#IZt z{P`34KQ{W{zd!!}_0JMA*zHq|{>*G_?Q1M*>9Dh#VM&LRcFg&rSx zqV8j@3jHsWhM6dpm8o)fUw9v_7<@n2sL#1iD7KdpOnpr=U%!@JyZC-ld^Fc=n`(Vf zM)_vzcntLu*&9zJ`cVqiE^F_shw^Vnd;OY=O}q7q&(Hq!q}M^-d*1oauSeN#GtfA# zk7;ij$t%$jUG;sZpjTY`IUxJC_u*EcN>L!KmzLE~zAaVT!R<_nr_a?GDsE&_@Fi0) zP6ZiKT(Jn3mgzU2sByQuS+|h%I9TZ5=Z4@6OjFa=YRMOo=jYgFK8Dl#&*oRJKRBG@ z_I8}AvEDoSIXO@!_2i5&pJ~T8x6MWzY6I`1vQ&e5FZPXG?Gj3b_pTduQ9oglMuoy& zCkj8zwnkoMpS!nhmFGdZ$xQRJ(B*-gYj@1vJ1tQaX2DAceSOW9(*3ISU{|5!cH?dv zt+%?PzS}hR?0d05y!T=k_|2RWuepBSI^6zp?fH3*`3A?ij%ezK^Ig0h-Pc%@lbn96 zj`T%{dwa>naB5Q#iXRr5w1rX=e8P0zWwDv48NsZ6LB_Q#pDb3(sGKAk`rdg(gp5-w zoc&h>qa4-vd*_GWI%B^?2tPE_xU{U2DtlIg`{84{qUgQ#VN{3O#X6@YRZ|m~C5;oz z&gl<<*Df2><)&uZ(j6Yn&FAaieo{^4_T0Wj_Aoi_AbG-jDJ?cpaXVJIGlomUYtLZ3 z(8xd4X{mSFTTob7doH8%1DPk$TFmvwE<2JshbOe6Zrwd;H_7GJ$Ev7d(LNOY{*f`% z`*=I~_U=>zZNAGodH(rc8>fqld3JaPHqKth<O{E%xZw@1v&Si_61$gmMUD=>Y9uD`AKSVZ);YO=~aOVtu2P{+n;3-nzPj= zq^!!r-LRq5JxbRW?`9|`52jQ4bsHROR#xT2suU(rD{t+Dw9#JN`J^&**M3L3o1ONk zTgD);yZhJ9VvqEIL@bVD@X41emc5x_p?8b#E6Dj`VXKRb5&m!w+Q%cmOmXgT&T7cZ zNWHl&&e>YFZ^qct{jSbU-k)lV* zTNPn%C^&YXL{>-d4t5*X^Y{~RrhF+$*>*De@*=JBezPuz6q>?|e%P3k1dY-sr|Py^FNW@{3|+gd_x@e} z!J}b1>N=H074qPY#Tp0x?nch=&9Fk3H65ID0(Di(d&jsWN6G;>XZP$z%RlkjjR`Rl z2ESk2{L!a;alv{ZdrqswEQGy8JxjIK{o!M+kd08K6j68qcEbTZ5%ouF_MvQQ=|Wb6 zxt59cJ#U}n$J5In&e=ansD5+-yLx2noWZ2qVUgodzFz3hN$<(b6*cASb9>)AxzE0_ ziw^U$cx$tX>V&N)a=aDhok?jyQ#tWYfs{G!OQP^><-vOO*^6h$FRu>QKCYN;4Y^2g zNK1cN3^p(y>!OFpj$HLFUMfR6wSn#Qi@bsIUVMLBJhk(|dIn>kJkO<;zhqI?Wc*4& zJhWzeT)+L3#)e|btZms-c6)1}?M~GB1E;#i2Nho@%;t|puZ_{+cV=axNydA|Y*z+y z78*j()BRa$trh76)FMKX-Gq*~q(>oWhX|JDhKbt8A$#>ly9x1pjRM=Jx;t>6o!2Fx z5;8pGf7565aesejr6Xyh{$v8P(U@SiElhC6`+Gaf{7nLHnyW<4B4bq#km9;|J0cNy}-r55^xow;zzj zB^tk6P=EYhjaTzm9BuqY?aw4NeIcI7;7>IWvyRQ6T#jTPRI$fAio%&1dy5V6QJO8T~s7rs(4oNw&7KK${cDNFc4aJ^?&w&$A} zyRzC(u>4)C=$OU%YA)UgN&aZl6f&wI>PDEp@kOIUS;FE6x^? z-S~TE@uMFv@6~x|6VQleex4pjn|8)%9PFvRce0GGcVzIP<|v9pD1^HXtjq1S1ybu8n#%bC+FPfSngyMl&03W^N?PvBi+_( z5*&97vHIivoQ^cO&)c5N(ogZfckzFrU^Z09wcc!@-GaCu@0uBURMexU-S{0-&HjyQ znqsTz_G8Pgs|z0^J_My62;2^MXq&M>p(Y1UV|1#mYa@@C+>{(TpmwMCq z*VVL^o*2&2thSTo{AB(%Yfp>US|?G3GddBmn40vpX~&~qbEGRlP`R-8G&d&9L4wXY zo=TDXwwWbtI-?AIzaUaK;g=cYSEh}ucNJZKPN24>hglm+{pZ@|Q)qMd6VK~9Y&Txh zxFdM9@ z&)XYnrlw-9@T%2^Jh$YqysOeWL}geGVU|=Hzw^FO8RDbcbMkHx822u;FTZf2CbH-i zGVAc*Z25%Qfl*F=#)y|*hC~HSRFUv-Ic_O|2z6cQEC2bl`Y4?$oP=%O4V(u^+{Lys z;2HC`w0C$WLo2ZTvz&H-n55=)nWX<|D|IH5*5HoYH9rlz;&`8a8YwXs+}%pQd&pRQ zAbPwv8_92ed#6>LN$BBxaK!BmNLw-+l)BqW1cYkTc9V5^6ZI)!MPccZ+oEC{LKW79oZM>U8E(-R)DA}A5 zIV7#!Y{DysP}?Z|9s(#k-jRgl<3)$#L-$&18|7p%+f|b`E02t3Nt!rp&N|7e(+?*d zS@J!$Dw;l(t=^$NDDKe}Zv5V!C|uIET04?<5Sh_DJx^&>h1L zIK8w?y%dESx5y$JrRU*tPJv>**U>kBLa{q)bVNdrhM~;VGxiEHj57X`bX_E?XaEb7 zrHgQm*88w0j=eV`Z`iN+((Oc6H?Hq3FUYaWy|!A&t-ikVI@)KyDf({b!7`rYaalX3Equj=e#2 zyz3Re>5s*p@5culoxvCQQO2n0c9vvy-EUjNCSmPwZ@jp_|6^%6dUe^alQ!6R(TBXG zanhva`S}hxspKJD2x4WU6xZWK-HX4AOVmNFs_xuP=!CCq6=6sw3z6|nFs-=$?#5)n z3xaksMm}7_zDh?ENnuEQC0A_kC$+UDFztMr8+V;Ne|^S?j3%yFssUFqS6^ zTLv&bd8wSDQPeSDcf`>B=s?lDqWnxib+rg=vg(@!U$m>fXf2t%DOMg`+8DH3TPqmP zntAd`Z&0bVZbBMM*{tT{t;fA4ujrZ}y=3cuOZ$>gq1EE6!69<5_R+waj!O*==1ts) z!J3Zw;UW_P&N;q?XP%rlTkv`c?Y(&8&+DEhq)B8qcqpuem3wO0uEjwPC_vi)r*fh42NZj=S&#j7%PinPJBl)_@ zTY1AlqP2Oeg&l1*Bq(kUr?ZcqVCsLqPoOoP`YPkyJBrxCbNb0`FP4dNAC&3V^RO?N zwdv~a8C#d-N%fG~Wg8}bZyg_#+o^u#B<1>(m*c15>#&u<4$oFA+7e@f%t~Ebd;W(8 zKQG^PnHDYogo}WRypKNR|HTv+B{qp?mqo3<}UUF>%+W3}v!+a0E z$^ZKHQn}wkCW7Hcl=MX^MWZhZ$=>-k{ktgy_O&}Pl)S$7vQN&OZ_e#E=-heL(}b=HqOsc@BV32x_V&wbFhQ`mPk z(E}?lLnGg1h1_>=^sY#CnFih1^?@EJAk=68m1rJlv9504ugKVTv9?!s*%wJhmCbUf z*4%5`c+lXD^&xOxgJnxq++b%9iqdGhhC<`^NapCmKFdNBxfhdwRXe?IUm8Lm{%u_bi0U2D8)xEB_HOGbkY`631^n91a87 z8ok*X92KA66`tuMj-!0&X{7yw)2#guL-S0kK~s$B8P;{3oo2j+!2$zrH*-7>olOS* zy&S{+(_(^?-Y3^vC=cmOpwG3 zQ3{p&Z33Z!>!3O7P!{k_2L4E6kJ%SS$X$XgRZlr15C_8>7F(Q@9Wu zL6VWDt@ih#gq0_iVr? zg7^pCZpg5WWZ(yS^2_&v(6b{$#&2KQSl{Q|aUPYzFEaW;kLJB&Cijm}GkgtG5kR^Sh!qLnC{QPZ2t{N-GUd=Eb)r^9}DN>495(<;>G?qAQ1$#TmS`Y z3FF^aMpAg&$FEt0KmGR$M5Yjz7xPRc(J51x|Cqf&I7Z97_1u30^p_+gNsLpvLjTc$ z4I-FlNLISy{QeElHN#tx!jBsH`=T+>S}gG1USClAyUsDt0deqVgM9g}{e5qAm@6n7 zHu<`fZ*Et8!H3A_dzTLp0 zXX!4rR2x{JdAwQ_KDV}>)B~GGkCkLYsn~!tMUxzU zID~`?)S3%Iz0%L7>-U<3~m&gnk-Nb>Afk*2&C{j z+PTZDD!#wSs9kCifmD7m-|yci*_6RBuz+1$P)Fo>{A=OCw~xfBC+>=~D;})t7l5`* z9(p`EWZ=yuXg3xPsb1uj$?-|!lhF$MPADYjXixS6_g6xXYYNGFy{GlT+s(Xa^FYHlhMcRn}pjL;ztUac{38b`5xJzDXF`2!QRwD2UHjU!02xKa=l!|beOYEu z;l(P8AE_%%-bxl5pA}aIj`ZD~H&T8^o=^8K`KS~(`r~xMZvEB~&9R(LIVoVa#&E+B zYX8V-AWmG{w5TevdW&vSNzX3oh;G`6^R08&FJM4T2==C%0EKXbOXxBE?NQonauV^O$xeayQs;++Y`Fi{4g(R_qA@UzcjR*bqcc_)IUZh+>ROBY*q zy=N~QF`*xWk5`HH&}Ga1%!sKIT3d&y2FwaV8|7m%irNpJn3lwcRpy4MG}n!Z?2QJw zq|i&y7iLz4uNk zKYMLPiqX9%K#+Fie=W0Kv6y*OU}hlL&9oY7i+biY^@00?t`8|=^9`sIYHsl7Pq$)7 zV$%C39OH4He-b=(V27l|5_2yK4>Q%hlsT9CZ8Wu(6LCXhyTEffbo{e+HbiZ;c6Kn!ckPb`v6 z?b0hD$G1>JOY{5)5mT-A@)UO4Oo@LaWAI^9$U}GYta_kyAb|qRxas$A?W)=8Y)hjR z1wejDrYj{}0tTsm>Zu*U@ehTE9_u)A7^#f7)Yn|pbwyW+d0L1pHW!jGRjMD%Df;qr zc`S6_-Qqdf9ksts_QI0+)?MTG*nXPcD&ON8K0O()cFifd@q*0x;lNk6A9{!O9*0|P zR#6vsTGiPT9rZTK)*qM|xXLm6eip{UXP|h8tM@TkT>Pmh6@-JrsHM~}vFu*G+D95` z)}!TlB4S>Ll_Klo)fFOM5CC3(*DEr9A~Kn!#xm;x#P2{=M$)4Wp}^P37J9t_kK^bC z1oHA2;OLNtZ@!`hM(>QwDT^`6g*xHWhz=)bdbI${9XuoIb?8>G_4vo)o0+_ORjMu4 zW0ggWKp1WT0uW9EO19{^x!vrY9}x(%KQ~%vB;5wwc`#qH_p!UP7jOivRr8+`avmal zk0Ze~&}CE>Xdf-o&(=hXSu&k5X6aue4`35}`Sin}ZiSujne*)5J||*1!Y8am(GCtP6p(s z>}^8PO&_r}huXW=1oJ0KGA^zs0(@v=7OEo_+=p@f(+$#3IQw$2?cv!axE;8SY$?PH876@*NJN5K7F;DadLuIT-8jTRVqJ85cKL~!M6tEJy z5Zpt8y7h_<^;M3C-IxFoBSf0r@ds8MfI-UC3y!BV)bfDUw>-+2Zqqdl!CQmz`b9xI zbv4wy_3BGA%IQ&{zchyp&z)s7#NE_DD$8Y#M?aXT$PAvtrG!11!xxR^i>5q1;?2*`qS_B0Z52 ziQ?2&-N??XQ<$1=frpO0toOhw2XbWg?8Al6?Xc|LH8V}z^-G(@rC655)u8n)z{E-{ z_=L%D5tGB0Xet5Ygyq&+C7#1i_ zE%Z190HY^tiuWess_j>GgE*q4&FK$M;~HWw4;@cO^?29cdR2dC`h`KA$1l_g3sGy_ zXZ)*%a~F>by1W=flsQ$qPw@5=GxDtUhKH_4Jg?s_CuVZU^`nEF8fKVRo8e>6O`+77 zSY3cIiGkYlsR1UNCRKVQ!9T;BN;$N2zH}^%0JHYn-N~-0jIG|A+*z)Ji*qBJ^u2^0 z<1xOw#L90+hH#T<8n;T%NFB&F(za3!vwXVJR6?jy?eXhDwD6(O7}x68u?(Y1*U6=Mo+~~*b7w_W|O1lEpwC6ek*li+0V@pI!Vptc6@7?XQcw8#f8bmLBA8oi# z#+bc#l`z9($cOu;gKDcf9hD&Qj~|INj(wK*u+zk7pL0!wUYJi+SkIA+K7TWikoNLM&35Q_vNnge)(uKj8>bq?aR)yioT0#K z5!A$`zV)fHZv1?X$z~qz@oBBMmduOZsvMRSf}$uM@sOn}xjYgVO*&2)qr9Xj%mi_Q zW9S#{Ur(c#$ z&R4JnkB^TIIzeze^m-OvC3^kJ#?q*d@a3N$q%jy!f(yJRt*pss4p%Y!Tx4Yo_;5Jc zJX#Y(F8=cZ%thbrXRE~*O!RENS3VKg-D;oO5Q_?ZI?wZX*Jg*({)hKngL4l>*8`I< z5i2V7=&Nk7qR9^hDO_2fwRYbK?a6X8AH?0mWa@6GZNBN>Whzo%vz;8*=nP&8}XQ$w6!Siw;ZP<`+ z*gE&&ty{?yC}tZu`mN3$qm7%FYI+21hEwdWQk1ue<41`n_GA5;$Eqfp;;c6NM7bk& z#w$mlFwg^s(EfUTSOj*k&9p9gVhX*)rcZ+vJL<x!Un#tu4m5t{jkG=h0UOt6(V=zo0CZPzLcJa8`e2Qjb(rs8m#xUU2q|WAJ z60PO=_BY*n%y?v+nO@{LsoX>iZiSMK6&5{GXV9UNQ?DBP`ifhyr5f+I)3We%3n)92{igaPb0467W&o`dy%5KLz}{`G!>m+ zpm!o}8jvf!p07LhuKr;CS|gu9|0z!r-CP%+J8XCJ6Rk;T0$*XE(0=--SF(hTT=Z*l z@jY?Xyol)Bo?v)=$GZ8RS0m^kCTY3b9sBW@ZZCGCp?&YhKC%Lt}pFRpOwu z-51QdCIu`gGhWSVqpvrWm6z@n`jMo{QW%|SAw0kA;XhJ*f|BMIwX_wGB1vNByZaj# z`pOeEuoCIFW7HEr^hqX{`tbDZr;I9PtlKD?Iq8}E7YP4=U*qQ(6UEPm_zZh6C&K(9 zNcN%((-@nBex1$d)!?PPNDMHTxUowL`7+%x+K^J_feg?m>RIsRexA|=4%<)dMPFaOdV_kiwr_QLEw^IynLR}Rp++2x?;(Z6%z!8sj! zf^$)S5qtAhK;y%9;yTKIc>qUAIOpZ~i0Hq7-FFxP;KM)?VI9t&wg{N(Gk-(8kY<6a3h)*2o6iSf3DLCE9s=5EDa3?tW2WD7!bMGnA_!VHxc@M(F zqCcaqZ4rhl*PPyEJ}9-R`B`SDP*QOEX801Ll8+cu#HG*Q|06n7iGenH62S7`00T19 zz0KM*{}CPRFp*i|oB!r9T;d&KNQxKg{EzvNzX8XbCrSE``3SlKb1hG5?*1Rqkp)xv zPO6FYzkvv3vMpyg!~P>W&ciY4@^1gbeDuO>Gbkm_{l|Q~V~~)HJcCvq4rxfc8T$>; zl?(ri-9J1?KsYQJT%-LC07UnS0RyV{Y;rV%xOZnkx6F!JK|#Sz94f9<}b z_L_fA)Q)`l(iCdEtd7tpq;2DMI~Wpi#0gGdy-iH=_mZ z_1QFY^KqR+;*b+bq zK*rMPw_j(^0KF;yVBs^@TxV=7zygY`p*&p{jcoM@QTOlizy8q{<8Gs!Euh*q z48W${3r`u-E-^#PYKvmuA~t$=AHYr|z4Yk|eD{OiDJSb^Yv!?o6D!)XFTEY8pm*R< z@;fBv5m5zNN?Dt70sI?tyvoC_-@hpcoB<$DgWxTRcQTAltHUa+if<*8OyNZ`iTFA_ zBcSWu;Z}VY1aw+Y)-cduBGiF1hTvPWT0W)5`f)OWsAujxxBk5~8d0}9q14wdIW!GS z@2-tnEuZdV0fae_F;I@!UE1bP-D;zA>nzc)eZ=Rweox@}_Wnu%hwkK~6)q5fUkK*} zt#TXG-P0+xpx2SO@$L?p)bD^rM@9dv?>2^aaItCUnPxw6V-y9QJ`tS|yGAw>u(5(- zaUG59)VV)?{j5J(lR4WbglpM*m~jpKJAIY;)T*^}Xcqn9_{hk&>AoPgaDu^!KW5*m zI{4OyniA7aGG;{~RT8uAuHPZ}7VNxRJe3EuSQPXPR*7iROSsM0C>#&P)-c+FL%Gk1 z!l|>`nVaOBx&{PNnqi}#D)_Fl4C?C; ztun94*Sif?adZjWp(@x+vQ(vHU_x*3=mltqSr6u_j*uktt zOe&1#`aAEFV?MC6I8!|nKgS%A(#3~FyD4u}B-NwH6Yqr?JNy1jl?T8dT@Ze7e{A3r z!TF|aD$a!gZr2ijCEu@8BG&^{D23Lnl1X;cB-;7(-k}Q%WFY|J#bC(E`!Rp~;i%-p z$m}>?lS_pT_X&YeH=N09@%%bE?gm=(BJbY-l_b%`d2qB|y@ARRq>xfhfhq_~ZpdvO zPo>y#R(8iL$}Qh~@iETB->d!vmnoP|A@=g?M`z!AYf-#FsggSEa4S&zoBxT_Sj^!( zeyGWrt;cNmOj98~89jt+W6xtN_3AuEbjfP|2n$3QnET-Jbt+w+yV8XEzTADo=e?th9G2+s|^GX1gdW8$?xColpkpTwm(f*a%J-OeaB#)8`2 zcuw8&TM$0XD2hPz7GQ8(`zkPd4`jdlz`6CND_s=!+%I)n?pHAToM?G3`KoPv6uX8Z z#Dq(bI${yx)VkVrBL-L(1>jxNJbr!WGik+lreWl$dN3E+lPY%;?jy~j=dG*=I4_vK zR@ieRu297n#IM01+YYo()MY&#voA@u!$n|mj5Ywbs`@d|iNyUM?j{*{?nL+J>E3-X z-^B)Q%a_kcS)PBGCbD;-eDWah&-!72iDmE;E`D+g*hx^q8C}PnV-f!%B9E&sUS@7S;{MS#8`;%pi)&EidcuO z50||#Qs-H{I##7tw>N_)Nu_K`_o?dP!aE4?y>Fy(%IXapv6hAkk`FgO5GsmF_(*)C zn}FE_nd1hSeO3j&K$2v;mh3izHYJ>4B;=|T9m!hx%=KqY+=o>z^Yme}Vx#{fNJUmr zr=HCC9s~2584(~6LcwbsV5t5UOg&cbrV%&Hgqg_wk3~KYtAhko@H*SBf@~z;KR3^A_D=xy^`i zf5PwOWXOWGw0a-k7M+A4D>oIO1J-Ls{#!2$OQ)gozWVuIa#vXTu1F(W7Iq9;E6Tj{T*mWVjC-@Ufm{ET$D(Xvh}Pe;HR_u@3Ue?$nlCO_8x zoXA;T>f8lSUG~zYhe>f>O&^~TXs)e-)D%o5tjbbxM7D-CT@Ju=>1$ce73;xUz;C6< zB(H^J_`dNj? zagBGIsSGoO^M2`wu1n+laRu}P`Ng!!xI9_!Evgyq%z#M}WCZmc@0eSD`s|w!%V2~c z#UY}R+B0OwCVUIEkpjT>=!eU|9Q0rLeE4fSIxO%r9wS|xj&J1_86@X7bW;v)YhoH0FX%P(u4lmK+rdp-Qj<-a&HTXY-CI1*IiJb(GiT(Q@0Dz$3bS zDgkCATZ}^3@Aam+`@(D2cfh@htp&H?I*>i3{RAZ?7dJXgjwY8Ba*8LCQ!=>x%2ABC zo1{RcFgI9H1OYk5j1nm;V=3MZSBS9+a{J(SQn61aQ zg0iXMJ`SOx(5I$ycnPW@&k;vl(xYWIjnkSFh;C)okE*+*wd6m=rW_nVr;N*e>l31y zeII-HB`a(sUFK_>6=QI`l%s#R1Jv_h2?PJn<1eZ`QPk3?lda#$BGmgujqKkuxW$ye(fBSx*=xX6z56JGRdS-_>u%$UcO`uDDKaGXbD6|u`+zE?@- zW=ZDd5b^t7SEXRx|30r8%2*>9338xE(IMRiPzki0dq@dsnP?Ioa;Dl-yPAin!Fs4D z{!$E;mJz?#N+z(a(2*d#bIRXH;sYm=`{)d@5{uCk)TEc#F}>48KtH$f$H}n-e?9{e zIqM@66BBJNj#}U%lWGAE6_2W5nqu*V?cMo(uCq2YoT_4^mPzh{M^o!bnXb zd?Zt3o{=EI7|azf97F%+szyMH=ulcU7FNt#-glyyEM`l-!h9LgLkc6y?YcsY3xiS| zC801Ffz%dky__tU(RZSnO-qVZVNit*2C)}8`Rd@C&37LcUxmRQc;CyzebLWQJY$D|gyL%KwmnJlavmQJWt)S{{JSfBxf3%W?2W2;#*qM1bDh5|OQP(<7)s4LijB-R8w**>n9vlgxW zXt@#qRVt(=c=zE0mESwS$w;!IW;QD~|B@7PLD!NMFcO?XW&<*tA#R{n7D)=dIR zr9dRuqi_EDK4;n*d_nfmd1X)mJkU(gs?nRP&A-(B#vkdE2`7|wy%S16anvY)idws@ z45pC(FJM<$*Z!&-M_YpD`kotm@`3_g4{#bxj|dHqOrA;rZXJj^ldkc@93*6;GFmJn zplGs1anY`rsYD=9#2y{|#FLQ$H{QS^zK0;srz^nt3(P%)CJCh137=X-9rhRtQjY=S zP}$kLGzP5q^?9_fzbKv7?L;B^ftH0TOXtV#=; zi4hsTIfGa*G&H*cQ8ow!Ci^78DBmE68B8`cTK~&`De6ujz)L(?txaD-bjND}>lnkR z7PW?R#IlOzcH`^-fx+APR31vOG^>H^-U@qzS5Pe?t_gPaOuCP>n1l-=&dY8cy+ZX? z1MFe$O?600SBV2V>&G}=6VwcSe$4$bKv{KV!8L*QJZeZ z%-b&}q$E);+Q5o!2pR;aKW??0yuZj2~{ z=k@2e%={}J)Qq_P`fjRGGe29vb=)fbnoaRAt;NlSrGf2Ybp_EHqb~(R_uCtIWt8Sk z0cb2o%tp-Q$gSBnj*|%R8f5kaLt<}-h-(UkY_N0c;z=JWwU2bAehQBihcqKlxN{ngWfFr)Q-rM^E)Dt4D z*L0`8Enjvk`MoM0-n}jKStTmKpM5*{iZ4@O5>tp%J2B_~M$)L8rYc+VKAvo)dCxV3 zwLa$t18N5>-8t~sDthv^ey&V7-Sjbhm(9o^r08^7RTD9Z3l<5TZvZ`p zcuc-tb*rtXuqsJ~QUcNWR62zqGc#rg#}XLa?jO`uMRB+kH}xrCAuuBgpC_&t0FqE-LNHq7`@2h~{*UApL6>4cO8cx>>ohtb0jiuz zG`}~J=5SB>Mn#>Qwuj)arQbMl6HI>&;ioHZjVQFregR^c~IHvSq#vE zggAfb<7|u>xa8!)LubMC`q=?HZE4UqkZs7zOvz=p73Hb8% zAM_^mg`NRL#M&??L&SWJDd`c`mwya>6j~yRQ>Rp3n|R}GfFD!G>yVbe|KuZxwMnah zQjwCJ$s^yS{F^w3eIyuyNY0R4bh;A%FD=&LHdnJw7aJH{_Zj+oO}aBE!~la-(GXRx zMeCQ_XlsMb0_h~7 z{tUWf0C)oh=u|$W!ItC-lGwnbfY%&K@UZ|oa;(~||0dsM60c`&-6@i)l@3PjV7-EL zcgJOQczQej^IiM#k>#vg@3g^127L`Tz^cOt{K89UmHG$)U*k2e4(hZBQkv*}D9RA6 z9Z3COtaDeQu$uYBpSouvEA5Rn!;zyEUS&tpz57eZI zPT!2-R3Y34kE1-|Dgw7$wvGMJ7%|W28DdP`@o4w|mA?(Gp*D#=yZ-%3GF~YSlBJDT z{$}a;lHe6jxMe$i&EKv~shcS<(~(@dS9c%}%c9Lb#WRDqn(xeVUq%=u+eoEjD&oLI zhk(nc^Dm^ovn778%HL?w`GUKSwes}VA5x$g(r}}7Zy^4KxA_`>83lMcH7^b~e>T}$ zAS^IKBa2EZIV;t!+j}#Lh2m7(8zV^_&AKWakkK=*VaduZ_h-#Usx5zzScwqHXcm%| zjHFoW()mrn$>SP!V5>1*b6r>WVx&lG==T#o2YfZP6csQ?9UcnQv?5OO;RI^p^LGI9 zw?}itA`AhLC+WM6!Cydv1SNt|wLt&+t81?Lxq8)_A)$uZ3*dlK9dWBRyWCb&V7sGv zDP8$F8JE@VOQs!Bjv`lS4Mk2j>L&RPTU}6Wi(o*ac#6BnxCsfSwSHJ`p8)w zd-HS!1;!s7?K_($R9j=8Imbh8x@xUUFC-d`P%9bnZyT@NhD^w;6o2ATQ18B-9{*md z^(*}VaK!k#>V(7DYkc@hKGcANBp6GcBb$%tEEa(B=-S6W*%vfL3wg6$E_43+=LPuw z`Pv393gvMXOqkqID#burXT!O>=Xbw7@&-Z($QmF<9{IpP@xyCClyd_ar+GW=Be~~OhouiP^F^on0jxExMvFhDxz-q{zM2Su~` z>ZVwv3>s|Ty}h|CzXNj|hO87%nP1?QfDf=3FN0Ajao>>g^bnZYGd&cNJiaUZn(`;? zzk;t>oNG|856$Up*mz`7V0Xgxvas9RobIi9Xg;U@+yoah(Vvucw^nak!`a@{xP^$y z%!osCx4}FeRxkU_pf}LTnK>Z6`3$J15Z)(qT*?OJGi}@ZKR39c>fJVdGQAhdOS`3rtJ>BA(}xI zyEC{7rH@ED|9Ra>cmVuRuvSX{!(~KVSocgxxxRD8A`1n`XHbuy(3c|ie4V7Oe>AT= z9R>ttObHW`{}5g`f&;mf=l|L2nScO46N!<+i~gHq!X+#q23aOLSRTAO7Vdn4zY~Wg*0a5mpSi-3{XSCPu_;?2F znVA(4zV;}#mumY!W8Hl4pg32*Rzzc@$l)#M-AF1gNC(UI5VB?q92uOqAo@H%0eQ)j zOc=x*$$G@12pTybfaRr`HSda3Ndljx`}7lX(bj2fk|2aX+UkwWmAggS@B!eK z@$ySZ4vqfuw0|YvXu&oJV2MZzAyl_Wt{zkg&hw(!T_4wlT|^JFsj&r{mI;XM7{Cok zhgt9*D4Q=q>yrmQ{vB*fgzdg?%_a9yrty88xkaBI+I!LxBkSLtgsAu|y6M=!_Vw*c zvH_^Ftk55R#}bHLM&OTQz%$Ioo4zezk92TAH5&tbb~|9vbE$$8Zqgv0b0Q%F8UWNL zJb&tC>6B%|4`4{>dmlM7{<>@6$qYC=8fN{BkW!+MieZ?>CE2G8SeOHKo{xV)Op3Zr zfqCKjqf)Sot3e)M2MkOelqr6+r2M_W`{*VT6@h@)&~_`vA*W{@g5DXVrM60F{iVbh zu{^n}fbkP~x(*cRRPSVky37o9-hg#VB%pWgc{VmqOE253&@h$XIfV>JYW!RIeTitr#kVOPt7@NDSJBy(4G!U+5E`I@^ zKo&7JE}3Wz4gpoB7?BvHdQx28AcCL^u0}9r>;|4^`zWD%RRzle{z=h6!e$K++x4!E zx&u4+{iTe7(la)XK)W!7xXA9cJ^u+1KWFy%p(&QiJlDNS2T7$io5?!8XVH&0YVRtI z8^vcCw>($ml(M0MWXt=&{0sjVQAQ9iqQRxf?ieJ5&qP#!1iAvQ&PO+%U^L^_EmfcX zAHYDW+=Z?cqb+Ysz9@|0?724r9Iq)y=$@FF@7s!$!YJ}Tn!5^@n z(v17PaZ&!Yh_k4vWuifuRdk!E(Ik-61O6_cbATOz!k1!;#qiwBsC|7j4vkv_n+VMm zt`6%oMXF#Sbt*j$A&vT53_lD8^iB;tdH)Zz(9#qkDfQ4OnQ4m0<`X6OT@$7A)WF zHWj~43U~sIbE3InP;SU_IK}^wByJ@eer=@ch+;DlbNbr$1LyzX2;y%0=5692lDV4x zI)NdJ&8fQ&8`Mdq;;)?i%74;BOYK0w$O>qCo)?G8l-bG`@67kYiQE+*v1hIdI{=y))L;HhedUl>9VKZ zC7TjT#VA&KXn=Q6e9FdA;?`&SxMY&epN`Jzp+G3E!xR7IB7;geql~uGNaCAtYFQ3PrL)jUeV`^xfSLZOE zviBWe#3dp|T;?1ag&jBqMOxIe9Kb9X&!PV>UEa^trSfzXnwCl^NarjCTD#gM??K!X z+(+QTpQ~C(QllNJso`}Dxwu#)zMGOn6bSp{3!>W>3x-+?TlKU0wjDrTn5~S-)nUSo zJBSqIjl?+7^h>MN!b}Ca0ew8rH)VsDq3tmvJXYx_z5agXvOpR3!d-9ws)GH5r(5!L z@yK>l+ULAxhIS)m>>?JM#)>g{&S@@JxlRb_ox0 zgB5;m`m^#=)^XY7{3FKgmnRGlyib05^O}@sNN+8>?X5MmJ@G}&yvDdY()<{&#EYJ# z5Twau_hk)AF~(LmRBj#8tdgq#0<0l7#-BP+`e?{$o|^C)EiqTlX{Sg=y+y1;R*TvW zxg9)vR~dTM_e1_~-+*j{>&t_ig{fqUqeub)IYJ);da zgx`7uz!V!zP4+R=rghCZ=IQjVL%^#f&!9|TD7(5ssfu-unZBjH3u+LOL>TmQ#n2QN zg2Bcu6Dy>sKJRXhg-YZ)uMYF7YM2WDsnJf5rp(m*LW>>-sKJA4 zcE4H|+l=rDyR60;jYHAVFZc^G;)R7D^3uQ#e*;X}3+*MZgde6mdP2TJ`ecyq3a5<; zJ+R+c-g^vPmUDoH5YGDIg2x)C-g|ZM&agrAR0RBV1_;S|s9i&u(`^W#9zuQ)NqqvO z$26$jA_c591^8wyzsbhZq6wgUiiOm5AsHx5dE^%Y)+#1rAOu;fRsJdDgNS^ch}i>M z_9P7fghCdgty;;i{uUue69L^i24HZT=4Woc2|odF9}ED!nJSFTMB$6jD-;DkkfM)p zI!I~p1dqJqrLtJocYle-%ri1AkkGSaPuHAB_4ro)Kh(W>IMjdpH_SAav5rwuqQTgg z$WmFyzGlr96{2s6P?jOvNTTdZDN9*eM4}``HKG*SktoKRJ(N_BV1fEka{0G!w;_A6|>^ePesiA=o}cnETh1WUiM zx!6w!fTSf7W}~v;&9meRze8{p*%`4pEM@bS{qHJ4AEwqqZC|MAy>r+JNzCX}f@w@n z!$DRZJS&QUu~*+5F+)XafL?HR&j;QgJ!-Sc^j`)q*3sDkBMBXjwbDco3vTm#d|+We zB0iukk$?i~Fsr1En%#B0u?GQ1G2Zw--}D&mAa|1~9JF45ELV)wYCo;E0$InWy|)L= zk~|xvaKD*ju}d)hyPMb_0-~6VK~7e)!Tr0Kc$*0l=7Q@zu=^WgS>_laK5d#jry3)2 zd~~6|eR>bN%9b0hca!!{(hD^}h!*zkjS0&;-CoRz}@JR($N#)Kly=wmjvo%PU`xNflI%Y#~nejihwyS zFzMIaN_`6VB1xTjWtx`wLVEH}vt0*x=1g^d4)HgzQ&yHE(%( ziKV>;A5+8W^+`|NhZ^l;A_Z+f)Q87n2)9pwM=#E5H5*!SOAk$Dok? z%IfQZX75_5XjZ=rt)D+?J!<8_SK*q-kyJj}`=H035G?Yt27KZwC`Q}sCx=Nh_TtHltiqKwBYOO>XOi(dbehjQ< zmxD1JkLVd&f6PX_KQ`iLN0V!$%VtzV*jZpOx3TdOAhzqu3d_2Irk? z*gBnOH|XW|zmEcVeUF48V3m1zUFn?B$n1G3s%0j#F@!myhbU7Kae?sh&V zN@)hrz%-PRw$B&Bi!7X6s|U*lzVK;f7Q zykO|LXR1bQ6>q2UKeS3_oY03D~KH z$S01x^$@&r?XB9u*JQXK5N?$XO8L@KQJA=c$dwR(sej$+rs5hlo?YI6&DX4u{5szaIM9C9O0fJ7o2I{SN)pOqP?SqwKSJe9VOy$Qw}Ko)UDXGRpk zH^7=(fZx_@?FVR>)a;NFxYW6-h_Dg$ThN+VBT2_VG<0#XZ_Wzri?9vg9ze-wJ|$k@ zwnI{PHCgIm_-C`BIc$MD8tV-JkzH~Qe8eW`h{21o_f~9pdOn00L+3wXn-5x}=MY<$ zElsT*e^Y#_#NwP4BKT00i;Tm;5R~V)Fp%^6=;VFg^@1BFV7i7%CZUSY;Y9CaHEN=J zhoyDL)SwcY!HTG$;UPd5)UqnZ_uJt zB7ghNDF(mMqtmn&o4JG7XUL29J*dK7irR*sobzA$bl+4}F2miWj_&N+tW{@rKwIGO zju>#{$=yRq*XzH&)HElfK_W33K$li%y0APIGb~76MN3~>jDoLh#!^JIuQLSph&MP3 zIDBaYpFr`Zoa+q_?;c*>N`2J&N_Ryb5GJG^eAc(J96G$O=-vK}9K2q_5yBIc+3J(2 z_$8{6`N@#|jH9v+^iEY47F}&TNpbW-HL#&dY!ZoH3JjaR_qmv3G!_o&yZT9Gb?rsn-b4g3f zwIC76q*z5TXUf$y2!|BPbqx;`T2JhvNW>8-4 zDfzRT8=nUQD+FdCQt(1B4%OvZ-9*2`-o_Nn@ssa=w*s}@k>C9_buYCof zLGaKsXCwgotj1P@RX{m9zRUY7!!Ni1LePOT_?wyJO@tf(DK_auI70%> znE1j;&&Ybe9`)JaJ>uuJkJqFh0W`)uts0P*l+C&o#Q_YZOcYXCaPo_%lOy>4m!iUr z|8GTwvJV^k(;y%cDZM44mzUnc!1)0np8FvX#R5%JSv>@^A;KGg$m3q4uM1;U+5gi3 zQF#d5U*j9!dVD(wvTM8B2X@NruD-Sky_8lQm*ovB(wUyz#WII{E2iuvoMwt`z%$Yu zu`qATLyHi63&-0FV})iu=P+~JGX}e11uDmn2O_^ux*#*gWZ$EGFbv2?|9W?^1k$vD zf!YYT;l81&W)XWa68JJuhbMt0im@UB13ClypL_2eHNstdk-M!GhCe6-={FH(*U8UJ;;`_7D*ij*2H#r?Bz=+MfYy} zL$W#o(ag!dMn&@A*h(jY&BxT{zy=cD0|T+z4&oew4zD*agvM!9`}gftdK}9eVQMoC zE;($md;tbH3Ol56_z#X^%YXDNyY`|8L~+zgrh_3rVZFFU9}0e&(x-nL@34SW?n_NZ zke-VaLP4Umfz8#c>R^o9gBt+^kPhCv^oFLbH{ej^LsMd_inkyC#xd-lpmh$zlT z%We6=5F##DuT%Z21=w@{+O>ddPp-xD;02%`#wX+ejW+hq=gc4gh#NlRolc?Q3Is>( z?WtGi>OYizP3&ARt{NZ~&x>D!YeHICXMeqJYm6<4X>%&{gS&4r4X`llY=SSJu)T3S z0F0SbY3d1uxt(Tae3q}s@{C3#pQw1{6lzaF?Ok5A(;uf5;SDs^FR}HV;5alXRh2Hiawp^cAWt zwZbbfq1Hh!X1z?#{LR?uRw z(K;l&Wy{;occlBj?JRkIyWWOidUW*8(ZaTl17U!cnbY^^LJDH8i}5kfWTr*CZ?0Ue zp`wPB_gP<7CvN=Ke(v6~px)b}xX2F6_{AzR6r-xw=6`&@OT*;zEVcsWS?R550==R~n zP;d;H;aVI+X2=aP!(aIdzc$@P6bsbqMF>Qc&})!G z0%(E*!@|NyXByp0k`I+`54t=?yuva7W#Hct1=`yOQIk%{9^6)RFt_6 z!HR!hVt@e=)Uve~o^lHooOVG{h&FX$ZjdgRdSUwU;=Oxo&OR&WzcG!v2w)4Xeq%yO z6EESsczeJF&7khbFbm9-D8jjh1U~U0xJ2Go?KimxlaJwXITka+z}tUQ4=cC))fZ+{ z)7OK(@cWcN_1`>0F)OAbFNhbu{?YU_fg>Yfn;Pd|#r<)9kTn`K8R1wGA1q>TtZF@| z*YF9|(uKrCbCugRe+n+4QkM$9t+;q6I{MN{^xEJLo(}`Z5!zFMR$1h?$iWPORMXRs zR-!)KIehNj9UEc5gin*cGI|~8qXeaFpiORAJP6=S{&h+1@Na2Am8*69w8;mF#yYc+ zztR?Et4nZ9#im}xTj}XYIlT=~;dJtoF@0=w_9sIqwuw+1K7So$OoxA;ltE0lsZY+oU5pOHp-`N} z_q{*-lx3Pi!k4v4n`)`cY=Q9*{?rm^{&_HR8SgzlrRGxOdwvtjJZo9y6wKX%Mqlee zLGyL{5yXut0rYhaC;-#{!e!*#6fH*CJmZ}=il=&VS}#L<=I-{|$B#609@ zLNN4a9Tk;*ZjR;7qU&UL*;v8|f4LTP6B9f=cE_l(B(+1g714E2xt-aK0yR_yaO|@t zB4G^P>L6b{2T>YDaovmfd5ey@d+43yoxUC&6J{~s)FFaK zL@@r;iJdu0dfrb?ZpI9PiZ#|l(z>P)brCr1#I2l7 zAc+leJ$*;4J2JU4OqKH!SI58Cyt*S?PB$d}jN5$#C~AZfoqTvhX_b{;#>Olc z#@uhn#QZTIT_2x}ig(mrgEq3a{}5U*kS$OMa1KFOSOFGb$DW5JMGa4Y@q2r{ z7fLnE{|+VaMiJg_!dHpo{T}ZW<8Lx1&a2eU0usnJV2*Z0_I^Lq)7Cd#dx_`U zkkh%@=m?8Z!@0{UD#p(Q(d>(KDW7oz+tcuOsr`J%Gy(_WUPX>9Ime%B-1GR!yE_qF z=Z0S7zTXqltlq_K%@esd31Q~82y28B(SCLFOfq)mIFVrpl5AZrlxD#g(%3WeYqRUe9Acs4r^IvB>3n! zFm&?V8Ur=w_}aPZR~_os6A4cmY}VJCxI?%)L`e%-|2**^KbF)L=c0qoT)6Tnf=Y&i z7w8*kTzqae{lT<+ZOk5JPS(9lZb|4ZDDW$Dul#%!jq*_gx=(IZicpUH6%-r@FZK>_ z$R?3_Tkn@^&9~dzf%I2x2uWXt&y%&-VvRC0@Lp#M3z0q6WefXq(9T}pMjNilWk`g| zme%*>rg#dnHtM0pPH^g&1NNAePigoB8KBsz?Kv`W^Puf3^UL`V=k34f=J&NabKmDzs*#<+-litkb{DXlBTSc~dnKFTKB!ome_B`dmkr(PxpWU9! zoZwc(nN8bp@#-be?AP$&58>=<_!eX7?)fm*l-Ii;Eo|@5H{rHpaUuk%cDL2i%zPY= zRug5LVBe+kE!Bac9VN%kQBKB^gQR7>^H)ng8B-_`!}h?#xf}|T&j?8r>95 zRsT~L(_&sA1b7MS{`IeCU=yO24Ys^qH7pV;cOXFfB9kxUONC#HLq5TPQI_sMjR^pZ zU^?;q${+J$RVPXfAkx^1PyT%lc76=S3CE9%V<4cbubbTflfJy`t&*WtrX_Z}*Z$ZB zqt)hzNPUegk=&u3ep#L7=AXf&-GqzhrWrLi4Oq;b}cGGQ=tM3p+I9W4=By7t2Bh3E&(ZWtM;_Ur<2F|!4p@5wDP@_ zL2eXbnWq)D)yx={NVyCQ4oZM7!44J)AqGP*UKA<`q?=y3wzaYV#%L5;THFRcyJv8l z2~W78GOY+-^Tc;y1zvj$%b;HKN<4P37)S{8_f)hf)E0Ava z31V$wiy9eVh;oi82pK|CAYNra0YC$5^K*&{&mdT0cPRuD&c)JH-ok<+mj~f@=0Ie1 z#~c=UDMH7U4JGwC46Y^68lbkx|H-k<@X;Ewqo+5uk3>9C#8? zllw56l=p;IYbCKCR0i4qywWY%dCLZy7AQwUty>=L8eqm?E+Tx2g7-g5rU7}?B!gCS zNdVa4<7fr?O>a;c`gB@>w6_GvZSy?Xo0AvYl(-T2=$KXkYaaNK|*^BU(S9bU|n@jA6^s(L_hAfocV=Hx((Q!D8VSx}&on zr>Dh!pw;~v8wMhbbjD9uZWno%*uOy!xZ1O)W%P6;;qrVYiWlCo)5F4)7>u(?T1x!^ zm+s7?b>}3wZj8p2)|sNr;UaziQ_s46 zHxHFd^L#n#Ax5U-8y*)e?dSf9MCXNHk3Oip1(#q7>LM>768377G#q1Tro>%X zlnVuFD;|e==xLTDX`(`p(r20K_-So?T3K?El250lyQ0wf#4ZSDUreEwUWgb20w!n% zZ|{L~3LgX?p`J+_hgdeS%O&6CdmxGi&jAk>UV5n^9uE&T1xW~j0{>6U`!tiIejyAoTVde9!uQY}G^z7|spJQnzq$mx!X9-EOH|D}x;j=VpGMu1GdOyaN|=$@)W` z`#C|Q1~=FpBi9e6j2B0O7e{{UA6-FKKDgB$qM$gnZvgs09(x?AYQD*lAEeM8V8 zAsM_2fbry#rsn2W$gOCsYd)BNg6S8mS9y4J^tC9uMfYn8{E^m`o6u@vNzkv`fs+); z$sr@j06gFdU~bAl$>ML$ATkR4lJ5n4W~Xx??XyUzrFa&8oii|9F^6vxTl8`<6Nqp(BF} zS{69y{Pi)2ZP3Zn;%WoVH;LMBQU1ZC4#()8;l%u#A<)`c27w^z%p=k8Fd?Ig zGEPa`(9IU2u)F%T(Y@cxkXz+Ni(x+UB?TPSKhrp z>S+(+K5uvmwD}X11*ZwDGvFN#X3RjK)y{+{{XN3muT0=bMVhu2{@CP|g=XD$9m>9M z8lH|K5i@$!ZCC??+RFYS4v!PNE}O3lzMG;+q3lbm{n4ry7d_Gfd^H&|~*52+U(d_|{lyG$}YiY4>#j!w+au)Voib0J2 z?5^_tnq44<=&L%CeG(=bWXZe?itV`$r+sE{8Qt*Im;%hKl9{>ZXs( z?%D>CQ5$rl>V3N^^>A2<(cH+3GM*EQb!g})zjq`ke^Ua?104XXlnAqO5$ zOOT)z4oEJ^u-WhT*y1)@%Nc+CAzDznUcx4LVSnnjklq&R4ZIhOf4in=uQ?5~A3b-< z1(V;6_)dIK65XY#CjuNFvM(`>Zs`TDqpCN276!>BwCSA8JsgB49(PCX1101nas++g zR2N?fiur|-edW72`?^JZF_rAlcE3S*9&{re^4vowgd(7>Ma|1;i6*$x%6Ygv==yI! z12R8*mDu)FZwF+z-y_&0JctRs3i{BN9zue~QdP5%rvGpZTfL>ABRP+|UJLKKEV<8( z@8Yq+r`*FE^dNY0%^Tcuh*<*51@LI@xzHK{fSz*Hx_fhmE=jcwB1+qM(47e`J4kIkFIBGZ3h*xHLs7Y&Z;@*feUdhxwhsRZ~c zy?ihXk3O-k^U!yMOh?veyJM>b&0eYw$82a|n5$`zp+3uh3{%zCV$XS!;2@$NALID- z-HFB%q#LL=QGw<{^M_7PfhWdst@uIJUbVO%tc({!u0_Mlxq?f=XT{welrgUx;M_z)k#NU)Hc#MYJ1)H|83c!DI=;sw;c{W3*T52s@SQ}VRH87o#Cv%s_IT- zvaZv=^&7-D$jpnN#PAcr&`0ZgvdJ;P2WxnHTIu~Egc*Gz(5ZXBpZ0oH@M&VXy%)zk z*M0b=8FZ-jbv~U8O}!aRV;LmNHbcyc_zVD9tzZ|c=lSsMOW#T~TL1&2H>F82KTTUO zY=RU(OCo?h>j7&5A+`6Wc$JM6=u0Bsg11Hz66UQ|p#t1xo7MFb>yEtf7?@Z|bW@j0 zd!Q5Qx_tEmjJecy$elrmnn@Gkl5}EMg?A%#19qTtG?QKPPjI>kDE{YuWhnr`U=e|(O_v6reEiqb>Y;ER>@|6Y0iPi-MXtx?S{@Szkr7RU z4(&Vyjzk)v&JCPdWi^Azk71ck;Gzr!iY|1g>O;dv4~hW2)@A4ofYHrZ93Nq26jzj5 z?9PASC;9-8=@_B_%B;a)a2bf}lzEU6oxb_IyX2M5Vq4%xs4&^YLlBLM3hqDs+t8<5 z!U0dPtD()@lvs&Hv3+3g+jS8+E`f#@o&dYTx_ygx)J7seVwD9=K`V5S-thbyX9gDm z`a{yZG0=%26*tPz%)SXN9RntJxSnNsLDg({8;PgcaD?1_XU#LX_YyME+zS2Z%d3y$`5X*l6bf#6%g^7(#bRqM~}e$G>PJqqN3D zXwwP&{KeQOkkLB~FgjXQL{$Jpz^>61qK#&p`#@T?>-_YjFYM~<36b;`?;a>k?j1*c zeD>6Z!M^3Q^^1?8>Y)%gil`xu96=RWC>k7>`|mNLdT;gIorM=i4%;E$5XyVr3`bTW{Ha6vPck z(REO{s5Nphd0V@fVjEmO*Il={WTlWJ1f9JA4TE2Nti9{IzZ24qdh&`Pf&IwI3gvHNC4i# za~SbA4U1jsj1Y~lC4Z`=2`KDtPv_-y)`iw}ztG(H+I8zBAhF@=?u*+rgLf=POcY9R zZ42|+Z#Uk)>33;QUMNhVXr^d`;W6SxeY^0*+jO&?1;rsCPeE^**n;Pk@>9iLxVWx*nF z*$)s&S?B4X#T9>h8QMizJ~hCBz4^CZEwQ?rEwNv`sbIatriW&%WD$ZRl&Z(qHNi&Z ze2OXF4W$WyRwN6hO~W%WoPQwh)w5ANKyw!s2=dJZU~G4%Ler!WS2Gd6egLaiKNIJk z!zK`wkOSbl1q6zqjjoSK+JT-wBqAy-&l5|a*#YeWOJOO?x#_>jT$>T*`7=9Mq{3GN}>P3G6$ z!Bkg#V5f#mW zQ!g(Jy|=VBoBJ^PYzCd{BEQLUaHi`UGVz{hOtUAEZcFP0DD&f-6Pf<`;9Pq5;9+O{ z!s;~S;c>&@?TJWUMO7g`-@R)i1*y0Lp3%(GGPy%YfPW!qL zzlN_CIps!lI8Mi3bi*Cgx%K0d-5dk~q8sBy%i-Cp1@&_8!cYnWOO^iDL&b?Sp!d+o zN)Ql`f*?bssj5yfmc%>1&U7-^X&W3}+rY9@%v~@ZL9!@A5u`eI%B4NH67Rg!+|3Y< z!8iPaFD4ZBvKC=Q_~j(wbNar~?{B9dfDx>yx&=~zz%iQ&^rQ%QalkpdWC3_NUtaua zwbIk$V;qo(Aot}EW8y;}$5v)IMBwQz23F@Y!W(Gcu~9#V1vY2|T?75r)3JA;BHIzY zTu<-eNQ`abcwi{B+Og2qP$4dtv)_=j_2BtcOcLU0tsf~ z=}^w&t~OM+*rD&YV~^@t^h5ds zHZcP@L%LZ%cN*!%fti%@fB5rCadJU->LAJ%gl2;QKtWpD5G2WFA=mhR?K^zmkKeMt z`{H`v8%MRgkUw}_+*zhLMy#E|4j|Ff0L^1q2N@Lg9N6B?z+Nx}*|wP|3yt#-fW;C3 z#K1~Jwb*!5ghZ2aA?Dc$~6n2 z?gLLB9`FHlXzT6*FZ~?$rIWw_BGMBGk?E zFr;d%+YZH|7z{X10mDcF=iv6KML_71Ae;G9j!xu#J^_Kx)o9U$F;EuC06qyRD=R;R z1~D1hr;QLX6+HB{=2iOGO%^BDOxMq73vhP?09PjKHb;I-w*?VpqwC^);6i69=xUy7 zeyJb4liWMmK3p5MIPUa*sc`q_wB+DwaRqcLP}BHnX`Jz<&kXvPt z?(6e=6*0LQ$*BGG?OeZe>;Y!Wem}U{O5EXsPJ*JUp`|Fe9%Wu3Z5gZqxE>#Vpe?8H zG=mrpbLqM8QNwn|Dn#K#PR{J?>DXe!;7jy@U9UA-KzA zbJov8CYK|3HaU+ZDS6!^39vu=4=#JRKkIS~AQct{~?k9`l zSdbhWxhF7gux4-p*{Wo|v;w=ZEA2C!Kv{Ib9Y;%uR%@yV(gI=q$+U0#<&XD#KI43J zqJ3D5eI8sh=7ypbSAgUhxbgef_RzlG?*126q24ZsgtSSH;|kpjx^TT1lQnz1vsy^RHBqNoxJ4wCs<($4uC=d>*r3I zS|7$rwaR?0Ptezu3Ff+KDt}KnI07yq(p`2D*m=eDN}_<>Q55cobz%*y=i*{QL{}RI zL)-0=W#8mZ`97YVEZn(z;Jd59k*Yosp#CR$2u{^>rv0XmtDFupfF9J11DO-o=*B!Q z6K;yZW<2>b!oq(^7op}t6Y(#wpUMP$01j`u{;&9plokQc{`{Y2$ZKR&4j91=;fyB} zT^LSv^ixoBe3J4rDmdfDX+dnGUmAy?eo8WhMFL*ECDzNL%8Jg z1elU7A|mP`)ZYQc?1EfoeWOyzDx2WI+W8eOe$WyGXSu+pk@IkX?$76hkGd1t`#(J_ z9`-P&Lr?7C<1+IC1PpoKCO_D72q?;@YRx4;tZnB%mzz_BGvng0Xgf=uZuDMAF%n1B z4!~p!5Mp`3k#{`_(sf!POVAt60-xXT8qr>tLoivs(m?Up#3i_v;-WXd(dUE!?6Iz@ z;*;_`d8J6h3obTO7;zj5(d-LJrJSC|^=S4r1B9OCCrprX$aS|DPchzo_lN>A5%*@m zC}su&2Ez7_j=hyY(msG!U7;hGzCoFWPP397KH4)uYhQ3hgE6MIcpvaa3qnFz&VJ#0 zAyXGXwqyp1uM()dErX~I-Mo_zVSbPjG@ivaF_VYV$BT-Ju%2uMgqa(TH!h$EvsENJ zP$Na|fB5ta$!F4`633@Il>WmJZwDv4C|YMr;Kkz?(CV=#Cn&eEm=TzltX#kxDG%?> z;r7wpHNG-H$2i^&K?DitgzW~r047*VvTCo@$8>_E)^lpPc;16dlQ*u|_eg|w{pE)^ zym(br6=v23-c9pWNO+ioSb-L3QMER90J@BpmeLxJf`14}Ls6s?0014zf;A0E`H1ZT ze=eeE<{*OK9J-Ve({wSB)?psF#}32sP~{l?rdvsztUR}(iLDY7EDGE?6NQXoF!T-I zZ>{37W*^W~a%DBp#uFLmtJmY%94oF@>~_Fb?6SnoA>`ALhdTP~5QnQirqm+Fb11q# zdZGW+%8goikAp^1mYN3)eJ^zuI2`gLDgMkP2^`Lv!$?qClDrs|Y#k1eiK8 zn%O%!bYSTRUF85beA?fU7Ubsgfx~OabSPs!HEh$PVOK?%F3B@iG#SPjsB3YX(yU=E zJMX#C(SuO#Qkun;>*5lz`esb84(kz>U}{3arw-w#L*myl*K56W9ca8?|EB+8JCW%P zz?zO?G?h;C2S%=qu{2=t%dcDs7rHeD44QM7x%i}oJ9sXRLh|k+#=TX+Bu1HZc4F(s zO+FaP(a|EKGbMoZz=zH6bIe&69c_kKIIt2%jn&2O;K0_HQbf8_cpEg;)C8oJe)FmI zDN81>!%AbfP$qaC`uJT_Z$Ds`Alnjb^5j8RO#8D+|*Ihce7U!q>-L zwxCz&jFz^0*QLatl7pP`)sf-==z;{F;5a>eV_w zU4;v6;IJ+Sa$LNd1+gI#E+9e0bXl$tv@_@r(G49CmH$Xu7sx{(h0{gIpf-mHhfNcJ zKp|9c>|*5i^Mkc;%*g2>zwVCU@vN7R{r?t-!%#6qCkr4A47mk6q<{&{0Hs`WI4re& z@;=J%{*HY>OnXDqh~&9wv**i!1)cw9dKiWsXo-M>?}2t#0#=>a{p4;;131nC;!8=8 zI>9jJ0L9gf0u+$C;;%6WC7vQ{LG1pt|Cm6iJ5iX)V!)<)lmBi&_+2n=aQEil@9ZbL_p(j}$j0{a!fiT|#KTtQBNTgRAt7&Z_$_`QL} z^J$<~eFoV7Fnq9R*$3Lyh0;}!`53B$lNd=uV>eh_%i@>}4Iwb4{U2bvuYUx8rCk4Q zk4Dyeps_})RB%ZkjWq-IxE@G#WbX+^yMod1fjj*lEHK6u!rw~b??OOaf7L^f{#@Jo z>B9#Q3*Ma?GXC?8OaOLdXh<{$Kd<1--s3C3e`K35?`8S@{{$DBCp-W#aRyTSfb|$T zZqUMx7L8*gNpm?!CItD>io26WkM@E5=?**fE9eQ(7WwR2X&^WVJEU*J_a=LURecQW zLG9YmWdlo_(7E{j7myJ{Z>;u6KKj$~lv~igix3jhp&|VgTF7m9ZsZR}D8+RQtKqR7 z+bB~@D~|BjH+;VbfrAtXWZe-qSF+Lg&Ejqp&8v$1i4DKK}k$9e>VKq`-%MWnAE2I|F8No!428W8F6 zR%rr_1|ZkU?&@+5mL9Mt1`P}hc+HP@+}}Q*+%LLM|2e|Ag03f^ss(?}lH7Ylg>Hj< zwG+fXqHRw%z=8}8d2LX_5;lP#VQo`9QD;q~Ou|Zx(;5SP=T4m7HHciciDj*lgS+n> zOhTpm@k(cM&I7#n99$DwId7WpEq-?dxbsu?6Td(I3Sc4@ytpUNR;=u}?v=x$XtdgB z!}fqnr{9saUVtQXm*sPCi6i7L_R?F_(*Q*6P|gu+m#(zP+NjJPb8`C|y?VI%GEsbl zp8?qyVRmc=TM7uVG1?uc_A|QeBY(t)l{{3{_=2m4gmZt_fT+JP_Pj7GiMC}fukCrN zlw9^t6RY&nfzM-1mVGDj4U-$O@46+w#dMsA&LV=ce|3|&*zgeG?WSY7%i4xB`unFdx!K!_1T!Qq#&7mKynUuvTvgP@2 z!P0)>m%FBBm6t2J)@Y39N$#}Y^Ma|Zl@bK#8wGrv(D0S$vT1&x?frQYGDm^e_RG zFcBlOOV*-i8FU>yE+gd*J1gDRO33IQa>g*835c%?F;iF^lU)|}m#XLAm56CuF*R#R zWMkWmTl}X?Ry9LpBCg9Zjj6~b3rmn?xOf=d_)Kcd4HlfuA=|e#ey%Ielo`YhdX;1v zJ85w^9AzDV#et$vk^xx#hK?Z;g|#}^lU^W2Bzuw64p=h@E7780`N*+3EkN`LBz^7l`R5N#PdX!R@X zii66%Q{%#m)wzrsfO?9T4-9%%Ybi|O2 z)Ai9Wz1mU*Xn_p0jL)FvbbJTUsB9kXX&>f%+5+S&O3!JKzTmxVL20pJ%MKG zMaw7>JOs_t=$l6%5bPX^&H$)^ku3sBCySt?cS@BvpRSH^6lx~m4&D1==HPL>S3@8VwVIX1+RjKH` zK+bC2`jLO({iEf}^|mvnexRj!xUziw9x!`^o6*ZZucAx^9uck4W%nZz%(^)8&_L0r zoBMRim8_SW%E6*gEwtRr3UX+f8lC|fHH-1n!FkyFw1~D)a;4LywbAPt*&RW-D zZ%_O6zP@nc+kH@EhTr^zbiE-en~13$9}b6`>)N0C9OZSJNSm>-Nb^URt-)@MJ&nsP z`;Mg6AY>cAC<~3<5x&o`-Bd}ELoHQQIQ03a+95=og1z}HkbW{wl5Q(#Km~Yz$2w}2 zTf%vm@r+P}h1$~#u*9qsmi5s-UAKpg9Ycvy1vX^~!3{Z+Xv=-*SIul3%FavK?3rhv z0rRm1piV_@u-m@rrHA$$Lz5y<`!FyCZ5u&BCh;PH6%E=cUF$zm$3YdO zTnahD5uiUtVmIV}(psRO*y`~Z8#Y>a3adLhl9hGrZQjLJ8F8jB>~;WsbZk@nZYt<$ zdc?fNpVe>*SbLU+>V*8pd#BI*gw=Z!RlB+5JZcGz3i#?@#?J?tkA$GS8$Sh5TjxGl z_IrHq%t|tpE150Zx|0icVF&Locixvf|M|YAta|qWl1d`YjIS0vK1XV`&~lca4v;>h zb3LUla4=p28odyo-*CD6Jb<`0pGeTES`A0R;sl@L0GSB)^i&@zuF(doOp2egGu%eq zy}++Ge}@d;Qa_N61^IHe>tFi7+EP)Hr2^{#_k(EC?D>TfEtry>@&LylQ;UYL(%~W4 zWv)lSyJI2uhn~v{r((kCb#PQ2Jk#Lqz zNa!$3#xJ)+SDU$~;*oCnXYk=w{386mnt8R4>r->^c)K!4Kd;{wLW>5xwZL_q;#a|m z)>5|EgEMFlD4>cu?tLq|P-C1{q#!py-CZT)aIV|?V2SDOE@)4@d0RSgey3AYoq9lH zRk}^6;!($i6F;N*&wu*8tifT{ps)1u;@X@>1MWt4Azs|HM!#coP&HY?Fn7Q7wSfM| zn7a>m^jEA4eU1G0z4)&>6puDsg*}ODSm~q`vvwIBpyU=Z*Yp0A8lY0FSQ}kq<3#yO zh6==IPUqNz=D1z9rqSUhZ2B2)7ps>v4LGv}T!o*sJ)b?zTLuS|to1AKjke>dVVhr> zsU*`C8jrQsuTsvlT+&b2S`M^C|&sF~Vo2C^aDi1dA^|r*2%Uhu~ zPUpH5f9*fsmZRJ~SNMsST^I?~56@s8l_u*dL}Tctn5GiJw}kTL!lmWKNG>j>uiVNsv`p%FSK5 zDgGZ(QHrWyDabpmp6DMmG&1@W+8AdEm_TR(?qkrsg3VSezk^l09~M4N-Uk>Sc0~&R zRPd$cp~r38miC<=Gz>bQQ*voV;G4R2Uxi=)z3IeVLw>wMda*QF_6#VZ>?7UB&1;WS zx3TP>en{|KlZIb|mMKu$c)b0s3YG)4Sl(Vp9X@)HeN^VruV${mjawdk%LqS^Tu_iK zC>Gi9KJxs>V@;U{zYq7n6WL^an;7J~cEF=8@pH*}msXDA}UD($H z>!kXRM4Q(fvYpr+uj4cy(j+eWkfm3h5*7L6Q-3YUlENe1Vih$d`hGPWnI0IUs|DTt zv?o&Q{U_V1<{CIq*?Zvw&J3SU*rpd7T@FLJb#K(79~wF9zh^jh*4807_kI1xLwhg_ z9`Pp3xA!O+#@4CkSZ#c-a>sAwZk0LVD{Q~GIw6_btMdGshZQzKAUULO*n+3bwQyU^?Tl%+|yzjMR``uVB_lhL=YYDM1-vofuQSS#;`km_r zGr!{XA8I{<&Jmw}9d7-6MBr|Ej2Iq9rNPT+-5FR=t{n`l<=?{})ikTcv~w4yzvpSMik@;EUO76(hx`uk(x^^oJudl%E-PU#*t+I-M zs$|YIQ5WXstL)$;6?#)8VCr!G`-}csmp9Iez))LhmHb&j;!bjHx}gwM>+pS<#F6T4 za}V=Bk}=VI?75d5xjxWw<$_X708YdU-|tdI&%pxdMvVA?aSB^THd+V22m!y*v+4zY?kG4)Q2#cv5613ZI!Z= zi5SjyYM0dJ-j?$@;ogx4T!qP@#l*8%YW&DIXsk<{DT7O)$wtbb2anQm#8JUVIKjDh zdXp7*=;xK$Psjf-IR0Vgp+X6yZ+%P`{(QbpbRcW_0R2#hwf210>BJM;n%{lBuqhsU zP_)#qDUB=sKHW$^?S@rCN8^kPKSXZe)kO$`r={+Q8Dm3b1I`aH>n`xo;FhOw+?6hn z-wq4g_e~w0Z2rXaF*m_@AE<|ZRZ)9HR+Ma(+3hS|=5tQ!>sL-9XZu%aiqiy5WQN79 zettnmi*v@ezk*ZmHJ(^Yc2{3Vcq#PphG@(>gKaquF|&RjKRmvXbn5K+&6suN$C~)D z6hC4ubk1S(Y3gmS8-6B-UZI<4OT4O^*(G)NxKs?yCeLZ6J$z9+hDd8{d2K>T*l@t1 ze+jwQx_Q!Lq73EJ%G^!I{8{76`V*I8M!CYJU_%o<6{0Tru6ao zfLZARwb|ea-H(=f(Q4sMAm}j=rZ}8*y~*D4GiV{WSF( zG{RCjBWFVBc}?36Xp*mG$-BIr$5*hOXTst#3RovfHGT)})hg%ujE`R;l4sbN>*p}` z_my`g^eJe!7q*&@`NyZ&Q)@d{I13)m>hfJ1vK9)Q_S{b3nu$K$=Ud;!+iJSJ6vEaL zb#}in=Fp7l*`b^>oPk?;cUlG4XgqE41vcvX0log@7nYsk?w6NuXWZ!2pXK=JT!BkJCkk$+sx8VZ! znu2Un)qi9wZaW`)nIJ)`6^0iy#JbC1C@r!gtubb76devNi3FJjOOph?ul#Nb9ZDN> zWno&iV^NVqHd*R3sTdFZ?;=x6D;oc4{g z&H6)sUHz-X=SjJ|7`swDN$h@l@?XyIFjA0&ti1WVJ0yV}25-JM$-ay%CaM|@eV^6yb}|Sc!~cVqyO6Q{^nwcAuFq4kT{?%e`fQ4u{3{2bv{Ja6$yP{|2sy5 zVwQ0D$L1S1sYYzMG~`JEodj=)nv5a<8mIZXA2OBY0ibcsi<3~t4iLnG@>%^G9My3rQ}2vgXj6;DwJ?hbi#iP!Kry(pDXOf4+|yv^0{9 zIQ>`kL}1qeNMe=TA>qGc(H9MZgI9>fNc|AjL3cfQ~JDl zEP(ab`+4e5nerZA`rhO3(}lLUr_8lAigk_Nh|vD13M00RV|mV{6d+lHlN4vbqA~~L z5=xI2gE{IPOc}0U=60)Z1$(R$04@&q7Ic9A7<7IwnhRMKjwlU6u(_#8S`+4-@n?^~ zomc8~vp|;fGv{`+L48UWA}j_#*SpyiELSDa&HB{c-5T5j`(g@zu?C00=m$zDap<^h zLoVr5@00TXeeLV0zyC+Tdn93WJ_q(yY2G&g+HFC*Do;r^E7oI^S2h3E82D6FP#>5A zmZbvdX?VmnG`efSZ`J+%|LW=5t#}G@+>|$LSDvMH*M402Y);0B6m&r6( zOGtyQXz7r0-PW|pamiM0p;;wLE-4c!!Z2j`*RGLDa?_=BZufNRuK9~3Rz2bW8 zC8z^O)}pW?*Kxh3KzMO?Z^%GZJ|F5WHdIPXu*Vsbg-86=etI#yr9ruu0U0`f-=7w2 zP`^wlo6<2Nmn}j7y9qe>UMOffC5TtdKql9{JERjBmbxpW-w)!G)fm8cI$#tUhPDkC zzRkPPanrn>Gvcwf|F#>Zx(&z`3|0{ zZqR#Qvb_>@RhpF|@jf$%XgG~-G=f3I6hKs!69b|aJ|fz74 z`NA{EWP%InF45)F=1ab5LSj zx{P#jvh=3LSNFFviTg1jV(}?stXeK&81e)*EHZ<20Jp}CMN~Dl^R@4OR9Ao|bIy5Y zNkFxr<%Z@#Qan#A>y7W8&?b7C4!x}+adaz$ERA9q8j#WQW%uDKeSKjjT7V5>4+8D~ z(thm1Yv7u*)74ERfLIu_Ej@`xdH_~TkvPE>+gKseg`^n?Uk*zyewgPdDeSTGw_clq zidYf_TYC%a0M8X`pK@Vz#bN||CvsOzG3ws#prLQD>r&CJwl<47{Oa-0RjukHVFG?t zEtq~k-W20mbutk=L-fEw2{37;x=lQpq>7s-GwFmh3Ax%SFVKIx>-V4c_Z~%@T12H(Ar-^1rWx0h24RG zuk%>N1VUWeAqNqf8i5RR(cVLQWJ-#FiO7(IDPec&W32xLYdE#~k0|LzoE%0w<4eBH z>*r@qwN)+Wh#yxR4!>USLZ`GO%Q9bxo2>#k70Y}-6j{bD=j&0|QkmwOL=b|oAh(8W zO-sKmA+&oFxpHaMrsz7TetoTE?%=e5B;mCRuWsI#saWwddqf<35JW`wB)bF*lR(kHd zr{0cl*g^LN2;55T)RwVGffVR7uf#H>u}~i$70ATCZR|+#k$UG|=!z4jFU`rAHG!KU zGraC@f4k2+i^7tKICDe;8DR}X?zI_XByEr9`N6c$nTIulE;th&8>Xr(YkE}AG6)Z? zoH%*qn#B||OTCwK2;Ogv(vVHN`nbX+LU8TjWTmFxt3%_TeM1=ve6c~kzqFLW-B*8# zcaAFwfkFgwXZGW1RGpju9QRbniV`ViW z0?9ptTbPE)Ya@x>JMn=10T_{?+Z$YzOu)Ap#6SCd6sG^1fo!<_ zB2Y%N?(aan&;GDAkVB>QAu3kpzc`no)wZ4u-yp~a*kLP|vkwTi+^?>=rngg%ACD3x zic~IdFfnn@*SwRp2m{T|7;#ZQB#Gi6jmC}x19J$y&;TPCm6ON`vz_hF5=B^DZHT%K z9O@66i*ixgCY?-$mL`V}L&ZTPrcaA(z=ROO7Q-|c+sM)0r8bgQ39r~*htA345(yYnR5`czI#;WUf zbz5I5a~FFHQKnr#|ErraZ()Ct Ol5(_nwX52}O8Ou06wbc@ literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_allow.png b/docs/extend/images/authz_allow.png new file mode 100644 index 0000000000000000000000000000000000000000..f42108040bbbf9facb9fff0c320428323f061e10 GIT binary patch literal 33505 zcmdSB1yEIg6hBBjN;;%dQfW}S1ZgBh8UaD1rBg{Mr4a;_loDxCN>UIMkWxZGLZm?& zY4%+8xBK7!?9S}Y&d%)4ym98ryZ65PjdRW?&V8hn=q!g+ci8JxciMc^7!KkZ>@vJuSY|M72eJ0*RBf^rxl2wv9?-oiI(lUN6 zFTW}e7?N}zG8p>)$&@ceomwRb_4h-s%3R0l@IfsB4KM!BhYN$k&kUDM4g>j2Tfh;H zC||MCpF=t1ZlOD;t;3&{XH}UwaPy)c z8F$#SHy>}Q*9Dx^o2<#69lQG;Z+3qFK~_~|J8;3M>4C4MCgG9zOY@PkTYBQ(s%(fI zd{4!$4aQdJC#H>uqm_$~U)H-unBT8V;e7}Hxr~4PB?0TMq(oj5kvgBv7d;u021+rs z>~>!YVoC%_c}jH)HAA%CJk(o_6v27QB#zNAf1z_slYPL`xp4U0`{6HzTHRxB9foKK z=|qwivI5PV%p3jmP7nG5JGzTZYVLJ3>{VKIPQ==Lex}Dze6uL=aYXm&+YvM2L8+tF zs^!|6gN^&Ye|^fn`Y=x=;le9-ojNz`nk5t~IjJ?3o zd3weAucUs$)B9aCYpBgreWGMPTv9aX`@GP!&MjX@w()N3Hl3Zv{`_<2uV1gVhG4OK zE`LpARVFIZEsX6+6ISNb={#}WxEtZ>MzTBPz}PQ-v^!!xH)LR*EFX66Vy4Aqs}2J9 z>VO)|qLe8DGO-6as|I>ZH39bS|!itj*|`F#IXAkVzVyWC6cZ?&Caw%Y;ATj zsSmx5OzTOK-oHM|o^BGi9-cRKuXCHbu|}X3VtVv)>6}}T5L?W@{#If z(FotI^;#RVUKO?NXEw`t=e1@}fA+iiY|Tqb{VC6~;fKfP^&WgFxE}MSp5Vdyw<=|Z z^`YVv*M=MUY8}<$Yp<=lC%SwN=2999+QjF&Q@n3A_zb8_y|z|Q7IB~NDbq+3qFL}< z8NQzHyE|kMBj)tw@p!!pg7rBp&2*6+%F(#dTz!igB{D^8XQvk30Ps<&+?rWfmv zT>t*A=CRgOp0pL18l{FB!i?dGGC#VJ?%z{^&gou9drk&1v{#~qJS9#DB|XLb58tdL zC^K$8tJcnuFN3AIlvidk&Yg;kS2-d68-EYK0Lh3TvBFCwkJ5rDLp_!T>&ncV%ZQTs zZjJ4?N_nlOiq~;d=S;oxE-$v}%j}4oJX&vFH?F!Hv(gbmcS-9+A%dxAXX(qkjyR_E z!@Y6mTY(4N=F#rYM4j&?$=VIFvC1gN(w7Au&oY&1$)0@GD>c5tI+<>EHp&v1r)=!u zDB?7hx3$Bew9uR3&TClFvSxLf5V$j-OsILqVff~pJoVkr&*sLO{NFjWN0Nr~*5@XX z;5d7=v{Q$?+sk~j_3KguQ-Hv#5(7!V!5kGU z(c0n87cIJ>i+Zn;B*Ir(&1?O3UvVowrnux3LBwFXSn=YanD5pdDha%gR1D}0lfDZK zxjhn|%e74qIff?d+k#_o+p@{>*Uwj!S#SKE>CpX@w+xo@yLjw_YNv618wXtJ zADDLK7Gvy!BQgZ|1^&^GUD^auTsyILU1xfCI%63+hz()=MPo8O*a#+#W8!wlRa$hN ze5sQg-LQdaM|Af#?Pc3|*Tu=k`+Oc!8<@DahpwK*&no+!ot{XH?ZU{@u^OhkPq(0Z z_8eRNhAl+5v>VPZIM}%vqa(i6&gn2*QkE5X`c3S3HcqzwZeQlw){ev?>8)?Ctz+Jt zitWE<9X)*J8*qBCp!AKHN#?T83#$}u+yb*E$KU6f95Z0$7L2YCO(b2^joy=cey1mO z!1L_XpGqRHKBFi1fwSk8!Z4pQlN!xi3M%jJ|Gq~hfiWM?Z!VK8Dy0w6Q<#X?=ym&0 z#chQVhw{b(DKsiRQx|rvOw)X)siwvRBD(rE{MHd))VNve8+FU_6N~ZA(!_~MY4qL! z$vfvdObHutiCyOv?J^|2MG~(+7a`1$vXV8Wd+RVF^19=NT;n;$np2i^ad#zn0Gqcl zZ&KZd(cNc#h=PbxJ9h}?kEP=d#-=K*B0Sb7Z!u_QO4apbN}H%X<*{Sq*DZ+lSRK`Q zb+hQg_CjAulmAf{)9D6|c))S3nmarqt5XG?YU#RFob+9aN4^f9$=yHWg}DrI3wpkn zAP$Y86FoQV>NxgBb?t03E|B8Qr_awt=iaS1J%Qj#Gh{#6P*=I;gz0`7Cv*5x;*IBu zzFmKIu!TQfQ{~gPp2gK~RUyO}^V`BUy^eRjy->tAg6$JcFMi|m_co%Q6V~aynr5HIj|<1)Ci`vSV_RZO*tFRQxMrAXbU@*JNX{{_qMePtrDJ z#Way@z&_~?K_){`9xh#g?Pqk$xa>eNe)A?r`i={neHO9r+~yuH+c60>z@{sABJ5+0sLQS*uS(xNu;6 z=9PivGHQTP(yL^m)>T2;MHOFq3_EJPqJo^^ozr;r#ch@!=ehO2>TMpz#Pqdf$j+jyrz9lD+@x}N`vdhSbjvXV*gS($mmt7n z?oKl_GOc>czettzzQ?LQadDbPGeg4q+HJ)*HZ5vPa+vK5vRi&^usS7uw?B)1*Hl)< zdrzDYp^VNwQAx|B5k<-SYLMe5XAOb0SE8}GyQJ@(HE+@MT{Vr2<>6B9G$HF7B3A5; zdr!X>8)z+$l$ZYybsS~(YftaDH8Wt$Oml8N(cfDeFZ?wsr!1lnPOvO+?JPgVjLh^4 z)II}+A7Ln>ogA4Nls3m!^48ibYiq^k@Dw_`wlTGWY*b!0CJz3a<32x^sY#UBiyH5v z4^@+^)n_Z_XRq#2ZE0M-^O5Arix0flsJ~4sYn@SF;8kkOS4+N8xc058liL6JwqR5S zW5)A1ocat-Eyj!bC8_k%em7&L1?mTWX?lniXlB^o!E)UoxGO0Y=lg4Rn5%Y;bi;yF zm^;G@M#XLGr|R3IF7EP=6}Q`VSUer*yn19O=eICt;=dl!cA)A=9UDVF$q0Iec>jb{ zq+fHh_t_Qo^AY3IYAJlYB=@COW>(6ZMara|FD%>5<|srY6*K8x?UVkU-mp7tviy*U zu>duFrDZLDQTx3@yr{F;?oDM=a-WXSUn+b?udiMIp6%pXoLT4ieY4*>m*A&Sjy%?G ze1S!lRzi}Op!xkVM&~Km0L}*jy-g5E_nhAOJT~%8cVa zEyPu_oM>uf2&i@Xf3JU-C+TKOCor-}tutG%!ZlIv zsqSkLPt+HmDrEgQiAzVGh)KrLV^SbFZ=5>!h*$#nHh9644EIOCy zD+!cKh)mT)TlvjNG=SE6z~8 zd6tLytLtMjcgjdZr9IO~Fj4CpgNn7pR0k$6_5yzo|9z)GjPzV@Ih*SSt*eg#BQ(aw z{*KXkY{nL$*Y1+ih8cvVp2~DROQzAnj5wE){&)2v4)>rS?fXSs_irwB&y+bg&+U?< z8;I#QS%pa|X&TaFQE_GJhYJaq$C3uHVJIcKeE(r|U=c#K*OCb}0NuR@CAcG63>pLx zwqHd@{jj{?rjOFinuWz6-GN+ubRqO6?xMbOYl zI&#%gzII%3JSlZ=U}0L_}3y`&iFiyC%708ZdKcj z24t-P_X)YrYwYqJESf0blPRFBhjku}E0Fp~hlW@>E|7Mu2FH{*s+^lyQ88K>qC94PBvBWm+m+x+UExs zNYH^wOnQ4Z02fzoagDYsnY*wI-;8e0@oGvpJ493bw9N)a`EVf);d8sY)K5qhL)~{O z$F@pD|6q7I4H&h~Yr<8Df2XZdEl8fJn77aG$sd5gP^=eSbGMmty`@$hu1fFg-g!s~hjUoFIQbmU^(?UwWHX^d8TrMD`~D?OyIfx%}$H zeWAB5AHTB~0?wV;j+mK`&wNi)S~h0d^zOfi+2!=z`kBb0fPGQtMd&f0fh8=yuPD7?G6Eh8&-H6VG!IryE(U%CR4zmYV3DA;Lq+x7=Vo zB$nBiJB51mV{V+xtI(59robeK)5aTd!BGv6>0Ud26I%LG7}r;SJyXi4H&dFs&V6Ao zje`QGzv}l6U+4w%`I(HCsnb2tIq6^=>+Si^Ti?B>J`EKc=oM}NoJ@g<=>mMynAKyb zOeTjzY&lwPF1y<-27s1rM3~}igema2(zW!5-<80zH+H$#nr;@|7|c~B7QO#@M6CA! z2!*5?``zWCLvk&rv!q8L+O}18J3ki-MG@%PTGiNZ(=mwZo@9pFMVcmpW@O@ zm8QdG?zs+xDConN0XUNgB}4H2l-6&k>7hqaQBO`!j=D=tYQ7TZR5|pFrQ!9Y3Mv3Z zF9>D)avzXS-4^QE7ehNbhOfU@!vh&d&ir)}MRxEBBm zOfM5QP$DoParT$c%k)jjH@I>d3bO)?`ycHR0JCT-hw=MxdHVf+?c_1Pszd!Y%4mbGN+^T)5@xJ#^lOgAgs=L>X zV6TR>_hrfQBc5a1romcZ zlJe31w)^vm*7BVn2lu;H2IQ=Y3A=F{DR1=LKf*VBfHlyYCEMJ5r$g{qSmEWs)t0!W z;_{|fdxABm?c`dxr|6&2-J4b7)}d}ZS3oX6HBn+D{0*oEHDJgLfH9033y*Kniy*>K z@W7ppIOlXQ?%Yh{6pE#nx*SZkSX7vKV~t3sV8bNZjb|;)7Tp!uT{gP z;nL?VjgKAj?tQO!JKlG_f)(aK$Y)r>&|<3*fv|?jm-Ox&?{D6CW2>|Dt*S7I@0Kx# zW;$2kej5`?_WeA`$18T4O!(u}pIJQ?`>*;dTqB*f*L1}Fc}B}n4`uy1N_5K;k}}T7 zpUtPc{8=9?{m^yQUEbvi7vkW_Xl=Mnw*+Hes+J4JFpI^e3R}S+SqJEvv_P@j@rH_$ z)bEthuRCodSM(BPoIATvq=hy^eLM^}!N4sbLkTZ^K_-2~e54Vw8*V&tlfhQ~GF=Bl&HIsho`9;QWU#;|J zNM=Ud!n-vKEC1$*8?4A%6*R3?nwdxSB+j0WL$8uw;~XppNeq?UGWj42b*jYJ+`;`< z;;CnW@uTC;c?qgyBJW*K*jnkwGwstzrIl1&%6~jf)W#}t{E*bb8YgwZW&IhZk`eVL z5L0q7lb<`MuL$7OE|5$Q+Ej3$H-;P>?yhvnX?(WuFnDXPC;CvZUO8xmo~3dGS7~!z zyV6qeZIn;#LrcK+yU}6s zL0U?9;}x?-qZ05$H}5YGoSyaKey7xLYi+L?Y#k88!KywCgGF%2QsH6i{-T z*!U_Q0816}1gAF~6>6eSCF2E%IRq9z0lPhItW-#%tEEaki0Bn2_LpArDf!fLi0cKGa@n&@aZYG!W=499G?qbYi&siSU9mD?Dz#Vx`HOS_4 z%C;#p`LiKBW5dy}cLAT4%{crHVWICle~PoFjwV|`jc&CoX*x$;Iqmn}_M|-Uq}Xlg zDZjbdcg5fjN`|3%3VG$Eqmk2f^=9L_k{KH=g6*6^MIt^+0*CD(N_|1W7?g6BW7zn; zBs(`B@BLVGmy*<#kfE5Wtzp971sXD&awq(PR-iA+SowQvyTvCHIpxddO%EoC1baJB z@LN+?XvJKz5f-adBdohic)FAHch`>>9sc=vShe2--pk`)@cH!>F5P}BrG%T>_n}m~ z6?GDKzDHbxju9i(r2Wk~ec{}gnZxjpRt(stb@OQJL?Mj9VcWEVDaAG1H(zF#nMaG= zez@QndcTx(qn|5$AN6LOv6(!zZI58%=d@<{!S+ITs-V@)1+6$47Ry?8O&sSq%{@or z43Si4sXFrQrBrj}LLsug9Y|c%*K%TEea9KFdnCSz;GVpyH2N&ansT<)Gmdlew&wZY z54;GtC%Ls^<1lKl@unK@6B*v z>wG)yB*9;FHsTYq05SdLFjni)^hz3YI19@dJetdAt1e74VjiNe=qt$rLm#F!FrLy3 zVDe{1M~M6EjLkf4Q()hSi?T7aYNa=I_EG!Fne~9o#UXovn2J`v2_bV&g6Aq{DA#hGQFQ3cqgGs0lqk#UzE0=&^SHJ51l(HR zOV(~whS0b_p`L5?d=&95l}U=M3^ok=jJ2HdUb2Ci4)J>HV}H{rA2Z2-$R}?6wdFz? zBym_yw)cI`j!*jJZ8u5iT-w{5TsTohCj~S;EEPXgdxwbXz%f5SNT#hN%dBeQCsmb+w_$8FC zvZ;Bh1P10Y|BUx}{;#bf@&e7}Ij^%0LQ~{~q|liK73HSMeC;hCM-{oR_IQ1aXIU=8 zZ3vEU4M53%56J253%FKlBlt_ZbtfXTpr7(b8K;lu<-LYtU5WLGoXk1iB}+w&N9bRDTo z5ZMMk-g|`}Hfgq|=%5PnH?u0L0((&_@bJkY+eOw)M>?Q)YOd=JX#e9Ha5K@uWPI*Q z{YPy$Dg?tPdFb9M+=)2xvr>yTpYQ+5B#Xv!DZYef+X)j1R+qUV!t2?0<=n;AuSV#fLT>!vFCk zB5-cC-6dMJf4Y1td3HZeU$o6%4L?4}372gtU&PhbwYrFlFoqtzfuVcOO|eYwDTNS* zLIhE=_L=#%AFk1MyQ$JlEu$6Di%2$i_AM$JLW>k2kCXtx#a_NQG0MlG4orlTRq?m} zwegzng6q$FbYjP5_)&SWFXA{%^PmV7RTH zg@peb61hbfLWRC0@qfb~WR0u6$oqxja_B!U3Vwk+lV3qk!oOo4WWLi1vRVF{Z$FsN zeqrm_|K^JZ=BxWM7U8DiG-#A=Gw-p?BWSe#{JiN1SPR7I+Du)~t`zf_)(V(a`xKi^ zHTmm3DAq4!1J2$|7D|wW&{)-mO|WjEAMb;aX?{tsMuz8HaQEAVPq&x#OawU<25q*scI(^@%wuvnwHPk zyMYMp(Eqzc^jTrF45NO2Oi(yS$@5YM)asEcxysRCE?9Y!m1?sPxBu9kd>{RyX*$RJ=;V=w~n zgQXz_7Kqx>-kSCP#!SeA=@>b2i58P!D_pK|lT05yPP6;zpyGW3j`eG~x%u9&@5Q}# z-|OF+Hu&gBc&)ya5SK6y_{&kDfahbPE{xA#{#x|Jt+cJNOcp>op3&)~P{>9sov3Cq zPVgS3fPHV$#V3di!Q2b5`^F@yC&sFNrt$ZrZrK@PfT-$|;l{)!0KikG0!YIe$dk#} z9#QYWqR}Tc;3~(sh?_3yt%=BWCx^Sb3;f1!R34oHe) zDf#A0?W=x!s}}(cCIQEz0*DjqfYsu%d9gmPxe@Iq2)x^AoU4-HJJ_P7m+(*l%)~)* zW*bVxAc6ZqhW4is2VT5fEp8rxK}XMsWbpC9wmJaz38K3n<6E6%DI~2rWA*r4Y6laV zWrIiN&I7;U#3&nJ+M6!UYAE%x4H)7ie5QLl7+B{PZFj*~;9BFqO*FBEd%9JEm@k0a z@MW0v|9#7P1=MUVL>IL=pRfjiNgdc4MQ@D@H@NUwAQA-&kWEL*M7!jsU73HUp2WpM zG9!15vHFE>5E=`=Np*>7quSDFdf?f3slCerE1TA7R`(K!%^j|d@xA7m zh7WK}K;6nN!2I#!2iQpXI(R+D_*lOPI6HoOt@ub16`3MeYsD&(ta=sBoWUHG3>2()v#Un<9W=R&y`o6|)3Y>ZMdL&KSpw0!@o(_7d~_ zeUN;M75yWKnVy1wpm+(hcDphr2sh4j$Su zWUlpCx`E{Jo2ABjh(-r{;%4!dV4#fmFVnXwMBAreE783YDi(OSR2X81S#oGzX&GkB z8~5;Zs~tm;XIX(}kt>^?=Y5-qJVeQ#;H`N=%BuA8MCCYVR5?Vb8`4@ND);1U^CtTt{3+)2o)1YD_hsmm*F_>#8%qpf)a9ROVz>Exf(6F#I-its%2y zghTU9B^`#XE>h#|t2-t-$z%|Jr4koo!86J@cL`naWvG|idB}pUG##g|f!fyEa zGl?GtkP>7({q92Mk+ASrhE7g2%WM|QiIC_@;)Ef{x&y9}n{nQ``LDlee(ZW;P! zT~Iv7{Y>RKlDk?sAu4NKmXgDG!8m3G8j8lB|Gfvn-pfI7j7|_^K0(TCtL=~d%!>_z zmR0eNd}PQ-;PFD`s;G}hvOkU&mfBu~t??dC#}l@EOwHONen9QuqMw4I z*94_t1N-i72y6cmBY=KS>;)gGr-=Y2+*5xpWV2OCkvA#a{F9}RXL13mzqmF5Ioeejwbzv!a zORP*Y^HI2jBvl-V+}9$ZijdD{y#+AoV6(&7k||=Pp4=hP;y^AC1$ux<9*evk_B~Q=FlRMv_e#D7 zQ<}K@e00JI1o^MHhd^>DnF2kIgzi5YI(W_yD?@=snk9XfnClA)UgMiWcYC;Qz4J;2 zXIPOH4iSAk1S+*^$8UMS$O?h4sLO#U$N*73S6I!w54+S1Ola{jkoG(;mF0c!jaOf{A1MQy$DITaNxFbn z{4nSCNLDgNzV0fLU`3!9sPV=06ApYIiidE)zCRO`5#ji$mdruB$!j0iMG9b8)xi*u z+kh_7nkjw%4v;}x0AJMsyz+r2kPIx7dahDTo>`-xy(ZIznXc_6^LxU*i!o1`v7U9dTl1FppX3(w9qN#^$q>dp1vFv?i9Dq_F;vd+ zYE|FYk0Y#hE$<#;6Ou*0fRs?I)B%3NsLh?)nyCO2u!*rlV2_6F{DBz{}?6T?*{=(?Y6CrE0i-r~1*|~45yD4)g zCy@O|h;h*>_gST`dh~pyPnpZid!_st#?Hlkq>v`HnGB*EHxJrhjt6m}F7T|09@5Zt zycWgsze&h^U18c2{Dfg3)@ZN$oj@#2Cu7@gg<_)U)@Ytjf=qhA*~JdjdA^&Il}DVGscHJNl<<5KRg`pJtkbrrQ`?!w|gZurWiIb%~-%S-a&jnC+e7}4@N{5~8a&XXnA{6N%0lmgbGa;brtR?`OY z7R~Ltc`MBcB~Efe4&fk8kBNhr8H_0*850G26+& zRQq)k6C#EZVn;N-De8adG1r?V?q2XrVZ^f|l7Y|5gZ4OV{`2!AzCwDnI<^1l-h}OoKVSC~A=lXSff{!~TLXcYh`#bNG`ELc+ z>GYU)gHD9d07WLHb0N%`k*OJgd% zHDn;x(M;jXcmUbWK9o2`YNF22x?BM=*;iX@xBuWaEIJpN;5%iZ9O4Wu7JxVH&(~(m>oEj< zPtbGedK-G>TSS$-3@W82u zHc?Ir!!SEyXrBNANiPHv?*O{0a8iyS#W>`D*euRN$S;0_T6006M?Qe`$e=@|9|Eb1 z2C_`At@7}D3>5R04>cj?pUV6ZTvo00s)?hzT+}g0Mvgi-J$+W0nZa6$y|(jg*_GKu%Ccw94X#Z9(Bi0B#JU;TKsF%Db0np~c00 zSoV|TZFuy0ZtwQkK7uTQ`W8dHzZbE@yku8dfe^2s7T(Gy|w;FUialCeF^!i^` zC9Ou^^T=Bq0#hKyf+nctcee+|Aqe$e1J1tN(leQcn#L|1m8^0y;wE#Ug<9Z?ankN1 zYV^nc?b=n$P+_t`uHr;z)M7_}EM1NyuVO&9CXHdXS#x;WtGrX!V0&9WgPdCMJS&;_ zJe>GaMTiMb)Q0axJmcDqGQu!kwW37h)`Y24g>hCF!H;>(jUIUd$!gVJW(`uJ3p3mq zlP+2BF&}!Q^xt_5gHgN0dsPytM-cQ*0CRUcsEg-6JbDR^_3iM+{^TE6C%2&zAjZ6T z_pgdBhoO()Q8o2`BKXb72Nx5n3F>Si^?%0|=ir`t#vxuXjmSri8t`WuS~4ZS)}kG!3I1kAZPs)4rW+fW;+P$TxR5b zglx*#rnTX@BSd(Lt(`Ey1elCa_`LFDSUq-YNE-13)%zi|qta}TE zVCc60`jQ0-ia3L6h6|Ywwp7102R1WkKJ z9gKzhEg^?!*pqjAS!8%R|_l@Cr7_i9vp0`Ld!rNuuZQab$e>A442M< z);b6U+R&$ggTuo;n}^MA$KMuW;>x!G*Z()cv8k;6;0>8Gj<|m-5UL$=eIk-c7~7&P z=Ke_xrO$|z3i{=__BycSbUhEMbJY-S_&s^QYd7$4$IchzR_m4^bcns@m&V=}^MDuG z}*0iT@*tzpJo`VH01tbStO5$b2YdQBmg02w+O$lRvDt@yG zr>O=^Z}E_3sgCze= zM_!>MU?U-fVX@V=Nn<0qijlg`r}N^PHv9hySI5=7ai!V=AaQM)?z$|ba2seQa-PLi zZ2(rae}?9}6y^a~&Qf8t9=ro}o-0quPQU@eWe^BPprMAq?DLy{SBUra{B^yCdGbi! zmpC6}NN#A5^I>>=kBH*gW=F>%u?5?%afU3EC&9J~9|dh>?Z0wekqv|z^U7fplt?^g z4Teyy723!2VilY#(8?;K{P{UuTo#vXLL0|Zy202eh61XP1BI4)IIbm;hGt^~io zu$J6Nx(n}@aEpLfq=7Dp|5NnRBqh}yHBC=|UCntT8MMM#Tc z?gbM;k;NEV@u@+HVUB+HY<9C$r5tJ0p*FU}IWC;2jf$B5QWH_%wYMX8pcJrY>Z5P@ zA?4hfMR?k^X_Hmsh7prHilg>;N&Y%>n!1*8n1l`foGAE95o0a^qtvLp{s4s}+4S3= z%|jLcQ0~4~#!l|kgYB5ri8@`gz?0oP8_IC{hPwTkjDGjMhRpY98ActfEu{WZY|#*1 zEAY`v$Pw$qnq}GRC#?5PuB5O==l4s#pfSQDi@c8Evrk+=u}b-7bk+z9mccz(1}@}x zPh(jHpjV(~p|ef8yDyO1BJ`w#&N;p_upBRtO17PGgbWie^f{VyIlk!a`E@Z2{^nz$ zk^5q0dTqKa&h9()Y&}B}%Lix@S^-ajL^0?G{&TwFvZ6OKmxF2JHcyz*la@e;iV%fN75i$znEaVs$zJcDnZb%vPyiRU616bBVK)*X0A z3Pg*HKr#<=5sGU|mHp%P6Nvs<6n;`ENH!p2mP^n=BO#8+fJC?&X@sN`9`gRUcQMx3 zCHFl5VB~gZ<-?7{Qqjz6x`%d&?pZv)8AIGtsj4ub0SW1d6gT4I(S6*w6I}7O9}uCt z%Wl1618d0B5)$Q|r;e1gib#%W!kf;|JvMxZK3TH?T>&(Dr4nnhv9$xWd>9EPk5$e)E*p)0&2r@57@=53QDeeO?!B0Z@7fV|NVx(r( zyc*SATnqHC{O2&FWVX<&EbKR#{*jG`j0qHI%od833u~Uewbh3Be^O~M7YWj#<2coH z3;3f6HGXhz@qowobR$CcBNza_$13_o$TQjf;EDT_IU(#L^o>vTRY0mghLGl`tzi@A z4^eR*s4$tqb|c2W`PT~fjs1|?#^(7F=AHE1-$N47f1iLE_As~*vq_Dx{-yE_MJVc{ z2xk8)4Tyv3MDKlsBzv4!(ZO=A@yb}R{-4=b!OerBoR_E#L-4_WmYoK^6qYwANUon= zmcJy z4nh|?cPcyrTZcA77<3k?R$6{QW#BTzn}fnQRBHV%r4}prRG9()&kllYBA7;s%$u85 zzrPc0wURJDS^KPSZv7)2Y+U$eG|#2)dm(N$1RRGGGnFk9tYcFz_Sd>vc@i8oZS5Gu zA(kj06!YQU2#cLj%HDkOe_LodENHGUhUqQ&#H80+R4Cs2?=0#BA0ZiZYr?(n6Zl~6 zoE{$tc`RzPn<1V6j_c33THC>CSCtQG|Bs;zaL|x1*uZqRzv5HzB|*m3x=t$RLd9d4 zjoJa=3kdrB(nxvMrSU02=PM%P=Ul@}C%dGebqVQs6S8PQ3oZq6Lr=t~`wM-_s2>lI z!Jd}DAh7Z`?j(wN{j&22L@K;_Vsdd`59LSMs34L&- zBNiczbTOS${1)A{sH(I%2Dvo;?D+Rtn>fU;8IND(r{eDPwJKRP?;V z(5txw-Q(}=!>j?C!!bGiCN)mBG{}--dyHnf?vlI?M8@ZCtDA&WRpN;IZB5<-8WQYf z^7gdS#aP6MbGNcJwg_oGt-H4f=*~$XP zVOPhLpu70IlT&fC(2Np45IqN%k-oq2CSRO;zNkQ7jt+fy8vSqcPc}>bt7+x0p+OtX zaD8)-xI^gsF#PU%&y|V9_xW)Hvey{jI}uS=${sG}s@(+cW2JhT-tq`)fw#CMbIzj` zR{>b&Uuy=a*|i23yP10&1g#9onmR5klksybzxC$9?D(T@d8U`&pe!9BoCVjwfcKRG zTWA=6>3HrDR!q|}y2;yH;W?Dwrz0m{5Gqvtsq)EC;Cf;Oa#%lVi#c=y0|Cj*J*ni) z(6<%MBy09-s=1tU{&tzP|KZMHgRlD>FE!B6(^8R_=KGYP9rL-UvwqY}JpZ8OCX?yj z=$H_GItZX~k z(I=wt>X~c)Hfphw{{IQF#>x9F>4wc1hJg^?$4M)?S9F|k zhI6oW7a7JXN+qiS#;-xC}ooJ8~Jcp8R_oHWF(nE4W0_qW&d{U>9I&BcAYW z|6yHWGPwYgD%YVKO8v<&Xn3JdAYZ3@r2X}2LmiL?$6bUiKmKJB=5%|2(%Gh7LSwoEv6^&%exs3-r3b*piU*zhlV0kaOvRKKzx{ z;^oAkHfU2#@%`^O38@t((0DRYt8&@UJQ`w<`)jna?ehfKfM4zd0*mw==Oy)qX?cOT z$n}nS(orQ^)Bh6@UD?$723WV27{JhkILNo9@DNY(V=8{W^@inN4e7IR#I4 z9?-#v_x+;L>%3?NN&OAJT`+=lL8uuLAMFFl1)}B1$6Vm;5WBrq79EVYDhQZbw6dAk zF*b1>Cu#-F(lb~A1~Nm!7?uZsbYc?Y3j7Wdi628m2h zzs&R^xOd}0?mFns?1bS8LC6FQ$|s3IvKS=3g)!FzpEdh#7nGd1Y5wO0px`q#ber$F z2uWU_bq7e?BLx;uy-&!Pk)57=u=e1^s$U(S^9f+kA{=etkg$eo(krgDRdG z5$zv6AvhH~wC!@BFbYCNsfjlYw#$`^qmA{ z?q6Wcvax#qp$zk0f&v$mlw=MJ#G8|^t)nJAj}RjC7;N)Orq3?hwX1$%RQgbR6@!ww zg|rciHke~HF6fKmWB{*x6ztmv&;fIFq0NVE<0=|=XJvgRt2R>*HEdXmivS8uUWwaA zGvKm$IM6HyExZ?wmy<^U6_C%!o31Uv$Qhi6JYN`NUyLdSShf?PVme|NSn*v+MYXHp zXdvvT8!V=vFTyN%Y~g6Pem~0Ta0fa}KV!BPA>88Zh)N*X1W5y~SR2rMGiZqdCQrGt z#_)MRJ;a1{h$oYE0#8w7Y;u)aB&5h%Hw=PZixDG7zN#K%={!}PJI6-CjRe%e!4jKM zT40H-Q7KateBM z^P;SMk$5=`v2T)-mXSHQoiu1v7|#EHMtLFE1gci>;tr7idgY`aim&PBsjvV;{{&g6 z(C%iD?3KfC#*S2>G%^or4H#LKqT@@9s?SnL-vLexzSs$-GuRpgE zSac4MAd+C&cR@{o`e6>1b?D#qDKW#ZE*2fD0Q1;?1@_wNpGRgR!NU;b-Vt~E86Sd$ z--Q@@5f>Q}0MDJj#Kz>mlhEpqL3~xP|B{xHsLp4C3`gjJt607 zko^V&G6#x7J?*_I*e5AS9|7{d39wJNNQN_*dzKkizR<{!(96fMc|$CLFY&d+NcV1U zdXe|RpJY_c89%sp1y=`a`n@xFI>;W}!Mhq#0hm}G2^4RMSu$h;n-G8nKZOmtWNvf@ z9&al9&ZpV10lL^4wTc5@h`~?j0}1JEfi~f!c(7jFf;__suhOWJg2HgCQ1(Q%?pXq{ z1Qi5igC)VRpx^1aEF>3kZJ`DJyM4?SY}OYjXP^RKLlZgDD!puI zk+)JFE$5^d99$3QnqmrsGAIuW{W%R5V`@tn^~ zJg#kK%236I&^3BNE=TmOH*k(gsVtZNDRW5Mk!f?lEj&J@S$OY3GQ3_y6^ObyxVgTK zRo*06?9)JerLAM#f)AJ8S;&?2-sF{;4ob|+`%Dc#5LWQu&r#V!!;BDe~7>jS%CV8^sHR$ylQpr)kEK3G-f#I3pAit z7Z$MeYywYu<6UqPX=JIyy4>KQRpIugKEKBcfecHc%m+Q451fvQg$pcOTtQZ{zjL-+ zTz*LlUkc&4Q(%!Db`3}3j00ytcjWc07PvS-<(%MYy9L7YBPnq5dkJQEL5<25EZ5UJ za9`+Q5Fe0rbq%hLP(*Bfv&s4dJ0^L|Gu#{Xh1J}D zH<@%9uY-p!gINtIB1`>6!39Fs z_C*@}8qlvfscBZij7VI1oGdG8l{QoPJ9tfDH=Xa+h5eL&m2=@J%#1AhC?~MUFUQ~CgolF`neL8y3jZ^(1R~Q&H zJ>uHw0V6ySwwnu3<-EK|$@8@Rg=~q<{_%F-kd#P)WICgE;8&Uv2GCZ5w(;6a`mOF! z7Oz&yGh86niFj#ok8uWk3`r1RU(FgyMOdX`c4A1Dyzj(y0dJK3kjP*wgM`a;OLiGG zBy;{~y-64ASsbVrCYQ-IlngWC9w2x~f+KZ2T=WHK$6b;nkA*@ik?}kT!fCkR!f(ZZ z*+90~j-n&`@pzZG#(7Gg)%R!2`Qyc&;J^}8$y>t;i>oC!Q+C;cQ#s;$!py}YBEUBB zGI6y<*#AMh?a?sKB>GO?RAHjM^0`kq0(>H9^X)b z9QKOJv@EFB3(0aAEX>)8GJ-Xj;n_ZS|L9FJaDD$5GoUI(2O0fb+ajUmX&06)6|8tB zbRI^S0sg!;YX9b&5NefcD=5O7bh;7ae8vLpTsOu10kruHli zsaisyYN4P=Q|=*E$&k46L1x}K<2(oYeZ*5!_1*R@78E$V{kWGwrewYfzX?Suh>o|< zAy@hPbGR~0O377F-D145&x^qS)ws?Mc#uV+!}q9d@8}C2qk1fvP6rJ!5PXM1Fsc}L z0ULTV>dk14{ZN;)1S1=Q=~a+tNEGRIllser(wl4PBXCIus;oUbZex3L43hiEz?1#I zDiZ#ynEr~}a5Ll+EC2;(1p1{al+M>Y> z9?jRFLAoc6K?_eM!g)E(#(0CEw0(%A9X1d8G$2HW*N)-+KdRu^yaST-X4hR5(ATSQ z-%d@Fssc(5clT?rSuupG%d-uj-f`zRk5`wbOW)V$oB=Q`7Y#*sr{Tk@ z`eC+sc%>9#S~-N*%IFN{samZre12{KXFFJH8oa)vN&`K+zy>N4(Tin2p`L=b1|@*u zQw`if?C_o?q1*3K$z`|O!V}@WTgY2{kg_b_yUuj5IWQ1$Yap&7cs~a}APYSxGbA}c z*NPXNdY}tgn>9{)FO`5=K+c8e{r>=SgRdimYo`li0|R2c2L44g7y~O0df(jEC#5|n z417P5?{3~Lywu;v^pz%)Twnk_GZ_jY&Uy`Jde6yY4s>o36+0|eTTDE$47ozhjGqS! zS>DAQs8V>d);-spiQrU_0X6G(AgirkS%H7mp~C$MRu(r9ri#oO>vxui)b&dYS!7T5 zbEUR>gw-ZM!o7%3$)k;d@A?Zk(Zdoj@y^{}8Rlir1WM~ER5dE4#&45cMC}HzAr%`^ z#Fm&hn`KNv>!d2Y?^jg=RAXz>ku#tIg z>L~|WKLxh(TbVeA)@k$R7ci%UmGcsX%6tiZ(sEYz(dFs;236(`JPVMMDh?hVY->jV zM}Q}Vv2PK}I0tW9(VITMab7?#DB%Q%6tBl{S_$!6gEG=i6zz2!p}Swd^=;Gz@x&qUQc2R&y! zg&7ik?GGNN{4dpgbx>9NyEot_50XlVgtQ0>qJVTtsDN}y3W79%g0}m%;Xd40h(E4rL?|o01jPQJ z9HBRTe7Bo#q1o+6|LJeBPyb-4NOlaVU~$0E({7obO4A)<`oc-~UKfm_Ec60}z7q|E zEF@Nc;NeA9R@r z??quvcY{KgTmzft8jEpcC14E!a$dqnYt10)mPWOCY7~;;2!__k8W5*RE!=CGGzOq7 zukc>$mC2(p5{&-V-F5bnCuL=*f&>HTn~OSwFta^A9BIOkgTC8^qYY-!WB zhIssd>D;?leSu-gXfX7|y&7Dhh(mq~=sH>65;BTHaqMu>B4J+`&~Hl(!JPUq8pKRW zyc>~Q_7xxbY9g--84Q0OzOFJ{ zQ)PD-^bnjeIu4704pR68%C{rhNFPi#-A8`Enc0nW;V%3C%MHj=?xo^gr?< z(24M>vo?-k0+Y!(qT91Ma+wY9Oc6DRvS@7pZ!6>!RLm+5Jm|3+UB~2;7={-MJ%k~5 ze@EnT?Sq_8>`P9A^GyAb01=8(x~HGi`;$4@i9vzQo|O2I6_ATUSWDc>&+BFXIynf7 zwrLv)k+)5mp!F4;NDjYXjLDC>=XKn_bY?#Qa>!n*xiQY(zDqA}3{rRsYU-H^54LRS zgt%a^T3p65jlw4w*Tqym!ZB;o2nAJ?il|5`Z6;A-M}RBE<~i(=53A;amE6ogt1+JL5%45iJoOCL8haJFyvZ2o}3h3rr2M96K!yBw0LD?W9 zC4WU7B%V>BOM|{J;9(;?weVqXEMC9@WCLAPY+M%*5+*h?yWsMwTko*mCR5Z}GBA@C zH4{jv`Tza;2#=U1thHsd0y4CD*IXV8)ou`|vy!3zB46yH3%K-e-B3vGrEP_X`O|Ba zQ>cYRe@TR4zE}+JN_W&{{rxUU8}dyE&A16GP%O9Z`EnHywS9vx7qH}j3m4+Ud85N~ zfotKf7{^X%SG2lL$iewPMUoHC<2@1|D8Fq_MkmF5l?iE#UcBvt-~zYFVPG>^YzQBe z{98?wWC#JWEJC1Ko;Xa_+JC()HX%yJ0(W6}E?2ki(L!fR;3o-M=PdX&f2jZaiq7;O zl=H#Yw#ofq%`U##!mJCj@Lb5qbnXVGD4u zA5RTQ-xr(*L6Hgq=oZN#o6tU>`A+rr*W_2Br&mG80!nKPDfB}d7-@q~bZ>oO~?0666%~7@3SLX=CmL6V5*#s<%58O*QkA0vcLNJWe z){42m16Uq3;Ir8PZHNcSq{2XfQ32*T)QebgZx3Xh0Sk|8=mg-_NZs6 z2W5$cytEGhMsaqDu7*==K;YhU>T=0>rg&s`d5XAOaksS>3 zfRBTNmF^`7A%`U<`ySo8j9BGI#*_po`g!wxLLrNwyAwU~gf`|IP&o zpkIrm0`cBJ2Ff__d2wfet;>gbt^yX>S&!v0|Cr8EK9(L|K~dXlHi-xO8#S6KD7)_1 zPl)Au2c6sGingyeL$wQ9BR>0P*Ka!N;ClM+3&?IP?q5<=Q z`BgK%4JpaE=zN>oza^BHy$a=_bg#?wzg|I-P%XK)`*HM%mi;q@7-O zN0Y4&VpUH`hw`Y;JZN-AqkIbGOZ{1h-Pn9lBi{A3yvAd*q)*>)JaRHbeCS@H1X}ZS z60j~WIuzCXY@)lJEo!g~Sg00bc$VW!;J8+N4>o*bA;j(vm8ywiMtAp?d>^Bv8(_P6 zpy7)K<)IREXj9;biiQq>h$`p!IYR?LJ6Mr-4xY*_(MM$L6(f}D>?0(>ej-@BJpPRV zWP2!Uw{hpWgu!yPwDkJ2`LlxE_q0!mDYwVDYl+;=MhMPu0L*2B>uebq0gnZ!GIhJx zfJP-7rP&-Lv9cZ)xI`eT^)<7^q>-%iZuZkL}jWucKdh0|hlxWp>dhQah} z->8*-?XW(aVu2)un1s|t=n-x{ra*R}-$UITj7Rzf*DT5Z6pecz5f;D6nfUj$P86VY z9=6gC{(LUeO>`WYmhivF@qpI@)|7g(e|X1Jz{Zi%Dp~w}J-|}CGyCcjd+Bd+#4ZJz z{ym@IE7pSU*FLAt7M(_+5Vk1e(>iy|`@wurKi@d_5)SVdNI3S&8WdW@kj{xO^S}yW z@#r41fFlZ6^Cn&SbHqBB5C6aT|4f$&5#9>?COaVS4lJ;GkstS+>7?hgf5$dK0>i2X z;#|X?2F3ZU6Ao2->!W4O{d@Lf7+CoR1QVzLaIAm}_HKL(-tEVMOiGH-L3~Sb9f<)= z07!doiN?7NgVpn2bq#bq@!%v1Yx@@a-(pgSsyqe%HbgH&}2pkwa}SrgwwGaV}p=VKaJcnV2?oP#?-c#Ie{uWE&vm9}2bB~=IqZABq5m<^D1yW#9_#1XuT@z62uT%bME zN2|be3dhX{i;DI9|5ZDPJi;Ow=O%F=krPXnKH5*?YJH5 z7XeFx?U37r;S(6nUjvr-dVuoA36|SFtTz@_Q--PmZ^mOh^R>7Ree}l|7D6~R>3kwMwI`tZQ-~s zkgS?OAkm!L0Q375WNjDKx;4pTIU@dF7~7F_7yb#3PrJJDO4{#;}kdGab4IY8~l)&Z$ zL}GmuoNIG+i~<21=dXG3He{J^?MqI9xa@U3mCJ{^ ztq=#U>ApkiJzU};R9B8U>V;VRO z@I!t9BacZ51PRU$%k_vo@b}X=D!h<^0)mL_+eEe6zcIVz-!rfRlem8Zl?TrXL_LP8 zG+pLPkOAqktSVI|*ypu9fA{5^zx#5@`65-aEjy4sj^zXsxPI@Id($05)rSEV6+zc8 z)cFpgNMr$YFdu&FA}q2VVHI#sm}$Rt_0}_esUc+42bzh6tR&yFBZV)|BqXl z@&~@2w{Vus9ya})u^82xzbH+R-bmFb7cLc}Z+ZUW{kvTJ>T&UjRJSdUab+{}a6%na z(`C-W9h5HCxDa2K|IKN7)w9XHYGd7K$@}2bf$!+(>ejOFW9LrK&W$G<-kt89&Ykq> zB#sih%Yj@g@B{oaJ;HZzw8R6Nmyhj{dDtiQ{Z|bvEOPE18>8z6W*a5%HTh>UdQl)x zE?o6K<$wt4AA-HB%-{Gm^eTV5$yt|uar{sAsJAfM@AFn6*cn)FWCHBG z@GVel-iE-{>{{A@j9*_KRsq-leewjkyL3^m=UOb^~EIZcg zM&+?ennm{z%Ck?$dnaAcuf~KObHVJB%r~RpTookNv0haH5H(a664D+Ii1aIFj2{?q*Il8i^H|V`2T@<^=FiF1g+<4r(rk^Fg089hWw+VD z>FR3K`qRtDj)ZK?hf~6wb=cII>C7Vs~?tr(I|v4 zQ*2(^PSX-?JJf5E5#wmA)#cuLbZF^gO2g#N9?|>BEpUj^@e-)^;Af`lDjoj5eiUgL zq?U~(pHY7S{Th>`9AmhL#p%Nd2}j8;cQ+vln`rR}P>Eb?x=dky5RfwIWL1^8tjBfl>ctfW^KkXw4r%E}7jDzU3Z zAazp=ItI_z4xvIwY5|+u2$C!EcbH*Nl=v%^{-QnC*3-`v0PD$uGUe2FO|S(jS<9I; zjPRY=w1o+%2pbWSxf{YPEHY0^0EEfyvI3h`pO$U%(e<4VEK!zv>0gB2Qv(ar8h@Gv znzh(ZhMqiMovTKxYy`bUTiTZLq6RJ07;tdKW;oW5Dea;Nbg(Arg!4V0?fKveK^afI ze)uY~3cx-i#-JzM@ zHFq6D6^~Nx9vk_b1{v&t@;EGe*+6g>LkrOz<9^WyrSi5G!g+3=1$BDu}9o zmhw96*N^>AAequ!K^UdJebq&MpJCgRj^bNWlx14p%XRo_6%Ho;xLBjFE8{D-Q>V=n z12$h~U0+lEfjr<`vyefd0I?G#ADcl=!vg@2Hvc|=LG=Qgz@^*(T9K1?No_?CTgPR4 zW9b~2Itq{wbNf{LD4WIiA&jB-y5^l8(*f_-#wnBD8LM^SWz0~PIsh%oA z8LhlegHm~}?xeH!Kk0?gg{f4}#Tb|e^WG4%k3l@_4QLBvcnI6;)8m$5rfI$S$slvp z4AFTI&Y@x4TjI30{;BQh!R{@wpmW&MeUKwzZNDlQMF{+G?Uq}{QC&1u?tWMCRdYUj zSymMfNT=@ajC4)=O?{WUeN;!aZ@4_n(S5FQ+6r1JWwzS>Fahib6sa4*61!3m%i9W< z-l&csAFyd*J((n$@x9H-sKJ1yFYG09M74xJ?xMbMXY{~uIWhngaLLkK?;TbL3kk-D z&o(=-fphk@&`%XZZnkZv>W1H65~2wK9jx`rJwbI&{7S^ZMIoE|R}_?GAu$GmW} zy$5t}$q`Kt)wdDLMUdc#=b&`e^^?wX(14Ox`q|YDoqIpIQPj*pd6mQXYz6S-NWgJ) z`*F>#oX6U1db27k@5l5&$xD|o%jwcL_Q61-OkbByq%Ky7Q)V};H%G68-6yHqo$$4c z!k$j*8J2C9WlzQmM8hh=YK%LrSsDW}uU2UPT!@HlkFx7*k{DmGfccFFf#pbjIQ7K2 zGY0&gS742pf*9p!SZ@f$-`j;|Uwd1RKTebB{({K4FrDCV zNN$p*i%BU0dwOBsjIYWsk1b)iC$S{(ULm>kkPhs5Ww7S=k>U|y4wc@ewIDS z7)tmRK$*VbNyBCSx>;2tb~mXd9a4E0o`T?$MCOh&M5k0x*uo`mYHM%rcLQB?&nlR= zzHjlA4`s~2GvEFQHPq`K0QjfOL1i1T1~sKq{kj*OcX;*kT z`IXUD7+VTF(#as3X{NZ4!wV_|&CdE@(kPm0&>WsXU*|+V9E6>Rr^5?okiiMKJc{*i z<~~OW^g@EfwT$8Wddl6lHmKh*(Y1tdO@J_mq%b`vx>eIQhs&B zc_&)cy#2$6?p>xEXE@b8Jl-1GLsm_2^6_%*A0*dR^z|$*EdcLtj$tURK^(ybO#M# zk`#*J{REqs*w8l3GT`)GVtra2pTu2%F=@*9U`PK7(mIssHzNNFwWtId*l! z?#;sMul?r{+wg++i+7amiqKIVvoS}~HsY6DWSiD^U|+AfzQ|uu&C5#!hD(J^y8d#c z6J_r|?-XjhucN8PfZJZYu5aXi{StWoJ}O~d!QMBy7eOwXbBqZ@&1CI~GnWek2uV5$ z{p#-64? zJF8o6YCYF_b(?hkQ}t(6qWCFRByzTK`S`~9me>Ihhf~0`3@D!%_-Yo)j+xd7F|x(1 zHjoKLyv0q?CeXr@#8r-<33g9gifBs&76t(91$78C4|TJKz+la6MRCsJFa<_(K)FH z<6J=2UIbG}y-8r^9SK8_0$4lz7G3+{*`p;cjR<8>$naf-f z`vfTR`A>8(oAZ#2i{`&tBHTSX^j=`g_rz>>6{ID31h}uL29OLxCi?T$?)1f(>7<2j z_rAh~7Q`jj8%d;Y2r?@6Oqy?ksF#IQY%N-Jb1aa&J2F%_Bke$5`SJzFWUTpW_s+ z)1Atzw*>nBq&r;_2VOViuy3&loziWY69^EMt{hnxka*ok_z(3rS)7F1fe%SCRfH~h z;2yyhLeaYHt)$T%D<3Q+kT&%N|=nd$J4bhu>3H1 zS1s4Ap<3r>gUROxvf3f$9L=$gaN+yk>693JXn4Q9#T}pzq4#Gij;MHQ9d)i^49B*H z#*vwH{Hi1*(sVeK8*q1?b_mV54doi(1w4^zn^KQYH zG+(;xgYdW5zj1|v+vxolS8m&S0-mJWN+Nvvmj&yQ`#+`jpi-VKs9aHMr_-)@dh1&0 zxJT1sp{O!3Hm9qXD%umrdEm1+ttowJ_oJZzb1Areo*EXn4O0wW9_o7CJX>pO1gX3A zTdh1`885bT3H!B_GUpon&kw$nCw`8YdNc`t@X$3j!VxksEw$qj+)Sp8hJi-7u7^tc z+RV3KRZS$RNE2Vfq{H}rfpFqx!?Qr0HiACU98u$SKj*aLn`ca(KZuneWbFuHa4X_x zm>%n~4%Q_}RvK$;WEgNr2>gEZovawxSouETdF=D1*R8I@W(~chnD%nZsaf25M-GlE zx2b<{IBI~rn)SFH`Q9}DQN`;IX>a^4F6&=c5fIG~eH z(Misw@eu_qZgd2fVG%J99jO`waxO1aFn;TthVY2>vR>nT&dsq}jgxln5rPa|&X=)9 zKKzREG}Z`C`M!L;$%3DTelpM;myydR{zx0C=;C;ne@&uahthI$%vf~r+^nQD#mCDw zK9x0_g=j~+fNys=@xQS8j6>LN?VFj@xrq19p+ehJ&Fg}}b9=meqmi+{j*m5U z>4nkOE#XkM===0oPAdeQ6(=uuv`pe0=Xf|sLVI3TTW;(enVMRZ-{|^`t4E&vg)OJ> z<6H^*g?+~af_2t0XYje1eRlnslI`q#27R1lmfVK2PV_K#QAHij_7qZBTK{mXG*wpW zkImt{nJ;b|dhGs$lz{c)NpJD77G?kyq9xTRYxD2wWDo~iyj*Jjz~KB#Ck4Z{G{Sc- z@1OVWjaOmrw9p{_9mATZf;H{1BOgDu8GeJV=m#N5+j3>sgKvI$Dtu*1l5$V=)*FWq z=pmU;P>Vl%pU$q^_l;6WjQV}7nOXIm=Ly#jF&B!Er!c+lw$uCt2Gs7u>sF29{D0$Z ze^RDpL_MW7L+6J+uaztE%CkA(IbG)<5~p{$}ke(8BA}>YDAGe;CgO6*z4l zi}1hcJ*Q>hwfXZf+P?w2n}n)8&0XPqe;7|DNQdotc2$!v^KklRC6j=OdNYylZ`yAb zvA@CIb<|QlyDF8~Q_1326KNja@(zDRv6>tu@i#1OBrVjH{~$Dw4z-N(59DOFVh=XD zse#4jh*^Zjpn)y5HXhh1ZpinzGDewjCxh^U?H3K?)oU~R%-~f=Gb_7m6D5V4XeP&~ zVa)y4Gnz#KOzAT8@RSLp7TsU{DZ$tumW!+MsrbXDM`^QC7TiB@X0*Sq3Ak!u819 zWIQa2Bqy!5U_B5N8lzVC(N0yV}`& z2v`K{v_KuEHu9D3xaTR{5pM3t777B%TJUNFHs!8{T+ zI{pOzq)5m`Z_EKSQr}=ne?-BiqyS)WHjJi_b601hSPjBZj{-=!LZN_zVDfZ6fM+z$ zx$}~I+R*Y4NM|l>WMo%abetG2u7JRf+rOGdsS2l;&>jOFqp$&vQ?GM!vm>9{_T4Wg z^Ph{4U~!oQq?n)5GiCW4&dATJ=lP^tkkkcx=-2@I~YXqb9;bZoeAbhb(2<>*t>(m zjSQpWb9i>Zk5Ov3+zy*tY9XsAaE5(eWA2pzbw41vs&88Hi>;d&vLi5zJF zt3)Kld5uR!siHfW$%Mzes)KPR#eOk+<(W9u^W!?d(|nD>P;EA3B;~;1n#7(kF?2pk(8!JTcvZ4^ z6s%0j!q{G)JB22Rok0Vj$)Fkr7|8`V?cPew0>fdH3H4lKmw1N9?d6u=gr6GXMxU^z z&Hr

n2L|QNjuH0gY?``98dz!+rP4sz-~>n0h2UJ|yt@5G3@|T*xn29kD-R9|CCI9twHC4;H|uCd8~Rv$ z1i0gpqA{5ZL|$$I!iAh?9>Ym}j=j0pSuQTv4^>>h*3HW|X$1PVY;dwU0R*Q6?qQ6D zdxA}U_R*+2SIvxDYn4!7UZqVJ%iiTHaQnecp?sd#B-4k9_pvqzT46?A8mgI2vR>8V00Sy@-%Q0`L7Ra6 z((r}bZ1IWwl|DS#t(R%aLo=YMzMEson@yh2VC%I*7y1?M$%`p{$|4Mf<<@lJMiO$xC-czcN^$OG@a?3J(6b$buiqzR`N zdk7_ik1UJ>v>A_pU-wg@ZXTCDi@xnES-|_0k!>|O__lavyp=AG0Z>V+%sG2jyXC%R z@%l%&ts_b=x1`;KCv|V{LgiIBfde;k9AJRC8NqQ>`w;jl3jtVX5vsrP-BpNV+zh`c z3&mvnKg4F71ySRygix%BRf7K3OL#IB_t9Y5Npx}p03HqiG_->usp~%Bdy{*SJ(i`V zd|`5KqFt>J5;vc&)A;6A7sJen%RyQh+4l88JB=Y;09dSm#au*cY#HxanWfdI7uqVT z)v!czU%d}wdKlabCsrkIgUd)hA7Gv-GR zoj8S{4z4PdYf3tc4&3Ut`Xi;L!mjMtZojd6Md>lvr{j7vw~)9lQfNXsRd43;>{xPe zn0UWi{7|Pi_*@-;PnNxXeI0PR?EuAE!5%p{aWOIJEuSxa1fuzQ;4pX|uK`xtA#t+e z$tf%>95+Q-DUE|Im8u$`d{bubv(QiX^NkWeyU@NWe}%3y3@Ud2EtAND4#Cm(>G0qk z2b_6GoZA+uRV3+*xXU1_w=J;b5K?j;0?E|D^e85`olqdJ`*Rnr2Tgw;Y65~fr-k={zlyfOERWx(D8w-jvNrt5Zvf@F-*aRyT&H&wM4sOv8lGzq$i7_aN|&5s zMhmxh^_PglegVeNKCSZ$kn_3blSB-R>#&UVv`A-_l3PNn^<$l6MqKDYUGgZ`m7xQ` zOkqiN>E9w9`Pn7DWAl0^rnlUk^x15Txac;BmD0j0prYX^J83azNASWTaKV`5Srt5` zb74o4UCT3q9}BnQ&Rh{ap^7rAz9z+=u$5>3_f$}T34aRks0DIq;19X9bq`AmyVmsx-HtJ33JiH40FuiB z1<_X4y*Fp@surD_qlyljW##kOZj)w-lW9_cS-^rh0SNkI@t)w2L%m|Bu9IkL&dyXW t{`B9d77A&U7VZb{!qJ#((6j~HnNeK-%)Zb~CNyMLr_wfZ6Ib^eZ5npA9(WRnkv0@}@nx)!0 zf;6vg_@mLt5TeTB#2`4yzqU$=OGYp4&il*s(vl-A8W)XAhMxBQR#^XS*Y~7vrzzd4 z(rM_O@zCBorxj;c-pDVr1Wbgqs26`<2xQjoykxnG9fk7u&zDY|u_rRh~JfX_t#|MQt`Kc<G$S()49>g@OyT0UJI2b91^yU{%mO~_LA@D+atkfKeEk+0*|Ngl3r)sM6y?)go{-yGX8YgKBFP2-_ z2zgq|Qs%AtB|4S&di={1H6IH!_J04!E_5skC-eLKima-1 zjFnr*=Weu9K9y%478= z{WzJCo5a_gWBKico}t?o2MZ}rb96qK$ua2^3I!E#iMefviyqDqy)&pOqm7``EYOJI zG8VP|QI`9Ayh@pjtK#v9mRQ5=oAZ;rm>T6bq8^E6WS762tUR_@`AGPrjxp!Ft!n*z z)%aZP+)B`CLF1ggT6_AuSvaZx*Oxqe!Og~u)tRx}U{+c`@6#i9Ny*d3lP&qC#ax-_ zdsKa~%#jYBFs}5$cj}ZQE5bXD3*E`>H4PrSYKyrF$ph*I8mtv-&;P(q2;1&U6`W;N z$q+B9=dp~S3NdyMi4-F&WV2K4K&CERhyIG?1J ztOi@cei30-PQTR~{#2im?6nq!fZgvZj-O<&`5(WHuZ*2a#uT;mic) z$D3bhC>IYuMyyXYl`M>!_?^KUyxu2nvnMZ!m?>5}j_1C{+E8vw7iyOzNWCIuWgqVC z_+>Ire(O=x$Y$Q(nwI=f_jw_toDj@C*~x5^`K}IxGEcihZ^>{?ulFW9Cz??RaH7YiU^o z)!ny81Jf-5maD_>ZM;d8u~Y9j+%x&QWKryNZ~W*JzLWDmqt6k2D z`e%P$ZyhMSz9sttkCKV2>7;Corl8StcSW=R^e8vcNUfgJpqlpkTSfIJw2^lAJ97Gh zRHE>PMRe#s2fwQomqZ)Vrr9@S$_X?1JG}=AX+oMWG1spbqrZXuR^WBKzn-XVN~N>>#O9-C*BSn z&bFJ|KU{o6c}yehHZ7aTnMT3$srA9pcApqt7yd9W$L%x|mq*FJOLGFEF)bH6qVLD! zb{!7n%IjV4YxFu6E~SfB=1y>K^gIl6DluxXnfv(U2d{Df2A=TKhTQ~H-Q;9EDxtwU z`;!gNF+TEIc3wLsvXRB?YJMWyw9$(D;I6omlaS9@J+u4P+=?5_X-}}Q$n!moW7CH0 zGi>X}C6^T`&D+CxEcD7OUM`1x$-XH+xErf1x+(M`TxEAYTRM`0w?%ET+97)}QtV7+ zetoQ>%W1xogiE*abWeZwL?Qr#{L8OT#AKf}jp@z5%XxNj@Ap-o<*6!o9(jL_VI;wH z+t?6$OEhY&e0^-I+3yne%6$jjftEAnQ5WNHyH49Ne&!Na`Zf-57UaB$9+XkzpFKU} zif7Z5+qmm>RL-GOD*kM3f;C0psTL`5Z|kE$V~?GF>F}SWg9I+&hYa1?H%zHWvDA8A z^P_L83Q1JLeMxD#box@aoe_VA}MGLJBnvje{i<_=DefQb{x@S5^~?|Wb?~VwpLO}Uku|b zQu=9LQH7rE-+n8rB^hmT1ls2>m8ZU(QwmG#Jv)h94y-0?3tDpbaHH7pAr3v3GH@wd zg*tW_+NZ9KE@?k3ab3*-@%4}L62 zyh&ix@FEZR2D@7>Pbn>otjNS9Xqx||`3l!noBpgI($DK~wO)K1+Qw5(rD;jLzgEOH zmybQfFDG+{Y$`5zC2|?>;n02ji2qrWxo!=&ao1_J(#Lj2i{kbmrBlD#G8*x%2UQ`% z?ZP+DtdS8`4l5_eVYW=}Vb`zDc=iuUpHtJ*SUF)oe>EpwTxTS|rivNcy}OmB{Uw|- zL4mgG&Q*D*ngtJ&s2m~;ZNIId#iWv<f~O%-RCQ5eeQM~V&2@q8bUU1+j+v=u<~EE-7+j3klMt zanB4J&yH4Cm`=O-`tqd}c0z=Um8fMHgwwOmRX07PCnQ~jJP%3Go z`thjIVLF<=ewi&P&?9-92mf}$P5p<$S>{)$ahQy~A{sZ%$A#@j8=hCqcn>c!J;PqL zZ}vr9QDvIrH*3Q(mqRJ}Flas79`2!elVJLLp*Gjy?~i?hJ1^2~ZK*pDTioJU+wP-R zER99oHZpJ`B*jVnt5weNC~vc9?XC=|D`e<&d&<IV7TaE1K9|P_pjO0PE>Y?Uw)x zs^Xn=@1Fl%N~0@j+y|*V<}W6RxywJ|laph9FdOl@5t3!#fD>Qd)AJ-<`Llk^j-tav zwL?~Xq;5@nm0LAtv`O;*Qvdy*gBcl{d)$Uvp4$rDho#S=RPrE%cP_s1;S8~e>{49T zf7^>mO1$tOGs-l`EF=57Q=dql_H0Gq=eOU*&NlJ+F>&UfM$`^}A*%CS$_ySX$4H-- z^1n-@MMp?Cv@5)WFLsjgR&ts#Nz|)WZ6Tj8%XKqMIK~hzKPSTrN@9BHi0i7BI;Yr{ z#_fe7GnV%(L>JU}xWX5sTBPtDdEzrMpZd*(aeWs*)cJ$&LJpUGt>Z=u?oJz#GS=I1 z?$S@^-H>jwB1~0cj`2lKTmEF&+Uk`h6S0PH@F6QU^-Z!ukaF&!N>ZK>* zx|+f~*S_69y1p)G{Tu&*IsUr92AZb96Q{f^4YJwHRvVC#Ke?fpeM?arq${t-esrr%Wu!y^47DBD{RMa(^f&T<-W0omiB8r*#G6)mA6XIev&zhOw{Kv z+HN1Ep@Nncy7t= zJ}5leU1etIk~9pcZNagbYHYCRN>HrX-PB^Qo^`&e!@aKEenN}(!o)9u(@?Kr7pK^3 z&>D($qo+NgCC!g8=fdl0aktRLBy}(;g^8zld2;=%;-XCyo6}w)3aRa$xk+q16l}Xm zJB_I&@|L@sVCGo`|5hMJshsL0eWdU&5{VMT_BQjN7kQsKJq0!OH8X-4wuyt2!|m{9p8Ds>FA*}$A_zC*(vtVq zvg2O{;sxL9e7LvK?=LR#*uJ5q0k`lq@(`-Ohoq@rZpRHwmZ9}Sa5uf|P!r~KIz4jP zJ-E6+$}<(}sxuozY#+cEje=brjIxfqSHXYN`dEjT`5(|+8jq4)XZ!hUw~Q|W`PVFp zrZKSh!9wXDkZi(^%4V(az{~gV@sTL;WP%%`fNubfapN@D!Qe1urQY)4Oc#HIb6 zKH36o6_~CCQJ-bPV!n$5qv5f7iw@pGlmN zYN)5H^I~o#dEsaoJCRPMze{m-2xdCF)?HWd;%FJ<(O=(7|B(H=@L8`>;+-`FxY(sH zj{XWq)7R{>+`YIA4di94ELhH8Y!kX(fOWEOL>piNrB(*t(SOfV zPQ=8gcH6WY%#S|eK25vpI(SBA?5S$|M&$YMYQ4%6&-2s$GWYET6^kC>)2r9`ZQ^Xk z;oIb0fLY{UTqe(P+x)dv)pV+etj7*&V(RFUn?Lr67LFa)ET(^aPjPnU3dLk`XN}vY zS?WBn1mjBq3ujApqRvYy@rg~1ubiOpV-$3pi$ZRTh2FGP&EoZjp4E(S%9(nrF_y~5 zWpc|oukWEYou3`^J{je132x5J>38=zKXv_4q+L5Zai9Hw+7Ck+}vL3lzY^u z$-9A~sKl^NODSF0y4>p?>gZ;Z{s%LuuIJy7n96m1GX9qJfZI#!wYobSkTg4?D4Ux) zKdU;T67|%y#W(@< zMhQ%L@|hGWrrcKSO%TLHpS_R5hHc#%Wgc7fat_dcVZ1{ z9HsEq&JJert;Z|hUl(#_SqpP}F^g^T4P9!u>Z$SW`E39fl3h=}oUbvz@ig0!*tx%5 zV%e1-Cy~gZtLCw@G{aa@v;ONf0BtRIpW*lV%dfX)hmvxhk%IqV%2zBOHFZvh3u1x5P&lD2D*T3e!(7 z2C}4ZYh2gUx2_{xq7|7p)2)d*u3FAsQ{0Xqj^inVBtLCsZTKjW`WniT8re-1OA>#{m*Q_Nym>#X+ip`za_ zg*>S3Wsr!SX`z>=3f5^`?1(#2Rwl#e{Hz_1@KiRh;rYH?-LpS&g~X(|b_ei%&F{=d z*Bio=eI$(UjU!s?1^=I^DLj{!ho%G?<4CLlR<$rF&`rnQ{XRTbk#eE zNJoJ!Yf!?g^zkqxZlQei(d&3m7SJT(JUIUxNWD|G58~TZ8>Q3dL?d+Q+fBIgbwEuN z{JgrZQmA$FN2%%kREEB^yUey@<*NH1zQ!`k|0uJZt#9{4TT_-<5;#*Ubr~F?7Ic&x zt9n|pW=+obkl5fHur7Oz&0v0}GSSrYx8q02sI8gou&R34X6YkrH|}8J>}Lj3mkIUU zdEukRqRZksae^dsD#B~#R)z|3&pvcxZ>QX$=^~(ifMu|ewC;I!xMY=1+(hBGy3+;iDDwEfE*VWP0wTs7oP_LsEJ)F0xFoQY2qc)gk?bI) zx;U2GcEjRf*AK6j7D{Bs%iZzP}j`B6;Yoyd96>1D6dV_ zh(KW2GpM$2VZS?d>??Y3yxznfVUnkwacR?Nq%Ld+ws2k)-xWz$OePdI`-w?Y$-aea*(?Dk$ zgP&RFR%tOQr$K(v!)+d&)dhqr8R5jVLN9X$y$yu$w}GDrF^Wn6!9K^vFQmX;dzcfr zN2b%appkhGz2fDmeZ|TSgy#ZHoe>%P$HjUTa=JqHt|{Lkn!C;uAi3JOS0kR?c&9{@ zc#_XPeMjX&C;W9q59+@$S^XamIiJ0!uce?SPn*7Ze{0ufX?6pRZkoht@fGRKP?G-l zeV-g#1F<`Wf`cMxIP`UT?LX0nQ{C0#mQGCy%rm`ef*VOV@lz;MX@dnLW#U_(zh!jX zBI6$?g}1L~btS4pukHj`W&e>$I3c$utBP-5A{24!7A+*gZ5ZFwNKp)6=iF`7=Q zs6|*w$l+IDritm*gUWF22R?W^Jpo?V7=d{v-ch|g$c+~}8kk4*Ew5+0S4bH#x8j|@ zkFIw#%t&#pn^Ak$y2(85mOoQQ4nDOoiyH=yO6wAZmcYWFUvf#tlC7y*m^7YOH=M(S zdstPNuMihKY)n{{RTdZX&V9<8wC}Z;<~q2&d-OrDjrkE{{y2Z;DdV>cUe1!J%AHG6 zD67@D-6bjN<@?H)%K(oQ6ok2(`d|kX{TZ(+bl3LzLiAA?zsz&{+o|Nb^b^#Dt{DQq zJkDY_3IlaHYyICvH7XZ2;u+h<HcHo!>o%*%p zDAG>OPn*t}12FLufQHlM9ikTtYqaW1w{_>le}IWP!ENeJ*GaKvsYAS^Um9cVwW~SL zc~iSMHFQE61rP19Gmfi|Uzkjz>VzJ(h+AOdS4C`fl?~sPmHR4bPg}*QCESu)#qEh> z(zU)B5qO2Kqe!n;nZLBEBGA|yR9Ak7qJDR)ypw)ZXhni1j2y1-6F|$uyKP=q7^bf= zhP%JHasNffQh&ClGlPM6BenpaF!RgaP%E5SIeYz<*M3{U`s4dIEkion^QoK?+$5Pq zWzRpgEmEXYdY~maeJV#}J?@{hC%Z2}kAZ5Z=HwrYE~_t-`I~FOan}QsDNW-SQds1k z$HeJ1QmtKH6LIB*l>2DiKaQB(9$dw4GHF307qCmEh_Wuk$H9tJT%KF!|InDaTeGSA z>wR#NSi2`LhR2K>*ZoqLz~}&>wb2jsTJhY?{nXhym^GiG5!Z?CV7LlQGQ4oVFWgl7 zeBX}C#7!7AvSRwjL*X;TW0qos00Wxh#*^HVp>WD+{ih9#d{(`ekAf~OY?ZU~eS0+I z$vJXg86}ilpa+n4fuLT!5veRDRi<$=5jBtd{!iB-&Qlo;mg847PZ}cpSSF>zYpfB( z$={*W%`_rX7N2`c1ByV3~cFgDbmB`5m!wn>I?3xoz5HESR+n>rQ(N#_f8C z%TB%5WIy!;C8AE#5U#IE*^?RQy7Be3y(9{F8`givKrN&Z*qZF4FTp89X=L>3hzte` z3%r`Bt5KC6yEeOFMdKXl8I&yxM48`qQW-f^-0qv)pk2Bu2XijDL!XJ?QYIdCZDKMc z|8pu;5qCBEW){JA7yZ2rD`z8@~u72eSVh?JQ^S-FNW6v9RkE|H$mzm1iE~6;%L-$T!@!atMJo z9-QAc4}M1~Q}LL5_QL0~^mhn2$CT_|ZHy4V^R|O_`kFjN^>lh5I%;n4Bk2j}OdV5f zbB*x*yYYD~i*=<}iy=^@pS#$5S=KIO~Y#n}dteXV79fSQl z3=iQ?`$Bfi=_py|yO`J}E;HfA)tkShXiMcB)CaHy=9of64Q9^G?KWKnJ1kIA2W5%XXmk^#M-$omte z5x4)|Ck^yCUh2ZO$3dHqDBs?(KmTTDmZl9?zFRMV@w}D>LoZ8gEdH0e^bVy$wlC_L zwEcS97m?Bk;y2Ip4Y&B-1QN)+V6Gx^WQ|it<5HAyD6VDF8@)m!Y9RG%w(y}sv^4GL z9`0U!2mRF=!dYxg1@cEDjm6mM8Gf#)=~#C~wX(JqyJJqjyiU*NuzsxS9(Ms6uc2*_ zPV;DX-%lYjyB)PMQu1`N#|4wfelWE>Kv4A-5*&tHLvz!=My)C3doz9Ng;cnpjqoxW zx~(^nZktADuD_aWUHOczEg`Mn5+e#(*_jG1n!{$oU15m;6g^3|;MwR8OSzPjuT^D( z6Xo^pm;dVp5RgY=UcV?Jaf2@QlKL#~8(d-NfpI8{Irl>olP?f%lz1y(TIN)$5@1V^ zs=i+o@CR2|jUHZrSaCFLW^u+e`)Bnr|L_g>fPWCmaZ48bhmT-L1svP@V^i+$*HH{-NIYg(~pIzBbm3@ zf3;GCUE{gue@bI@x2`l`RAc^Nhc6{GEdNQ2JE0pU`lyU7QV^jLdpek}niIkSB*Uv~ zx|gtP&A(hZ+;2LkTC#6jk8JZMF<#=g8I1DSUC}in{qyIE?jh}!+RQfyzm}~ylcQuy z3{_9=mIP-8R)Vnm#YPPjB%xd}M|5YK8?T1jymy;!QvUgqI(pXS9i!vF)|l=zux95o zoc?$6np@z~sV43ahhMz>2`_*CJS!5pU}gefCGJ)`uuxpQtb>?E_qKX`9RS6+XC1VGsk?&pCwQGy5^1kLWep_wKryJOVkhx`x4Qg^gm@l*H zqjph*vT+Wiw1G-neSUEs-US@wcqMt@eqgYfc=l(E)%BLaz)=+(A0>AwSJ1Q-p-CNA z*o1h@4^_M4TnB-lneAw4X4x+uvdd(rAm_%{oq)2x)r!m20gz`NfYCsWla+3@eP$C! z!=7Q?vzsRADn}icJVZ1x0TPpqO%aqKl0ox$-++gSMhE??A3 z3}9?fYxHzSIwb%#OCiacNa3vzS|0|Te(*S4iOc=q1GWLw?Lg1P-Zb^$qm}?n%ejtd zw9jc8%b+F`vaRG~^*O?R6sKk6VZieXruL8lx_e&J4fv<}%0Hvm#JuYrDLw{(M1eZu$ zh1jESw2Z4zTC+geC}2fWxr-FsVwn^r@i|(xxN+%_5v~dATTH}OvG3u}U!Py$5m>4XjB){C zyfcWJh=?B04=;7yfFpVG3sa>2E~olE>&m4kq!MThS6@MF1_RKKI@c1Rhw+Or*BU5e znfHSkypDD{=dY068IHax9Bl4pBxJA*?9V+ZD$X&EV4#u{;YQ{nu%kp=tq3=%1Sf}l zy0CkW<9w%DiIE7nkU~Pw+C4V`H*1QMEz}fhy)P&ZGE2K<9n(A%v9vW*s9l)<hCI^vOPT}HI76rue!V84U;u|7N8M*W!eAm{F1+B-1D_c3+L5&S-Sf2%?! z{l=(yB)0$8tTaC2UEtrc(`O>~YF`vraa+OinqXBwkcCaE)-c9B0I$*XLo_`J z`qsNrp~Q9%T52I&2KvE@UZedOvP5g5_026$@M3&OE;k>W?>wJY9G!}!r$a7Eyo~?| z+-j!m<{2h<&+ZpTBtk|Ug7z>&H3O;`B0u9)e1;Q-TMEZrp|l|-eu?NVK@FEYWaYz- z61PCozrXjgcWy9zt9N&r75RCC=_Bn+%VQOVVUz-#MllAp z&a6g@%ySk_s@f` z&hIRLe{n4spGGTT6)5j8uW$Sifg~6f23nK74y@~=bgiTfVR$XQp0ixt9*x`mMN>AW()?{AG zm@KJRod7#bJ3c(>r;9fRf<*HSLJrv=eX(%W(OT0NIF!Xs%N0`Z@UReMd;_ltBa+e5s&0G)v{c#S z5za#(!Ym-I^7@>4)_`;8){{|{bhlsF{K0tWLT(!-F4sp48$2{Qjp|t-Q^f)M*$J93 zViCc1X~dw>Q#ajfPuKcqZH3;OyROAMD?{?$N6XB)Dw%Pa2M&|zBF`-#o|ORr>=KTo z5i6d1^8Iai4CW#|z6cgJLtgRS#J?+Phg?Y}>h`US`YY7J`9-G0xQzQhtlqG`;>$F< zOeLgY(twl_b#t1|jtV!micv3N$P{S2%YjUZWC8t#-V=4U*Si5xo(llPap~I)ua5OC zz%``RO+0TRgn9bvK?b!{72ElCe+&4}F7VX6!@#y?g= zW}ia`@|0>&txZ?uJB7CCB#~5j;npm=byt)IdH_pVQ zkc8lB9Gw)7`^L22rp*dl?TS^3{b5cAXb34BdR6${uJ`o>2)pw%NPt z&lBWd)_ESXcbi#fq4Z3J%MAgveHQWWP!e{Ar<3*V@1SfkL3ip~KnV7XypEKb6~d4) zdDZe6=st7@Y_LUX-O`G8q4R?f#t0$oS{hEei)+uxsEsCVCf_%)I1 z4Gv0*pwnANKj1?3d&pNOOjtkN#0z>F|N547RuP|$g~|cY%W6(dkmf$rDp28SV4#g= zY=HKmZuy=*r0D5XqrbAWnGez@diVM;-$IY;@6j8e%T#oK1ev2ibc0)5sXH=~T0~FC zWra8|wcia5ZHt?$3F=Tcv2?Odye$K5_1RDT`Cqa;VHt!9^eYsD zah7-jw+%8v7KYzJR`2XQI+%^z|#6<0$XgI&=tks5CQEjG=UBSl`V57BztY+mu)Yjik{WuKM}|o9KPrYLokK!H3ixlkg&=#Qm|w zC1#~=C?N0UsB3U9zs&sdShg7-^?kO;Ro2Gq-C&dX}@@?+>zr7t7eEpEy*`yrOro|$B;D=RgK_pwNbSmVN? z-$vw{h1?3Xe*al`KZL>mU;ut?{~}J& zO>!Pg!VqrwZGvGFVu3Pji+?dEAu_FtdL8ATLJm#@_2lqW3hMMus7B@hupJSbyB}yw7OR=pgt z%*tAe{Exngmsxf(f$ulY`ukh*MdUpMv55J{N2^2t0`h<&Ea58rJ*Uyd1%+CWpp#|X zIGs9D8tLc(A{VJ4%L0M%SI>J5(7XdmryOtt+xTW}_KoflJ1@lbAcw_;{oky_-mm#$Eg+I1aSfGz%C& zA=Mn=GOE{MT?5~NELZ^>w!+q5XbNg=WnSl_xh&^AR`34NXg&t<1PCVEyJ`<*(1sBv znyqrT&Zk2w?^f6h$(g*I4HmSHyKCNl*^i%^rKRD@D-Xb=F+wgj5v#s}pa|_=ZzwyMNpRJ5kJk>#A6@B-%5ihbNY4 zn+_F|p!viO@cb+z0kIHaoIO&^p)1Ht&CG`xrR-|JBXa{OnbCdg+@%-H>dH04(TNuj z)!{EhbzB|3nOl0-WhG}~7W6!=?THqxHaQIB)z>-9zF;%j-i$}*RaZS$o2jINSXdcA zw@vHELjNsCmWTvh^x1D6zt$ka_UpSzJm!O*JJ41VKlJ?k%uBb{`Q4L=YE^>fW|XyE z&`%4dKfvzo0z>r>PoZ!Xlpb*t|ADL{xBi8!6xN?!;2~?fLYFBqt&7WlN`6*Zk@a?U zoQ7&m{=dS})z1j-#nAs}F!qu72PH7FPuoa4AA%66sd4*?-uz1wAw)*zWfu3i0AHRW zdY9z^sY&SQID}GQ=yp1*G;MN|1!DkmM;@|&nG%II2aRej9z=}$j)kI`qr*YPr(pa< zq&K0|^wV`t*3YjP(0*&5kLZP9e96{43gS3-Ww`Zt%KP~J?}-}qN%zIHZ^Tz44NhyC zXHdU64gO}{^!=)CPLRP!g^TTqEI`vx7Czr~UYge7u~xlXX-%iZ&9)r!m={L@jlq}f zV>90RTn-6;`dur!{}$8fctK zuh&|GWr;vH?Df!|E&4n21L;QW5l1>8Y zqV$!oH+sI|ArB4VH;AbG*qq25LyL;oa94cbz>IR83p*6W_#q)NQ~F;w+ZlG?Jqi5= z+RshI6O=Y+04-x_d`?u=@|37nvg)_;82k%XiL2p`abErl%#yG7vu$TB4x zK)44MOtB3J@SYHqb^I}5$c?%3g0K(PnHDve-~brfIb;$I{^Ii=mi7}v_!9?c%ZbY& zfpe>!aHRfo?te*IcCd#}%PT@4js`}y+vyt;0YtX zsi(n>@(ueCnLS0kHU0tZLV;Y)(9@y}+$b}ca$_Jtb^)CICH!`KNZzLmgNZY0g(c|h;JGatPYm!~PBSN`Y**b<^ZD(-|z!KKpFP)#1IDPT>$q*78{9q?#B04KY$yLA0=7fX*$@Shf*y^ zhu`IVdhr?^x!Zlc5CE|mfzH(cUiZ`MR6xmdJt-qBInt5TompqdHa5s_a))mri?V4I zu>iNIy6AJdJAzht(jzcMd_CYE(tuKy4=MD98~Or}z=I!PmgNETSpuPALw)_}FMms~ zjcEzsZgR>TRqxDUJ9l60wUX)zExQ!|vg*0WRtLR*EqzNU_!cXbl|Bd%=|zqT2KnH} zJ0B!}{L3%N+(NA`oH3dNs`GX53x3?HB0&-k9TnSqrlhQDZJsi=un`RGsU{W{c7XGf z9WK(H|A!NL>0Vhhc5Gwlg0wvQO+C+OW#mTg8+);9=!VT~`KWYYNW6*~ggD zWDfcEHkoD7W8KWaOYy+jjOnb|RyTv;i2ZBqc}b5u@X7Og((OV^U2MvTDf9wNiy=pH z{}=8-3C>A=AMYTe3YLH$1cOdUq#7vXao390!EA;b$F|^GVo*c(#{1|UlBz(DHnkIk zfVIw#e)bcByet?6bLq z_$Y5W|K`oo(vS659tJEKNJ4Qwwto zsw$Ij$1kW&SIQ0dMz03r>@Gtzc7 zU5#=glcvx;(#kw<>}6*A54F?7PjF4#&^AD_orz%!Dr;C;Q!3E1toau`VFH@EgK~kq zA+9r!pgwSACUH-Y{~ME|9-+A*I9?OoTz+I{ID8=k(q;jml6Kx!bwjQ=t=}Ugfa`hR zRr^oUx&>zpu@l9)i0W5+khrqOMOWbB=n&-5{XNn%7l|n=8JX0pI9DJ1{k^OGa5R1G zA&c*CvfA&gOjZ)C!B>{XfW#5>3FvlIfSfAtV?;1J&N^nRUn_=V2U;x zLbeZ$`7HKQ@svDIYr}&Dkc2Byx^?fNI9k8#Gyushpn&8cbX$VEqhD8cI23VR3Fcx3 zPjXX+_8k_Ig-4o`-QjeWP^yDqwVWaOqtUAlv@_{4i?4J@SP1X|w=vQHY->oqvh&h7 zl79kJd(C~gKvNzZv9?c~f5ZNBIUZupj3k8_XYsGguFN2dQ3{G z*R1@$MA9U}7L@~Q?;O+>NamKD$1D)3s{#o}2ev+K_aA)2#GBmIAKo02L3}WXjfsOkq}UyqJ;#D&>G* zkAYN$-K!kW7OzpHqpI6gngM5KqZF{y(AtG@L4!8uUjjFtT^kvdk*?jZ2{%(T_kGVU#UDUlG99bkY49KyQHyv1N#zx4UvS3@Q?`^rw@f#e> za?sg^xP+oJSX#ucUCe=Ol|@R;$d37v)oLhC#a#uCK_XTXXI}!J{Eu|kJG_CE9&lDz z20~!FH8x+pe5tzKrNt}$jz+{|a5WB8WElYPnzs%BlW@UVHTr)+g%A(MYcJ9wHW-y`mlhOVrj*1Q#-_GyeO>m@du7W7~d#cG;<%{?`;4hK9wjZN6 zgm-&(mUMf?`a zBkq<7zKQWhY%NE?jTS%?HwK5<>U1X=KZv_MF{jcA9KV)WYT)YTBQlOhIbUde40=>& zg@Jwa8%3zKPkt7GTWgc?>P;MqU4Uh-%`2a7z9*v2zQ<-%U#(W|Ax69oNo)7*2rt;G z9k&-0&5dE}FsS9BhsoIo!4?EQ5^IL{vFp_kCiFiMq~yi=L zV?czlD!-H)0OfQx2KJpmnDP4*{$`T+sB9}s*7C!Fh!~`pr&fj16)A_Nrz&kZ zufqyc`mankcNeEO-DqDfR2G}RPsaFvg;A@dt#Vn_vL48J z2_pJ^Xi;W>PU31X?+;Y}0)OBNUm&teb&v7cJhX7I^})tvxXo@2{U7jr2e$r0Nk@zW zw0^MrmK1OUh(|u`LEGio`s7Vyh8+6>VGd|y9LOvvH$T|V+_MJT0g^U@(w7MyY!L-f zPafo93+Ts3E&gSfN1*fZVYu-Agp(Xck@=9h6BN*S7 zM?YkLP%f$8Y^UV)*m>0OF2O!|ZLC5bq5$UZ*H);8o>T2t`9(HoUyiGAT{nXEfdCG@ z@{T(sodrO&Ii4Ii=+-@Zul81Duyk$n_aFGe&>@snT9&PflhA_UzTU7uh`BKHHW-su z1-$1mjx#jSTA-Wgw-sa-ppDwO`QOYU(SP8%B5{m3mLV-jY>EgV?Fbi!hni%sV6R`o zV1&*h?GcQECQWy(T>-?=V{`@Bbu7TmsRfX*EC6L$=yLhGk^4N!XR1pX;$J72(b6v9 z=2baJ?{xktJRFi~$j1mk&qE?Sx@6TbL0)G9NpnK4K6}?D_}nXmjcI-l6(*Kdnht3d zYAJ#&mV?hKPaR4*%(L(*S_{wznAj8k4^m<-@l0(H<2aObvY_=*m}o$YXh+|a9B4*y zQh1isEK^7!2rBTn&))wc6q%mGWkKT(q!yP9>4ieP`I4W|yIBK_;*#NFI&fMHEC`A17+sf zg6pn}RZp2py@57+9AcIXE^Ybh>Gx`I`wf{Edd{?`^FL0Y;AjrzgbtbRWWFyQ@aU3q z;Cfv%luD~JN4>hf++N9zZ93AAw_?oeRp+|#SV1AobL%T}==Zzp_1PGFCKy3fPwq`+ zWL-%h?Sm4nG{v_tzFg~@{g2n`@Z5mGlU306y6c9HsvTzIG_e@%;!R%Cn|kXpS0@E1 zmWCl8yy90;NZ-KYiuagm{0P%^7t--y{p|&ho@FR=7J776;kgCs$3IVAa)CBjksph$k0<6bXi>yuM*qSiTw2VTToZ#vm} zWfp|m3X>sR{um&f1GLFT%bK1%g(u_0wYD_k ziynxxD!&iB#7|u0fMD-Q6;$u}(Th|k{V;^4T*W5(u#5uAp2O`qWD2^eS$npm%9%=Q z(teP^P5nw@mKNAzZ1U9G<1Kn&T8C!E6w@vjkB;0V% zsy3T7JNkRSSqFB+-I;Hv&y}|p`Ll;^|AlOTz-vgtKGDN#94h@|_B4#Cx{z0rsUNO75_z!(AG#is< zAG8T|wq#tgyb1jK`_(5`ck$3eq|-7=fql%FYw;AqV77k@s*;As{@Q!*BOxF8i{ay5 z!jN>Mzp;e3y*uVK{oIawTw}0B<(DAT5zmNZ7B4(H z%3_*J+^uh5F(M?&kbEx>$=HAXMJkxIPpJOjg^VE+9f#3>7hHNvx9K}_F zXn85B&Nbpw_s8hrjctGV$V{ebHb%T8jls+{wf+<_66c-Dn~xu$DKSn9F^0d$$a-eu z>RfD~@vYaEQtkm-ze01o90i`76jjg9vB_FhiSe%38psL=W!T3QA^G_M>nwI6u{dMz zZ}xe_`to5_>&a$gmuEeD;G}|+NARr##pSg@9MaZBg)k_|>F-tt9dns!4H|%EIT!jVV4kjnLC5QLDX4mYZv0WS9jd-#fywVd`tvnG zM$%y(dnDy{3wAMNZYJoNA^m-hq$a;Bi@Cf$_&aa&069;ceWB;8v#_L*4f&KP2rcsO z)N*s|XIi($9+zNqd<3Q6&5+-+6PV$g5R2|4Riv;5rkhw`|Jh$1K(AbT%~)9wI)q>? zB*gkpU6RT{2YoOqmja;Gxf77W;El_q;3%R6K_(Uoq)zCS3zN@y(g}nQc<-X1*p{co zr74^5PFAmVUFQT^BmvT5p+rcNr>WuI|Cz(oNY2OX^nkX9T)UsOThQjz37R6_D`zlX zAt`W(x3A(@RPX(->b^Uk%f9cMpFNVj_g>keNcP@QRz}F?M~F~<_Ff@Mg~$$tl#!KP zl$}k=3`N7J=X2;hulu~N`?~Juy6@NPc|Ff_{&$|8bo`FraeTk;&wG74li4bh%tvNR z<*8jwZ{S44m=qf-_29=dgrBH+1^1=^^fw8p=0Z5c0ss(cf~JBXmnDDURr0Mm`2TNxCYbt*J{h9)m0ld}8J0*s`VKxFZpJ=hB0qy7L+ z!ltDC$@*)%rGhw)8`BLJPP&h?!+0p6ANe?p5@|Fmha9fJIeu4uX89LE(D3E=6MU_v z+31@pZyLYN1cf#J1*hyH5y<}nr{LmH@T-to1L)$VuTk`TKLNa}sX+jn7(Zh3dVbRI zu%h=#=?jiQ8zI~0-izwsYt3XVM?jhJGatuVAQ`=W9DEr_)DbIg!zu!Lv{RuxYd>NP z+Y@2n6|>-#`Q;}W6u;&=*JsmjccYvD-8*67J{ z3;q~*tX5Qd)1numK6e98{qDu3m&96kPuG&tiz%9i0uy`-_-MhxdYC{Rd+OZA^o<|t z3ef*|5DO9AXTVM)SSVJ)VTw^#u-T3TZvlEv&^biD_Od;u?AzyUdw`;_+)xU}Sj*ha z$ET;)Y&nOO`5{6lwv)*};uH&Dsn!N!-bt-7#%Y7r1A^~M4 zRkligw_{W9*i+c~@LkX0TYflNbQfh<=nf^=MBCUYJ*k`Rt7lBzUTgJYEPbX7% z<4Nx~P&AAi1H7fp6jNxZZIFr{Y*MS~53Qsh8uAV(#5aVFJAnPbOZZg3TzT)4w4oe= z(Hr2P<)?8gl_r`FzfP`- zeaN;GUo_(6zrk=8S1=dX<-}xn*v6CS zcbDAEACj&UPrdHzf_#C5QQx;p6KaQ-XK{5}DNr|>K1M*;ti1S*L^gcVlw;@U-#sR1 zN2n$|)FlfnTIuT;d?S*~2h*6Nb(c7K=-bF3hF1HmZ%8~}*rwSErm_d9ytzj#U`u{%XlG=Q zBWt8~jw^uV&HGmFvS9J9#2CNya3z9Xi^hOzi?KkiRVHHcb9O}BM;Kb9X@>0m@ReXK_3PqWa!6vd*w1QN??kBB+|;>5-Z86GYt7*kk-Isx$IynPt&?BVL%n97bLC0o9gKU&@JdM_M_us+YS_ich!udyRdQH? z)FoAn?!1^>PkQKdVv#dc0n^-ibiG;-OxP3s+{nOZO6TkOY@O;ijPm3FwlFjE@|oYr zg)<4*nGR_Nyx$rtngnQvr!`V*TEF>jZ-noD)Wd(|d`me(L6$W;u=N&d-Q-1TDtAwH zB1{>S0J>Ot?<^bb78U?&Nxx6o;Pd&tR>ns_IFxrXq+EwJK^e?g4(2=+j#%6Oj!ChA zE}?7)ls!xV|FhMZ4{=D-Zo`hAaHZCfXcLC%3uKUt7!SJ50QVIB=vSYBWa*)c*aM9Z5# zu!3Af>Px`qno7x|-3`3~;2nV(Hd?u2)hm=d6)IEtCxGb5$G3F81Gh&D5x!^Qo;VHk zBCQP~n0$R0{*kqK?C6CkIbvH-M{LWbf62B09Lxmvy9a5xQiw$K@XE!XCxLz}?Nu~K zT1Ub8)D8abgmZc25TFr6sCGbif{w&$$JST}Ht#_-#9gTUv^C|WHh>-OebcuYs`tla z&d6h7fYrE_UiJ-?r{3qJ>7vw3jl_mYhVC$7r$G*iabo7VpN<$gMxtX5&(nd_f?eh_ zVze&hYdlV$j&S!EnTnB3ab%W!xspa~8q%c0pbm(c{eZ^DLa;)*vfvPq`lPVo1Awq? z0U*Ip>E0jGwuU8hG$!f7)31grV{aC7`eHp=DUHuS1SpWv2H{&(^>Eij;Ofq2E>^zy z=^EIqFb*vnjvno$73V8%LkWo{n{lQ*wM_XU7+p&ak!F*B68$lb13t_s(;X?+Fpa@h zhr+YOgM$jh!)`uo_yO}-52_k-!2E2!4uv%7km41LAI`ad?@MIP^mD(Pa51IWxbo!1 zTn^B$laju z1I&!(Ri%oNw;w-;dt-lcHoZ&Hrt;P@lgaRHe8g2h&kM^XEX`JRA55SfL;BZg!0KN- zYZW}FWF2XgV=Z~64pPO=R!~{>xAK~GvW4uVw!9ljoyls50tx&uyn&Go(<0u#dwU2t zjQ%^-qc?eU9)2}n3R>tO5&)*3QP>uZdO22dj?W$o=fczf;9*{-5j=(t2boOf4=l+3 z>GB$94XLL|TfAj+-KbnlG4v&=C;CSFV;R03?E01vs;@^jv6Fz5>j~@6YY`X^VkNWZ zL&-&PD1ZG`iesdBF(GWhTNvXbIKywYh%N%<uqiP-4X~aO2EoK)6ET39WxEpyozPY$Y!n0uhfHDWudl@;+ftQ*Y z&XUrzGd&cmaZFaC$H#HD?2c*<#geaIJ>)s^di?>+N(w5dYocbrk+)k+b1yz>B+|*1 zLNOAHpE+JgDHWt`Ej6ijmS%`>LtT%D)vZ95E=4SDx-W%!(LmZ=q=-|0HS5^{Pm{hj#Tu6z46MR6W`o zu$B@uvT|DRAhO|XM!(h@??4f z?!Upoc)GZZ{*m2UK$JJDn?|-rd=B0(qfK8WMgjQAiJ(RbznBY5#TlwH%q502TFQ){WX5 ze+hsM7m0ryxWDN)HQNakuqgw-8@OH`+qL7HgPqJJd)&B;-WPtZZjWzH+g%;QZyo+I}{tPwfUE^ObRsB z8Kp+g{};?c<{LM5{YWXI0nsslm8WoD^13FwxM$Oy!hYR8ML33t%ovN%!QUQ2R8 zvfOkw;?4nFU6Vgom2MFjmAV1Og92~dfZ5gOce)E9#k}6($tqPlkJ~mj*Vf0y!Piye zlhJ-LiEV1LDjzD`x$uao2XB?}SZ$pz0a|pm%Cs46KkyfL0$|DBf#iY&K0h=Q{ZnxW zT49kZ-VN<)oNY{M{tteL5?z}%O8oBIhmA=r$a@=osr;Fbfqvkka%Fz+GF%;IqAx?I zTVFD}hdCa+(O@GgzsRzZ_{`00ZG7}>ec84VA#=M`e@mY~3H^tE>UoE@dlXW$UB;>|f6a z=6M+!SyC*B1BqowiLOW^JFkSJF!d5iQDYmSH<3Pc?Rnc-s2|ErFFN0bVI&!qMfH#u z^6~ zIN~x;rEu)P@L$xRfMG&J3{sW~6zc)(s4aO^_?{I*>%wI&Vt< zqMW$vRU)$P<6mmB23R21c-tm|XL;b|28x=)<5>MzUJj(}J{8ueSU3|(j(xgug<^|- z*-Q7%e#Q^xM>j*WvG!xL@y8Z*T1+#2m4%nTqn3WW(F>*Z^L;Anrh(Gu0^%Z(e?gJ9 zW{5HJv+SMF^TUL}zhiEOl$s2^{dd!4?u{f0{kY=Q>31x-iTsgyZ?F3x8-}VLis^nsWY(Y^RKGGZd) zz*y>1V%_w7Z_mtG+xAf&ckYWB>I)3+AL|LX>uE5*W@r2qHB5cy=LwW`3ZglibT+Y7 z_+!O=l`{xA)hJM%VVsN^BK)RG*Nl`m^E5Y`v^QC}+3#WzX>}ibY={p0=s2DEN@U&h z(;2?NO}%GcAMONRar)FG>E>}(va9{>qS^Gu>xw=K^-v153$B@FgYOsbc?RTP zZE>d_88F4!NEZX>^^F6U=3vp6F{=9{l4nzYU&wKK&~}|PM7AJ6fh1H?>OY0xy&RD zxCj%NPo=%1-Kh8pCb?g+kVTIHjwL2pVNWv$7HCaux>^`u6DMH^jNP7G5gK*g(c05fQ+!)}VPGd|0WHPWnc`>WG7lpLugtI>;pJ$A_mAWA^{#E7vC8G8!@oS#RL5m*WImI4FSwlk?FM)dr$$b z22^neDB3GHS9qDHn}i`dk0&n}x<(LB3XyMs?+jdEJg|XraAeGs!%@Mj zs+t;+mzUQ~mvsuvMBRMp3~O+9v?DI4=z5H#&f-Xb0otO3SrvjcZ#Do>i%&@*_Xf?Y zFF`hPh$~c!4%ZwukCmW1;`ZuseeTI#SHV$g!KQk4_*bpx^le;UP$oOTc$ikuxYryn z4}>23#sCbMkIH8Q`0=w*3ue4uWKkR_M3@!lY>fskxOPcmaeZf-C3F;Pi zf!$XSA9!6Ra%wxXq^|#1x>fVp=CMO;z+$3IxuC19ZLaf3X+}(PGT$qVY4&=0OqZ@4 z_6qw+eXbr(PEJ*2Wt86*s8mc9o{%5Jb(hQXP-gK=NSL_5dgdB8bojd&fk)Q(tkO}t zt2eN7+`_gu!$Mzxi0eG|h-D%U@wm-g-}k_#$2!L`6j@sHs&~uNI5H zIh#Aux<3>0?INb9nBx$*T$E5bI8olX63bVk8X(l3RxtW?fIHjjJGO^EpssCDp$LPa z+ZDe51Y+@XUoD-@wn8{XyKj5Z32lI{>|8_ZZ0p$A1&F}C;Y2rVFkB5!2d&If}! z?U!XL+hA1QEIpUoTJdM==e}CL%Y&FsDFXuo%mIvJ1wLR~)#0IvvcK%Q=EO*@e{E4p z*@8%EJn$7oPB=RnFuRvXHf?k=wK8oROZY4}Z`B8?c=o@4Ck!^b78XLa;(Z9kpm-)* zRae0ZWWs#PCn60A7UqMj<(rIQD-|WY`fw~63!kfI4NP3MU&C&a>+%eCeBa+c$JWwv zT#`6K$o*bi+N2Hp;2Rr>%l-V<_cvtTK)Vq3$nRR$4uBp#5#iSfc6r^G%uWaZ6?(fE zYB9baew$T$ldq-*Z<-o_pqG%nR0SRo58vX9`Ow+jm5V=E%bW*;Hk&jq5Gs-nKdDQT zo)zIW2S$+`Z5D1ox<`aN1(T$1Gp%4?mU(VzC$5g(g5G5@N^fASh(o@-l1c2~6(u~x zKlA}uSdgLGWQPiCzcZiZlyJp$)v0rJ)5PwrIwdOJeBTHvPOGIdy}iAAE-}KF{R7rV zejAuwR!#{D#vx^|`U5ZqYPZQqNTju?C@a@&!31V#27OXBVpHW7wQ+6m=$AxlsHz%{ z!nE1 z#Nq`@oNk&I5b^2yvol|pMX|B5JD$SSCm}9w6y#)PMjr|1zvz&-y{i$3gyw2}lJ-}i zHVRD5&f1U|=>Hm)D4gqj_w~uii#~B_DJkqgatk#zHWuvFEiNt=K7IOhXKO137F?~i z*Jx(LRInVdGh^;@o|}EtHU^KNx`syCdoTrVcW_BbnUF{}5*S8YHSVT%@f%o0g>&&osLo57T|8k?6LbF6o^1K( zF0LOPp(~pAqlRMOu)_6&EMk%&{PQ-*gKiv9UaDj$6YjntYIbG~RypmR7<91(4ZL1S zp4HFsl`5VAy=JkZI9*W$%SDPh;(jk!80jjE`bo5bFW->25}Jv^eGv}+RAadk(>hkV zpy~DoMn(p;!AuNkEu$c-g`#X@n}0L(nEwj!E%U8UVmo{9{719A#p6Pjx--` z`aS2|x|7`N6;aca{*1oPYT#Todg^P+CXDY5AC>*|>C;*bwa#zC?g3LR3V*Bq%@gZI zotX;cCK>EOyL96QN@)LKD&9QyfZT9fuShSv;e}2@Cmco7C_ zYedQf53zHyu?S9R3_>|(M#PB-glPUHnXQozc2zg%K}NOLX|*jcsoI>K=*mYl7E}l| zm2{S!vG({zs6xe4IiY*#Zqq}<$n58gqc2(ZvrC@KovR-rbdl9l)I_4$w#V}Y+SVD% zJjXYLDc0lJ$J4>?>nw5V^1W*kBe8m?%2vL7K3vmK^mt($PP*OH3c-rLLui^TY5K9U;WNJ3;NZ;?FJmEiQ{-=A_cEGymq_|W!2x)BTBR~ zIgXDQn8*f`*osE0j+;oaPgDutSZ^}N2@^f_$##?4+=&|NGYU|-rQA^pH?Y5S@P7zCu zref8lM<<~Ko`k)PE0&LnwVueTZND6Q zLZXLbx0Rdkc#)qxs_I#*F!rWJjm+>%^1&^o(Ay>kgH@r!#678pYQL=gY@n`wwa<(q zpEN3jILK2;c8X6r)!5HoIP}$MN-hJ2?*JsmmmuV5Q%TgWhGVtW38-}@#Y%UW7wWyH zDPnEAP8uY^d#!T83g2xlMvZK>9<;+>CPY-sh2nb#+yt^bDE&oQd_Ng|%J6CO{R>FU zk~`Ec2vt{W>m^xt)=t_XHa9Bb0nyjpRC7*9#0?dl6lAEMaHxL5V^{iN56LiSpoUgXnw`X$HmzkU$q!APcdZ4YYm zm$Nj3NuugJLQY?8h(5>Ctd}L2-u}>zrTRXz{tYu8Aw9y^KL}pkKWn3tbn@io585kF zYV#S2q; z2Na=mvb(G0w%IhHzdj-6p?jtoMwuy$29-m`kuHvIf4~-@5RrkG(~Et@4`@he?u#q%)v~hl=5INUNCu{ zCj2AtuXuG)VFt+8V%|A7wtZ$DqtuP{*BJT7G)|7(qhrp>1iXVP~B-vrs~X zhnTnRfuZZAmqLZJj~)lvKQ7wKD-qgz(8PbrPxS?c0grx;sdYS-?OV$k{!{kFO}zub z?keoZfV)qlY3u1MuWJ-BUGk=L2Yp-r-uLgOlDuOoEyzZCca3B2IZr1UK^AGctIY_m zoOABQP~JAyv*+`_KYZVN*d>2Wzc%fpjY*Q@?-{cSAKRIvN{`0h`An-5&Ty_0PKem7 zoH$C~@uvT!oOXP^EVu68zho{SxrP7pUvZKX4Ivkd${olx*m6Y=-}HclPCj(RnasnJNa+ct zIIS~Svc(VK_TdUYIeNs{YdKS3=%Zj|6#t4e&8i2)wLFQ1O)M*dAJUMF9;5&&7}USU(__WlvsO^ zdI0K8K3HWrL7&fullJn#T(JPj5G^wi%esUG`7>wKfZgnX`k9*V;Dm_j4Z5G{JsI;k9hIotR0*q zcp2k^1Z;I~K(5^g&KO#V{E1=)vO{GXj^^TI2HOn}BvE*km6iFYm<>$7L6paEa_%nD zt_N}od2@5K30dRu-3tznSWFrHq@yRnoS_QJsVdB!nm5U5X?ZW%+Y6MPSfZ3|xD$Ky zI0vy0LacrQ!ZDXJvaqmd2;S<&)D6+4Zph5^A{G+FwF0xW@jVRTL1Kcv$l=4F&J6~^t#O1&#W9+U<5-L%olx;i` z62{FCo%#${PMs_dxVT7Gr!n<{=wNFuL_TllKH)Anv3&t$hrDJJ8t^bdj!k=@JNX(Y z{Fd=HRkoNM8822Ml`Hk&LOeL@7(Hl8McR}CBw8l8V2Tssy4Eb^Pn`Ofb50u3T25$ zUgk?1Q0r9T2UOu@QiY5NuE~VqmD>x)2Aqn2=3Tz7Xw$ASlRf8w@xPf{L%d%3ujR)@KbCwuKV$pg>qTSl@tSNE`xph;OdV5zu%( zJdNq7kU~UNbv$PRiT-L3BX>2&Ad>7h0QV`P_&#S%?9Ykzb`?oC#Y4|h7Srp(DTla%x@h_oP1tQCucwC+7T{Bf{y z!~^w6K|#R*UU^aI>f{pM#wUx7jg32T5y+9sGxlq@I50(=ItQi`yYi_Q4o{1VNqi+M zsP3@LP`ZJ4)*jbV8dsK=WcfIdkaY0sa5ko}1~4w<_7;CE+B?=JylwKs#InTc+^+Ok z5_AR{Ppr6-$$aObZ&n$>3$sYAT~fa0&wQ?U8~N_n!xRk-Q^p@Vk`TGbQqRLW9dZd~ z_tms~>rbO~NFbdqu z#C>g!u6o?j0A!WPS8%ww1@1bIp(4rAw78jr<-rg-$OlI?w?oEOTO6YW2Ik-sOcC{` z9(N2`S=)Pmc=txE>C{veqsGM@k0>oGm{Lo;7(PE#sNmZ~c2!Ts{^x zB6mdPq|#qQ))dh7em}CiM5sOtIIIW<}7H~s&cTNzPdDF7ZhEo1b34M z4bDD=iA$yvJ119Q(@7Z`1*1`qof{OrNdOCNamuz#{R~uTsUG62P}qP(DOlvgECyyJNo~(Ay0)hN+>j&hZ6W3U#qb7q^u$ zCh3x=A@U3*Uom7yh7?KX6R-mzTw_7ezpb7RWmRccrJ3K%jpJ5&^DL9U0BX~`&fVO{ zB%JOmB-he9?kraJ9^1Kl9t#VbPfJzF#8X={{H%Nz69@syHHsJ%CE3L|_)WI~VQATs z5j+8deTq|qV1u=0I}_t>*aLRvCD@qXLuPXMZtoh~%u+WAskk>LoQ_wA-Cp?}rGdSl z3}3-E>y-`XarpH9^KE!-)b+5g=mgZZuQ563JbTOjz*NUmzvB-uW{6vL{2H_-IDGV9 zj?alXwcdyBBqk&r)E76T%SZYS5qap-Q^7RoWnwZ}Sf&%R zWWDp9;V3n#^iR8Vn5G7s8%$`TBUq(bOW3l;WI2z2oK;v^MMVvqsmsGpp^oOEh3*cG zliyR;W6gg9+c|{T=da2c*`tmAfMoR8XHdTgjE9x7LPN4EvEUyqHGS1b%C=$u3#2zz AiU0rr literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_connection_hijack.png b/docs/extend/images/authz_connection_hijack.png new file mode 100644 index 0000000000000000000000000000000000000000..f13a2987b28d70f81bda7096a54ec479f20b2690 GIT binary patch literal 38780 zcmdRWcQlrN{5QHQWZa>Wd6P|cLS$x-WJmTWTSjDrtVC8;86`53k(E6|W|FMz>~$kq z;dx*6UFY}5^T%_Z^PKaX<9p8c+}-zeeXh@Vzu&L%z8+mylP4ymBE-SLAy!n7y@7*+ zrw0GA;-7#|9MZp9Nu8EL$=dljUBrC8qOcZqwAp>oGp5>Y6`LFzYZtTam6~jq zl(O->Yv}U0x4S0Q8_JNEaN@t7u9aS2%^r1;MNd*kcE>vaFz z5`!y2a9wzpAnfBm*HrLpe6;=dC}A)=ihhkB)vv3_{WAtU3bpE!|9-z-9X{|4FysIC zNnSWjiBAaQ9qex>ynipe^-RF}iRHnK42757b01?{H@w+Mb#iW8>t~h;CXy?v@!A#| zE;Mf?No$QjeRCr6Kf|fhR%So)ZE^ao@hDl4(_ChP5&vrOXw|)at|5biD%SkSd0{MB3haB+ zu?K4qm?#;BPH3G{z`Q2VdZVK%dT%w4(xM~&;%j~3(skto?zsLpx&}HtpZ9wV4(NS% zOkdSl_2f85QNz5RfcZRj{IUXOfuO{3+z6X~jm2MKO~P^O!D_`Eiie6I%FhvxCU>(d zl$|lO9+v)9us$IL8z$#QX3X+XvGUuSr51<3b{*b+Dq^_vIU~C6=+MB(bLI@YUf~Oq zI`7?$jrCUgs?65z-dFN*bIWjPD!8EerNH~Oyl8BNWO6$H|*flr<}TKpI@tT-lk7%*fg_D^WJ#G!72ZQ zhC|$g_&mRCzG1~wyx1q9*nccoaHfEc7Ig9x8!xA zC@7Ur?a=r2^(cw(Q4WjYpvF~pgM|&Q@z0qiFnHWl1j6Bxb-o9NjOj+j z);(o;w>G|i?BD%X;X*xk+2Px~QXEH|^V@Ejx}Im3Iuu5Vt<~eWuZ2%B@`*f#Vnd303ey;0#-cUPNOQTi5y4&vQlKu2>sck;hC7b3B`MJy9J5J@w@I%jnI~@r;Yob1TCDB)r zc{-f$eU;xBbjsjsmPtcE{ss9bly2+gDG^kc-gT;PzbrqLky8BFayzs&*edw2^5}5? z;!^G18Kr3s$E~$*6~)dg#t#lGGf2M+%+VJLfZ~CI1p<%M^Shc4d z?RzOawF{Dwu@7ar-jJo$_o++Rch>d_tYCd%l|3GfdrT8ocy7Ml`wIKk@X`gFFWDM$ zqMSURS4PTb;3@FbyCSGBTm0Htm)j$~x>G+WHwF`;u^&pVUu+e@rTPhIXBvy>}WvM6J z>3f|sjr~43tQgCp*q^K2Uu5jNdo9)Hr+%sZ;k4uPAD0eahlDInG(229q!hdzWj9u> zFZYCoJ<-{72W#qo@7nXa74PlQdvPa~jkkY}ICfOJEoI0)BHN3a@yZ_5J9~I^@@Qw8 zG)dSw-Fttfe53Ic52a$3T8fR`$;#7h*6qeEscwE*d6pgV8w17)%-X%>A{|!P9bi&cq}6dT_zs3pLogx-*SUu0NOc zUmaGtuiTxaz5)kz`srmyqYEoRr_P0Y$MYDk=E5<31LuUh_Utd7+fq!0%3apN-`wuEdD{_mRDM(phX&uM^$Bh4wZ~^a{#+TcoNkF| z=k(XYTIdbcY>(N}1YbT?Jc{lK^dP1z<1u{yqRjS7Hbdx2Z*b)qZo@$~8G7~3Dwq%- z>z-8YYxVnY2CEHwK zULTSo3w*BCk@5o3z(h(HYNt=tjvw^qGRJ)~j~24hzORl}afs}lU$+{bun%}yH+lRR z>4TA+P5kfTU+k*;i4W3=jV}ApH@$jxeREsjOk2%&{c|27rErK99XC=?G|G^sif%Q9 z`Afdhd7EM0`efatK4neBFl6Eek%mc|oI(3j_TxN6Cz+J3#Q9f54yZ_J6VF=BzOY8A zc~J#Y1cwJ$)-Jm2ZdkrzBTDk5TFeev7wbvdE~RYdE8gMH>K(b6ZA$Mym}+&^ef672 zPv6uV)zou4N|AoP_V`vmiEe+LyY47(!Fc7%!0`<_8=r^ak@z~pRWug^nyCDPXQ zBv-@w4eHN!CPnnk^`u=c+iF?p%i6NKKIYTU3R_lAMCX)X-8PS;u*fSBnlC z`0H`oW9vPiJvnN0=5$_waqjLarSX}|O_Svbl1AHZ1q^USxl<*bC_D}0%$;31{24zs zur7I3^O0$I4=akCX#4A6k-3j{#GE1~4{DIcu(Y6)k8$wQx6oZ3Jau`8yH^i)rrY~o z){xiIwTvysz%e+3da}7gKq7X%1maK+KGi^L%pNT%`+=swxX*UV{lR&Dz94E&<9+kn z2_p2xIEJ;}6;!nk=T1s(plvBP0!-ORQt|Dvvt7v|nrlic!=(``8N=__YWKtQ1`3QF z?aYtic8Poae9Er*I(zb~R|gx;4eQED*#+NlS{*B4x`V!0UP zPB0RT?=V~~6<=>Aw;?w}1v7jWbz8&_R%TFh8ZD?h%gn*}o;-G-a=dZS zFa@fEoXH>e#ysU|TMFsF9)A_>XHFn2em`aFu{hqWzT08`bYPMrsTkIe?@E?JG-FF8 z?|mPZPRS~t{W3h8)QG?&8LPfxEH-~->ceC1PBrQI*xjJ>R`=z!3$6NhHR(Gu`j*BL zmmY3C#x@q^=@&09dim7d-^!)F>|!)XCw{-g`Jxp~ne9OL+k@F3aU9xrO?}wy{rO1* z59lBKnru%J$g=d|eT8kMfAB@y`S@owLu-d~WqcjzS$mzEH;VbFrix6hURz5nGU;p= zmG2pJ<(~Q>a0cIeGfz}p`3w=KR`!d*X@RN-N4}#JwXuG$@IOs|*Hi2WJ1&jW#7Sus za1y`iMCgGOhPRsJf?W7@eTM8k*o;I>1{W@x*$CMWGmTcdX%;UQ-rcznDVDBtIp3f> zKbl*M`u0ftMS~2P5R#ZnwmO%X9%<+Eg}*-bLPOp-81t>5c25awUk8c7X3M^L`GqP3 zkGlrfsAW&*x0#sxILYo3y?`z5d?S#8iBL!TZ_#pL(<~;cpl5f+t8GvYwI2 z7bJ6H^cQXuX4MIIW5gxHs^O%cXvCM%*?B}PHc@KoiE0)xedI5Bw&}~jrua{ML7OiM zSNE3%`klNNc-19C#i{K02nNX~?xG;q#iM8xO@7Sxwm5p0I*i_Y`!Z1P&GA!bDH|qd zjjQijT+quYB(LXKhYBmcN8-q^|G4tK!~{otujBf-WKI~|`^hwe?0IAaJB)=;uiW{X zmo!lCN|aM2=Iv5Cpzsj+@~?`*zB zZKIc^(4*AoL>s)qTeSeYAmjGk`>dibd=nWS318}b(G;%I$(RB4_aOO;){m*AJu&~F z3_mG9y5D8>-@rzpJ{2b@R-8VS`Cnx75I*q_K#K6c{!G*%G=7w9ApRGL6tYTA zTJ(fiEB5p*2*%^|Zmy~LKUYS-aN@u2^zdJ#wG1$6Xme8k-z)RtXYiIu zzt?yrN1UVmMh9Ox{B3$LG4)vkVs{VQb6k2Dm)XwfyEE+~)vxnXFXOo_eu+LoOyzNf z_LK7%Dek11*(tn+YWj=s<{0Qvjf}~)2R{w&4433Y=B~R+cQUJCdN=TbL;kZUvG6n2)2siv|2aUFM)g?Uf7i)_41UJc=<}cZOT!L0MM=l< z?;sH{nkxHV{onf=;a?Z_#BFN&&ma*<8;igFpZlMJr{yv6Nc?w@Vg#}&PJ4jHW_IV& z5}M!Th(41mp;EDz?7$K8q%p=t0KES4-Uy^C9*{hFP zQ#~3DV*DZv4(e>+AacxgCJ9{K^xByWiSW`MFVr>F1T!v-q5W1mQcO{dp`JmPI78C`ifRUv&~k7mC=;-Hzdt)7JD~JRE&`Am`y#!hUPBQ!l=*tKhco;27k;@Ct49EcIw0J? ztYzv9u&QT_j@{TtdxGm-OL}Wqa&PsP#1NqPq#Ab<{KLIp{7x_ zFFo(q1w`{~hF-+usreDJakJ5-;!YpSE>dg$+TC32Xaby+0oncCy|uBy0J7<~K_2t`ajnHaT@T~x02n@SBfK>B$%OO=R6 zRG$lVEuFlm9M2WQs!rEXcrpUT7>!e%gU}|v2YVlZh3J3COg7RW${Jn|@stYyoOUCD zDIstYFE{F0xym!@2m_Ajk9I;fqI@-Tj~TTez^RvMlcb?V-x+@Whdgvb~u9P0`Y+bKGqbN4x6mI(dnA zFPPuXfJ*=F-j6%wT)bnXy<2l>50st@*?$rC2H=z_7eR%mTVD2r_DVaT)VJ=Z9|t+_ zkNfVcrp;a@H?#+n<{mT;xV|K-X3i*>ik# z8aD46F(`-FA|(#rQ;9ZzB3yOzxT`T{9x2YU^KSYM;6E}CO2RW78Px=B-B&Pg zZ5PkFYY;8JaJ;uN%)GVOPyTBSH|y9*5oOA zgQixjt(@?G%uR-&a_4Kk;%jJgrghq!nHYW$~Rzs2a4C4 zvyUfbiI~k3iKp})B>lMW|KSmEb8lV&cT_G+J;?CQ`27O_D+O{~pB#u>E@Y2x9#w01 zSJS`EmR;x5rBp$wrA_DqWj85(ceL75;Drjr6KA(kh&*51E*g|+?YBnK>+U1f-2Phi z<`aWjh&j5gBc-<1G>(uVMSpe*^iZ}I7TNT3xK+zX(zUDuK7V80_gYop$#L5YgLV(K z3-r3J=Rdv7KRnn!N!$>6`ofn3*j|()Vn80Ie}1c1B%S#NjtM6>clTBksl@wR<+pno z@K@OaoyDrD`<3W7LPEA*Rr*iv+Q>YIl#%m90$YD;8Q@849Osa>lkKD8#~VGoh96 z;8flCu;5h)G>y#>WGbMA>e%=qw^9qWP7*b-7MhIihZg&r-VGkYGM@}idcfN}gIPt( z+}8!^-ca=l`{Ui>=fbZ}kg2?oDO^*r+s%yLHIJIjtr%TrQM#XL7W zj5(s`v6D@~^9|=Td#<{6%pcV`J*#t_>TrHSBZAuImlXu za@$18%m_(5hTa}yMCK}IP7xBNm z@Zp@XmW=RBM}ls##OYsQFZAqR)^2OaC(bR4+@e32L8V&W^v1(K?{|-8>@;f7i>mfI zBZj`I1m7=s;icw^<(C1EFijsZ$~8R$LjUQ=t9#kcgvrCjRyMHZ5jP8`l}qEK2Isn- z4=hqdeqck=Ii)7Om`7nao%qeJPkx*OeOUGLw$CP-0=Fc*&wuOLc}*l0%plU!-@->t zGX4X7Ie`7Ef8wF!$-BgTF_IV*`nf*?XQxhM&EuEcp)AdFrt>X!*xbgF5Dfd8k4we< zD(N7?MHhld{Kbjg-TIpnyj%{-kJTkf`zDX*&HTiOQ{_6f3`=fDBh1C4J1eyQZ1rS# z!+E~Bt;G_q!;)j`kBLL&rM<~y@3Vbv-z<;Vc^4t%Fj98WC>A4HdQxaikuFppPvbPr zlb51-_)i9j^AtBXdnBsd*j~RiyD9k9sPP8Q&g+#9-etdfrM!nz3n`5f8&`ey+JFrB zWmMQ@m`YcP{Yl)K96!Q@#>Wk17JDHVPWjTz7b_skOUoPH6FS5jv9~o~tQ+!U14q$w z?Rm%7k)8C1DgT#*TxxfW6nJx1eRxzgjCj2#J$Gjk z`puFPqeFTw`Z^MmmNn;H`~J4`w(t`i+aKc9tL{gSF@}X~%)(|xN}aB<^xG>Vy0jsQ zxVFu)o`lbv59_2uQDO$@wwoDV=EiStUOJ;Tx#Ep!J$C0pt=+@}@v}TF33lENyqX#h zSkTLA7jRWE9Aav>iGnX+S16=W0^C^O4*gs4$D6UwRTEDvUG&pF##aX-g2BNmZM8mH z#pC*v*VRSO@TSN%W z*~q}{6QnryJFSsHJd@5J!bO{RC(CJyb}+suYTPN|Cp=Vz8I{{^K4RnRFZL!w=usV{ zFKBo2CTRt-Fxg7+TnnBA+tv_2k ztrr=%Q*_Oaj2DDC@ZErg@@n;=<~o%T@xG2%i|E7-*QBREJ!Oj!SJa$b;bM<>vNDF` zjc!ooeKe6No6wgoB4tkt>a)vbK_t{dDqXWQqORHYqN%Pd{wCYTg_|xcrER%qtX`PC z*CbV595yc#s&>K-9-lrL>sI^7bM7FP@_s{wkd2sHLqmh%AEDDM0;k$W@AV01RV$f1 zOdiyevrXTgIOA!MPg6`Ew9Wnju1rxlItMK`E+q;FZe>n*N_Uc5Y0C&E=~AehoUY~y z10Z@$!}=ZjA0!HR37c@JaYu!>G52TU+lWU}iTdZlX2|ghE*0RP!zIjeingx!^VKI9 zzrjz|X+AnNG#U&#jlBq3pINx30W3dRrPF!xH|(uf1hAg`g*xxl->}>76)+znm0o%x|G;k6(=VLv-f%4B z{s&P%VF4cC|IUC+_oK(5YP&*3#3>K?$= zjQ8}f(3&wq?A>RAk7nn06p8@QwZoy#gai_O!*T(b@Rj|lDVEfu^f%Zc7O zPf#KNLJc>Rn-k;<#TE{~g{n`3^jfPw|p^VrtPp zOhWRjohI=LdlXFZs&hEe+kSqsM!7TH?}L){crtvp%vsd50$U|11xU z3M@}UN>lDX4glfDusp?k4@(*TrY@X#VR=s8BYE?W&_2Nh%hSC``Zf5UkoS4Z@SO4vv@EKi?58;M{{`bbdg}RZ_UrreEm^y+nSjZ09~83Hzn{AhLNL?#|6X zQCq5d?jq|!#Ys@MxnOIZ3+)Do(gqAbCNRpME8T4MAs}jj&4S}Uy{{93AAjBVz_&-=eC8@ zf9J=3tSW!At`msq%s0a z)@b{1<0KI!MkjCL8c-RzQ~p2DY|k1%K+ZI*xSPx6RO!9@PJ_o$Eb0Xe_jBEY2N&LY zHhnAC*{%lz!38yQ^z*#Xuf0vmj(y+frNf1sjDQ{kq0+;2%%` zMb&b$F-T6$N(U5iP3b#3*hw4VQvr@3PPe~#=#BYxxK;4rBk06WaUO1;v}!#zCm%t@ zoFeMR-%ey6Y8$IAgBdb4U_#d0lTi)5R-@-_r+l4L4x?8+2zg z-d=wh0Q!K;qcC8S43!eN-v)a-kr+Ki(5yjx%Q}z^_q)jPX1;%nU2Gs)>juosq|bj; za%K4E!S;9ua2O^}&$P8XcdX5ckr7O%2g-&O!BBLi09&(N)Wx@+ssNvL(V&Zvt`d|w zg-2w}PbmXF!LSmj$PPfwj$>7BHIiymNfy?aF>gI)R1N#3D(+kxiJ0p|$o(Dj!*84u zSMGdfDqpTY9zeED7=c7RvlKA&kd5N*Zp>)Pu+A>?Sy0$fOZZk(Z8Uy5zfn%EQn?&( z{8Vn&!;_@#Mdq!jCbi+!ZxyeCm$BEo(>3Y`##4A2(~-BO`!MfJ&~gw0<2L7d zsCkGcO?Bu{&A^M!$l*2J<6U8vE-VwG9kfqirEAOP41Ul1D`b$4NOOT1-?hdyb@tvZ zmgZQY&@(+AH29AY60T%I>L!u$0)JuT4s&wr(x;a)qF1(7*{NJpX1h~3HD9ZoCfc5) z3dQfDwwL3I8xrKaLO1(^U`WoJ!t4WY^}-z}3}YcCaF}NH?79^ArCGf#XsSxBux8fk zT6nLDq=8;Fzee_9uff+oSjKt-e)nZPfBeQKIzDytmT--ARmvyQonL^`?Jqe(D(!Dr z9J#^`fD%vHq(1!w)~jEnN{m62SFH5jnvr4xcM27EGa2DvFoPuBTw!W#Rs(s!Nl`KN zVP3&tN(PT2%nLNp0Wd$=49d-miNsP1l+T(vWl9K|Tf~ zpAKTHJTI+9Fg(QhHr=oR6T@SUb-dT3%uF52t*h$Vne3lf}QwUp>ZF#*b} zUQ>@)zjR(j1zdoo|H(&wdc_Y1ru4NX7b)uej)~5=`1ZF~va)MqHF5L)a|A|}U;&ck zx9-j}YesvJB9D##M81OH5|K+fv0ga=9g7N~B+k_*y8@_1eeNA;F@y6TV$YnRgQ79? z5}q?RZ8Dkd#qHvn#IF%aH#~$}E2W4`l5lH6t+PE%SZIRVup0vt{{AKb=af$g`dqBm z#n%a$`1EuBRU`)~+3VfbXANG#Geo>k!!n6W7ABQyNJ%GwZL@0@TNgOBRnP1NF8JmFX}y85bhh5YZ+AP+@e_EIR;e5eEwmFYNqobjSv zOc6s!a9!}4Y1TVMxDYPXfrOOjc`*Ki7vRfYKHXu$)au?iXG}<{4tC@OhNUXzweBP+*y*fD6E0uQ`J4kM~@%Jf}X?F7{aWmBw%v{bxbWZ^8Q+e;l0(nfFv+9lScPzSpYssM+P<`+XuNQr_ zQ_0~03t{!{?DKc73pYgEmlOR>#%q1rn_QrLo7RIe*y4Lz^k=Y$h=6Ud#CAZ4vQm)*K2U0l4rRv^V{Qa#(#jkn#ncxuN_t|rGqe;yLk6s%m8*O67hcgp39hjVnL~AE{H4rE zsVm+y72w{>fw*ANovaV2n|-=9@*K#&?puO{Gry(pW`G+I#eV78>_N)U^x9s%@!rYQ zbGfMXJHnE@*+uJ%xXeC<$>em5tDT2RA|C27joPC_Fa4VPrCfT2l2AvnL10e+yGwQ= zTj(=DT{)ZYSHT|%#pZcLU0nWFp;`NDLs^Q-(Q@LM%Uri2R79Ded)at_)$uc!IT1DP zX`BD`07{ukC{5cyLAU5lzgFf+C+e!uUksr=_o_mqSPAbB9raOnB>YB6a={i3n;DCY zE|a50xbTYo1nQVJ1*Yi)&m0ycG?fXs3#!Sf}Els7@DChpHnMYMC~~P$Ek$ zs}GLZ?>p<$@tzA0(YBWDPc>+wtRVwr0`TE?nN>-VO*xg33Oa=5O5>$ytWq>%5Cz+H zNsQQK$1%H-=hVpQo_H)tXiVUdiegN`saD0G<<`oM>ZKqGY4F+vU6ZP{lbW*Zoj~Y$ zd+-RxkCZ!hG(0@nkH!RlFBt`;lz3>t2cO0MV=QX|Y@Wqi9oyurYS|~NcfUiCR^YB7 z-)!HY`4kl93OFTNSA6;pU~z2{dy`p%X(u0jK6_gnv9+EDI;8g7`r7o#bwWrv|HPRFt}De250CQ@3?J_qdae5W={%y? z!k}6e7B8yfD>U;b5pM|^+g?2vF@k<;a;jk)D0cj*Z^#*pP_=vxC13!6XZAM9q?BTO zNkfyJ#gD9MTBJ&1T7vtreqo=^LWaeapJUq*)Uy=Lb~tQXpa}Z@mvzBApR-?zc_g7j zN`W;%jJEcvh$yw<&q}f-Tip*1;7kApSC`I55TIQng$sa3I8m*2O>n~m8SGy^ks*9m zoJgW6JuzaVbDh2q+Z3M*y;XT8sOb5E7Zox|-I?C^HBA%F*} zEhaGa-2lBO=T=o25uMl|f6eZTa(>?BkdvYTJsRa%X}&i-lQ^_)F!#R!#tRTAh zOpr7eh1Ji7>KTF_vW-B=<|xBh7S-SWZOeR3E_-u#hUzGEERwOv3NN7NHqM|;@3|2r zgC?E$jQEV=IyK*?i+1S`@nt)#QG7;v>9|EUCD_xU@%!xFDE+%Pz^}ueLm=>g4C5b) z@uR#%RPY%=xAjke!=R;2MT&*zz;>5n@WXij{!sQ9#GBjm z*UJ1+G%7ghh$2gp&+v!rID`Yek#b88$OxF>c-^B)9*kg~LQ0r^$#y{bx8h#uP9cz)oEArGIyvX@U}uLjB1C zp-v!t{0Qcq`Lv#uGG39A-xRx!SLzCj{~-9LD5wuQbz&qH2(F6-QCvO+H^*my;h`I? zG?>uJD7I9yobn+ZvQS!NO8N%u*a-rY{m!a0@FX5b%wCdYurrhx3p?T76Lly|s*v5F zBn=W3(Py9*NKdC8!Y6YQ!VLw_84UYBTL#%~NFl-mUy}|<3i^xv`v2}blAqCkmpXU& zt@Gj0*I}+d$TS{<|DD7{F)N?de`ldsolY%y8{@P8Q@i!~C0iC}XXp3=#%p1r1)FV5 zV7GA?$Wnc-1IB*_sDhs8x--#a8`V@U50`S1<+h8)NNRz6~C|b z?6M;_xJsTPF@4LnGM)jDHDG7qt0_CxQk!5x^7ogj(=6SNW>^9-;)@VhN)>4TN}Dfg|L zQ4&OqQ=qu~jL&rTo@tLW+LS#|OBKthz6nsODx8u}8gnpzn=bUNi9^vP!UqcBk+EVO+L=P>my@ z6-_Kw!j|Pfyd~nk)py+X>izLmTA4)r7Wr^0us}rsZmZXDOdjOSBZr7@ zP9W&;O|Pq#`igs=Bua{sVfGH3B#znz@J+Q}aRm#NewkfTo^hQ~>2Wh+nTJ4o-@u(h zJfBeB=eoFiIt-U6J_?8CHYKF4aAFAVFC>=n6YDk=XB|v1_GS0_xvVjg`e1p{1lb;9 z9oT!uk6B4Zc9p{yk`Y@YyLiKWBnYB#o3Ve02872}f9aC15Z~@J@_LBD#1FFvTmYrpb(3srZTVQ6lYrSZNV_!d9U?unK+7Ql~!; zmB(=FGhO_%<@{BmgK_4%5^wJQ5|>;5&cpuNh0I?#fo{V6##3FSBox0>jHf$_aU-aLRh}cQ!D@)9WOc35~Y8 zDQ5bX3tGgIpwfu-3URyOkI%qsKP*^2{-DW;$oR+-hx8L-SKHV*@@ia-5B6xRfx>_l zF-5Db(td^A6j%>CrAN3Pf8mXjZXZXB;s8OBTEopWrMZvy$nb@bVW~dd{iDmER3uM^ zp5xTvny~f61lzXt&IDW$3PPguWW$k{E_)MG~eCG-p)g=*2~zTQ4H z!mRlPG%)<#N^u>$)+rC>LQq2v4c zbJC3sA?5PL4>WPJx>7x#ym1yiO|Sh%m*#lJkV7)E?1^;E-WLiV|fR1s5; z8rc~gWOQQw5~dhKaGLu7_h|>cQWet>zSiy2_Utcrw`4O~(pyt7jFm7ewBoqF#_!35 zS>wX%WlIJ8zeHzY|Mxx8@nxTTOx;DDcH1v{_vd+-ChYJKs~!MBMd*qc<+zzHOMpZ? zc%W+_U(C33$GF2~a?15||GKo3us`I4Q(wQfL*G&LGxvqh*Ty`iBRpC!EWm;?ZJk|G z`aSECcqFU{(b3TH{92`xS$J%D>WuSW*MP~vLcQ}HH?ZojaUjqQB2(;TelfMbe$IqO z`@?Da2uKaDlI1g*rtr0!kwvwCet>&gA(kbAP)Dww-w5nOaZu8=E%v`T)*kZ6haA%N zec7JG%~9#ogsczP_=tbYd6Kf^J3yU%2kgHXl!Ha9LG+9P;4bf4m8`Q1)_u{pN>T-+ zL7JbU`DOkLH0yL5NMZ=ctU*vdID?~?1`0ZWg|-AzBIbfT$->vS&yluE7WGsvARXdh zPL!aX==cJd?0CQod}*m_tyZ&<6DZpod3t>QY< zz|IWqs5ipEIaJ>#j}k#0}m;Q`DY3!YE_JN5_(lG}x?_4h9W z*L$|h;C6i0ttt^yhZIl8;n^%UDj-`v+7)hnIeT1Bd!%QDB)l zSS6JxKuj>MbYHRQ=Fcr-PfpLhGDP`j1vv5E8kEOFRg`_sEP28WkzanJ+p!w2do!MU zV5>rssS-L9;U9GJ*pQ-b0ihcFw{yXF9N>c1Hz>2yNX-HPhS1l@qTUa8(eOXn{futV znbJxBJ;3*t&7r5;sRJ7|=pE<6+db{(E7R>9EzROqL7qJ|O!BaN0m=St?1v80Bc~;@ z=EbBZDN+vFb^gq4BECsBv5YxnYb|G0?4F6jM>{atXjP&45YaE~#(oG1Etz2mgGBKt zSQBHQ5*6{lzM#3{u3~RaUw|Zm=nL{0I&W_x9m*YW!s5yaWW1p$(jFR25lAo}`|4K8 zN#0}yjL~IdmcOci`Iib+V&pe;UC#FwcEvTldU#GiRHzzemIbk&+@RkXC#L-_?T$7?WVLUHI&+>S9qRGom&8izm-eTV(2Bnxt@B60EPduN&c zu0=y&W;(B(AF0eo%Z;@o{W;UbU8Xzm>D?Hvp$7*n;8VLH6W^wH(tTv+5ssJRisUV~ zn(#2W$rIvg8iP4@WPeUzBS8rW1wU7-Nrd11?FATp6VS}SHIWI*T#4j7=Jh<3cKps$ zvK8R-#9XH-Qi2T<3r2rVN2F`+_<&Rf=I}Pu3hzKKJd+&Z9euW8o-)0uY*fVfZ6PQ5 zx@u^zwAi`DO6XOSK}#!`&EJJw%^~Rxj@aB3wjOSEV>eZ>$D4(0C8zfTDk`-e;{H8n z{;1P_)&w#(XmYeLpnzxyB01-R$DJII8sO^!&@=JB+v6^G8p26W$pe?`-MQ|Iu(8hm znplB~EfY2$hcNX%noBs?#KvwUWwRfril0vg?LU{zX|=)d`VGKZ4&~4?#M?R)5IS@d zI)?dN?<8auR8Glz@Bdi10CoC?_ExAk{oLn=&rXj7X1o<{$%0N%3xxSw|AoO4PxO85B1>yKrzVQuF;PP5d7t0fPOU5gs_dL(lZj z&H$Fd>GVWi<-hh9CbSt|Xiqxx6C821N7Jou{>m3nt%gEGCvp>d{&z%YiV_Cs`Ty=a zhByMwzdU|}>?pru_!HYbAR!RK5MI9MPr-`;iv1d8{>>;R z*Pze~BaHlWLXjH554ik4ZB_q#O9!SWFB?nwdxyeTfWrrT_wN6r3;B`aK^+_qZ8E#7 zRcqgFKo77wE%f|XCYFzqIfnqQH$#XByMdx9cXJHNGel*g!{TV(%Qb=~S5?1iiozJpg3yce@)sc1vSEzBIvEu@`?h}BI zM4UEEU}AJ1+63}SG8BM(5We7J27f&!oL&n1R5Fj}+{U$BNZ5rg9R&#dPfvnL!n5p_ z_r`Zc_M0Wy22{&Ub^n2zZV2B|}%6By) zj(SV*OxVO5Ay5%%a$sD9f<4RW?yDABGJGCnRW_^_n9y;kbPd6muQ1O5?=hoAfQ#TY z0xmxadHcX82gk4PER~0IL~DigQm6s!;Xy3E-q1>g z2rKM2GD6GkJdlt3{j0D+NP>-@)=Vk1s}8g=mHU9 zx%4q_#%p3_!g{5rY%iN5uafYUcy3zNEd5LsvcC!K0b$NukkH6fo08?4-~|w;)%@tP z@Yf`D(oo6LnBi-&n!?3&$k|{963*z5D+QGvfMXSG`jbYSNuxK#4h4~e7-+zu`bmT} zJi_0378YrUt#SZU>@^IOZHrlU-kH;$Wd9lq(brJJ<&He?+LPif;f42UY&JsAV^Ao_ zuPnfWDg-1aF%zId01*c?MOQ>gs*u;`@pFAi1h+@xY<&v|F|rb{IP+`fWl<<;=@-yH zO`*=f&?OBI-U$75?8!g$g$QKNlFSeEgW$59V9gJsHlKmie`h0NIv2dhwa9MP+NgUX0(bJX@cA96fgU& zOMeT6|G9YLA7&BV$(3{t373HgqcHfX*aV?k(CWX*;m{Khq20}|kblN#xntS}&CA+h zDj-b3FsNl|^)cpJFeyDZ5F&9fk;*{X&3t`pfY)&BwC7Cw8MmZc&!LqhJ-;XZ;&Y4y z{`IE=|IunOR^po_=96!YDv6OMXCyI5_)|<4NPizNLEe?oH2po^ZoKvu68V2SwrpzQ zEi>!D?Mb3l5_zSFC*ydG$Gny7)2`f8gLY0fggAtk0NnKbwJz@ojYV-gP)3Jr+&w`0 z9FXR7f&$hw?;DUTSdk7kL(n2`xzRL*qV*8&6sV+jP+b|ezqv$;?JpP0i5$L;&{gd7 z2MwO>tO=f}3nVszKy>?8s{xi!X8cT0^L~q!&6HwyvBMW{Rcc|hpnvq$aL7whdx-Pz zEaBPErjsQK@9UaAhFHQuYd#Ac2-(oMM>bHam!m6~4rZ_cOv>>pp3cA6iBr#cD8esizq>b{ESll$AN7h$ypgYhOk12F#g-awz=qTyd!C``BV|Ps3oH&rO(q&enJ(`yE}W4500e*pjU3Qc9K@JjV(7nBDc}|pjk6tt-QU)!-R;8$5Mg!XpndCrtrSN}ca^ zNKnE3g=*=8C;Q&8c2BnRf}fr008J!R9I>+$_LSZR&f5mh1%Jg7-ck~sEaK7*8*=(g zcp{@mKquaD={NA+n;XzuYtS3mi8ipLi)kzZE`cz34C!%1b|)yc*(>KF7}R!vlx~ zNkK$(k#v-Mc3YZUGedy^kbX40I_DO&ZZJGJ*h(N16b_zPMtW?CXP9w9Fjy+h9ad-+ zEO#6~i_)$`oRLp*z+7NMX$P-n;=rLUQUkb&$ zi6v*0g)n|}W2~VXsDTSO^uhJsL2k12rj^*h4i{xo9>V;Pl2 z3109!za-Sq6e#JaQrF6ttekBsGzdV`67Y?Z3I>OkE(z?U?CM0_Qq?Auxd2i=BWYOa zwr6d=jLKIq8h=?952Z#qpQJLeA9!t@>M6&eo1bSz%IHCxda3!Ispm;Oh}vLmy~=O% zg{?f-?7hGju5TRZK-#b@jHJa`NU*0>G2Jugf1U2_BJiEz)21&@fu`x)V0KD;4;D z^p>2<5`dfh)5kCIlRfp9Uo(@2yk9!|V&zj};2RZiApN5xo0uuo2~e#aj4=p7>Q2oG zPQoY@lwV}H+K;njEgF;+foq&k(|r`${w@X?M^qr{*WTMo@VlxYZZLKb$jEd)BLU@c$D>noQ&uk7Vw4s90q%+(^ zqp{nqvy{Gbwn?nw#S3GKYn_(zZIk%c6OF$m0?MA=?c|?}`Ezl|Rg_~(=~n=Ihhx5& zoZuP<9+Y%!AciF|24QsjA1fWI<)yO+x-7IlO8jAC+(|IWV-QN*9RXj<{KgDl;;fIKt>NOqS zGjt`~DxPDF|7kf0TMwsPK>)p=fvD7{+;?kmOWmok+25);&tEr7Aa0N}8?3`b5+`H) z9MwiAM6`n*H3hFHYuj;xjb9vxQ^!#z|_4I0|0bD}h+FH7Kk=ge6 zW?|vWL%Ls}C>X(;G2e+59eg%3xp`(d!p8vb3Gslus!ID@#%Rx=6Ik^XRydps8n^1^ zeSWi9?x`3y*F{tH37w+JK3TUKU|CHMhxWa^X87y-2jVQ?G!O@4zSA#{0bQLFBp`|K zM8#zi$qpw45Az3yw(<5~hR@8wwg*`UE)e-L$%o=)kvdI%+dXUUY}Hu+`M%b2*b4TD zYwi-Ud3yDsSBDnt7NXvxfwe0%SWJzjdg)tmqwde5UG`W$)%vf-1ust*3SQDpmH?fFY!pO|ABVyN%j%oJ}zeScC65P8OOFVZKp&JO%9On?F4Tm@X zm6qY3O$*~0TWL)=mN8f>Ugy0I>JX{@EZemkJ8Z|%rh9RPzA#))8iIIhxo*a?Rte>w zsMkYteiUfFZ-2B8oSYH7O6KMXLhk|w46cmM(-YhOu(EVGT_^Ue@Sp8ElYHXw$-$2U zPR-fzvPj*s0o6Z7+0!Wi8rC=P!4EgLLr)-6C+|@C2;~)`m+Tc;WW(8$W8a>fndn&} zI)Gh~-tbkm!y8p?<%H|uUeyVUU|{WOCly^rZ8_l#=!Rp#QeJyZIYsqSy|87wWOhEveI8W;W>VvZDC~zN-_K*Y-CdH37JJj&sa}s~ zG#h=~0z^^Aw#(prJOO7etyVzUH8BIQu%2K^$(xzUZ*o>X(PQAwXXGP0`c4RC&S=m- zC>)sliY|dnJGlJs=|5J_o|C-ciiGj`U8dEhN*&v?cdA$o;s_`p+r&*SaTYySp211< zqtm8Px_={LnBmedtUc(Uto^vQS2S*cv+P|w!8PE_7f@U#ur#|6Tr4~~XBV*iS~#*C zqF1Z8YKzM#L$eEJ-vH#2p_{zgGwA*0ia@D`a-kRDb~S1oNvq{BgT^Isk2KNZbLsG-11dZj^xd_db)r$_|3(ZlHzTE!HsUzwcZw!26ExZh9qT?jCxsSRijP!h=sjf5>0FyzR_}O_=?u-1=^R%z-|zB7<2Ij2{`t$hO9m^F$ED=-%jE-$+62ISly*D^&*!*i%(C? z)-FcL?s#=?Px16zmu9_}Ymkp-gMF9b0Y1`WN?q$#O{vq zT~GL#G>IH39B55E%Uz_X+1n+!@%iLY$c0ps^USgj{2#`V5 zksRMr68o`>nX)Hvo;Hnr);KwVeXB~n31HSIz&{Pv8qm3dP$9xQ`BVEmfdK#sot=hVV9w71n>8xz`Cr7gg|HbiO^ zEyxMW0>1@|(ru!F+S#5H$~bE}Frp>FVT}$;G-^Z3zPDMj<4?8AZA5GrPKPPW$ZCg0 zTo}xAPO`m5_ciN9_NvK;A@zq{PTFgy5Dy@Sa{sTV1wH(7)?rOVd}gNdLE%L!EsH5G zh~uupnx%q?;l<7_o=fv=JC(})M2Z;i?kQQA;JnJUY}dNWbD7~ek&bqY)(9>fSO=3q zn+VJcXLVP!8$bE|Aj1l!>o>Nz0gL>x=kRHcVis*RVU%oHMNOiKe15MP*xr^B7Z$p@ za-1toO)XV}LmjQada5C+iB4G{*?E*Sha1a&1Hj;zihF@>LeuCE{x$pDqgOwe5S9;w zX&`-V1{lXodQ}@}^7;ovm~uE93Q--Wn|y5T-E7x-|2EQ(O=25|60bQ8cpXPR1Z4TwGAP$W{J4ecD(duP> z91%gqXFx1my{Ssm z<88_61n_hVMo^tnNTof!5EQ=9S6PI+OOH#U(jMc85YIHw(g4K$9=Ak zt}cA|5^~kOcS;7#I2aTw89y>6e2&m84lu9F_md)2-r6NvnX&vqY9Bqntx8(CDvpCE zLr9Y_@7cwZy@^f9J=NZa_&QfdCkc({`BPoS%XGIIYxLfCZK#?sq057xwDjWBAD$^$ zC{I+KOY_$kHtXcMcT=@|H}qo0gwq5j%tgFGn%yo6t%KA8zYo?Lts%nZE5m{lhltWD z>r=p(L++)!O5OC@EQ^Bl%}kO77SohR3fKqE2{t%>1udBqi>HlaZb1|e>Nt=sM%#W8%qGW5*EDL~rg^TyYhz*I&g3QLVTZ*&?nrUC zrmXfQp0n-`D+Le%H-mvPdg&{#ZL@j6>X$h_y@66>e|w+L8Hbk&qRZz+93k#>w2YD& zZm}zF_(m**0%#vgg?*cMV&}oXg-VDD9Sd8;-&{&8}~13Mg&D+lamIKjO!yn3sg(Db~S`2Hzjb zhh{b7|3uDoJF5X<$EO0wRHQo_VJoKsi))k8Q)HQkt^-MEPmLuc{pkOZm|2kh`Gol9 z%h9O4s?l3_L;<6EZn)@3yE*|NPkei8jGmGhN%g;?oh}LTd<>q&8u$}gc|s^MJvkPI zu)hgF=WT@AeAu(QeFrLg2|1F-^jPz4gw$j37_Y_e=$nfl9R45!>@gat2y>qznr?1v z%_U2Tk#-vN!QEg6!mN5FX~KVb8WXJcdU=LtI-Sc*;>)3azjI3NGjPYd+M$r8+*gPr z<06q2Hk^(elk{T{uP^dGy*4_iQ*jy!m|d41EbOu?4P;C&L|wb%vKtJUi4vy2{n>6| z;lAdaoWfnO(lReQ4;_xVw)bwY;j=4k3vC!LlUfJ$U-$iH+chJFFg4{8MZUr1UQP78 zfbnCHk)^4D87qri{vz@N>6;XGfqvATeV+b7p8vQ2<0vd0fhZG^VfwLw+Bp)arvUG! zVm

#p8!y=e%VSEz5A1!{}AOuQ0US)*&9cEQaDFTK4dXv*Ss2JHEZmqF*js$nh!g z7(PsTXuHQ67S_@q_Z*)uCN6R;d{j;IJ4VvHGM16B>XM=jEF?5eHM4mdr{asBpRXTD z-^GxJxM7R1ouXq#q|^xLGIbCUq(55SXt?xM33H$-@wfl^{X<6I_pCysXP#LpuhxhR zo^%;aF?uNv;)&ky(2Ff<{K}6w{cymn$bDnuNKiS@#<&?6elVBqC{bbw(29dI_9F@R zU|=cxcNhVVYnu2gxNRKb52_ym(j7}$7px`TQ^QsHCkM9oxDa?B7J<#xOKcVI+W9hU ze8-_dm?jN-7W^L4IjFt;pNggR^8T%+@utFqo(7y{VaOFsuU<>q{|-IOTka!)rPCoI zkt^(-YD4E3coMQ|@J;GGd{0n*! zxwy=q?<3)U7VsR^;NT)nPdpFb`Wx2=(GRHE83by29O-%2^d?LAeEi;;ncvg=M=d!{ zQ(5koaBcsL5*Z9)d1= z(}}O&GCvN~-p!C4*%;RdGE-*0RmlKfW!zfl%Uq|4>6hEauJY)E+hV^aE;e2EV2xJi zJ{{5w3ps2rj>EQ8i$>dp!De@!{aNJxI{Z@OrL8uoO&``;SD2tcH+@vtZA#ZMWV6vq=&r;om~9#2cM?*N|YfX zDc%r#TYK`?;9&FJ8_QG7Mwy<^0bNwCaWH)7Kdw#@*kEch!2kAnD2~KGvQ=R}sTIDu z8uE?3d*5_1`i;P37kXVu$zN>W^bmIKfTkXP#u(fgYjfsn&%)d@7AueEIz9QYigh^f z^f{PvOBDAo#Jyjixs*|dfp+EhYQh}Nq;BQ5ue~nvHVvo5M-!0CmHn{=F7z(6_9qk% z`|AW#-FWN&_$(au@|r+XR|oMgh!>omy%MMBsa_9KU_{730g!3FWtoprh>XqV0Wjb z+_-do-4VSP;rg9C^1CYL9#N9YjB*%&H$b48wNjVLxzAdRz7IIW_%CbWp$$m^p*USt zKh*t}YiR0@u$aF#zkao%LWmIO@vfFZQPknl?14^%J_3Ei;*MWhrhCgbq&qH&PNs83EiZkbbD&k2@2Mq`!nf$? zJPl=p{Y%9enL`+)m<)Jy{JAJ%Wn8~~iVEINZ74IU>pwhr*tBkI-QD#cc3GvEJhj^3 zUC?J~oZaU15h7N|?|NOD<6ZrNqPVyFss2mC0y%rZQ}2*7b4~Bm5lYcqRr&4j{gbpo z+czkRo#V!zZ(?o%gy|IH=q`4;{Y||{W+0+jRcd& z)BjldMpEhEUV`3sXzDW<*wqrB2z7N^NHRoz{~EkPat0(sB&Sz^m%}@3y2hN{yNMXY zKx0oSMV>{Qw*bUAM|UDuC1xhn+7rHoNT1Tgf0CjP4X|{nZb5~;Av`jc28i;K*lCx+ zv0NDb=Ga(@#x8D)&p6bO+N>nDA%?_v2E_xBUU>BJrOs4eS+n1SU*%Xb=_$`qcfj}n zE8S6|Nk#T08FFB?AcHKOUH{sqX(QRAgwT(Xl8A23VyaoWt zqeTKIZ!VwdV^11FfP)K<^gBTLUDn{{T-ER?!dbuGZUfkV zkvQW>=vf;+k*wiW71WZJIkq1%bP?a2Av|b{1;KU89V7mnM+s*pYm|?F4wH9&yE4fp z&))VZKoctVwq53~CvU3(PIE3@>I@0+n@3PlO2%8oNxp7hR8%1G;03X#XqrH-d1J*# z?_ax;pd1dens!t7>Z?z)ko@#=UHiSvq?jAqFAVO_JvnBa&~kiKC7QSI?hOmKSN9L5 z{h4OuYfPaw1*PSI8ZQ0p4tUa=kzW=jF9GHo|0zWieCV&d!9Rd%F^}x2ty@Nfl54%) zK;KgF98FcNOy!!fcW`K8SbV1p3T+X@<0vG;8PY{)CdkTLAN_pnMb-A^^2?0d_>PuI zU1W>R)|t-iJ7SU%@g~_&l%%p2XZ-pO+)la7y*eQ-o%zb;TQ~kZP3I(N+E*Ce-bSDK zNLvb75G-2!`v!dd=cM<2ZC|`7`59bVaID0^cu7a${C%Z9>1N*6}@Z^bc^$>cIFmHJ}_FfpH0dc+h*```U|`HCJ$eu;mC(9Wn~(-F@dP9la#bewn5O|XFu6qM>@Ky>DS7_F5)u;qCsInqBDpB;5y_OEYH3mVX-0Sr zdZ~-w-fFjlme0t>c+sbq=R|I|@_MtiDd_0y2ajfcv5ej+IMa?1-HMFXj@15r%r#&_bpC;s2G- zhB&ELl*ligFQ2$DLxXY zcuzaMCQUV1k<#?%*2&$g<5ON8fjQFr-E-idE^FYhoHx@io}e61R(TelD)R4^e6?}g z#|*U1{xk0X!kyUJJ*9LjX#R59NN^Kj)mr;E*85W8Oc4`*w1o)F2Sw#T8dk~w`xnF} zHiTVkU}+hBsP$gW<|k;MW_%kS9=2Eixv88RFm$dsmjBT83!(Umk{A2QZq2SnWqcdN z1HxlUzAUg3y>ek*-1R2<-jz_%2p2NWdHiEfdmtMojE7^W3L<6hfkH~?rhx3Vp?;G^ zw~wQYnha=^xJ8IB_(C2V^tVbx@1jgsz)@VM!j2U(kHD5_bf~*zC~VIFguK$qEgsS^ zG7XI|PKbE25Jn|h-%i0CjTmI9jZ}-0=yDar&9G%E=IG~0z`N%PibffJbV9C>cm#w) zbru{@GAykDf{a8*DlO8z5qw$EQ-sf|qe#nVjDUTTH2@*I7BKv6FmlsPh-?uLH?!BZ z%u;Y*Zex^*|0vxU-~iFX7sh9?IzN;v~?`7_Jc z5T*B7vTh>vD}#t#dYHt+@Mybxq>oGVD>c{Gc)c&ULhVXy)F)tnGoX}5CM>%Bn*xXZ%S=;q5C z8+H`I=iRZpxV~ifi4*rmw&tfxv1FkR_7qovUmDL!8?&rZlz8z3amZ?LMyrrCNcp$G zzB;9|TQ^)%EV2Mw|MqI1%O-T$VjX%(jI8uY(t!HuM9I9C8)!^T*5F)UnR+U*9<9iC zAh+3-_n|GCkLDLYl!&8!O`9D{P6%4}lDS6&hXTSK15%#>Np*0lFo<1l#@S&Wi?5$& z;j1o#;sMHgC&il|I-UonskHmPPDZLl@JsI`wJ&_z{GV}ISM3R94{De@kGVuN9;g%> zaBoC55>v>FhHhx(r=yK?PlQSO zHLk$3cE9@WPA*qYKky@-1G8DBA9t@k;(Yqcbg>0~EiE=Gn3SeqfWjg-m?ovKztcck zjN>&WH=Ska+I>K{ZmHys;GFp`i4$hputQshr^{JbwtS=27HnO0LT{EOw+E zP-SQ>=lc~s{^?lJn?H+VO;;R?dMc4X~uHZS6alTo6a#>JCv{y4{Mh)u(!I3GgOTEkS?>DsTIB;n} z)_Vo#4^eX*nJGLaF9>h<&(C{J9IH#W%QL!FLvQ^eL1R{lktosTy6A%_QFL^5C#Gf5 zUZR{2js&UIv#0KFc`^Oqoy*xDbbrrv_g&(ULW+prZat9kwM6H5;oprV-ix5bCxlwg~>ze4X) zs-(+p)r+s>SVZO-IX>=ANC_rg6YmMX9G-`hKL=z9@s`M0P7q!$XcUGZlRPw6{8-xt^0G?LsobyzoIsguZ^Jj75n?Ocx%Vi^b z6c6KQCL=+l(eYQC>`fvzj=VytYSlX4Q(x1Vx;5q&-!E4pQYrKPsgMz3E}TSV+ag)w zMAQYIKl{JNurJ&QaI!G_od;dO!LIUiPE0wVxu4NZfW952g~TGZWs&f)BXhC-=ei<* zzVa1m%eCW*qu=AukzL>dPQWoRjL7^*g+O8!w!_r+Pa}Xq2s`sqi#7Ecwe+Sw^^(+TtNdrv zKii1uQBFRgpX3~0ZvwTP45kxoxdpUKj6^9gjG6Va=;)1D?Yj>GV4TbB(0aFQfS-v0 z&OS18WCXy@bExzh87v9G4M+L0D#4EdX{6a-yGWbG?K)4TC1u?(Sj-zjk3X0lT>qD{#j<5%lo3DcpsIK7gfHg_xF9If?6(ggMnofwW@HW}_-mn#lqW-i7N@Aoui|@=cvq%wlWC5>`yZQM zj?0w7FKbHknWS?R!|z1zzaXugSwtKZkJZoM;fuxwey{8|$fY+!4PQ7b7RI&b$%19y zX!d_c(jryHTx3O*J183RI+Mij(@(i*oQL$)ErY!dd-*C19<^snB5Vs{xjL~1vQ`k03UusA{>r^ z9Zi`jn>QK=sIO(;@&*TfNnd(wEMK0&wG|hQLl`X9@;~79EeUKoK~3iCm$WO=@4sSl z?xLRWv~wK4jX{M^cnTk>?m5w22>oPqg z+zhOA`#XSsZNH%Wx=NZ7Y1Fleu2r%kCdT15dBb87puBv?*H@g;0K_r8{o{8z!5q>b zocnon9j7%%+e19UiZ?cbut_#Hfrm(f&Q+hUc!LX9_P2N_K1()y?v{gU;v9TYf2#%G z^CWxHacFSgL{Z%LPwTJ*aUr@|FL^lH!1cwyrsEzXTrZb^=~7cej*`s3pQK5*M~D1+ zDRZR-%xk!AQ`i4X<29A>E_=0QOWf`MD+n}D5PU!4@D|;wKLMs#0+)J-&QN~-4gZU~ z|8BDd9}#6NEd8JH@iYl;&%(F;)ji`(y^j0HXB4aWvoQGoqXdMTUk~M-=W7o;ZhSaI z0)M5t_(#9E4X#JZ=;`ZAianNXIQ;(678KYr7i~^$RTP%J*pJFjY<*XV3^vBCYi*n{ zx(V9Fch#dg9QlGcS9S`XdM_d(#w-yuJb&q89R_rSiUvD0;nc$-;mYCsYxF>VTZL`S zCR6Xi2x+G%DuV{J@E%%)kBQLkU>I7SM=yGI%Gh`M+%A#Zj2-;XKzs)mz|ufBc(3vX z(ELRhk2|agy9f5a2^c(Gm3?_8Vv&343iU6AMba5jS`v(??!5+dC)iHyy=yk-5;V8o zl3WPx<<>V%<}dGp7y{_UmmBA{*4*;P8DM3(cbrw7-%`pb9wqgam}CAcVOL<;Ob+q( z&3P4)H^W_`hab~l+-X%U(_-q}>o(DFe@TQ(-`1x49v|D6(t`NRhE=y0oa zY)OB28F`ejeWR`}jF>N_4T}A7tIWz_^LHvty9T$qPe5(HOcthP#;vaGpw8c^FXaXH z-qnLc|J!V^m1OVvmP!5h@l@n|AW{6kyP!v}TwC)U&#PeyrFoH;2jPa2nh`xwIWYdB zklJ|MJkXRLgXGQ7%d4>9BOqS4GYDHbCu)uosWLJv<6|LrjIMCIgVbiFq1)c%bL#!k zV31sN5tv11r@m+qJ<{rw;y(Hn_3)`_l0!3vTguQs=lAa-;;|z01sJnd4|pkcZ??K6 zjQvuvFJ};czFSCt5(-5~2|b8WHH1FPa*laFt9!b_Ma{70OJKN@MRYi&p;b!OO1DJH z6ZNmSgoGYTnOP9`;cUoGs?p(eL_<0sGkOT(1I<)HlP%m=hz1kB2AM@P=Df?8*>}*9 zU;<=XYWnpoTpFdGO^uxQG>nCkloZzuH?@OvvxIm}m1zq{vvxicT5beS>9K#%}ZBI^SO|~JwB#zOx9tg5E z#>HzzO%Tj5(dT!&`OS^=j#2}d^-Ct1Fda5{zmv4#h|=eV$LFiQMrcnHscIC$_aY+) zeagprFa2lzk(>E1>UKcEZJl(P@9|Ekx2HhJCGgzXi0fFN6>)LsZ+j@0pQz*RvWHss zh!2F!$qQG}jlXoRW!QTRiI3-`%w)zy4x}Nq2hjeDBelNmv}$7@ugy641jCZ49pRS& ztIrK7Zw>IHlwPFKvq(xxnzJp($lNhI#wt&MC2WVO?pf^k@OS%EoH&7Q3M+wOv7NXHWVD zDql?Sa!-SoW&7cLft@MJ*Z1}&y5l1eK2zH9rAwH9{KQ4PevOQ52Lp?sYU2-T>C>BK zNN$oLz;c>gc%sx1o-fW#Zm_Xkmb? zo_2>#{=R?Fg^N4qv}7gEFB};MS(09q1uJfm#29&wGX#nmPjS{U+aI+&w2}wffUInW zV-Tf%L&u4s4y|WFy23ufVK!as89g{RP85}fSfYty^}Dn#8>r3*D2a>2&%mnO0}9#3 zQ@@~eA$C(*KVrE%*7MU&Ki|!8|AOiu*oedvs_(QKY7GA=;`HE2yc|@oj;l+y2yHm9 z;xoSLC?%tgb$fX`6q!YP0Y=^1^!Z%b6$Ta|zMbP-v(3+DZolfh4%Wb`lJFyJL-Jal zUZ7Q*Q@t4y$e8UK1C*uHW$l0y&_*?c9QXvr53Da_?&m6T+_n8PWR`I!PbO%1% zBpJp~0P zP?5Ko9x7dZ)92>w8K@qdD(S{571zbJdO!OzZMe^8RKKF4In0t_1>b0%9O^`NZG%cY z&|G3*Yh0FT2Rde1M6%-_uro^g0ptGkAINq+YIb2NK*2`{>gvYOVo!G_0`e~cY{Z2m z``5STy5rvHA@#K)C+>Z`ui5Xzp@9b3OFe(vH4jAi;5E^wfg<^CG*rz;V!q>kEWg5u z>@d!6e$t@bTJ-q%m-G{#i=6e_$yBsUZJ3Yn3180*1#sMwvwd)g-*863XwghsEOl#_ z5&e3fCyXK0nz@hVcqoW>7JvVFcgOPdlhDj1!PQF-oL`n{?~^ECMFC3bA$JW&W5Sbj zX!`q`i=e7(DJ-!G)@A#72_ML(tk%4dpx zZ`gRrHFDXsBx9e>UhA!4%AGDt-muxEJ8dmyVmG428t$YH>|t}?QF;A0A=vj`QejKl z{@HhrGsUP=K2mcNSG{aBT4Fh&6+CP7gdnf&yuDB_frDxl6xmV~u|?u|Do9DH6>dhe zy=I^qyM=6#q8QyLZZS{4I;6i^cEhtzdhM|X5bv!W*n8pA)1R}7)a@a>ZhR``sX}y9 zA5hw+w^~H#P5ZF@zL1;FDjKJ1Cni{OD%z~dA&7?AqvJe=2u!tEH+(%Y9*05p0u`kO zi)W~Gft(UXA}hj#G;BMLM@Rd{G}|Z#a&9+j`s%B?=YdFL@eH3#+K}m$f^Q=(UBTWP zsNKx_ZTdg#)%Rtu#71j04wqin?)x;MW4*tM>w8yKGC@=*-E{h_e=z3)#$+*~3Qg@g zA%yxYX%43CCbZH@wV$U;tpyok)wOan2FvOe(w<-Qf>ut2!Db?u=Gn4wLHkX0yNlHi z#qJAakC>QI5E4zu-Y@pixs_TvdPSv}qj*&EXI=pJ$zy>q3ToTDlI0dq&G$de&*?yr zGt9TaU4G0(TTaUd^U8Ls4j+qyPoLKApo`&YpOY76>gl$x`z?R)s?CAq-x)=Vd*3`< zeeu1}0{LUl-L~?XxN-${-h=sj$r%wpx{uXVRr>aGcC_lt8HToaB4jO)sA>yUp%m26 zsU%JI@>;b@ za?Wi3Tzq=KN6}6g_9SNU#dX!YefIL4HFz6H(t_ZoLD!1*{eh?n#y2{;!k4O309oLtUKF zl;C>itVJf9^oL(JMg0f+RI(z@sFYMIlnqci6ntMF4ZBcW@^M|qSuVDT*gIzzDEi+} zZAuDLjmSBgp3^pb?&<9g<&XCcTHJT7WV>puurVu1sb$!FK(07f4uIA(DckdAblKE6 zYKl5Icmz4#e$(&f&URlp$Nq!<*BO8NP?-77R9;tLQ822R_^x!J$<_T$Ks&=_8%eR= zrK%zxv6m@a8)=I8WAjU$#7B1-xs@F7_gQ}D@C(|cbOz5rVU^!Sk9T_S#P~?J-S#>Z z)$xx)E#DT=UT;%)Z=z}Xrt>$ux@HlrXt+(9GO$+YC$5l2q< z-ea67ms2VhVw^FgHdxr_6ql7U`gi_ zzYPb81~6w~MZVR_^OYYrw=mHhX)U3j}izPk*X~ww&N_5|rl+mNp5iE!nnf(RiM{({(?J z7vm%Qa>kg_SDiz9O52tMW#nBID54JSFm~;zj8r^%-FtTWmp4B}t!nAApv!r5djWa% zoo9R_@aXZ7(t_aU<%Qu|aunrwL|)5*(`668`o1gbG z-7u3$w|LrjowKsPLoJ(ZbVr-t76v0LyW^XF==s+){aRqD8}8qEc7J`MmeU%O%<3}l z^X)sz0#-?%=A@T-!`j?w7yh6s^-0qyPL?HyHZfjk$>fq1d|fG5IOJcz=CClc)AX}* zi`dw2rqx`0@BIBT!$&py2Z6@*m~cv|i!^ha>lYsCc(VNZl277Gf3aH2^qr=A^_@j5 z@z3p-oS=L(*%M+{LBD)C^(56kaGm_rKJLpU3y;|~X}(ARd}#LUbnxqyd}L->Rx;^P z_m#EhK2q^XbjbWztmiv^tBhw;qT>_4ksqpzzBJZLzY49dNL_I9d!;tpb$$C$%q$Ze zs)z`elkW=UbsPFoc0|Sght?E>(z5cSf^q^3Sb_WMMG5X^(?ezK{sW85*xx&km)A=3 zSt8Dly(~S>dEb#qv$to$FR@qfL!PvFEB~83YIJTE)0sO=ZCZ8$OWLn`hHieosqa_J z&m{kw*}t6=O-FY;*|@PUWaey|J+=60;1~Kcc6?O~j2?$uZfo=?U(w__nX~EOm>o5F zW1jS`mzL5Ihb-uyINr?5%X3=0-^eJ;QB_Yh%J;*-{)cM&&6(x0a_M;A#CvWxcaCUk zXcJD@Nqb)$Mk>`PsOx}?f) zg z00;hWT~M;|D#sz`Miw!QH~`0l-iM{rP*_#9)w<%H(4h+cqow`dBRCgP&+%SRU@0@W z#j@|eA?q;ME>0^5Ik0cuThXAH4XeK)tGtmer^C(e7%(Uoc%vpJx%aOJ9#)?zSuP1KySf2M+x#s|7q3>>oQ;{ z!=_xn5yUNv?+ltIh2yFwsqV(~m3-$3qQKHxUHvRohG)FkA!OBKq zWeMfWb+5BJA>Ee0YhoBWpF92iD9vNzHyCJh~NyFilk zk-H6vVj7%{zj&b^PPDao>*G^=pRsR@iReb56GJ32qz8&JD+-*j=Kvm_>YRZlMP@ZB zAYu;i7$&`vXVCLChpVz*p6@Y;62G5gCYJvn8gln&lkr;DK9>$M^?l85_nF;gd`eKw zy$B8H~V|x4-*e4kxoS4J|xx(%OS#a ztEu<`dX1WgcS;|bm4FV&IeR!i?DJa=sqIK|JLoHlC@gC>`p}XtM#eaZOR{XllL4>V z1X@v}o9xws?{Isxb_q}r^62IyO)Ww}`|3Ah<|CNdx+Udxg7te?I6AuU~cGUdl@ z{&cTym~~u!jjl5)8No`3Cc{pDe_X|#+Kn~i0YVlml!FHTnk^x(2J=; zRpSi%lxp6Ef6~e{B#qRSR?8U3L(wabw(0!R-T# zHTr56Xd5o2J{pHOK>#@LmSO9GUe2Lke!LG+s|t`R5QL6zH46ptKqh62TgoxqjEGwP z<&z{i%pXN+g6`Oj<;(Z}eG>Ckjx<`nLc5Y6e@>xnZ%k2FUILS{_1dc?j;WqRIb)oJ zio8ygPHqCm3ReiJkZ6pvzkOX3M<&)e$qa(k;8e?Kv1uHhOV{o}JBXTA6q5!yjLPXDP6yn3>>L9CbEg=cnZeS zCh^GL`=S4wVpGL+8MJ*C1>>h(+Z^{tWf6qNkm2zAOWwiAMMc@fZ%eN1GdChL{3sA z!eHRV2lM*Y;Dotx{7^v0Qelf=yOI0f_g$@KF5{pLn~L#!WS1X9li(BiMXvZ4qaH7d z@+b~CRXq4Rc;u27neJxWu76kU!x7h2xLebQG3xA=yE9I%1TOh1TRrPhN!`nbEW4KQ znW_e+FKKwAveV-4E{N6y;YWP+Y7;<+0+sEcA?<~w73rUtW(}Bp;hg%VB{ecw@NIR8 z6yI`pGK0m(^-!g`=Z~@}xM~lF%wmVe3@_3j0ZLPgVlF!17jfnMO7#zXeaS9gzSe(K z$}||{q&{uOmX2;5pEsQO?$?XNdd07nHC#>rDFGJp zQ^!v*=&F0=rCVn?A+EHyz$R-L4O;;af8>d@rTjobG=h?B@F3(%{Me3)wjMpayy&BQ z_DE8y4v&kK>|SKjFcifkvGmoCwN8y@jwSj|FsZP8)fp0O;fV(}`Wau`gU&-a(;@yf z%>`vzJA&ccQJiX%cT4rG*ZBII*-{#qTfYo(|O!GSu@%Cl^ zC&kl05B0=8h3jEY^ae1}M8P0lzM;!BJA;Hys9(sRKlz+ z?&1Irb}PZ5p{lj*cKZvkGt!%mepc9c(FX`>5=w&F1E82iZ11@fe|ju{E#qBBCQBLd z^R>P3is;K@^e_UD4;rG)t>qWYXeWxz z9Cz7??Y}gGxVs%HY(dG6*#r9|&IKlIsx~!x*Wm2MXU8TvLKtkkXSbia+ZkweAa%*& z1;zYDczPbR z!LxJZOWhH>wIbGSB4Tvj{eaTnGzPN) z-PS&R0`H5u`vw~lBP^01sC|Q<&RqCE!(bO954%^?mbAvk)=5*+m~(^y9Z}eSVb&{~79<=wxrS^Z#E_0V5ay literal 0 HcmV?d00001 diff --git a/docs/extend/images/authz_deny.png b/docs/extend/images/authz_deny.png new file mode 100644 index 0000000000000000000000000000000000000000..fa4a48584abb3db280b8226d18888cb0539de89d GIT binary patch literal 27099 zcmdSBbySsY*Dnf)5~8$(2q@hlA|NS9OM`Tygn}SQH&QM-Rir~Y1VN-Fln`m8B_*Xh z_Po{i+k1awpYz`y`w4>|{)(SmnBm_`vP$p%`-!N8Fx9lYaf;>N>#OjT;s1Lx zpD;Aij7t}k<^H~tPiU0qzn?H4W4u7EAvq-JEXGqk>Gi z80oWdry&CguGZU=9&|Q?1@>}#Eo4UJn`31bFJBtT>(^}eDrlWvy=|))!rnq0)XyxV z#^yfhxo7oI=l#R`) z?Q5OqQKswG{N}J(#MdY~ndle|sWyh7U|Z2E$(U+M4Nq8->25k+U*8!r>iF^gk--?F zax(is{sU)wh4z*t!Dr@;e&^*vC^rS(87+K!>m>T|*}`QjuL*%nbHTg1a*WjRn8Wo= zWjY>LLPL9)(!XCwHuYiDHb(!WgPf^pdgtp~3ew?JEU$PiaG%WQCb|V(zJ}?Y z%kfh!ynnvoSW?pGgpE&M@-1tQn&kaqg+nZS>R1}DeTg8;v8qzxgV~tF{&#|c2Qh#Feh32>)UpQAunIX$j7lRR*`Y)mte)jbLcG{N6trxoib-BB*ddA zZpyEYR5%GJ+i<eGv#=TI>R^!uR?-PY7+XV6U;q3cY^EtGN4WB4W4;CBQ9h_{XoH=eh ztFLn1%Jx`q#B%suyN5d{-It@x^#YfaD6`&m>*3y%!@;J7>)yEYd7IT=M#et;E$=1` z?yB0w1c`ZGG>Pf$zE3SMExNbhRo3>?6_-FIR)da!hNf6b>g@hoALX|;p#k}hj zsJlh>lG>gItLLgQ#3d$LzVKctZF3x6`SCvg@sBc{jcUim^v~N%104=GEp)4%rp-O% z3Ob=Oq3ww0P^aKEe~Kq&(N1#h8yS}J+;_q_HQ)ZB^_tZWqZ$XBb50+}Ix3ADF=EPR zHr9WB&VyS}+0UjB&(3N$Q8RLFLHaVa@H<;rmtK>6K5Eh@Ap6-XU;uLvjtx8giyf;%WV|V4p z3=s!GOO^eM^gyk<^YZnZmfvZ2PWC4ApDlb_SjK^*s)5XVXdyn+3}WE z(iIL}?Slv!u?dH6*imk9_XgT%ybJO)^B;a3{rm>&La&TVK+DSpWwDdYzU|D&a4Ny{ zy5miS1hFIk&2M*pmkSIO8R#v%3|fAa;+Z{Ippy}ZPh%U`Qlq^;J(WwvY@E7Nu@#U$ zQf?JfwD7AZqmWlZ?kL*(bSp(`e`z3}tp>X@Swt_6U8mQflC4#%P}j^VQdW>8+n}Lchp?sL2MlR508JOpRC}c z@J065#>wDF24P@VWQ#Mw>x2Ki54K z#z)_hdc1t&dqhX{>b+`}&aK+C_k41;OAsFCS~q>vboYm`W5_AS0hC-)k{knEQr67R z{=O_Xj~BY$eFza^WL*+!%xo(9al#*-E;20(R$dKc-^8SzQQko%gAt~(vC|rUn`EQZvXD#_9ETz zLZ$O>!)x=0JIh+b;>yu_lcA(+x;my|{p-z#uWmPvOuxGM$T3p4!g|q3pP1UlwsO+z z)5rcc*gV*S>$Jnu{uQiS36YVh- zG3~JM*4a8Cj_rXwlp#!;p8}@iKfcYtiT4x39;AE!sM|Jc`&ULpJG=@PoiysZO+;Fe zv)WO7{fY3$lMal}F_DC+n$};w3o&`@ENMP*wZ7ZH-5S%-Z~>n{Saz(%z=TnTaLUAj zj9Dhw^;fchebZ*3Z_uaVak#&WYpa_+^(}q==cuWS9p>6eS1*1}4WjWHW}x477lo6$ zxz#PEx5mBFK!XZNRg7W%T3CE^!Q*pZZP!@Gqua)QJt3bOroJ+J?KggD!#)iCBoV*y zeyFYLKwY}VQoQuZ=!g4PLz>&#^rX2-Sv3gAtz3y~NIKu%d5~({g~hmD_ChqQ9#DXK zwsHz*p`cn|j@o30Y$|~+ zDzAGaKRsWatr#+V77|W&ky)Pk9!lE;9oI&an?t88^YU`O5+_cz0exlN?sEr)k!1_0 zJnY`ujGr!k*c@ii`|xW*L2Z>!Im=#Q zs-gwDx1Wq;7A9;&Com{nxrXPt`|43L;kJ?cIF!A$XuKwqSj$5f=H<6Zb#r!N%C=_? zzMo%45=;g2&}-K+S*J*^_GAR{%Q;8zn13MUH6%^q5x;>ITlXnB_h5JPChuKwg< z*xJhCaGXEI$&Zs05iXTfpIBB+`RBhUI1*V~u2#GX_4iBWvx<(>I62%YeD=(gE6Dzt zkm)FTw(9j^swa;urjMchm3g0ec|uuNrVw&`w^1gox{FmW;_WQ}*FehzL%5Ouf-qmi zwL94|+Agw^8DvL=g=-mJ-tgj zLc@hV&u|f!qHo?(=J81h+oAE$o$qxP#qVoVevFTItLJ-vOzgbNJT?0UOifAH}tjQ$>^Zkpl^1>I*sRclu~zd%V`C)uJ8Iw zbh}p5=+?7ZQUe9o=-DVacZbk(g-!|Knk9k;8j$W~kJIhKGI? z^_O}R+k_5Zd|CSv%eNj6t1S2?@L&&$l|JZA(64b( zjhH9AY>^~jpB~GiF3o+&$Hkx6dV4jbXXeiDnv5gi!R7T{|L4th(sU8Ro52BP6T21n zy$|DA9;CMQFB$p9-+cHvXreAia2N+g&eK%Zzx?BU&UnY1$G0RwErG;UNhpt*dtMX@ zV$T9t#CEx+M^?Ytj45L-*UGl2?I#bdwB;~7H8h`K__XPH_b>}Pcw zMKJZ->L<;WH!kPet_o()u2s!kCbJQ1>lAS@KE#VpD?w{vK7WBIG1Kbhs>wLSjM{y93*9M|b*!-@mvc3Z3Q{hvqwMo5h68iSuK_GVR& zj~|#`!l#o$;cjGIE59l2n&=kLcyTFN-}ibf53QelOr_UyG8x|Ub&)B?`>$nxHob5X z+P~W)i7%3wqr)T5=XXnJ`*-Dpn}YF}KNmw$(P|~9kpr$~?YULG>=j>XI27k~ zziD4exo~(}i>=!&1`TJQ5$)cS!SFY?Y_j8Ge{17OU_>eS*r1|DpR)_%1fb9cZ<$a% zTN@LsEo?I1-tJR=qfRYHYmM`?nY`kDM`)CIFHXbv*1D9-ECcUUB>^7I^=Pv%3i2@G z1kj)t@dO&ZDUqU&T*55j3=>JyUhwy89tv9@P1g3$h!ig;h8i}am$*ashfiV$qZyrH zG_^GRWrICRKB52nqoiiyRpg0(m@DlFjtY;o`^EQOf1ivFL=+F3>>PX`AfQrr&{OtyoG#~qw_H8&yz8w;C*QI@f82?QG!({7#X!Mbj1EX z*$12cVCdF5+r<70CwWV3JPO`so2J>{q{>n*ht_6BamF<3_kvIBu<0xD59P(wPGXJ= zHIBbCrUhpIINDies`FxuIhOn^nx(dr6XqHB!n1+w`k^}=0ZQ~2$BFE&LcHnr&U&~x zcdpKJ;dPJF9VSKVL0?yQCIZy%MQO?j1-VH}#)i|Gv1)xY$A^+Mf7e7#4>KOf zPb)y2YWn}*2MOXsH-?P7;}(+LmD&@yxBx4~QwciAj#oQbOgD!(Y|Yxdl*X?2ft^Gy$L00sz)O8&4ZX)%fY{2+zuvHDkBtJ* zul3={X#Dh?X<)9*)XyNALC>F2GSoP_*M-M#a_D{NKcEux8ha$_wA5dEn6sFZ9PW>W zKWOE;G=!ZZ>gk5iSYasqyk@O<2_EAqES$*_HL_nHX%f7=Ml3geMX4g7_o&tBpbob!CPBl?IRJIVMv7!yhU{3EsY(v- zslB8@xz8uNuWh*fE(W?(WLV2ear2=gOLnc%=b6^9s}9r6I0uV4Dch4p2A^iTZcjRf ze11vJm>SJ|74D=Ke4qsmlV1enzQ*N7+`Ug(cS=k?J9?7~JWUn7Ov(3!<=y0((q5-ogh=+5 ztfJ!f9S&u5nF#8_K9&bXNkY%&QJ1F+jp!Pp85A@E@hCI^ZV#kM202pgJqpdK@%tIC z*3Ftz7Fc#BFh{|%W^yyG+VkkSE1+Zq)|ELfs^w}GzV29-2hKY?YF9tdmp9WEk?=VP z|I_}Pmq2agLAp8L=My_c(D17U`0wUYesO!tOR^S4_47HS-P)s!zkQiRJBURqo1xgg{nM_R4901+BFT2b5E$c zuT0k0?@USPQTzZ>>-g)tgmQ(&mq-aQ+2w)!oSfSG~dF3tTp@^oAbC+Lbt#!nhU4+_Z;DndM7@6X8@(w?mX}BjxNx7k2`HQUagss z|FG*dV`7z9{xB8?U#=Gse`pUgb-}1BF}bh!S5i*>VLk2+{M*bVS4mi@LqS>U9n!v9 zpqddlm0fmmu%CdsZEwO|dBSa(1f3!D!S~G}uS>E3YjFHMVSu0qircnH zW6qPFeprbH$5!s{`<5bYmV}!T-p7(OX^ynSjEdjeGLN?wk{cDe5%H7z(Dw=YAwg_A zkj!iEn{`c&c2{@shzwmiFXo$J;!^|N=qb7vO0qF^HIev`yrP^l;O1?dj zNAFwJ+gJRP-(~vgsV#W;i-^7DG7_y7eY&$W;1fDqNBtcHwYQfCDXtrQdOk3OhzSbS zTk~CgYpbJ`m1AAHZGFmOPd@(qbBIhnIsdl(c&9Ky8$H^78eh-52~!Qz$cBfp5JV!& zE0Z2uVb865GeZpBk3nixB=9_*e#zx1aHqq#{2uUD$LdRzd@8r?CdxNK`^tH4^wA+} zZm`Dn+3Id17LB&8z`_1ZM1?yPbB;ank%`b)RLtMnQQWk%JfsOHRui^ttWISc<+Ib{ zgB{(ZBcarzlSjRu4=LKQilmd)C_dN0wp$we_>8fN^5#R7QU>g~_%}y#pr|+{y8o_y zU{L4rV_ffGX$SN=~)Yvh^P2Eo<^ebEgHo zR4aRU818PT+9C8a^du)t6fHRU?@Vr*Hvd9XH{i~=9N!>xNxd9l_r{dpImr!-Gq>OZ zEcK*~<7cRSGM2~kV;R&YGh21uUC{}{Q12y|8-vAgT5`Ju7_$;b_kU<$?%~wT)96Wf z+D7sr3RHw8b<~aPiC6SWjD5Xq+BEVElD$t|MK(`1+Guu2(#E+4N2%#Nv@=c1Bcg;b zZ8|}f?R#P=9rOHQqfn40^Fs!dG5>{W(X!j8i55*L^ufE2x|2O$GUVX|%_Q=eZ8MY5 zsUDkP1`*!KcDW9BhN$3xLB?1_%=^@1yplW48q=u>6<1HR?D^eDDGe{7+MW9Gt;8@Qsq<<~&o!ReCg2UPVQO{sLClcnNEH=>o7EqvUA zeJiU^#AIr;ZQL{VxkaqVgn#icgw-~4JK(&~c&F0Hpu=3_xY#?sIUOo~wD-F*meSB` ze6))p?X0$c#OUz`Mlz_`5@#(hiL=jQ*sk(F{(drE7jD@bz>~I^(b~26%P>KE*3G;-&a;p@Zsv-B=+T7D7P+^fU1(Ir^m%&t)^rzDa#C`NL9!fd`O=#Yun8;l{5 zR9#rO#tjRciR0_DYt)yAO2@E7He@eVmZ+k%&5_gN0iviODV@+hv=1W+XsvkU+zo6u zx3XVUjzckazoei%iMdYOgne-^EwQ@Ul3+Ek$2fvoWZO|MOFpjr78l+nV;^JstWgY$ zfGWL|*PDq6*E=;-P9xCbNx47!^;|;5kDcc#;)W0Cv58`@tu)5`wf&!w zbzW|$-h0Va7YUossGjy!&n4=^VN;XlzJ+t+qYVLkA+GzB*U{>2GHveMz*nNX)Ta5E(u{SO zeNVY58>rd$dKjotT=mCgim!NEes?wO9(QUa_l>5BJkCs!q?k70bNr=JJ0IA+D}l1V zFw)7%E$F>kF;pJ&LzJ-nL&?IEZyTmB>^w{h2?NQV9{}OndDQT|du)hx^{43~m9#ooS9CdYF|yLV5$e*$uN`3H)IUxQZl?=>T7 z-|p+lD8`xMQf63_pQG^QI2V7FIZBfDu{26NkZaQT5}{pDVtb>s!5C^^i4M=4+rXbo z(__0w#*4Uhlq=i7xHca2Njo>it+bix`vk3+a;`D=KGJe8>3~r&@rdkF20k<4`)>hJ zSET!0Mqj9skX)w@F4=sM>bV||CwWydw`l}_nVnqKSQqEb@6|Wik^@(&xv5W0NM4c^ zfK{~=aj7Al^aOkIv{{l}@Mu%moUBKQ+gQTgY`%5$1g$0D{+@AA*M7-m=})_heKMaU zb$q!7pN^A0aQIEZo?&brnn1lZJ%f647@%U3aiJ@KVrGFf4}bzg#-FEfU|S;Zgm zjTNOr&@kl2g^BV+xQE|-c>42h?OCoyt{SUZD~c5WT%KXA8}u!?N{@2`o)J+;6SV#K z?!|^~iwv5e=+~0v%SPSE1EJCy8YAbx?W5qaJ!=yhvL#TkJrma9&D8`QW>tpI-Gcf_ zRzIu1|9LQl+c&mxW$$kFZhcjifM%VC>kE;jC8WBWuHd6L5fKr|E^T`8Y}cCy zzpu4Yi*aK;_~cUkC~26^5V0mk1bz<|zEPNzqDVSj*Y`>ET3x)_AmkI|z1B^FiIW+Q zs9rm*@6ji(c+E!1@I8pFu%Btw3SaAHWxA}?O~aN(pPVX8(v8{MAjoof)h!J@d*yG`u7_JWsR2g#+igB!%cNnB_pkl<*)ep&j+@MNcWK2E5GLZDv+}jb<#%R zJzT&eL>48AFnaLr1mL1-u(q1NH5(T31`jG5bs5zqP@!g^j^^89O~n?|CMn&I&a<^NR^rWCT-#XsPZr?+ z^@;!2AEeb0yE9O1B(@N{NbGHJsmNJJEm|4%%)XHF>Kyp}|D=Q+EM6*n)UHf3HwXO*Vu2b#|U7EB!jiC1oh za*F6kEjUJg_*dKy*}!Gr-;}ul{=P#94H+3zh4n!6`HNS^qMhjl`U`Z_`RyjCaPC=9 zAYqAy7Z9Ad7@ldMm?WUO`+KtZ3_vI&zVNz_Q|-ep%8fZYhZE=ug)SWfgWHmmXK z3VKFHd4#{xoryd`V$6KX zeyTcv`px&&#xX1gGv$aybX!KKv%NtId6#2~Pc6(`s9O<@7(^gHv%J0I91k71S3R23 z79g*aB5~*yD{*^jK?kWktwL7Mquqgxh0a$QadWUA=fG{Tmm?+v8M?kD?DC(g!-;8o zF~Kk3nIN~D8&E@o(EYJz-5>R_X3~KlMROVmz3qQ-kt|c(7lYTTm%i~`pB%T-%cvPB zwG5Jhc$=+P?S7?#O&7m5{;31}ZiHVuv|V$??hP_7-v^c>z+m;UJ4I}gr)#3ty`v{X zX6Icw$UwXR4dmVo*6$TWh#tzSh$;%I|G7Y8e=g7sc+2?BxsFc(ozV<0JDj*Vr9mml zq35Xx*{)@giZKF}V|x)^a1Ka1hS_yGP;9=d=OqrZt7Qh;54J(}<&*v>7x#VKjrfPq zuFRMEbKNxV7&o993_4G^$_PDQu~crKIT$R|%X(03h{z|bG7$m4WHIO?Ut=D-w^L7qh8nqr|NaGDv(c$dh2SJV?8HUeNQfd?$!SOS__pclUSw+4Jimi<-d{b8xx3 zHg5;soahZGC2Yr?3tar+b#f>Nno2LpQ3{6P3C$qmte8K{_}8TktQyKn4%i0 zTjmvCzIJxzEG-;0+R5vZfnPu2yeG3`m4_7>s_?^$c5u<*+Rz4Yu=LWzgHVQLp^*F*HI& z(R$uM1-J)lem1l1!+9ki`JT7f#ZU;<-oT$)SBm2;C})v>4(rZ~5hn+s8*ipHWH9}e z(P4=<2B+?FyBh!iDQb}Wzu3GF=CX$f(t#d|a5A(2e$C~Z_@j*+^e)$1%V3dC`Ct5# zi^G<|pyCNM%a?L}^^PQ?##cv#DEKK{IKl}ZcF{jcHu0yho(l;~>VJ)iKab{q_W|!V z#;h&;1>b_oEMX|0B(84=(yYXZ;CM2kzYf{#Rp7$$8hHpby!C6bS=Z}Z76286Rsi8T zQiA!j$5vGDyW&bE@Kk)k{&3sMJHs81ER;{=p_I&@ZGR6IAs67i*vs&W|H*6Nt$v_xy$24n zUyx`;92CKeUWV;@f`y8%cM&V*r<8b$4T?(Y1aoRs&5 z`ynsY50NzX%Iy1Z^5BCmEn}R=eFyc&Fa=!KNuiY(Wzk%vB!M)r0J7oQWq_QU2L@*b z?iKcT6JAG8oFE5LVb#Z|nJG)S11Fu8xENaZ63F(fJUpr(ENXWp^322wBaL`Vz-cM= z8{LBaY%6Tg3K^dS@b|Eq98gl zDICe^H8L08eSzyX3waptug3>l|LHQaAc$u{8lxBTRE@uSu{T?V9!=D^W&sf@1LbdK zZexGkc@igEt5h_Uuhh26VNPDefnC3<CpGp*zNbB(4BL+8pFasQY-s$t~g` z zo)>tnlX4l3g@!G*1K#)|c6OlL(dFf5bhDS z>y&Z{oNqx+?$dV@M*1|MmZfQD5H^-=VhNYC6Q{wDCaCV56~|1a(q`qOm$r* z>pbJtQ|F|lj1F`|g1+Kp=b31{IJE~0p#6c7XI4(u=S&y&3&0UlpvIqITKJiCG=V3? z2sf>Q`~^lm5;iN&m*BaZD_)|M$fC~ZX?t70k@*WvWA%0E$W}j6`w~%{IZLDZ?bvvA zYL(CuwTg?7-?$ zCws)-)IBG>J8Z@a$-G6_-{o3SJ2!maF#!5D?lYA-uW;YF@g2@!hKi}Zz5OPrkjsSh zv7I277w*V5FZX7>tbv#augCt=J#+n{()oS(Fdc||eIA!26470khKji$jPgN2K$%R* z)mZm=Px?Jxn?d$Mv=p}`77_UZ-HJS6=ukrVs4F~v#|H~3JG%7jX4jRhOwEJ&HZXix z`oI&@INd4sUKCTdcu86bRjM5h_x3MAt)))4XmLIQ*>j^AmhXZqg&N4G|I&@$9ONfW z!D!h}GSX~Bk5=BhS(6?au&^l5>K>}JtC`u-HW1Psq2mNZiUWc&2a))Cr>`7Tc1)xN zm4xt$K+d^A_cs8#$DBwC8rvBP_0NgB>?txB+}M6;#4iw=+3!B|wPC4OhF@^RS>`(u zlp!WewONyh4jp0z*bJ(cKjIgqB#Y_Qi*=o5XO`kp4?HdNW-Hz zm?<3>of{8t3{tQ_8H|#ImvtCn%~r5}5DM-eiu_86>er}*Uc_=$W(zFaxrqEp+ zNh%5Qr7?Qsd?w|?zC91y*A+Zz))aUER_tRnuoSHZ^5V9C^+=vI$qOXh9_eVc%L21L zsK^T%Z#aj#5~D8`jmP)p;8O71j=4Q}DQ*hkL^c#+T6#QVBd!j>@jcjDSf-J0wY#VG z2LNZm2hAq2nkVK)e>|1rfJc6n4$uXC4BQ`HbY3@092LR9crJCgnpJUVq%(zzzla`U zzN7kFAfZh~6zl*OSaqVIM@@va&C*J4&>yoY1^#lsl2Bn^4CDD6Hj@XM&!3*t$Ot?A zak?T`CML?E`8N->n3r&-D~~-9t?w{>gD0x=#~Pu54RhN$tjk{@fqJ8wnq3%fuL}_b z8H7h)WNk{IwQu-hQLht*PW(Yn0)R{!UhzNyLVyAZ02tJCtHl8Fp?okgkRXD<&DOtJ z{a{mMnPXK2lz*Q*1qJKeqgF8ck#498N&-*l)dbCdS&1b008^dh{(l1wp~y|MpDDBc z`($4PC?!#tA#VyehzAzIY^$#o_V>vM*73b*hWD3RGb10lC9ESaN<08l7e*BX}Y1ax5;18R%C?Phym9rczE4Amp*IkdUL*fU6>TFMcsl#8=pLQx@vS zjuC`<8KAQaI)OLTl2D(UDxwb+)M9p&t_1O+WXiZ!Ai8(P|u?x2vLDQu_Pvvf!uxt5rh>XJX!`5DsaU+aUSnOxy_Ig_jEJOAbHep7|OuRP-1HA z=a3F(dr(Aj{A=^7UCniH84^H-SAf=M0WcCtIDVW7zk|6bnMw~Vb0#)(P83wpHa^ul z@Zy$K7a^Ey{_A@h2E&!d8^|SkoM0t_$c3cLNkfQ>p?`ZTJw3!(3Dk5fp zEU{ch6Im`gW#;xDFSoWpQhO(sKAS(P?)D4IO;AY_MPX2&m`cb|)|lOV7UB9cBozLXq8}dbUvHw`b2L5q^uPo&mUopg-4#+aR?c4S=jTdatsG!v5m;3dBn@_ z(OGds$VCc!BMJiO!K&qJCNB3e&Va98-El6Wr>Fm=n1zPM_TyHIpAy!zb-69f_?l2HZwaYT0=v_TR3J^%F~5-6Pp`aqRI#R4jB&H z5Vi6plFjH$7fp$!T`Lg#Dn zJARpP!X8CJRaweODXE*Kk@S&5nCqo}vP7R8P?-+z3x3}7QP2H3K+WEDhP_`VPtQ!W z9)y-NbCTA~y@n|ex1WFm=%f9o{LH%D};%Mi^b3%nbpNiuFS_w-LRWA9DgTMW;DZb$x zzA!CIA^h(1hdU+4f4duPWN&*^Xw*sg-*&cwlbPv|X^MBW)_?)m0veu=xI9bSXfOy1 zf7)zQ92)zJ@hk}EY<{7O_%d8-mIdMZwTZ8F*`|v8cNzcmS9$_m3}otTwmoh+KJ(`X z_dQ#kIuAk884kx^|DI5R*&0FfTl;oaEVRSc9x*{`NRM6?Y5hAF7c7Ld@1e{Bg1_w> z!Dfs>)H8pVg!#4s0zca_)BedNDL88XxqhZ^#6h>x7v-e>pQ>;(e_6BBVh=a z%kyyC0Vcv^A!O*d|GVx;iTluFF)IPnOC=o|Yd>Av9|&yXH4}Eb+z1l$S?+slytZQk z63Gx|8KC(j&7|9Q$z%3y*A8=12(D$c2stfP zWjQWes4d&u$g}*u+mg81`3lrI@c^O*4d!f4WcXl+l*jsWY>ELyPO`bu8sy{It@e6{ zK&9`?5K;U(G6$KacmxCf6PFLEhW4+5?Sm*;FeAkT$0-gd>0-}~Y-6#bm78x-x5eB= zcb}Kt$kb$$@Ncq|A56D1NvYK>L>(f@f z*WAVqd06e%?G~hbFO&sL6-}l(p|;TSW0za?Nk3m3%ME*)D!|=N#bX+9`hjV1TPa19 z1LiN&^y|G0u1*^lazh*KO}}?hgc51VwA8T#B$J3zT`6I5?B_^$rByWXZi`^u)uq{m zmsOR*rFFr#)CzLa3qZfFAQ3er;jR$=uO4lnJiEYg-^c=Fr;H-H-J98=@5m$f_g%oI zt0#YFwXLCw+3HS0BrD`AeiT`}Ak~VZ7yv=&+Qo9vdpfV@!T6_+2>|dc>!G5;vPa!T zz!{4`tVxI1TkeVX!5o<;=oPC-Nw@{Xn{5K zavq3TtPpR@gGa5wNfx5+b=rJSbUGQ5QzkSQ!i-4mJD^|ZSofsA0ROz25(Yz*oaNKTc6O4#b=Mjhak7qjREIPRf;7w$1Nzr5=vpmp4F>=5A7l6fbUHK4zx>A%;Xhr4 z9o7HR;0Vt$Hb?!d`CWi^GCn%#-2VGyAB4c5P(ZDb-je?+eAXySm~}uF4vF9WWB;;wiu?D=;C}$F!dj zi6-;khoKw)I3iH`Q%%ij{t`u6^s8X+UByfO+giYJraSu6_n`F+L)RHGUXT6P`e42r z>+f1+|1ZsV)BWjS1t}LH$@~_Yrvl6ma{_aFxe{8s4Yz;=PzujI`kE~Vl+0Lvciu8m(^DtHCLyLc=CT(Rgt^INA8;>p6UnJ~uc03inj2r14XacGzp z{)$ALF7^OJ%7pNNo78_)>D|%yk0JmIsv}b~Na&Y@MNLY`d5s-HUf8_vA*1$6(hpFg-&`7 z-1B1CEM(_yg(b6wPW|}J4je@lPQ4ROB#)W&pX|wPeLPA&*>T80kDNfO$}?{CTLNuc z3uvAuoLDs=MXHltM}>zl*-N^)+Lsei4P(>tii#TGFls^HQ~^Gvg2cDMv=~|hm~r3Y zYjjI>9fSHC|4LzSesi}x=HebWdI7U>`gjVj?N|o{yk~B_#KfPo4|qtN`xr3%VFyhu z&k2vM`>WwXsa{C5bms_Kpa#^30Wd{kD|tB~5)ZSrYS+ZP^bMe#rWM?O&DUe)wso!> zMlIffyLau%5b$g@5EoP-;r9S^M{5xBm%weX2HwvKPtWr{J2mW>R+FG1t2>O0__ZBw zf-#XIs)`X9%Ckw?J1Tieo(GBdK$qeqeWr#&#{ZemqLCDhA2jKxC(1oiTAJqf$7`VH76Rls=!hfvcAX6VaTD5#&5~ z#~~^lZ^~zHfRTGvOpavfYa1@<4;AV?I^Pw?mT(9${2B*eF+*IW6o0TC&$At?GHKbH z4?0$%!6x|)cT_WuRnt)jR9p?vT`PMV<&aFB>2_-egI~l&JI^mN8g6X@zN2?VZde@B z6DVT9%_LR)^j5@OOIh$RX1CO=ZM*z>lWE}wfC9I}Z_Ty#Gl9nP1K8R8VGY3xAV42( z=H?_v!(ClssYj%F@TC>Gi)Fkwn;nY{I(NW-L0qNhT%NUdlcF760TonZO2z@0a&LCe zw3Ny$Iv&38xw^Dah=S7?ldGQ7jwDF?Z9sCFd+R(wsQA)tOCd$HHrX2HM$^B20_(@& z^w?G8lE+j7I*VaVmFDH7SZ39`M(%7c6)+7W#e<v=IB?C5RBx!ld$tX+}my+}I4AqcY(!B|tRvFmAC4TM4(_wAb}9Rn128(luO@ia(P z3BsaTl0laTvGLHo_J3mifN!B79rTJ$5(YJ%?vC>s3He|Hwj~U;@tQVc`5;Wt)6saSA&dhRyKVnUbK6@hg!7&=!>L^&4l2BoMb~Rh6OKwS z1_#}W)O8B{j_m!a)nsW|FnejEGX_ya5&vxUZ9eT-4VI;R-$e<>`*w6K@E2JwZ@j=( z+JrfKy-`v{Nn_ooFlyw+XEo;H$mru_SdcRDtXo8jR`I@Bz*vS1eg}sjQ#L*W)1}t1 z`xr^`VO`H7f_2^1Ls<;aYqU)aHo;#TaKAjh)z*fY?6E25{K)VQ0b&$Ry!mMI`NEUq z_n>y@)Vftn7%dUJ_(mUT3LAP2|shA~$n37lRY(jmrU-hqZyZHW;h0y!l$A#oi z46h)WGdV4d^$3HMO)K{2=O6{>sXLTUK{+}fU-x|3cwu<^y6<-)kc2D`cz5WDumiI! zh~7UeXTCvjj_samzwS?UQl+dwWxW!5TTE(?b%`ejzvdqsn1&JJi~i9gjGai5BgK6H z-<-p2onnzO1`E*bkZ2U+kL@3+1`nu*@@?THqVcb*3tm8lA)qH(4!Zd(GzyV&P}e0p z5^}*M8G{bLp~dKka}z@&++>iJ{W6Y}Afhd)60k*fJsv&qyjqy?uFW?|%+y0<1#0N}u5qNxNs)A!?a)q^VG`{yqm zyIuH?(L&d1Ol;xCa`&&=E*n*oh=cqAB3KvhK78``OJ_U)PHdwG{}JNFF-q^j0zQnq zsrU;U*Cjwb!J9WO2ae3^B%ver7#kQcR=;1JVg6 zil1YWLEmTH?+^Y%p`~mZQu6)d*$MSvge{Z*H+J2dqfC>q4CQ&r^Et5GIH(h{h&qkX z?8~RJn*X|kzu{nAMa(}3O7!gZ7GBg9HD&JtisxStv`Dd8!dQl9`rsN5&O3m_6@dIS=V-ZCqsHRzL zI4+OH1as28p*?y!`ZxCNe@;+)NtwW!ufv-C{K?P>f)8%Bf&(M>{$ZE!4gvIm)YL*SfxF@zLjk!YCmzwWO1hW|dJQqsGvBDJj0Br$J8<*sS<$3VYt z>nxhV+{gBx_5M8_O^M=)Y|?mD>4L3n{;%m4CVEI;rbR;;U8WIb2MpGGKm`svn<(lz9HAna&&)F5qF30fz$O#j zn|Fp7&sP`)<45yD?*^<5o$1!$qvT{zrjXuFtvdZErS5Qv4bjydLu%s~d{}vyk&2Va z>5G~uQq_i&Z7WPp}t)KRT@}nQ!04AMn-mWA4^tH}2zM-c`SSHWl(}(tC zkxzzL&AK$GN5U^E7M`H&K$gmRYclJ1TxmCHd%gpvRe3K+a&IGv%cg2Gs3jUOmq$8? z6E(D4TwiqQkxB~qH2HAHmicN9;CqP-L738=140r9p^qJ~YKH6Oi#99P_?wzy8m62@xQkEkty-vLwEWF)5mR(XhK(am6cY_tXT)% zs3~4<)(&-F>St-3=!D>mFzN)I6^zknTi*`f}h;%@*Y|}qi75q#Xme?XJx7reH zVUm*c$$%-6-!@GY?__QTYwimm(AseFkJ+T6&KO2&O*tIeL`1>2FnHd#27AJ6Ap3Vc zQq#gH9iu<00QwnDGs+Y*8jIS+r@iN_t}*hD^bmt-D$kStBA3dnEaiPK@~f%azKQB6 zTFJRz&S%ErfBK6c22U<|PrF7q&(M&L3G%D;m*SnQ(%;h@w(}#n|7gk0B)*ASC`+ph zDzm%+9i6}LDTBdNOfGSuolvYx>Tj>2^`-ZgO1fGiIqK@+`Pk=JvO|nPZEv8puDwBY z>LtO6C!wq@&32qu<{{H^$L)asQQddHQ~AgLm&|0oDcdm$;n<<z ztYh!&y;s>QTU!d*LK(^4gwONteO=$b;QPbp^MjwxIoEYN_v?PWo{xDLURdqz`QJXt zp2?&xJ{*U)b^d)@Ix&7J#r1sFx2rJh{1~rZ?@2SsV=W94gNDuZ_TzeHtdh>%myNm6 z!xbjH(9K19>kCiDJfxaM#j8YpLB9kz`%)PWksp8_6TqOI@6niS{)@_Eej{ zZcQnmwP{G(ZPBNw_4Y7(7V}TQ{|lc7uwsHN6#JxM^n_kJ4Ie;MDL@1xRP~c)ez)*8 zn*ny);j$D^wz!&b9ab^-mlDt>_YrsatJ|K)UVidtOY>N36E6ELh>mDL#5Rv;qYerv zVbJw~Gw-m+6(~Swz;mT$oVCOq4c4oT3oMm;C-7+gguhteACfG_j>Mz@+_G_Lq(vj8 zEvJAfR1pp!Nunjh9Perj_sc-PGv>hqqef>2#i3|wgf z*b)*XU95KC-3p~+;89oZbMh_lc*jfkglDJB`KlfFQIvK@ z%q+__?P5sCBq7J;=0d*!f;531Xl$vV;nh=XgYGry^6Nf-JS4sLtHNJ5BBUt|j5~s4 z$|U4xai5T~h|0FZ5~SGWgQ%{fmi5&cWejNNvAp`|rSba8wgvDLm03G=R+n*q1xN_| zi|GKz1bpTW(7dbq_fo+G2F*x`h}ul`(kv!`CZxXv<$pn5*gRJnzPBq#&r13YkZM>; zU)=cjNym$`EFY2|?QmwyuK-ueaeA`%IvoxAQ&Fp(qkjqu=_gPMUy=++dtUUg`u;I*$=w}!?d!YT5m+bu5?qx_K>ubudLPXe z=o5_$7Mh-^g7b9BMp^0YwJ=Uy{|NM&3Y@z;gXn^eb`}9GgFqj!+Y<=ruR8x85qb)` zPCjt=0WaA;A{-tVJdFbm$9K^&Ioo(EjZo=l3k8T-r-2`VOp>}vUbxc$sCfAZ&x!41 zLd3xG_gHP|6YVm4-~f#fw4E7`3`Lq!8_#@va6t2|HZ7wwoqnH%uv3=iQ)=I3o7vT= zRwEdfi!v_*;->AMhBF+k4`;f%pp&m&HN|}!tz*u|G`aVr4C=S82;%w2@`5v=9+SxW z(T`3nJ-B7>R0cWCTm zz@|tqjje&?>#al>>21Pco_UXGGYA-c+DUxgF4JR6{3Y->#bBuOf}IC3_ChqE8PSCh z6{!&vCwvOQ=KZg`yXynk$N}~rtf%|IG73v_h_)A{afE%tF!Q7k8tV;Rr9r5J& z+GW+~-mx{%KN$!L@`Pd=;9^WZW44Jqbgpm~+xy?k0nW2HM_?wB=P?KD%!7q=w-f~V z>%hr%JRDi9(<}uoZ`dCQf%}U`q1mjgURr6tO1Z~|mBWpAIC&fhn(@!LD=QxY+{jDs zez-%|@%Tcw_XCGNLhql^pDPRclp%d|QB~(3@;)KU-o<;|N;&&o#N-@`Y8whj_m;ty zYGgas2Uv(E2lJ~##J`@=^ZH&MKFWHXra+H&+mjl3^j!LuV3EHQ(Jq?sX4O2b{>$E7 zm^*_eReMx+50g*(XtHCdFW*Vl_7)ZAWm&C8!}@bEM_OH%mcc9a4xCmMW-L|HR7Q5F zrly3%{ zgkR!PJLS>`6*}vv*_(MFkHk{p>+OituLG^<>S3bvhP*t_5pE?uznem<;Qy}Df2*>7 zEqvheaF`d#a1ClCSRA~#ecwI4z6#eSau?pO?u*!UDNB<3@G;!21)llk*8KT7QiJzm zVSR$CkBlT*IS*bK3Ub`%T2_3e6tSb#-0eI*mv6l~(a1nFd>}<)ibd3oZv#>XTzwx~ zwi^n~52Ky>t_ui0}-pv>G!{?qOzlmV;%-cz6rV7hO3hZnQ z!xQsdy6QK`({ljQaLZLC*LCFh5O(}7hn(#CR`cql4$ZI`?=d81w2^$5e@1n&?52=g z=m2VrEVqM28oRhu2NbcbohcndJ8+nH>IvQvJJrjG{S*?onOvL&ED;|s<~czwBOQMu zsp)<5AJ2+i=wLo~txKU7beb@eze8BQKoGD79VP(-~6OPt;WJ zPr~?y%B-~$iy9`{m`$KQN;od(uxh&72XyX)*5wwRWWzWksU=Ex{LbEeLha(@U+wp4 zciVv=aX?pLw8%ufe@l1kk>J$4ND5{!qJGi_tGTLDbdV2u2=iyybgKW#K}xYyVf{_d z5oOt6;qERQ@Z4cPl4Y(0(#>6{Nio4VpN!d6+H$?}W{p4HY-+iCsg23-h z&@6uS`(La0KkVaH#NRUt%&xnvu{4RU2+z5`&{{obsF|bb= z3nc_bG&Rs~q$|8m~-etE+C*k176>Zd!=5li@NKu;qB$Yq<&QJgRTC!G9E$gtMU`~4t^|k47 zarNL5ephp;KV94x1q`G;cL!M+{ciCdQnek21{*l<{(Sldos*7f57_Pf5K(`0HjE^7 zwSzo$!yFu-h*IVr{1VTsY^I5~;>VPTfl|jC8Ug9~p7$g`7Y*&ihR6A$R{aO51fo5r zzMo_Bi0Ny^Q#Es8ZBGuNK8qjQ^psP_6H_H_58Z9pQz!z?*8%-r7w9@&$m7Ae75C(T zu;hq^W3N@dh;`Sr6$&6q6ho=80(vp*w{(C49s*2aQsS^n)wnj)DT5wFIB@g~GIT(W zCcBF05$vC6pSO+c{|(21qiaE~E50C_m3SR&p7)fH_U70Dkb!7n2us&g4vb3{Aq7hr z+PEQW&vxdh1gX&oi=C(wJQ~(3GceB&jRQ~3vh$qHC9l&>Fy!moxOn}u1xrvL`cHFC zGAg8vv+NzBQscM&wzLR7?AyB)107xCxVXMK;2GR9ZyUIrf*XM5v5$v@@JAj$K0(b^WDx*A6(#xvTZ!}PbI zsvzr@!Nb4-pAKb?&Z;19C-d5fB^wd@$LBAEd*X70W;3Ap(n({8<0wkxq{`6J70v!P z&5aT!#!zt7#imRLi{-?8OZlt*w}U(3fAx$04~|(l+CFKhNaOv;#QyHOY`yF6 zXiJjF@d7&ctP68U@Rt9Y@_{;eex^{1jD+PsDneOK5qu3~|Nngr&uIN23q>kpTm=}a z>Z-oPMJq@)YeAK~Nev>mJ%Iw_+Y%SWa^5NS(B1pk(}@@`{H*a{vyFxEP8E_@(nMLM z)A+GFDiW_<%>Pn6DF0|`KMUXx9hJ9261pWVa0!W4(z%K4Fst(ezxeXkH&C3BIX(4r zN#Pl=;Ud2g;3jSTiHXOkt_=T1T0+OE#vgUo4hrZiMgShdF8~nFcbJ9)_#JH$C%`ki zl_W7cJp@qzlAqdSg&Tg>M6(UM{s17n?fL`2 zQ`z?KuL+LZiESx_$iskC!!8WK7|qb(0NSp)7^PRth=7Z>isHy2!r7XCVJD{Ao zHA%3@c#VWD%Oo`U9Aj-qsU*p1IK{gx9p6P9h?rScNaDPFu+(3Stb?O0WX97 z0NXtQ(|aQf(vdIXokMeA@hgh@1I=C1$Kt8>Z3Z^UtMIm8(6O(LSpS~%%bvDi0Zr;w zbv54H68G*dg>JA%Ukf6}ARGV;U_#T;YO^jdrEamIp`mvab%4tr1kHO8$WmRny*glQ zuSHT^y727}=G+P&got|i;oe#$yr}^3Yah4fd&5AA7z&0U_gf3tOU!;uH2vbN9OBp9 zQfg9qPR?BN-maoG-{GAptf};2*jU{r5rH!oVlXDiPaTv1e*~AMj))pR!I=WlON}E> z)$#b_w_KSzfQR&9f$BpWeAPRI88LC+2T4X_FA(OyQUs)z8@^Hk9YIAJ$xvm!PZB7F z);klo-y`>ch*DS2kTUG)=_y7!*^gLdybKD1uvuilV=arQYwrPuxGIPmDe*&O-3oOWl!pL4ZQqngo|(9JU_>ruvq9PBv5}L2A8(1Y`oSQzeUmoK z3C_aAQ?Oi=3tN3CT9?ZN3JW2<4*_-5$j%~E%+bLeMyHiKG$cqsHqm(_pUykqfu$2n z^!4v#AbO@-`$*S2V!vns91Z-ZaOaIbrb%k^q8yCDcjKu#n!SkAf+lxxS`;B%jY%tZ z=$$n@4<&GfA09%kRf*F+3l) z*5)R(pK7!?9YMpH4j{Q92tt(GaNle?{Wi7JqOcdV!SVWCuZ3_Nd)Q~3jYaI;N3muR ze7Kj@nA<03d3rh{Bh(%lQ1V>M;}*v8$c7Q}Dg{0^#AIJst?&POG*MwYabr!KaV@Q| zVIhOH-}}?T#GaH(#)8Pyj|U}3M$K7@^K=Jk=t^pTvoJY(`*IIY&wmyc7A$;yGZpNi zWrLlP_%1Q+Ovm9NHYPT9Gxs=WChS{eW25AGmxm7@7GGUwlco+0Nlj%m7fvb47)@*_ zE-TBsYbU`}5RFps*{>d7>HO&&IyGe)laYa@ymE_p<;EoDF?$(~60NGMOB-~wZ@Ts< zFVnvN{d?^pR9?uAKgug5E{@}a)vK|o(NUF$&uDiZ_l?+$Gm{f7=_DNU81P*jn4T7B zF|Cxiaf8D+GAb(Qqg5`8hklt}sh>UH(9n>Don5%Y(*3)4v!)oAencj-k`60+W+o;U zJpVv7cKn3-sg&7{BO^UMn{-M_%7h69y)RCAK1k8WM`p-waR2dnZD3+TyS24t;q0u%#Kc6*G1)$|wG}t?MNCXgw>4_#)JTCV zJjSQar1T%lnSHi)?qQM7R(Yv3ipK$M1_2WsX;+v;r-JM1t~oh7D|lqfuH$viMrOop?yq{o*Jp5Kep4{aAv%Brru zx9=t6j6*&AzOcBcTvk~bUQlqw%}RDsQ}<6HWz@;+tcAZe-?f^~PKCp>PWWbGpPvlt z>+6>shKG|nucChaURi;CjfDL(E-H$2yNiUfkeOpr;&QfZMj!PB!bastQgKa9%}2}{ z%zR3Rjnn_?r|qwIiT8{b6H46q{d<;; ztu0$$FZnsH+i5;#W@ba?j%iqnJZ*=^W!GBWaI2NUj2h|5;qD*TJ9s1|Q>MyoOq?c5 zc0fm# zGZtGCpOC<_va*u*+(m(mj7+iUsfWj1Ch}lmQBlPee>uNRSJWpISBMwAGWQ#8h9yGD z;+&j-j3B0;o12@LHp#~150@4fv%L6~g-{|+Ed9!|$BNjL^mJWeVPTyc(VpE> z&CShA$8x?kN%8T~>1u5xK?+i2J<8T)jo$_bxh*X%OUUEz5e*pKuCbfum!4pcAs!`D zljrPAi?JqBA=#{VH^*^KQ@U;@DSqGM@QWZ%y+V5J-@#k&HFsUEr+1@=WH?xk>+9>o z-v4ATd?1_>a{1j468QZMLIQgHNLvi%J88@VEcV4USqa6*0U{zIB_!CUfu+^eNP}$D zmLC&d9q)}xHEX|rbL*8Ryph2~X{f7v+kX4nU+23(m(7a%Ha~CWY%D1$d0ttF4?fAo zM9$mJ#L3(|sev=DGg+qVnC)(ntHT>DDlJttG&0Jy7B-?UGr$o%->*6odqE~k&O%a& zBfWaQVu5ah32wSi{kCIsTiX>i|M1brdhNXgMzov0zwdOmH$*1Bj8l|jM^Wt)QMK>U z=EV-7fbai#n_)Jes=qyv$qFivHO`FbB)MrEGI@!-8@;UeK z-^Abla@yPbeBDE;bH$yD@5)%Hs$R)OGQ&hzP@H%qerHBX_QyUfE*32=FaI+-I=cNW z7>mVHV|IJB ++++ +aliases = [ +"/engine/extend/" +] +title = "Managed plugin system" +description = "How develop and use a plugin with the managed plugin system" +keywords = ["API, Usage, plugins, documentation, developer"] +advisory = "experimental" +[menu.main] +parent = "engine_extend" +weight=1 ++++ + + +# Docker Engine managed plugin system + +This document describes the plugin system available today in the **experimental +build** of Docker 1.12: + +* [How to operate an existing plugin](#how-to-operate-a-plugin) +* [How to develop a plugin](#how-to-develop-a-plugin) + +Unlike the legacy plugin system, you now manage plugins using Docker Engine: + +* install plugins +* start plugins +* stop plugins +* remove plugins + +The current Docker Engine plugin system only supports volume drivers. We are +adding more plugin driver types in the future releases. + +For information on Docker Engine plugins generally available in Docker Engine +1.12 and earlier, refer to [Understand legacy Docker Engine plugins](legacy_plugins.md). + +## How to operate a plugin + +Plugins are distributed as Docker images, so develpers can host them on Docker +Hub or on a private registry. + +You install the plugin using a single command: `docker plugin install `. +The `plugin install` command pulls the plugin from the Docker Hub or private +registry. If necessary the CLI prompts you to accept any privilige requriements. +For example the plugin may require access to a device on the host system. +Finally it enables the plugin. + +Run `docker plugin ls` to check the status of installed plugins. The Engine +markes plugins that are started without issues as `ENABLED`. + +After you install a plugin, the plugin behavior is the same as legacy plugins. +The following example demonstrates how to install the `sshfs` plugin and use it +to create a volume. + +1. Install the `sshfs` plugin. + + ```bash + $ docker plugin install vieux/sshfs + + Plugin "vieux/sshfs" is requesting the following privileges: + - network: [host] + - capabilities: [CAP_SYS_ADMIN] + Do you grant the above permissions? [y/N] y + + vieux/sshfs + ``` + + The plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able + to do mount inside the plugin and `host networking`. + +2. Check for a value of `true` the `ENABLED` column to verify the plugin +started without error. + + ```bash + $ docker plugin ls + + NAME TAG ENABLED + vieux/sshfs latest true + ``` + +3. Create a volume using the plugin. + + ```bash + $ docker volume create \ + -d vieux/sshfs \ + --name sshvolume \ + -o sshcmd=user@1.2.3.4:/remote + + sshvolume + ``` + +4. Use the volume `sshvolume`. + + ```bash + $ docker run -v sshvolume:/data busybox ls /data + + + ``` + +5. Verify the plugin successfully created the volume. + + ```bash + $ docker volume ls + + DRIVER NAME + vieux/sshfs sshvolume + ``` + + You can stop a plugin with the `docker plugin disable` + command or remove a plugin with `docker plugin remove`. + +See the [command line reference](../reference/commandline/index.md) for more +information. + +## How to develop a plugin + +Plugin creation is currently a manual process. We plan to add automation in a +future release with a command such as `docker plugin build`. + +This section describes the format of an existing enabled plugin. You have to +create and format the plugin files by hand. + +Plugins are stored in `/var/lib/docker/plugins`. For instance: + +```bash +# ls -la /var/lib/docker/plugins +total 20 +drwx------ 4 root root 4096 Aug 8 18:03 . +drwx--x--x 12 root root 4096 Aug 8 17:53 .. +drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 +-rw------- 1 root root 2107 Aug 8 18:03 plugins.json +``` + +`plugins.json` is an inventory of all installed plugins. For example: + +```bash +# cat plugins.json +{ + "cd851ce43a403": { + "plugin": { + "Manifest": { + "Args": { + "Value": null, + "Settable": null, + "Description": "", + "Name": "" + }, + "Env": null, + "Devices": null, + "Mounts": null, + "Capabilities": [ + "CAP_SYS_ADMIN" + ], + "ManifestVersion": "v0", + "Description": "sshFS plugin for Docker", + "Documentation": "https://docs.docker.com/engine/extend/plugins/", + "Interface": { + "Socket": "sshfs.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Entrypoint": [ + "/go/bin/docker-volume-sshfs" + ], + "Workdir": "", + "User": {}, + "Network": { + "Type": "host" + } + }, + "Config": { + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] + }, + "Active": true, + "Tag": "latest", + "Name": "vieux/sshfs", + "Id": "cd851ce43a403" + } + } +} +``` + +Each folder represents a plugin. For example: + +```bash +# ls -la /var/lib/docker/plugins/cd851ce43a403 +total 12 +drwx------ 19 root root 4096 Aug 8 17:56 rootfs +-rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json +-rw------- 1 root root 347 Aug 8 17:56 manifest.json +``` + +`rootfs` represents the root filesystem of the plugin. In this example, it was +created from a Dockerfile as follows: + +>**Note:** `/run/docker/plugins` is mandatory for docker to communicate with +the plugin._ + +```bash +$ git clone https://github.com/vieux/docker-volume-sshfs +$ cd docker-volume-sshfs +$ docker build -t rootfs . +$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created +$ mkdir -p /var/lib/docker/plugins/$id/rootfs +$ docker export "$id" | tar -x -C /var/lib/docker/plugins/$id/rootfs +$ docker rm -vf "$id" +$ docker rmi rootfs +``` + +`manifest.json` describes the plugin and `plugin-config.json` contains some +runtime parameters. [See the Plugins Manifest reference](manifest.md). For example: + +```bash +# cat manifest.json +{ + "manifestVersion": "v0", + "description": "sshFS plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["/go/bin/docker-volume-sshfs"], + "network": { + "type": "host" + }, + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "sshfs.sock" + }, + "capabilities": ["CAP_SYS_ADMIN"] +} +``` + +In this example, you can see the plugin is a volume driver, requires the +`CAP_SYS_ADMIN` capability, `host networking`, `/go/bin/docker-volume-sshfs` as +entrypoint and is going to use `/run/docker/plugins/sshfs.sock` to communicate +with the Docker Engine. + +```bash +# cat plugin-config.json +{ + "Devices": null, + "Args": null, + "Env": [], + "Mounts": [] +} +``` + +This plugin doesn't require runtime parameters. + +Both `manifest.json` and `plugin-config.json` are part of the `plugins.json`. +`manifest.json` is read-only and `plugin-config.json` is read-write. + +To summarize, follow the steps below to create a plugin: + +0. Choose a name for the plugin. Plugin name uses the same format as images, +for example: `/`. +1. Create a rootfs in `/var/lib/docker/plugins/$id/rootfs`. +2. Create manifest.json file in `/var/lib/docker/plugins/$id/`. +3. Create a `plugin-config.json` if needed. +4. Create or add a section to `/var/lib/docker/plugins/plugins.json`. Use + `/` as “Name” and `$id` as “Id”. +5. Restart the Docker Engine. +6. Run `docker plugin ls`. + * If your plugin is listed as `ENABLED=true`, you can push it to the + registry. + * If the plugin is not listed or if `ENABLED=false`, something went wrong. + Check the daemon logs for errors. +7. If you are not already logged in, use `docker login` to authenticate against + a registry. +8. Run `docker plugin push /` to push the plugin. diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md new file mode 100644 index 0000000000..aa4fa4a9b3 --- /dev/null +++ b/docs/extend/legacy_plugins.md @@ -0,0 +1,93 @@ + + +# Use Docker Engine plugins + +This document describes the Docker Engine plugins generally available in Docker +Engine. To view information on plugins managed by Docker Engine currently in +experimental status, refer to [Docker Engine plugin system](index.md). + +You can extend the capabilities of the Docker Engine by loading third-party +plugins. This page explains the types of plugins and provides links to several +volume and network plugins for Docker. + +## Types of plugins + +Plugins extend Docker's functionality. They come in specific types. For +example, a [volume plugin](plugins_volume.md) might enable Docker +volumes to persist across multiple Docker hosts and a +[network plugin](plugins_network.md) might provide network plumbing. + +Currently Docker supports authorization, volume and network driver plugins. In the future it +will support additional plugin types. + +## Installing a plugin + +Follow the instructions in the plugin's documentation. + +## Finding a plugin + +The sections below provide an inexhaustive overview of available plugins. + + + +### Network plugins + +Plugin | Description +----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +[Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. +[Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. +[Weave Network Plugin](https://www.weave.works/docs/net/latest/introducing-weave/) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. + +### Volume plugins + +Plugin | Description +----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- +[Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). +[Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. +[Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. +[Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. +[DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. +[Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. +[gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). +[GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. +[Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. +[HPE 3Par Volume Plugin](https://github.com/hpe-storage/python-hpedockerplugin/) | A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage arrays. +[IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. +[Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. +[Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. +[NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. +[Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. +[OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. +[Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. +[Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. +[REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. +[Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. +[VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. + +### Authorization plugins + + Plugin | Description +------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ + [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). + +## Troubleshooting a plugin + +If you are having problems with Docker after loading a plugin, ask the authors +of the plugin for help. The Docker team may not be able to assist you. + +## Writing a plugin + +If you are interested in writing a plugin for Docker, or seeing how they work +under the hood, see the [docker plugins reference](plugin_api.md). diff --git a/docs/extend/manifest.md b/docs/extend/manifest.md new file mode 100644 index 0000000000..d47efce8e8 --- /dev/null +++ b/docs/extend/manifest.md @@ -0,0 +1,222 @@ + + +# Plugin Manifest Version 0 of Plugin V2 + +This document outlines the format of the V0 plugin manifest. The plugin +manifest described herein was introduced in the Docker daemon (experimental version) in the [v1.12.0 +release](https://github.com/docker/docker/commit/f37117045c5398fd3dca8016ea8ca0cb47e7312b). + +Plugin manifests describe the various constituents of a docker plugin. Plugin +manifests can be serialized to JSON format with the following media types: + +Manifest Type | Media Type +------------- | ------------- +manifest | "application/vnd.docker.plugin.v0+json" + + +## *Manifest* Field Descriptions + +Manifest provides the base accessible fields for working with V0 plugin format + in the registry. + +- **`manifestVersion`** *string* + + version of the plugin manifest (This version uses V0) + +- **`description`** *string* + + description of the plugin + +- **`documentation`** *string* + + link to the documentation about the plugin + +- **`interface`** *PluginInterface* + + interface implemented by the plugins, struct consisting of the following fields + + - **`types`** *string array* + + types indicate what interface(s) the plugin currently implements. + + currently supported: + + - **docker.volumedriver/1.0** + + - **`socket`** *string* + + socket is the name of the socket the engine should use to communicate with the plugins. + the socket will be created in `/run/docker/plugins`. + + +- **`entrypoint`** *string array* + + entrypoint of the plugin, see [`ENTRYPOINT`](../reference/builder.md#entrypoint) + +- **`workdir`** *string* + + workdir of the plugin, see [`WORKDIR`](../reference/builder.md#workdir) + +- **`network`** *PluginNetwork* + + network of the plugin, struct consisting of the following fields + + - **`type`** *string* + + network type. + + currently supported: + + - **bridge** + - **host** + - **none** + +- **`capabilities`** *array* + + capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) + +- **`mounts`** *PluginMount array* + + mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) + + - **`name`** *string* + + name of the mount. + + - **`description`** *string* + + description of the mount. + + - **`source`** *string* + + source of the mount. + + - **`destination`** *string* + + destination of the mount. + + - **`type`** *string* + + mount type. + + - **`options`** *string array* + + options of the mount. + +- **`devices`** *PluginDevice array* + + device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) + + - **`name`** *string* + + name of the device. + + - **`description`** *string* + + description of the device. + + - **`path`** *string* + + path of the device. + +- **`env`** *PluginEnv array* + + env of the plugin, struct consisting of the following fields + + - **`name`** *string* + + name of the env. + + - **`description`** *string* + + description of the env. + + - **`value`** *string* + + value of the env. + +- **`args`** *PluginArgs* + + args of the plugin, struct consisting of the following fields + + - **`name`** *string* + + name of the env. + + - **`description`** *string* + + description of the env. + + - **`value`** *string array* + + values of the args. + + +## Example Manifest + +*Example showing the 'tiborvass/no-remove' plugin manifest.* + +``` +{ + "manifestVersion": "v0", + "description": "A test plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["plugin-no-remove", "/data"], + "interface" : { + "types": ["docker.volumedriver/1.0"], + "socket": "plugins.sock" + }, + "network": { + "type": "host" + }, + + "mounts": [ + { + "source": "/data", + "destination": "/data", + "type": "bind", + "options": ["shared", "rbind"] + }, + { + "destination": "/foobar", + "type": "tmpfs" + } + ], + + "args": { + "name": "args", + "description": "command line arguments", + "value": [] + }, + + "env": [ + { + "name": "DEBUG", + "description": "If set, prints debug messages", + "value": "1" + } + ], + + "devices": [ + { + "name": "device", + "description": "a host device to mount", + "path": "/dev/cpu_dma_latency" + } + ] +} + +``` diff --git a/docs/extend/menu.md b/docs/extend/menu.md new file mode 100644 index 0000000000..4b9955038a --- /dev/null +++ b/docs/extend/menu.md @@ -0,0 +1,15 @@ + + + + diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md new file mode 100644 index 0000000000..5801ba9151 --- /dev/null +++ b/docs/extend/plugin_api.md @@ -0,0 +1,192 @@ + + +# Docker Plugin API + +Docker plugins are out-of-process extensions which add capabilities to the +Docker Engine. + +This document describes the Docker Engine plugin API. To view information on +plugins managed by Docker Engine currently in experimental status, refer to +[Docker Engine plugin system](index.md). + +This page is intended for people who want to develop their own Docker plugin. +If you just want to learn about or use Docker plugins, look +[here](legacy_plugins.md). + +## What plugins are + +A plugin is a process running on the same or a different host as the docker daemon, +which registers itself by placing a file on the same docker host in one of the plugin +directories described in [Plugin discovery](#plugin-discovery). + +Plugins have human-readable names, which are short, lowercase strings. For +example, `flocker` or `weave`. + +Plugins can run inside or outside containers. Currently running them outside +containers is recommended. + +## Plugin discovery + +Docker discovers plugins by looking for them in the plugin directory whenever a +user or container tries to use one by name. + +There are three types of files which can be put in the plugin directory. + +* `.sock` files are UNIX domain sockets. +* `.spec` files are text files containing a URL, such as `unix:///other.sock` or `tcp://localhost:8080`. +* `.json` files are text files containing a full json specification for the plugin. + +Plugins with UNIX domain socket files must run on the same docker host, whereas +plugins with spec or json files can run on a different host if a remote URL is specified. + +UNIX domain socket files must be located under `/run/docker/plugins`, whereas +spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker/plugins`. + +The name of the file (excluding the extension) determines the plugin name. + +For example, the `flocker` plugin might create a UNIX socket at +`/run/docker/plugins/flocker.sock`. + +You can define each plugin into a separated subdirectory if you want to isolate definitions from each other. +For example, you can create the `flocker` socket under `/run/docker/plugins/flocker/flocker.sock` and only +mount `/run/docker/plugins/flocker` inside the `flocker` container. + +Docker always searches for unix sockets in `/run/docker/plugins` first. It checks for spec or json files under +`/etc/docker/plugins` and `/usr/lib/docker/plugins` if the socket doesn't exist. The directory scan stops as +soon as it finds the first plugin definition with the given name. + +### JSON specification + +This is the JSON format for a plugin: + +```json +{ + "Name": "plugin-example", + "Addr": "https://example.com/docker/plugin", + "TLSConfig": { + "InsecureSkipVerify": false, + "CAFile": "/usr/shared/docker/certs/example-ca.pem", + "CertFile": "/usr/shared/docker/certs/example-cert.pem", + "KeyFile": "/usr/shared/docker/certs/example-key.pem", + } +} +``` + +The `TLSConfig` field is optional and TLS will only be verified if this configuration is present. + +## Plugin lifecycle + +Plugins should be started before Docker, and stopped after Docker. For +example, when packaging a plugin for a platform which supports `systemd`, you +might use [`systemd` dependencies]( +http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to +manage startup and shutdown order. + +When upgrading a plugin, you should first stop the Docker daemon, upgrade the +plugin, then start Docker again. + +## Plugin activation + +When a plugin is first referred to -- either by a user referring to it by name +(e.g. `docker run --volume-driver=foo`) or a container already configured to +use a plugin being started -- Docker looks for the named plugin in the plugin +directory and activates it with a handshake. See Handshake API below. + +Plugins are *not* activated automatically at Docker daemon startup. Rather, +they are activated only lazily, or on-demand, when they are needed. + +## Systemd socket activation + +Plugins may also be socket activated by `systemd`. The official [Plugins helpers](https://github.com/docker/go-plugins-helpers) +natively supports socket activation. In order for a plugin to be socket activated it needs +a `service` file and a `socket` file. + +The `service` file (for example `/lib/systemd/system/your-plugin.service`): + +``` +[Unit] +Description=Your plugin +Before=docker.service +After=network.target your-plugin.socket +Requires=your-plugin.socket docker.service + +[Service] +ExecStart=/usr/lib/docker/your-plugin + +[Install] +WantedBy=multi-user.target +``` +The `socket` file (for example `/lib/systemd/system/your-plugin.socket`): +``` +[Unit] +Description=Your plugin + +[Socket] +ListenStream=/run/docker/plugins/your-plugin.sock + +[Install] +WantedBy=sockets.target +``` + +This will allow plugins to be actually started when the Docker daemon connects to +the sockets they're listening on (for instance the first time the daemon uses them +or if one of the plugin goes down accidentally). + +## API design + +The Plugin API is RPC-style JSON over HTTP, much like webhooks. + +Requests flow *from* the Docker daemon *to* the plugin. So the plugin needs to +implement an HTTP server and bind this to the UNIX socket mentioned in the +"plugin discovery" section. + +All requests are HTTP `POST` requests. + +The API is versioned via an Accept header, which currently is always set to +`application/vnd.docker.plugins.v1+json`. + +## Handshake API + +Plugins are activated via the following "handshake" API call. + +### /Plugin.Activate + +**Request:** empty body + +**Response:** +``` +{ + "Implements": ["VolumeDriver"] +} +``` + +Responds with a list of Docker subsystems which this plugin implements. +After activation, the plugin will then be sent events from this subsystem. + +Possible values are: + +* [`authz`](plugins_authorization.md) +* [`NetworkDriver`](plugins_network.md) +* [`VolumeDriver`](plugins_volume.md) + + +## Plugin retries + +Attempts to call a method on a plugin are retried with an exponential backoff +for up to 30 seconds. This may help when packaging plugins as containers, since +it gives plugin containers a chance to start up before failing any user +containers which depend on them. + +## Plugins helpers + +To ease plugins development, we're providing an `sdk` for each kind of plugins +currently supported by Docker at [docker/go-plugins-helpers](https://github.com/docker/go-plugins-helpers). diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md new file mode 100644 index 0000000000..2c3d79e25d --- /dev/null +++ b/docs/extend/plugins_authorization.md @@ -0,0 +1,256 @@ + + + +# Create an authorization plugin + +This document describes the Docker Engine plugins generally available in Docker +Engine. To view information on plugins managed by Docker Engine currently in +experimental status, refer to [Docker Engine plugin system](index.md). + +Docker's out-of-the-box authorization model is all or nothing. Any user with +permission to access the Docker daemon can run any Docker client command. The +same is true for callers using Docker's remote API to contact the daemon. If you +require greater access control, you can create authorization plugins and add +them to your Docker daemon configuration. Using an authorization plugin, a +Docker administrator can configure granular access policies for managing access +to Docker daemon. + +Anyone with the appropriate skills can develop an authorization plugin. These +skills, at their most basic, are knowledge of Docker, understanding of REST, and +sound programming knowledge. This document describes the architecture, state, +and methods information available to an authorization plugin developer. + +## Basic principles + +Docker's [plugin infrastructure](plugin_api.md) enables +extending Docker by loading, removing and communicating with +third-party components using a generic API. The access authorization subsystem +was built using this mechanism. + +Using this subsystem, you don't need to rebuild the Docker daemon to add an +authorization plugin. You can add a plugin to an installed Docker daemon. You do +need to restart the Docker daemon to add a new plugin. + +An authorization plugin approves or denies requests to the Docker daemon based +on both the current authentication context and the command context. The +authentication context contains all user details and the authentication method. +The command context contains all the relevant request data. + +Authorization plugins must follow the rules described in [Docker Plugin API](plugin_api.md). +Each plugin must reside within directories described under the +[Plugin discovery](plugin_api.md#plugin-discovery) section. + +**Note**: the abbreviations `AuthZ` and `AuthN` mean authorization and authentication +respectively. + +## Default user authorization mechanism + +If TLS is enabled in the [Docker daemon](../security/https.md), the default user authorization flow extracts the user details from the certificate subject name. +That is, the `User` field is set to the client certificate subject common name, and the `AuthenticationMethod` field is set to `TLS`. + +## Basic architecture + +You are responsible for registering your plugin as part of the Docker daemon +startup. You can install multiple plugins and chain them together. This chain +can be ordered. Each request to the daemon passes in order through the chain. +Only when all the plugins grant access to the resource, is the access granted. + +When an HTTP request is made to the Docker daemon through the CLI or via the +remote API, the authentication subsystem passes the request to the installed +authentication plugin(s). The request contains the user (caller) and command +context. The plugin is responsible for deciding whether to allow or deny the +request. + +The sequence diagrams below depict an allow and deny authorization flow: + +![Authorization Allow flow](images/authz_allow.png) + +![Authorization Deny flow](images/authz_deny.png) + +Each request sent to the plugin includes the authenticated user, the HTTP +headers, and the request/response body. Only the user name and the +authentication method used are passed to the plugin. Most importantly, no user +credentials or tokens are passed. Finally, not all request/response bodies +are sent to the authorization plugin. Only those request/response bodies where +the `Content-Type` is either `text/*` or `application/json` are sent. + +For commands that can potentially hijack the HTTP connection (`HTTP +Upgrade`), such as `exec`, the authorization plugin is only called for the +initial HTTP requests. Once the plugin approves the command, authorization is +not applied to the rest of the flow. Specifically, the streaming data is not +passed to the authorization plugins. For commands that return chunked HTTP +response, such as `logs` and `events`, only the HTTP request is sent to the +authorization plugins. + +During request/response processing, some authorization flows might +need to do additional queries to the Docker daemon. To complete such flows, +plugins can call the daemon API similar to a regular user. To enable these +additional queries, the plugin must provide the means for an administrator to +configure proper authentication and security policies. + +## Docker client flows + +To enable and configure the authorization plugin, the plugin developer must +support the Docker client interactions detailed in this section. + +### Setting up Docker daemon + +Enable the authorization plugin with a dedicated command line flag in the +`--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` +value. This value can be the plugin’s socket or a path to a specification file. +Authorization plugins can be loaded without restarting the daemon. Refer +to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. + +```bash +$ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... +``` + +Docker's authorization subsystem supports multiple `--authorization-plugin` parameters. + +### Calling authorized command (allow) + +```bash +$ docker pull centos +... +f1b10cd84249: Pull complete +... +``` + +### Calling unauthorized command (deny) + +```bash +$ docker pull centos +... +docker: Error response from daemon: authorization denied by plugin PLUGIN_NAME: volumes are not allowed. +``` + +### Error from plugins + +```bash +$ docker pull centos +... +docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZPlugin.AuthZReq: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. +``` + +## API schema and implementation + +In addition to Docker's standard plugin registration method, each plugin +should implement the following two methods: + +* `/AuthZPlugin.AuthZReq` This authorize request method is called before the Docker daemon processes the client request. + +* `/AuthZPlugin.AuthZRes` This authorize response method is called before the response is returned from Docker daemon to the client. + +#### /AuthZPlugin.AuthZReq + +**Request**: + +```json +{ + "User": "The user identification", + "UserAuthNMethod": "The authentication method used", + "RequestMethod": "The HTTP method", + "RequestURI": "The HTTP request URI", + "RequestBody": "Byte array containing the raw HTTP request body", + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string " +} +``` + +**Response**: + +```json +{ + "Allow": "Determined whether the user is allowed or not", + "Msg": "The authorization message", + "Err": "The error message if things go wrong" +} +``` +#### /AuthZPlugin.AuthZRes + +**Request**: + +```json +{ + "User": "The user identification", + "UserAuthNMethod": "The authentication method used", + "RequestMethod": "The HTTP method", + "RequestURI": "The HTTP request URI", + "RequestBody": "Byte array containing the raw HTTP request body", + "RequestHeader": "Byte array containing the raw HTTP request header as a map[string][]string", + "ResponseBody": "Byte array containing the raw HTTP response body", + "ResponseHeader": "Byte array containing the raw HTTP response header as a map[string][]string", + "ResponseStatusCode":"Response status code" +} +``` + +**Response**: + +```json +{ + "Allow": "Determined whether the user is allowed or not", + "Msg": "The authorization message", + "Err": "The error message if things go wrong" +} +``` + +### Request authorization + +Each plugin must support two request authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. + +#### Daemon -> Plugin + +Name | Type | Description +-----------------------|-------------------|------------------------------------------------------- +User | string | The user identification +Authentication method | string | The authentication method used +Request method | enum | The HTTP method (GET/DELETE/POST) +Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) +Request headers | map[string]string | Request headers as key value pairs (without the authorization header) +Request body | []byte | Raw request body + + +#### Plugin -> Daemon + +Name | Type | Description +--------|--------|---------------------------------------------------------------------------------- +Allow | bool | Boolean value indicating whether the request is allowed or denied +Msg | string | Authorization message (will be returned to the client in case the access is denied) +Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) + +### Response authorization + +The plugin must support two authorization messages formats, one from the daemon to the plugin and then from the plugin to the daemon. The tables below detail the content expected in each message. + +#### Daemon -> Plugin + + +Name | Type | Description +----------------------- |------------------ |---------------------------------------------------- +User | string | The user identification +Authentication method | string | The authentication method used +Request method | string | The HTTP method (GET/DELETE/POST) +Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) +Request headers | map[string]string | Request headers as key value pairs (without the authorization header) +Request body | []byte | Raw request body +Response status code | int | Status code from the docker daemon +Response headers | map[string]string | Response headers as key value pairs +Response body | []byte | Raw docker daemon response body + + +#### Plugin -> Daemon + +Name | Type | Description +--------|--------|---------------------------------------------------------------------------------- +Allow | bool | Boolean value indicating whether the response is allowed or denied +Msg | string | Authorization message (will be returned to the client in case the access is denied) +Err | string | Error message (will be returned to the client in case the plugin encounter an error. The string value supplied may appear in logs, so should not include confidential information) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md new file mode 100644 index 0000000000..7a107e2ce3 --- /dev/null +++ b/docs/extend/plugins_network.md @@ -0,0 +1,73 @@ + + +# Engine network driver plugins + +This document describes Docker Engine network driver plugins generally +available in Docker Engine. To view information on plugins +managed by Docker Engine, refer to [Docker Engine plugin system](index.md). + +Docker Engine network plugins enable Engine deployments to be extended to +support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN +or something completely different. Network driver plugins are supported via the +LibNetwork project. Each plugin is implemented as a "remote driver" for +LibNetwork, which shares plugin infrastructure with Engine. Effectively, network +driver plugins are activated in the same way as other plugins, and use the same +kind of protocol. + +## Network driver plugins and swarm mode + +Docker 1.12 adds support for cluster management and orchestration called +[swarm mode](../swarm/index.md). Docker Engine running in swarm mode currently +only supports the built-in overlay driver for networking. Therefore existing +networking plugins will not work in swarm mode. + +When you run Docker Engine outside of swarm mode, all networking plugins that +worked in Docker 1.11 will continue to function normally. They do not require +any modification. + +## Using network driver plugins + +The means of installing and running a network driver plugin depend on the +particular plugin. So, be sure to install your plugin according to the +instructions obtained from the plugin developer. + +Once running however, network driver plugins are used just like the built-in +network drivers: by being mentioned as a driver in network-oriented Docker +commands. For example, + + $ docker network create --driver weave mynet + +Some network driver plugins are listed in [plugins](legacy_plugins.md) + +The `mynet` network is now owned by `weave`, so subsequent commands +referring to that network will be sent to the plugin, + + $ docker run --network=mynet busybox top + + +## Write a network plugin + +Network plugins implement the [Docker plugin +API](https://docs.docker.com/extend/plugin_api/) and the network plugin protocol + +## Network plugin protocol + +The network driver protocol, in addition to the plugin activation call, is +documented as part of libnetwork: +[https://github.com/docker/libnetwork/blob/master/docs/remote.md](https://github.com/docker/libnetwork/blob/master/docs/remote.md). + +# Related Information + +To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. + +- [Docker networks feature overview](../userguide/networking/index.md) +- The [LibNetwork](https://github.com/docker/libnetwork) project diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md new file mode 100644 index 0000000000..f9c93a3de1 --- /dev/null +++ b/docs/extend/plugins_volume.md @@ -0,0 +1,268 @@ + + +# Write a volume plugin + +Docker Engine volume plugins enable Engine deployments to be integrated with +external storage systems, such as Amazon EBS, and enable data volumes to persist +beyond the lifetime of a single Engine host. See the +[plugin documentation](legacy_plugins.md) for more information. + +## Changelog + +### 1.12.0 + +- Add `Status` field to `VolumeDriver.Get` response ([#21006](https://github.com/docker/docker/pull/21006#)) +- Add `VolumeDriver.Capabilities` to get capabilities of the volume driver([#22077](https://github.com/docker/docker/pull/22077)) + +### 1.10.0 + +- Add `VolumeDriver.Get` which gets the details about the volume ([#16534](https://github.com/docker/docker/pull/16534)) +- Add `VolumeDriver.List` which lists all volumes owned by the driver ([#16534](https://github.com/docker/docker/pull/16534)) + +### 1.8.0 + +- Initial support for volume driver plugins ([#14659](https://github.com/docker/docker/pull/14659)) + +## Command-line changes + +A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: + + $ docker run -ti -v volumename:/data --volume-driver=flocker busybox sh + +This command passes the `volumename` through to the volume plugin as a +user-given name for the volume. The `volumename` must not begin with a `/`. + +By having the user specify a `volumename`, a plugin can associate the volume +with an external volume beyond the lifetime of a single container or container +host. This can be used, for example, to move a stateful container from one +server to another. + +By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. + + +## Create a VolumeDriver + +The container creation endpoint (`/containers/create`) accepts a `VolumeDriver` +field of type `string` allowing to specify the name of the driver. It's default +value of `"local"` (the default driver for local volumes). + +## Volume plugin protocol + +If a plugin registers itself as a `VolumeDriver` when activated, then it is +expected to provide writeable paths on the host filesystem for the Docker +daemon to provide to containers to consume. + +The Docker daemon handles bind-mounting the provided paths into user +containers. + +> **Note**: Volume plugins should *not* write data to the `/var/lib/docker/` +> directory, including `/var/lib/docker/volumes`. The `/var/lib/docker/` +> directory is reserved for Docker. + +### /VolumeDriver.Create + +**Request**: +```json +{ + "Name": "volume_name", + "Opts": {} +} +``` + +Instruct the plugin that the user wants to create a volume, given a user +specified volume name. The plugin does not need to actually manifest the +volume on the filesystem yet (until Mount is called). +Opts is a map of driver specific options passed through from the user request. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a string error if an error occurred. + +### /VolumeDriver.Remove + +**Request**: +```json +{ + "Name": "volume_name" +} +``` + +Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a string error if an error occurred. + +### /VolumeDriver.Mount + +**Request**: +```json +{ + "Name": "volume_name", + "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" +} +``` + +Docker requires the plugin to provide a volume, given a user specified volume +name. This is called once per container start. If the same volume_name is requested +more than once, the plugin may need to keep track of each new mount request and provision +at the first mount request and deprovision at the last corresponding unmount request. + +`ID` is a unique ID for the caller that is requesting the mount. + +**Response**: +```json +{ + "Mountpoint": "/path/to/directory/on/host", + "Err": "" +} +``` + +Respond with the path on the host filesystem where the volume has been made +available, and/or a string error if an error occurred. + +### /VolumeDriver.Path + +**Request**: +```json +{ + "Name": "volume_name" +} +``` + +Docker needs reminding of the path to the volume on the host. + +**Response**: +```json +{ + "Mountpoint": "/path/to/directory/on/host", + "Err": "" +} +``` + +Respond with the path on the host filesystem where the volume has been made +available, and/or a string error if an error occurred. `Mountpoint` is optional, +however the plugin may be queried again later if one is not provided. + +### /VolumeDriver.Unmount + +**Request**: +```json +{ + "Name": "volume_name", + "ID": "b87d7442095999a92b65b3d9691e697b61713829cc0ffd1bb72e4ccd51aa4d6c" +} +``` + +Indication that Docker no longer is using the named volume. This is called once +per container stop. Plugin may deduce that it is safe to deprovision it at +this point. + +`ID` is a unique ID for the caller that is requesting the mount. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a string error if an error occurred. + + +### /VolumeDriver.Get + +**Request**: +```json +{ + "Name": "volume_name" +} +``` + +Get the volume info. + + +**Response**: +```json +{ + "Volume": { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host", + "Status": {} + }, + "Err": "" +} +``` + +Respond with a string error if an error occurred. `Mountpoint` and `Status` are +optional. + + +### /VolumeDriver.List + +**Request**: +```json +{} +``` + +Get the list of volumes registered with the plugin. + +**Response**: +```json +{ + "Volumes": [ + { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host" + } + ], + "Err": "" +} +``` + +Respond with a string error if an error occurred. `Mountpoint` is optional. + +### /VolumeDriver.Capabilities + +**Request**: +```json +{} +``` + +Get the list of capabilities the driver supports. +The driver is not required to implement this endpoint, however in such cases +the default values will be taken. + +**Response**: +```json +{ + "Capabilities": { + "Scope": "global" + } +} +``` + +Supported scopes are `global` and `local`. Any other value in `Scope` will be +ignored and assumed to be `local`. Scope allows cluster managers to handle the +volume differently, for instance with a scope of `global`, the cluster manager +knows it only needs to create the volume once instead of on every engine. More +capabilities may be added in the future. From da6fd41d88bc2fc6cb63a800e5e2dd72da3d25d9 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Fri, 14 Oct 2016 15:30:36 -0700 Subject: [PATCH 152/534] Convert Markdown frontmatter to YAML Some frontmatter such as the weights, menu stuff, etc is no longer used 'draft=true' becomes 'published: false' Signed-off-by: Misty Stanley-Jones --- docs/deprecated.md | 19 +++----- docs/extend/index.md | 19 +++----- docs/extend/legacy_plugins.md | 17 +++---- docs/extend/manifest.md | 67 +++++++++++++-------------- docs/extend/menu.md | 19 +++----- docs/extend/plugin_api.md | 15 ++---- docs/extend/plugins_authorization.md | 17 +++---- docs/extend/plugins_network.md | 15 ++---- docs/extend/plugins_volume.md | 15 ++---- docs/reference/commandline/dockerd.md | 17 +++---- 10 files changed, 85 insertions(+), 135 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a591880ef0..43dfc29275 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -1,14 +1,9 @@ - +--- +aliases: ["/engine/misc/deprecated/"] +title: "Deprecated Engine Features" +description: "Deprecated Features." +keywords: ["docker, documentation, about, technology, deprecate"] +--- # Deprecated Engine Features @@ -192,7 +187,7 @@ The single-dash (`-help`) was removed, in favor of the double-dash `--help` **Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/)** -The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor +The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor of the `--changes` flag that allows to pass `Dockerfile` commands. diff --git a/docs/extend/index.md b/docs/extend/index.md index 4ce1eac1c5..9ca4dd9c42 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,17 +1,12 @@ - +title: "Managed plugin system" +description: "How develop and use a plugin with the managed plugin system" +keywords: ["API, Usage, plugins, documentation, developer"] +advisory: "experimental" +--- # Docker Engine managed plugin system diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index aa4fa4a9b3..4e4d88a871 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -1,14 +1,9 @@ - +--- +aliases: "/engine/extend/plugins/" +title: "Use Docker Engine plugins" +description: "How to add additional functionality to Docker with plugins extensions" +keywords: ["Examples, Usage, plugins, docker, documentation, user guide"] +--- # Use Docker Engine plugins diff --git a/docs/extend/manifest.md b/docs/extend/manifest.md index d47efce8e8..e0fc3c3bec 100644 --- a/docs/extend/manifest.md +++ b/docs/extend/manifest.md @@ -1,17 +1,12 @@ - +title: "Plugin manifest" +description: "How develop and use a plugin with the managed plugin system" +keywords: ["API, Usage, plugins, documentation, developer"] +advisory: "experimental" +--- # Plugin Manifest Version 0 of Plugin V2 @@ -47,7 +42,7 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`interface`** *PluginInterface* interface implemented by the plugins, struct consisting of the following fields - + - **`types`** *string array* types indicate what interface(s) the plugin currently implements. @@ -55,9 +50,9 @@ Manifest provides the base accessible fields for working with V0 plugin format currently supported: - **docker.volumedriver/1.0** - + - **`socket`** *string* - + socket is the name of the socket the engine should use to communicate with the plugins. the socket will be created in `/run/docker/plugins`. @@ -73,7 +68,7 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`network`** *PluginNetwork* network of the plugin, struct consisting of the following fields - + - **`type`** *string* network type. @@ -83,11 +78,11 @@ Manifest provides the base accessible fields for working with V0 plugin format - **bridge** - **host** - **none** - + - **`capabilities`** *array* capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) - + - **`mounts`** *PluginMount array* mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) @@ -95,27 +90,27 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`name`** *string* name of the mount. - + - **`description`** *string* - + description of the mount. - + - **`source`** *string* source of the mount. - + - **`destination`** *string* destination of the mount. - + - **`type`** *string* mount type. - + - **`options`** *string array* options of the mount. - + - **`devices`** *PluginDevice array* device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) @@ -123,11 +118,11 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`name`** *string* name of the device. - + - **`description`** *string* description of the device. - + - **`path`** *string* path of the device. @@ -139,15 +134,15 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`name`** *string* name of the env. - + - **`description`** *string* - + description of the env. - + - **`value`** *string* value of the env. - + - **`args`** *PluginArgs* args of the plugin, struct consisting of the following fields @@ -155,16 +150,16 @@ Manifest provides the base accessible fields for working with V0 plugin format - **`name`** *string* name of the env. - + - **`description`** *string* - + description of the env. - + - **`value`** *string array* values of the args. - - + + ## Example Manifest *Example showing the 'tiborvass/no-remove' plugin manifest.* diff --git a/docs/extend/menu.md b/docs/extend/menu.md index 4b9955038a..8403872625 100644 --- a/docs/extend/menu.md +++ b/docs/extend/menu.md @@ -1,15 +1,10 @@ - +--- +title: "Implement plugins" +description: "Develop plugins and use existing plugins for Docker Engine" +keywords: ["extend, plugins, docker, documentation, developer"] +type: "menu" +identifier: "engine_extend" +--- diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 5801ba9151..edf05872cf 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -1,13 +1,8 @@ - +--- +title: "Plugins API" +description: "How to write Docker plugins extensions " +keywords: ["API, Usage, plugins, documentation, developer"] +--- # Docker Plugin API diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 2c3d79e25d..77749547f5 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -1,14 +1,9 @@ - +--- +title: "Access authorization plugin" +description: "How to create authorization plugins to manage access control to your Docker daemon." +keywords: ["security, authorization, authentication, docker, documentation, plugin, extend"] +aliases: ["/engine/extend/authorization/"] +--- # Create an authorization plugin diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 7a107e2ce3..5313be8402 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -1,13 +1,8 @@ - +--- +title: "Docker network driver plugins" +description: "Network driver plugins." +keywords: ["Examples, Usage, plugins, docker, documentation, user guide"] +--- # Engine network driver plugins diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index f9c93a3de1..2f53047f33 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -1,13 +1,8 @@ - +--- +title: "Volume plugins" +description: "How to manage data with external volume plugins" +keywords: ["Examples, Usage, volume, docker, data, volumes, plugin, api"] +--- # Write a volume plugin diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 0233476ce8..6e020ff6ad 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1,14 +1,9 @@ - +--- +title: "dockerd" +aliases: ["/engine/reference/commandline/daemon/"] +description: "The daemon command description and usage" +keywords: ["container, daemon, runtime"] +--- # daemon From 3432e4d835cb6d582ed09c16769f6d8b5afee80f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sun, 16 Oct 2016 21:13:26 -0700 Subject: [PATCH 153/534] Remove Fedora 22 from RPM build as it is EOL Fedora 22 reached end of life on July 19th, and will no longer receive updates; https://fedoramagazine.org/fedora-22-end-of-life-2016-july/ With the recent release of Fedora 24, Fedora 22 will officially enter End Of Life (EOL) status on July 19th, 2016. After July 19th, all packages in the Fedora 22 repositories will no longer receive security, bugfix, or enhancement updates, and no new packages will be added to the Fedora 22 collection. Upgrading to Fedora 23 or Fedora 24 before July 19th 2016 is highly recommended for all users still running Fedora 22. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6e020ff6ad..37676b9652 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -949,11 +949,6 @@ these resources are name-based, not id-based. If the numeric ID information provided does not exist as entries in `/etc/passwd` or `/etc/group`, daemon startup will fail with an error message. -> **Note:** On Fedora 22, you have to `touch` the `/etc/subuid` and `/etc/subgid` -> files to have ranges assigned when users are created. This must be done -> *before* the `--userns-remap` option is enabled. Once these files exist, the -> daemon can be (re)started and range assignment on user creation works properly. - **Example: starting with default Docker user management:** ```bash From 74489d8d81281e9452cd5e76fa6f2758414dc073 Mon Sep 17 00:00:00 2001 From: lixiaobing10051267 Date: Mon, 17 Oct 2016 17:25:58 +0800 Subject: [PATCH 154/534] fix wrong output titile for docker volume ls Signed-off-by: lixiaobing10051267 --- docs/extend/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 9ca4dd9c42..e8b964ba15 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -97,7 +97,7 @@ started without error. ```bash $ docker volume ls - DRIVER NAME + DRIVER VOLUME NAME vieux/sshfs sshvolume ``` From 15f9c65d65739536929e3e33fffbd210c789973b Mon Sep 17 00:00:00 2001 From: Ding Fei Date: Tue, 18 Oct 2016 18:01:46 +0800 Subject: [PATCH 155/534] Fix code block fences and typo. Signed-off-by: Ding Fei --- docs/extend/index.md | 8 ++++---- docs/extend/plugin_api.md | 1 + 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index e8b964ba15..661bfc9f71 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -36,7 +36,7 @@ Hub or on a private registry. You install the plugin using a single command: `docker plugin install `. The `plugin install` command pulls the plugin from the Docker Hub or private -registry. If necessary the CLI prompts you to accept any privilige requriements. +registry. If necessary the CLI prompts you to accept any privilege requriements. For example the plugin may require access to a device on the host system. Finally it enables the plugin. @@ -63,7 +63,7 @@ to create a volume. The plugin requests 2 privileges, the `CAP_SYS_ADMIN` capability to be able to do mount inside the plugin and `host networking`. -2. Check for a value of `true` the `ENABLED` column to verify the plugin +2. Check for a value of `true` the `ENABLED` column to verify the plugin started without error. ```bash @@ -73,7 +73,7 @@ started without error. vieux/sshfs latest true ``` -3. Create a volume using the plugin. +3. Create a volume using the plugin. ```bash $ docker volume create \ @@ -92,7 +92,7 @@ started without error. ``` -5. Verify the plugin successfully created the volume. +5. Verify the plugin successfully created the volume. ```bash $ docker volume ls diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index edf05872cf..8e796e569b 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -121,6 +121,7 @@ ExecStart=/usr/lib/docker/your-plugin WantedBy=multi-user.target ``` The `socket` file (for example `/lib/systemd/system/your-plugin.socket`): + ``` [Unit] Description=Your plugin From e555bd0063b8bf8203c52c12d7a3a1ad67185edb Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Mon, 17 Oct 2016 22:55:45 +0800 Subject: [PATCH 156/534] Fix link for feature deprecation policy Signed-off-by: yuexiao-wang --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 43dfc29275..a2183f99e7 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -9,7 +9,7 @@ keywords: ["docker, documentation, about, technology, deprecate"] The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, -see [Feature Deprecation Policy](index.md#feature-deprecation-policy). +see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). ### `repository:shortid` image references From 8456d6ae12a81aa5382fba9e86229cf4ad015adb Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Thu, 26 May 2016 14:07:30 -0700 Subject: [PATCH 157/534] Add config parameter to change stop timeout during daemon shutdown This fix tries to add a daemon config parameter `--shutdown-timeout` that specifies the timeout value to stop containers gracefully (before SIGKILL). The default value is 15s. The `--shutdown-timeout` parameter is added to daemon options and config file. It will also be updated during daemon reload. Additional test cases have been added to cover the change. This fix fixes reference 22471. Signed-off-by: Yong Tang --- docs/reference/commandline/dockerd.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 37676b9652..3bdf2a47ce 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -64,6 +64,7 @@ Options: --raw-logs Full timestamps without ANSI coloring --registry-mirror value Preferred Docker registry mirror (default []) --selinux-enabled Enable selinux support + --shutdown-timeout=15 Set the shutdown timeout value in seconds -s, --storage-driver string Storage driver to use --storage-opt value Storage driver options (default []) --swarm-default-advertise-addr string Set default address or interface for swarm advertised address @@ -1118,6 +1119,7 @@ This is a full example of the allowed configuration options on Linux: "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, + "shutdown-timeout": 15, "debug": true, "hosts": [], "log-level": "", @@ -1194,6 +1196,7 @@ This is a full example of the allowed configuration options on Windows: "graph": "", "cluster-store": "", "cluster-advertise": "", + "shutdown-timeout": 15, "debug": true, "hosts": [], "log-level": "", From 03029668c6daee2fcd781b25212ed025ee20e8d7 Mon Sep 17 00:00:00 2001 From: allencloud Date: Mon, 25 Apr 2016 10:51:28 +0800 Subject: [PATCH 158/534] support insecure registry in configuration reload Signed-off-by: allencloud --- docs/reference/commandline/dockerd.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3bdf2a47ce..10e782e878 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1241,12 +1241,13 @@ The list of currently supported options that can be reconfigured is this: - `runtimes`: it updates the list of available OCI runtimes that can be used to run containers - `authorization-plugin`: specifies the authorization plugins to use. +- `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if these configurations were not previously configured. If `--cluster-store` has been provided in flags and `cluster-advertise` not, `cluster-advertise` -can be added in the configuration file without accompanied by `--cluster-store` +can be added in the configuration file without accompanied by `--cluster-store`. Configuration reload will log a warning message if it detects a change in previously configured cluster configurations. From be5334e561b7360375a531b7ef625718ab2a4894 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Wed, 19 Oct 2016 10:25:45 -0700 Subject: [PATCH 159/534] Sync docker/docker refs with files mistakenly edited in docker.github.io repo Signed-off-by: Misty Stanley-Jones --- docs/deprecated.md | 9 +++ docs/extend/index.md | 9 +++ docs/extend/legacy_plugins.md | 9 +++ docs/extend/manifest.md | 9 +++ docs/extend/menu.md | 8 +++ docs/extend/plugin_api.md | 9 +++ docs/extend/plugins_authorization.md | 8 +++ docs/extend/plugins_network.md | 9 +++ docs/extend/plugins_volume.md | 9 +++ docs/reference/commandline/dockerd.md | 83 +++++++++++++++------------ 10 files changed, 125 insertions(+), 37 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a2183f99e7..8f767a4ba5 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -5,6 +5,15 @@ description: "Deprecated Features." keywords: ["docker, documentation, about, technology, deprecate"] --- + + # Deprecated Engine Features The following list of features are deprecated in Engine. diff --git a/docs/extend/index.md b/docs/extend/index.md index 661bfc9f71..8316307cee 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -8,6 +8,15 @@ keywords: ["API, Usage, plugins, documentation, developer"] advisory: "experimental" --- + + # Docker Engine managed plugin system This document describes the plugin system available today in the **experimental diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 4e4d88a871..725dde9bdd 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -5,6 +5,15 @@ description: "How to add additional functionality to Docker with plugins extensi keywords: ["Examples, Usage, plugins, docker, documentation, user guide"] --- + + # Use Docker Engine plugins This document describes the Docker Engine plugins generally available in Docker diff --git a/docs/extend/manifest.md b/docs/extend/manifest.md index e0fc3c3bec..dab94a67b6 100644 --- a/docs/extend/manifest.md +++ b/docs/extend/manifest.md @@ -8,6 +8,15 @@ keywords: ["API, Usage, plugins, documentation, developer"] advisory: "experimental" --- + + # Plugin Manifest Version 0 of Plugin V2 This document outlines the format of the V0 plugin manifest. The plugin diff --git a/docs/extend/menu.md b/docs/extend/menu.md index 8403872625..756f371b70 100644 --- a/docs/extend/menu.md +++ b/docs/extend/menu.md @@ -6,5 +6,13 @@ type: "menu" identifier: "engine_extend" --- + diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 8e796e569b..7b75c563e2 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -4,6 +4,15 @@ description: "How to write Docker plugins extensions " keywords: ["API, Usage, plugins, documentation, developer"] --- + + # Docker Plugin API Docker plugins are out-of-process extensions which add capabilities to the diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 77749547f5..1b0f83b510 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -5,6 +5,14 @@ keywords: ["security, authorization, authentication, docker, documentation, plug aliases: ["/engine/extend/authorization/"] --- + # Create an authorization plugin diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 5313be8402..9546195bbd 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -4,6 +4,15 @@ description: "Network driver plugins." keywords: ["Examples, Usage, plugins, docker, documentation, user guide"] --- + + # Engine network driver plugins This document describes Docker Engine network driver plugins generally diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 2f53047f33..e4a867256e 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -4,6 +4,15 @@ description: "How to manage data with external volume plugins" keywords: ["Examples, Usage, volume, docker, data, volumes, plugin, api"] --- + + # Write a volume plugin Docker Engine volume plugins enable Engine deployments to be integrated with diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 10e782e878..944bbc6b45 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -5,6 +5,15 @@ description: "The daemon command description and usage" keywords: ["container, daemon, runtime"] --- + + # daemon ```markdown @@ -279,7 +288,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### Devicemapper options -* `dm.thinpooldev` +* `dm.thinpooldev` Specifies a custom block storage device to use for the thin pool. @@ -306,7 +315,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool ``` -* `dm.basesize` +* `dm.basesize` Specifies the size to use when creating the base device, which limits the size of images and containers. The default value is 10G. Note, thin devices @@ -324,7 +333,6 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.basesize=50G ``` - This will increase the base device size to 50G. The Docker daemon will throw an error if existing base device size is larger than 50G. A user can use this option to expand the base device size however shrinking is not permitted. @@ -345,7 +353,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.basesize=20G ``` -* `dm.loopdatasize` +* `dm.loopdatasize` > **Note**: > This option configures devicemapper loopback, which should not @@ -362,7 +370,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.loopdatasize=200G ``` -* `dm.loopmetadatasize` +* `dm.loopmetadatasize` > **Note**: > This option configures devicemapper loopback, which should not @@ -379,7 +387,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.loopmetadatasize=4G ``` -* `dm.fs` +* `dm.fs` Specifies the filesystem type to use for the base device. The supported options are "ext4" and "xfs". The default is "xfs" @@ -390,7 +398,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.fs=ext4 ``` -* `dm.mkfsarg` +* `dm.mkfsarg` Specifies extra mkfs arguments to be used when creating the base device. @@ -400,7 +408,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" ``` -* `dm.mountopt` +* `dm.mountopt` Specifies extra mount options used when mounting the thin devices. @@ -410,7 +418,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.mountopt=nodiscard ``` -* `dm.datadev` +* `dm.datadev` (Deprecated, use `dm.thinpooldev`) @@ -428,7 +436,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. --storage-opt dm.metadatadev=/dev/sdc1 ``` -* `dm.metadatadev` +* `dm.metadatadev` (Deprecated, use `dm.thinpooldev`) @@ -452,7 +460,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. --storage-opt dm.metadatadev=/dev/sdc1 ``` -* `dm.blocksize` +* `dm.blocksize` Specifies a custom blocksize to use for the thin pool. The default blocksize is 64K. @@ -463,7 +471,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.blocksize=512K ``` -* `dm.blkdiscard` +* `dm.blkdiscard` Enables or disables the use of blkdiscard when removing devicemapper devices. This is enabled by default (only) if using loopback devices and is @@ -479,7 +487,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.blkdiscard=false ``` -* `dm.override_udev_sync_check` +* `dm.override_udev_sync_check` Overrides the `udev` synchronization checks between `devicemapper` and `udev`. `udev` is the device manager for the Linux kernel. @@ -519,7 +527,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. > Otherwise, set this flag for migrating existing Docker daemons to > a daemon with a supported environment. -* `dm.use_deferred_removal` +* `dm.use_deferred_removal` Enables use of deferred device removal if `libdm` and the kernel driver support the mechanism. @@ -541,7 +549,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. $ sudo dockerd --storage-opt dm.use_deferred_removal=true ``` -* `dm.use_deferred_deletion` +* `dm.use_deferred_deletion` Enables use of deferred device deletion for thin pool devices. By default, thin pool device deletion is synchronous. Before a container is deleted, @@ -567,7 +575,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. when unintentional leaking of mount point happens across multiple mount namespaces. -* `dm.min_free_space` +* `dm.min_free_space` Specifies the min free space percent in a thin pool require for new device creation to succeed. This check applies to both free data space as well @@ -615,7 +623,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### ZFS options -* `zfs.fsname` +* `zfs.fsname` Set zfs filesystem under which docker will create its own datasets. By default docker will pick up the zfs filesystem where docker graph @@ -629,9 +637,9 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### Btrfs options -* `btrfs.min_space` +* `btrfs.min_space` - Specifies the mininum size to use when creating the subvolume which is used + Specifies the minimum size to use when creating the subvolume which is used for containers. If user uses disk quota for btrfs when creating or running a container with **--storage-opt size** option, docker should ensure the **size** cannot be smaller than **btrfs.min_space**. @@ -644,7 +652,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### Overlay2 options -* `overlay2.override_kernel_check` +* `overlay2.override_kernel_check` Overrides the Linux kernel version check allowing overlay2. Support for specifying multiple lower directories needed by overlay2 was added to the @@ -672,19 +680,20 @@ Runtimes can be registered with the daemon either via the configuration file or using the `--add-runtime` command line argument. The following is an example adding 2 runtimes via the configuration: + ```json - "default-runtime": "runc", - "runtimes": { - "runc": { - "path": "runc" - }, - "custom": { - "path": "/usr/local/bin/my-runc-replacement", - "runtimeArgs": [ - "--debug" - ] - } +"default-runtime": "runc", +"runtimes": { + "runc": { + "path": "runc" + }, + "custom": { + "path": "/usr/local/bin/my-runc-replacement", + "runtimeArgs": [ + "--debug" + ] } +} ``` This is the same example via the command line: @@ -844,35 +853,35 @@ $ sudo dockerd \ The currently supported cluster store options are: -* `discovery.heartbeat` +* `discovery.heartbeat` Specifies the heartbeat timer in seconds which is used by the daemon as a keepalive mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. -* `discovery.ttl` +* `discovery.ttl` Specifies the ttl (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. -* `kv.cacertfile` +* `kv.cacertfile` Specifies the path to a local file with PEM encoded CA certificates to trust -* `kv.certfile` +* `kv.certfile` Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. -* `kv.keyfile` +* `kv.keyfile` Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. -* `kv.path` +* `kv.path` Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. From 95e65ae675ce959c17b115558728dc371bd5441c Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Thu, 20 Oct 2016 11:12:02 -0700 Subject: [PATCH 160/534] Rewrites to Developing Plugins topic Signed-off-by: Misty Stanley-Jones --- docs/extend/index.md | 217 ++++++++++++++++++++++++------------------- 1 file changed, 122 insertions(+), 95 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 8316307cee..e0689f776b 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,11 +1,11 @@ --- -aliases: [ -"/engine/extend/" -] -title: "Managed plugin system" -description: "How develop and use a plugin with the managed plugin system" -keywords: ["API, Usage, plugins, documentation, developer"] -advisory: "experimental" +advisory: experimental +aliases: +- /engine/extend/ +description: Develop and use a plugin with the managed plugin system +keywords: +- API, Usage, plugins, documentation, developer +title: Managed plugin system --- -# Plugin Manifest Version 0 of Plugin V2 +# Plugin Config Version 0 of Plugin V2 -This document outlines the format of the V0 plugin manifest. The plugin -manifest described herein was introduced in the Docker daemon (experimental version) in the [v1.12.0 +This document outlines the format of the V0 plugin config. The plugin +config described herein was introduced in the Docker daemon (experimental version) in the [v1.12.0 release](https://github.com/docker/docker/commit/f37117045c5398fd3dca8016ea8ca0cb47e7312b). -Plugin manifests describe the various constituents of a docker plugin. Plugin -manifests can be serialized to JSON format with the following media types: +Plugin configs describe the various constituents of a docker plugin. Plugin +configs can be serialized to JSON format with the following media types: -Manifest Type | Media Type +Config Type | Media Type ------------- | ------------- -manifest | "application/vnd.docker.plugin.v0+json" +config | "application/vnd.docker.plugin.v0+json" -## *Manifest* Field Descriptions +## *Config* Field Descriptions -Manifest provides the base accessible fields for working with V0 plugin format +Config provides the base accessible fields for working with V0 plugin format in the registry. -- **`manifestVersion`** *string* - - version of the plugin manifest (This version uses V0) - - **`description`** *string* description of the plugin @@ -169,13 +165,13 @@ Manifest provides the base accessible fields for working with V0 plugin format values of the args. -## Example Manifest +## Example Config -*Example showing the 'tiborvass/no-remove' plugin manifest.* +*Example showing the 'tiborvass/no-remove' plugin config.* ``` { - "manifestVersion": "v0", + "configVersion": "v0", "description": "A test plugin for Docker", "documentation": "https://docs.docker.com/engine/extend/plugins/", "entrypoint": ["plugin-no-remove", "/data"], diff --git a/docs/extend/index.md b/docs/extend/index.md index 736e225520..7c82d56928 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -141,7 +141,7 @@ a `plugins.json` with a single plugin installed. { "cd851ce43a403": { "plugin": { - "Manifest": { + "Config": { "Args": { "Value": null, "Settable": null, @@ -154,7 +154,6 @@ a `plugins.json` with a single plugin installed. "Capabilities": [ "CAP_SYS_ADMIN" ], - "ManifestVersion": "v0", "Description": "sshFS plugin for Docker", "Documentation": "https://docs.docker.com/engine/extend/plugins/", "Interface": { @@ -196,8 +195,8 @@ and two JSON files. # ls -la /var/lib/docker/plugins/cd851ce43a403 total 12 drwx------ 19 root root 4096 Aug 8 17:56 rootfs --rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-config.json --rw------- 1 root root 347 Aug 8 17:56 manifest.json +-rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-settings.json +-rw------- 1 root root 347 Aug 8 17:56 config.json ``` #### The rootfs directory @@ -219,17 +218,16 @@ $ docker rm -vf "$id" $ docker rmi rootfs ``` -#### The manifest.json and plugin-config.json files +#### The config.json and plugin-settings.json files -The `manifest.json` file describes the plugin. The `plugin-config.json` file +The `config.json` file describes the plugin. The `plugin-settings.json` file contains runtime parameters and is only required if your plugin has runtime -parameters. [See the Plugins Manifest reference](manifest.md). +parameters. [See the Plugins Config reference](config.md). -Consider the following `manifest.json` file. +Consider the following `config.json` file. ```json { - "manifestVersion": "v0", "description": "sshFS plugin for Docker", "documentation": "https://docs.docker.com/engine/extend/plugins/", "entrypoint": ["/go/bin/docker-volume-sshfs"], @@ -250,7 +248,7 @@ entrypoint and uses the `/run/docker/plugins/sshfs.sock` socket to communicate with Docker Engine. -Consider the following `plugin-config.json` file. +Consider the following `plugin-settings.json` file. ```json { @@ -264,8 +262,8 @@ Consider the following `plugin-config.json` file. This plugin has no runtime parameters. Each of these JSON files is included as part of `plugins.json`, as you can see -by looking back at the example above. After a plugin is installed, `manifest.json` -is read-only, but `plugin-config.json` is read-write, and includes all runtime +by looking back at the example above. After a plugin is installed, `config.json` +is read-only, but `plugin-settings.json` is read-write, and includes all runtime configuration options for the plugin. ### Creating the plugin @@ -279,9 +277,9 @@ Follow these steps to create a plugin: using `docker export`. See [The rootfs directory](#the-rootfs-directory) for an example of creating a `rootfs`. -3. Create a `manifest.json` file in `/var/lib/docker/plugins/$id/`. +3. Create a `config.json` file in `/var/lib/docker/plugins/$id/`. -4. Create a `plugin-config.json` file if needed. +4. Create a `plugin-settings.json` file if needed. 5. Create or add a section to `/var/lib/docker/plugins/plugins.json`. Use `/` as “Name” and `$id` as “Id”. From 58ab2c697d77c87263d2eece9028f42d6d66e7d9 Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Wed, 9 Nov 2016 17:49:09 -0800 Subject: [PATCH 175/534] move plugins out of experimental Signed-off-by: Victor Vieux --- docs/extend/config.md | 6 ++---- docs/extend/index.md | 4 ---- docs/extend/legacy_plugins.md | 4 ++-- docs/extend/plugin_api.md | 3 +-- docs/extend/plugins_authorization.md | 4 ++-- 5 files changed, 7 insertions(+), 14 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 91dd281475..31c785c8e5 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -5,7 +5,6 @@ aliases: [ title: "Plugin config" description: "How develop and use a plugin with the managed plugin system" keywords: "API, Usage, plugins, documentation, developer" -advisory: "experimental" --- + # Plugin Config Version 0 of Plugin V2 This document outlines the format of the V0 plugin configuration. The plugin @@ -85,10 +86,6 @@ Config provides the base accessible fields for working with V0 plugin format - **host** - **none** -- **`capabilities`** *array* - - capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) - - **`mounts`** *PluginMount array* mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) @@ -117,22 +114,6 @@ Config provides the base accessible fields for working with V0 plugin format options of the mount. -- **`devices`** *PluginDevice array* - - device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) - - - **`name`** *string* - - name of the device. - - - **`description`** *string* - - description of the device. - - - **`path`** *string* - - path of the device. - - **`env`** *PluginEnv array* env of the plugin, struct consisting of the following fields @@ -165,6 +146,27 @@ Config provides the base accessible fields for working with V0 plugin format values of the args. +- **`linux`** *PluginLinux* + + - **`capabilities`** *string array* + + capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) + + - **`devices`** *PluginDevice array* + + device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) + + - **`name`** *string* + + name of the device. + + - **`description`** *string* + + description of the device. + + - **`path`** *string* + + path of the device. ## Example Config From ab466cec326acca5051c165dab1cfebde6a829bd Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Fri, 25 Nov 2016 04:07:06 +0800 Subject: [PATCH 187/534] Add options for docker plugin enable and fix some typos Signed-off-by: yuexiao-wang --- docs/reference/commandline/dockerd.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 0c7a2f0019..b8bc8b712c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -22,7 +22,6 @@ Usage: dockerd [OPTIONS] A self-sufficient runtime for containers. Options: - --add-runtime value Register an additional OCI compatible runtime (default []) --api-cors-header string Set CORS headers in the Engine API --authorization-plugin value Authorization plugins to load (default []) From 10132c8e7aee5dc1ca296987319407032f952c97 Mon Sep 17 00:00:00 2001 From: lixiaobing10051267 Date: Fri, 25 Nov 2016 15:23:48 +0800 Subject: [PATCH 188/534] modify invalid url in plugins_network.md Signed-off-by: lixiaobing10051267 --- docs/extend/plugins_network.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index e551fc9d66..a974862fa6 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -61,7 +61,7 @@ referring to that network will be sent to the plugin, ## Write a network plugin Network plugins implement the [Docker plugin -API](https://docs.docker.com/extend/plugin_api/) and the network plugin protocol +API](plugin_api.md) and the network plugin protocol ## Network plugin protocol From a1122bd73da789941b9cc4374a9c3f82ac9280b0 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Mon, 28 Nov 2016 09:45:00 -0800 Subject: [PATCH 189/534] Fix some redirect loops and problems Signed-off-by: Misty Stanley-Jones --- docs/extend/config.md | 3 --- docs/extend/index.md | 2 -- docs/extend/legacy_plugins.md | 3 ++- docs/extend/menu.md | 18 ------------------ docs/extend/plugins_authorization.md | 3 ++- 5 files changed, 4 insertions(+), 25 deletions(-) delete mode 100644 docs/extend/menu.md diff --git a/docs/extend/config.md b/docs/extend/config.md index 53f52f7b66..4aa4173e1a 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -1,7 +1,4 @@ --- -aliases: [ -"/engine/extend/" -] title: "Plugin config" description: "How develop and use a plugin with the managed plugin system" keywords: "API, Usage, plugins, documentation, developer" diff --git a/docs/extend/index.md b/docs/extend/index.md index 66a0f07f3b..e00081bffa 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,6 +1,4 @@ --- -aliases: -- /engine/extend/ description: Develop and use a plugin with the managed plugin system keywords: "API, Usage, plugins, documentation, developer" title: Managed plugin system diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index d8dd4d2e48..6ac914e366 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -1,5 +1,6 @@ --- -aliases: "/engine/extend/plugins/" +redirect_from: +- "/engine/extend/plugins/" title: "Use Docker Engine plugins" description: "How to add additional functionality to Docker with plugins extensions" keywords: "Examples, Usage, plugins, docker, documentation, user guide" diff --git a/docs/extend/menu.md b/docs/extend/menu.md deleted file mode 100644 index 756f371b70..0000000000 --- a/docs/extend/menu.md +++ /dev/null @@ -1,18 +0,0 @@ ---- -title: "Implement plugins" -description: "Develop plugins and use existing plugins for Docker Engine" -keywords: ["extend, plugins, docker, documentation, developer"] -type: "menu" -identifier: "engine_extend" ---- - - - - diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 2f2328bb0f..ac1837f754 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -2,7 +2,8 @@ title: "Access authorization plugin" description: "How to create authorization plugins to manage access control to your Docker daemon." keywords: "security, authorization, authentication, docker, documentation, plugin, extend" -aliases: ["/engine/extend/authorization/"] +redirect_from: +- "/engine/extend/authorization/" --- -# Plugin Config Version 0 of Plugin V2 +# Plugin Config Version 1 of Plugin V2 This document outlines the format of the V0 plugin configuration. The plugin config described herein was introduced in the Docker daemon in the [v1.12.0 @@ -25,7 +25,7 @@ configs can be serialized to JSON format with the following media types: Config Type | Media Type ------------- | ------------- -config | "application/vnd.docker.plugin.v0+json" +config | "application/vnd.docker.plugin.v1+json" ## *Config* Field Descriptions From edd0e36869fdeeca88044336d165dc6024148368 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 15 Nov 2016 15:41:52 +0100 Subject: [PATCH 196/534] deprecate "top-level" network information in NetworkSettings When inspecting a container, `NetworkSettings` contains top-level information about the default ("bridge") network; `EndpointID`, `Gateway`, `GlobalIPv6Address`, `GlobalIPv6PrefixLen`, `IPAddress`, `IPPrefixLen`, `IPv6Gateway`, and `MacAddress`. These properties are deprecated in favor of per-network properties in `NetworkSettings.Networks`. These properties were already "deprecated" in docker 1.9, but kept around for backward compatibility. Refer to [reference 17538](https://github.com/docker/docker/pull/17538) for further information. This officially deprecates these properties, and marks them for removal in 1.16 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b8e6af1116..1298370ba9 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -20,6 +20,26 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). + +### Top-level network properties in NetworkSettings + +**Deprecated In Release: v1.13.0** + +**Target For Removal In Release: v1.16** + +When inspecting a container, `NetworkSettings` contains top-level information +about the default ("bridge") network; + +`EndpointID`, `Gateway`, `GlobalIPv6Address`, `GlobalIPv6PrefixLen`, `IPAddress`, +`IPPrefixLen`, `IPv6Gateway`, and `MacAddress`. + +These properties are deprecated in favor of per-network properties in +`NetworkSettings.Networks`. These properties were already "deprecated" in +docker 1.9, but kept around for backward compatibility. + +Refer to [#17538](https://github.com/docker/docker/pull/17538) for further +information. + ## `filter` param for `/images/json` endpoint **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** From d44d4459aff2864cc8828fdb02194a9c1cdf9ec8 Mon Sep 17 00:00:00 2001 From: Hongbin Lu Date: Thu, 15 Dec 2016 20:00:34 -0600 Subject: [PATCH 197/534] Add Fuxi Volume Plugin to the docs Signed-off-by: Hongbin Lu --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 6ac914e366..e9ab6f40be 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -65,6 +65,7 @@ Plugin [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. [DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. [Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. +[Fuxi Volume Plugin](https://github.com/openstack/fuxi) | A volume plugin that is developed as part of the OpenStack Kuryr project and implements the Docker volume plugin API by utilizing Cinder, the OpenStack block storage service. [gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. From 5e755d190df130f0e6734769256f8b83ce887554 Mon Sep 17 00:00:00 2001 From: Zhang Wei Date: Sat, 17 Dec 2016 22:01:59 +0800 Subject: [PATCH 198/534] Add missing "--default-runtime" flag in manpage Add missing flag and more descriptions in manpage. Signed-off-by: Zhang Wei --- docs/reference/commandline/dockerd.md | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b8bc8b712c..24ac77611d 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -684,16 +684,18 @@ configuration file or using the `--add-runtime` command line argument. The following is an example adding 2 runtimes via the configuration: ```json -"default-runtime": "runc", -"runtimes": { - "runc": { - "path": "runc" - }, - "custom": { - "path": "/usr/local/bin/my-runc-replacement", - "runtimeArgs": [ - "--debug" - ] +{ + "default-runtime": "runc", + "runtimes": { + "runc": { + "path": "runc" + }, + "custom": { + "path": "/usr/local/bin/my-runc-replacement", + "runtimeArgs": [ + "--debug" + ] + } } } ``` From 544745a4d8f271ebe87d5cb57d842303d1dc4438 Mon Sep 17 00:00:00 2001 From: Tonis Tiigi Date: Mon, 12 Dec 2016 15:05:53 -0800 Subject: [PATCH 199/534] Implement content addressability for plugins Move plugins to shared distribution stack with images. Create immutable plugin config that matches schema2 requirements. Ensure data being pushed is same as pulled/created. Store distribution artifacts in a blobstore. Run init layer setup for every plugin start. Fix breakouts from unsafe file accesses. Add support for `docker plugin install --alias` Uses normalized references for default names to avoid collisions when using default hosts/tags. Some refactoring of the plugin manager to support the change, like removing the singleton manager and adding manager config struct. Signed-off-by: Tonis Tiigi Signed-off-by: Derek McGowan --- docs/extend/index.md | 161 ++++--------------------------------------- 1 file changed, 15 insertions(+), 146 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index e00081bffa..fc31e74ce4 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -109,93 +109,6 @@ commands and options, see the ## Developing a plugin -Currently, there are no CLI commands available to help you develop a plugin. -This is expected to change in a future release. The manual process for creating -plugins is described in this section. - -### Plugin location and files - -Plugins are stored in `/var/lib/docker/plugins`. The `plugins.json` file lists -each plugin's configuration, and each plugin is stored in a directory with a -unique identifier. - -```bash -# ls -la /var/lib/docker/plugins -total 20 -drwx------ 4 root root 4096 Aug 8 18:03 . -drwx--x--x 12 root root 4096 Aug 8 17:53 .. -drwxr-xr-x 3 root root 4096 Aug 8 17:56 cd851ce43a403 --rw------- 1 root root 2107 Aug 8 18:03 plugins.json -``` - -### Format of plugins.json - -The `plugins.json` is an inventory of all installed plugins. This example shows -a `plugins.json` with a single plugin installed. - -```json -# cat plugins.json -{ - "cd851ce43a403": { - "plugin": { - "Config": { - "Args": { - "Value": null, - "Settable": null, - "Description": "", - "Name": "" - }, - "Env": null, - "Devices": null, - "Mounts": null, - "Capabilities": [ - "CAP_SYS_ADMIN" - ], - "Description": "sshFS plugin for Docker", - "Documentation": "https://docs.docker.com/engine/extend/plugins/", - "Interface": { - "Socket": "sshfs.sock", - "Types": [ - "docker.volumedriver/1.0" - ] - }, - "Entrypoint": [ - "/go/bin/docker-volume-sshfs" - ], - "Workdir": "", - "User": {}, - "Network": { - "Type": "host" - } - }, - "Config": { - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] - }, - "Active": true, - "Tag": "latest", - "Name": "vieux/sshfs", - "Id": "cd851ce43a403" - } - } -} -``` - -### Contents of a plugin directory - -Each directory within `/var/lib/docker/plugins/` contains a `rootfs` directory -and two JSON files. - -```bash -# ls -la /var/lib/docker/plugins/cd851ce43a403 -total 12 -drwx------ 19 root root 4096 Aug 8 17:56 rootfs --rw-r--r-- 1 root root 50 Aug 8 17:56 plugin-settings.json --rw------- 1 root root 347 Aug 8 17:56 config.json -``` - #### The rootfs directory The `rootfs` directory represents the root filesystem of the plugin. In this example, it was created from a Dockerfile: @@ -206,20 +119,17 @@ plugin's filesystem for docker to communicate with the plugin. ```bash $ git clone https://github.com/vieux/docker-volume-sshfs $ cd docker-volume-sshfs -$ docker build -t rootfs . -$ id=$(docker create rootfs true) # id was cd851ce43a403 when the image was created -$ sudo mkdir -p /var/lib/docker/plugins/$id/rootfs -$ sudo docker export "$id" | sudo tar -x -C /var/lib/docker/plugins/$id/rootfs -$ sudo chgrp -R docker /var/lib/docker/plugins/ +$ docker build -t rootfsimage . +$ id=$(docker create rootfsimage true) # id was cd851ce43a403 when the image was created +$ sudo mkdir -p myplugin/rootfs +$ sudo docker export "$id" | sudo tar -x -C myplugin/rootfs $ docker rm -vf "$id" -$ docker rmi rootfs +$ docker rmi rootfsimage ``` -#### The config.json and plugin-settings.json files +#### The config.json file -The `config.json` file describes the plugin. The `plugin-settings.json` file -contains runtime parameters and is only required if your plugin has runtime -parameters. [See the Plugins Config reference](config.md). +The `config.json` file describes the plugin. See the [plugins config reference](config.md). Consider the following `config.json` file. @@ -242,56 +152,15 @@ Consider the following `config.json` file. This plugin is a volume driver. It requires a `host` network and the `CAP_SYS_ADMIN` capability. It depends upon the `/go/bin/docker-volume-sshfs` entrypoint and uses the `/run/docker/plugins/sshfs.sock` socket to communicate -with Docker Engine. - - -Consider the following `plugin-settings.json` file. - -```json -{ - "Devices": null, - "Args": null, - "Env": [], - "Mounts": [] -} -``` - -This plugin has no runtime parameters. - -Each of these JSON files is included as part of `plugins.json`, as you can see -by looking back at the example above. After a plugin is installed, `config.json` -is read-only, but `plugin-settings.json` is read-write, and includes all runtime -configuration options for the plugin. +with Docker Engine. This plugin has no runtime parameters. ### Creating the plugin -Follow these steps to create a plugin: +A new plugin can be created by running +`docker plugin create ./path/to/plugin/data` where the plugin +data contains a plugin configuration file `config.json` and a root filesystem +in subdirectory `rootfs`. -1. Choose a name for the plugin. Plugin name uses the same format as images, - for example: `/`. - -2. Create a `rootfs` and export it to `/var/lib/docker/plugins/$id/rootfs` - using `docker export`. See [The rootfs directory](#the-rootfs-directory) for - an example of creating a `rootfs`. - -3. Create a `config.json` file in `/var/lib/docker/plugins/$id/`. - -4. Create a `plugin-settings.json` file if needed. - -5. Create or add a section to `/var/lib/docker/plugins/plugins.json`. Use - `/` as “Name” and `$id` as “Id”. - -6. Restart the Docker Engine service. - -7. Run `docker plugin ls`. - * If your plugin is enabled, you can push it to the - registry. - * If the plugin is not listed or is disabled, something went wrong. - Check the daemon logs for errors. - -8. If you are not already logged in, use `docker login` to authenticate against - the registry so that you can push to it. - -9. Run `docker plugin push /` to push the plugin. - -The plugin can now be used by any user with access to your registry. +After that the plugin `` will show up in `docker plugin ls`. +Plugins can be pushed to remote registries with +`docker plugin push `. \ No newline at end of file From c14c535c968e3e5be9e12a3606d233d0ce1f6cdc Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Sat, 19 Nov 2016 11:41:07 -0500 Subject: [PATCH 200/534] Moves graphdriver plugn docs out of experimental Also updates some of the structures being sent so plugins are getting all the new options. Signed-off-by: Brian Goff --- docs/extend/plugins_graphdriver.md | 376 +++++++++++++++++++++++++++++ 1 file changed, 376 insertions(+) create mode 100644 docs/extend/plugins_graphdriver.md diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md new file mode 100644 index 0000000000..fdcb34d7b6 --- /dev/null +++ b/docs/extend/plugins_graphdriver.md @@ -0,0 +1,376 @@ +--- +title: "Graphdriver plugins" +description: "How to manage image and container filesystems with external plugins" +keywords: "Examples, Usage, storage, image, docker, data, graph, plugin, api" +advisory: experimental +--- + + + + +## Changelog + +### 1.13.0 + +- Support v2 plugins + +# Docker graph driver plugins + +Docker graph driver plugins enable admins to use an external/out-of-process +graph driver for use with Docker engine. This is an alternative to using the +built-in storage drivers, such as aufs/overlay/devicemapper/btrfs. + +You need to install and enable the plugin and then restart the Docker daemon +before using the plugin. See the following example for the correct ordering +of steps. + +``` +$ docker plugin install cpuguy83/docker-overlay2-graphdriver-plugin # this command also enables the driver + +$ pkill dockerd +$ dockerd --experimental -s cpuguy83/docker-overlay2-graphdriver-plugin +``` + +# Write a graph driver plugin + +See the [plugin documentation](https://docs.docker.com/engine/extend/) for detailed information +on the underlying plugin protocol. + + +## Graph Driver plugin protocol + +If a plugin registers itself as a `GraphDriver` when activated, then it is +expected to provide the rootfs for containers as well as image layer storage. + +### /GraphDriver.Init + +**Request**: +```json +{ + "Home": "/graph/home/path", + "Opts": [], + "UIDMaps": [], + "GIDMaps": [] +} +``` + +Initialize the graph driver plugin with a home directory and array of options. +These are passed through from the user, but the plugin is not required to parse +or honor them. + +The request also includes a list of UID and GID mappings, structed as follows: +```json +{ + "ContainerID": 0, + "HostID": 0, + "Size": 0 +} +``` + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a non-empty string error if an error occurred. + + +### /GraphDriver.Create + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142", + "MountLabel": "", + "StorageOpt": {} +} +``` + +Create a new, empty, read-only filesystem layer with the specified +`ID`, `Parent` and `MountLabel`. If `Parent` is an empty string, there is no +parent layer. `StorageOpt` is map of strings which indicate storage options. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.CreateReadWrite + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142", + "MountLabel": "", + "StorageOpt": {} +} +``` + +Similar to `/GraphDriver.Create` but creates a read-write filesystem layer. + +### /GraphDriver.Remove + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" +} +``` + +Remove the filesystem layer with this given `ID`. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.Get + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "MountLabel": "" +} +``` + +Get the mountpoint for the layered filesystem referred to by the given `ID`. + +**Response**: +```json +{ + "Dir": "/var/mygraph/46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Err": "" +} +``` + +Respond with the absolute path to the mounted layered filesystem. +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.Put + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" +} +``` + +Release the system resources for the specified `ID`, such as unmounting the +filesystem layer. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.Exists + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" +} +``` + +Determine if a filesystem layer with the specified `ID` exists. + +**Response**: +```json +{ + "Exists": true +} +``` + +Respond with a boolean for whether or not the filesystem layer with the specified +`ID` exists. + +### /GraphDriver.Status + +**Request**: +```json +{} +``` + +Get low-level diagnostic information about the graph driver. + +**Response**: +```json +{ + "Status": [[]] +} +``` + +Respond with a 2-D array with key/value pairs for the underlying status +information. + + +### /GraphDriver.GetMetadata + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" +} +``` + +Get low-level diagnostic information about the layered filesystem with the +with the specified `ID` + +**Response**: +```json +{ + "Metadata": {}, + "Err": "" +} +``` + +Respond with a set of key/value pairs containing the low-level diagnostic +information about the layered filesystem. +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.Cleanup + +**Request**: +```json +{} +``` + +Perform necessary tasks to release resources help by the plugin, such as +unmounting all the layered file systems. + +**Response**: +```json +{ + "Err": "" +} +``` + +Respond with a non-empty string error if an error occurred. + + +### /GraphDriver.Diff + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" +} +``` + +Get an archive of the changes between the filesystem layers specified by the `ID` +and `Parent`. `Parent` may be an empty string, in which case there is no parent. + +**Response**: +``` +{{ TAR STREAM }} +``` + +### /GraphDriver.Changes + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" +} +``` + +Get a list of changes between the filesystem layers specified by the `ID` and +`Parent`. If `Parent` is an empty string, there is no parent. + +**Response**: +```json +{ + "Changes": [{}], + "Err": "" +} +``` + +Respond with a list of changes. The structure of a change is: +```json + "Path": "/some/path", + "Kind": 0, +``` + +Where the `Path` is the filesystem path within the layered filesystem that is +changed and `Kind` is an integer specifying the type of change that occurred: + +- 0 - Modified +- 1 - Added +- 2 - Deleted + +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.ApplyDiff + +**Request**: +``` +{{ TAR STREAM }} +``` + +Extract the changeset from the given diff into the layer with the specified `ID` +and `Parent` + +**Query Parameters**: + +- id (required)- the `ID` of the new filesystem layer to extract the diff to +- parent (required)- the `Parent` of the given `ID` + +**Response**: +```json +{ + "Size": 512366, + "Err": "" +} +``` + +Respond with the size of the new layer in bytes. +Respond with a non-empty string error if an error occurred. + +### /GraphDriver.DiffSize + +**Request**: +```json +{ + "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", + "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" +} +``` + +Calculate the changes between the specified `ID` + +**Response**: +```json +{ + "Size": 512366, + "Err": "" +} +``` + +Respond with the size changes between the specified `ID` and `Parent` +Respond with a non-empty string error if an error occurred. From a37fb8ab657b043afaaf2e1ac6e236fbacd2da43 Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Tue, 27 Dec 2016 02:17:35 +0800 Subject: [PATCH 201/534] Update 'ID' field for 'docker plugin ls' Signed-off-by: yuexiao-wang --- docs/extend/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index fc31e74ce4..e7a62ba758 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -69,8 +69,8 @@ enabled, and use it to create a volume. ```bash $ docker plugin ls - NAME TAG ENABLED - vieux/sshfs latest true + ID NAME TAG DESCRIPTION ENABLED + 69553ca1d789 vieux/sshfs latest the `sshfs` plugin true ``` 3. Create a volume using the plugin. From 1d7d202798b613fee98d07367d72de0d54fe33ee Mon Sep 17 00:00:00 2001 From: zhenghenghuo Date: Mon, 26 Dec 2016 12:49:37 +0800 Subject: [PATCH 202/534] fix some typos Signed-off-by: zhenghenghuo try to pass test Signed-off-by: zhenghenghuo try to pass the test Signed-off-by: zhenghenghuo --- docs/extend/plugins_volume.md | 4 ++-- docs/reference/commandline/dockerd.md | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index c060bf39b1..0a79efa022 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -169,7 +169,7 @@ Docker needs reminding of the path to the volume on the host. Respond with the path on the host filesystem where the volume has been made available, and/or a string error if an error occurred. `Mountpoint` is optional, -however the plugin may be queried again later if one is not provided. +however, the plugin may be queried again later if one is not provided. ### /VolumeDriver.Unmount @@ -257,7 +257,7 @@ Respond with a string error if an error occurred. `Mountpoint` is optional. ``` Get the list of capabilities the driver supports. -The driver is not required to implement this endpoint, however in such cases +The driver is not required to implement this endpoint, however, in such cases the default values will be taken. **Response**: diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 24ac77611d..8d4423e3a4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -237,7 +237,7 @@ drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that is unlikely to be merged into the main kernel. These are also known to cause -some serious kernel crashes. However, `aufs` allows containers to share +some serious kernel crashes. However `aufs` allows containers to share executable and shared library memory, so is a useful choice when running thousands of containers with the same program or libraries. @@ -658,7 +658,7 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. Overrides the Linux kernel version check allowing overlay2. Support for specifying multiple lower directories needed by overlay2 was added to the - Linux kernel in 4.0.0. However some older kernel versions may be patched + Linux kernel in 4.0.0. However, some older kernel versions may be patched to add multiple lower directory support for OverlayFS. This option should only be used after verifying this support exists in the kernel. Applying this option on a kernel without this support will cause failures on mount. From 380631b0a5437ec03dc2ec2e40fa9789c105b28e Mon Sep 17 00:00:00 2001 From: Ovidio Mallo Date: Wed, 28 Dec 2016 23:13:17 +0100 Subject: [PATCH 203/534] Reference new DigitalOcean plugin (Closes reference 29729) Signed-off-by: Ovidio Mallo --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index e9ab6f40be..0de2ad0633 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -63,6 +63,7 @@ Plugin [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. [Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. +[DigitalOcean Block Storage plugin](https://github.com/omallo/docker-volume-plugin-dostorage) | Integrates DigitalOcean's [block storage solution](https://www.digitalocean.com/products/storage/) into the Docker ecosystem by automatically attaching a given block storage volume to a DigitalOcean droplet and making the contents of the volume available to Docker containers running on that droplet. [DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. [Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. [Fuxi Volume Plugin](https://github.com/openstack/fuxi) | A volume plugin that is developed as part of the OpenStack Kuryr project and implements the Docker volume plugin API by utilizing Cinder, the OpenStack block storage service. From 2af9c822073c68cbd1f51347126cd6c1e2aa61fd Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Wed, 14 Dec 2016 00:35:34 -0800 Subject: [PATCH 204/534] replace no-remove by sample-volume-plugin in docs Signed-off-by: Victor Vieux --- docs/extend/config.md | 85 +++++++++++++++++++++---------------------- 1 file changed, 41 insertions(+), 44 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index eca33803c2..538dc6ac47 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -171,52 +171,49 @@ Config provides the base accessible fields for working with V0 plugin format ## Example Config -*Example showing the 'tiborvass/no-remove' plugin config.* +*Example showing the 'tiborvass/sample-volume-plugin' plugin config.* ```json { - "description": "A test plugin for Docker", - "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["plugin-no-remove", "/data"], - "interface": { - "types": ["docker.volumedriver/1.0"], - "socket": "plugins.sock" - }, - "network": { - "type": "host" - }, - "mounts": [ - { - "source": "/data", - "destination": "/data", - "type": "bind", - "options": ["shared", "rbind"] - }, - { - "destination": "/foobar", - "type": "tmpfs" - } - ], - "args": { - "name": "args", - "description": "command line arguments", - "value": [] - }, - "env": [ - { - "name": "DEBUG", - "description": "If set, prints debug messages", - "value": "1" - } - ], - "linux": { - "devices": [ - { - "name": "device", - "description": "a host device to mount", - "path": "/dev/cpu_dma_latency" - } - ] - } + "Args": { + "Description": "", + "Name": "", + "Settable": null, + "Value": null + }, + "Description": "A sample volume plugin for Docker", + "Documentation": "https://docs.docker.com/engine/extend/plugins/", + "Entrypoint": [ + "/usr/bin/sample-volume-plugin", + "/data" + ], + "Env": [ + { + "Description": "", + "Name": "DEBUG", + "Settable": [ + "value" + ], + "Value": "0" + } + ], + "Interface": { + "Socket": "plugin.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Linux": { + "Capabilities": null, + "DeviceCreation": false, + "Devices": null + }, + "Mounts": null, + "Network": { + "Type": "" + }, + "PropagatedMount": "/data", + "User": {}, + "Workdir": "" } ``` From 87290f018c2c9c01c616c6fae9e092305dfd445d Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Mon, 26 Dec 2016 22:27:56 +0800 Subject: [PATCH 205/534] Update docker daemon to dockerd Signed-off-by: yuexiao-wang --- docs/reference/commandline/dockerd.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8d4423e3a4..cac84708cc 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -715,7 +715,7 @@ with the `--exec-opt` flag. All the flag's options have the `native` prefix. A single `native.cgroupdriver` option is available. The `native.cgroupdriver` option specifies the management of the container's -cgroups. You can specify only specify `cgroupfs` or `systemd`. If you specify +cgroups. You can only specify `cgroupfs` or `systemd`. If you specify `systemd` and it is not available, the system errors out. If you omit the `native.cgroupdriver` option,` cgroupfs` is used. @@ -730,8 +730,8 @@ Setting this option applies to all containers the daemon launches. Also Windows Container makes use of `--exec-opt` for special purpose. Docker user can specify default container isolation technology with this, for example: -```bash -$ sudo dockerd --exec-opt isolation=hyperv +```console +> dockerd --exec-opt isolation=hyperv ``` Will make `hyperv` the default isolation technology on Windows. If no isolation @@ -746,14 +746,12 @@ To set the DNS server for all Docker containers, use: $ sudo dockerd --dns 8.8.8.8 ``` - To set the DNS search domain for all Docker containers, use: ```bash $ sudo dockerd --dns-search example.com ``` - ## Insecure registries Docker considers a private registry either secure or insecure. In the rest of From ee7f634e91496cf740f6264328036ab62fdcdb74 Mon Sep 17 00:00:00 2001 From: allencloud Date: Wed, 21 Dec 2016 18:31:05 +0800 Subject: [PATCH 206/534] support registry mirror config reload Signed-off-by: allencloud --- docs/reference/commandline/dockerd.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index cac84708cc..80d776f91c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1287,6 +1287,7 @@ The list of currently supported options that can be reconfigured is this: be used to run containers - `authorization-plugin`: specifies the authorization plugins to use. - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. +- `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From fbcf3abbe66b75b09b21a15f6b74fadb627a486f Mon Sep 17 00:00:00 2001 From: Tibor Vass Date: Tue, 10 Jan 2017 11:00:57 -0800 Subject: [PATCH 207/534] plugins: rename DeviceCreation to AllowAllDevices Signed-off-by: Tibor Vass --- docs/extend/config.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 538dc6ac47..b98be592a0 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -153,6 +153,10 @@ Config provides the base accessible fields for working with V0 plugin format capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) + - **`allowAllDevices`** *boolean* + + If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. + - **`devices`** *PluginDevice array* device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) @@ -205,7 +209,7 @@ Config provides the base accessible fields for working with V0 plugin format }, "Linux": { "Capabilities": null, - "DeviceCreation": false, + "AllowAllDevices": false, "Devices": null }, "Mounts": null, From 8fa89f42bd34d7a7ac44bf63d82adc98d4b96854 Mon Sep 17 00:00:00 2001 From: Zhang Wei Date: Thu, 19 Jan 2017 15:29:28 +0800 Subject: [PATCH 208/534] Fix some typos Signed-off-by: Zhang Wei --- docs/extend/plugins_graphdriver.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index fdcb34d7b6..e30632692f 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -33,7 +33,7 @@ of steps. ``` $ docker plugin install cpuguy83/docker-overlay2-graphdriver-plugin # this command also enables the driver - + $ pkill dockerd $ dockerd --experimental -s cpuguy83/docker-overlay2-graphdriver-plugin ``` From 08f74aee039057c148097bfb8ae8fc865c2269fd Mon Sep 17 00:00:00 2001 From: Tibor Vass Date: Thu, 19 Jan 2017 12:25:16 -0800 Subject: [PATCH 209/534] plugins: add example docs for a rexray plugin Signed-off-by: Tibor Vass --- docs/extend/EBS_volume.md | 164 ++++++++++++++++++++++++++++++++++++++ docs/extend/index.md | 4 +- 2 files changed, 166 insertions(+), 2 deletions(-) create mode 100644 docs/extend/EBS_volume.md diff --git a/docs/extend/EBS_volume.md b/docs/extend/EBS_volume.md new file mode 100644 index 0000000000..8c64efa164 --- /dev/null +++ b/docs/extend/EBS_volume.md @@ -0,0 +1,164 @@ +--- +description: Volume plugin for Amazon EBS +keywords: "API, Usage, plugins, documentation, developer, amazon, ebs, rexray, volume" +title: Volume plugin for Amazon EBS +--- + + + +# A proof-of-concept Rexray plugin + +In this example, a simple Rexray plugin will be created for the purposes of using +it on an Amazon EC2 instance with EBS. It is not meant to be a complete Rexray plugin. + +The example source is available at [https://github.com/tiborvass/rexray-plugin](https://github.com/tiborvass/rexray-plugin). + +To learn more about Rexray: [https://github.com/codedellemc/rexray](https://github.com/codedellemc/rexray) + +## 1. Make a Docker image + +The following is the Dockerfile used to containerize rexray. + +```Dockerfile +FROM debian:jessie +RUN apt-get update && apt-get install -y --no-install-recommends wget ca-certificates +RUN wget https://dl.bintray.com/emccode/rexray/stable/0.6.4/rexray-Linux-x86_64-0.6.4.tar.gz -O rexray.tar.gz && tar -xvzf rexray.tar.gz -C /usr/bin && rm rexray.tar.gz +RUN mkdir -p /run/docker/plugins /var/lib/libstorage/volumes +ENTRYPOINT ["rexray"] +CMD ["--help"] +``` + +To build it you can run `image=$(cat Dockerfile | docker build -q -)` and `$image` +will reference the containerized rexray image. + +## 2. Extract rootfs + +```sh +$ TMPDIR=/tmp/rexray # for the purpose of this example +$ # create container without running it, to extract the rootfs from image +$ docker create --name rexray "$image" +$ # save the rootfs to a tar archive +$ docker export -o $TMPDIR/rexray.tar rexray +$ # extract rootfs from tar archive to a rootfs folder +$ ( mkdir -p $TMPDIR/rootfs; cd $TMPDIR/rootfs; tar xf ../rexray.tar ) +``` + +## 3. Add plugin configuration + +We have to put the following JSON to `$TMPDIR/config.json`: + +```json +{ + "Args": { + "Description": "", + "Name": "", + "Settable": null, + "Value": null + }, + "Description": "A proof-of-concept EBS plugin (using rexray) for Docker", + "Documentation": "https://github.com/tiborvass/rexray-plugin", + "Entrypoint": [ + "/usr/bin/rexray", "service", "start", "-f" + ], + "Env": [ + { + "Description": "", + "Name": "REXRAY_SERVICE", + "Settable": [ + "value" + ], + "Value": "ebs" + }, + { + "Description": "", + "Name": "EBS_ACCESSKEY", + "Settable": [ + "value" + ], + "Value": "" + }, + { + "Description": "", + "Name": "EBS_SECRETKEY", + "Settable": [ + "value" + ], + "Value": "" + } + ], + "Interface": { + "Socket": "rexray.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Linux": { + "AllowAllDevices": true, + "Capabilities": ["CAP_SYS_ADMIN"], + "Devices": null + }, + "Mounts": [ + { + "Source": "/dev", + "Destination": "/dev", + "Type": "bind", + "Options": ["rbind"] + } + ], + "Network": { + "Type": "host" + }, + "PropagatedMount": "/var/lib/libstorage/volumes", + "User": {}, + "WorkDir": "" +} +``` + +Please note a couple of points: +- `PropagatedMount` is needed so that the docker daemon can see mounts done by the +rexray plugin from within the container, otherwise the docker daemon is not able +to mount a docker volume. +- The rexray plugin needs dynamic access to host devices. For that reason, we +have to give it access to all devices under `/dev` and set `AllowAllDevices` to +true for proper access. +- The user of this simple plugin can change only 3 settings: `REXRAY_SERVICE`, +`EBS_ACCESSKEY` and `EBS_SECRETKEY`. This is because of the reduced scope of this +plugin. Ideally other rexray parameters could also be set. + +## 4. Create plugin + +`docker plugin create tiborvass/rexray-plugin "$TMPDIR"` will create the plugin. + +```sh +$ docker plugin ls +ID NAME DESCRIPTION ENABLED +2475a4bd0ca5 tiborvass/rexray-plugin:latest A rexray volume plugin for Docker false +``` + +## 5. Test plugin + +```sh +$ docker plugin set tiborvass/rexray-plugin EBS_ACCESSKEY=$AWS_ACCESSKEY EBS_SECRETKEY=$AWS_SECRETKEY` +$ docker plugin enable tiborvass/rexray-plugin +$ docker volume create -d tiborvass/rexray-plugin my-ebs-volume +$ docker volume ls +DRIVER VOLUME NAME +tiborvass/rexray-plugin:latest my-ebs-volume +$ docker run --rm -v my-ebs-volume:/volume busybox sh -c 'echo bye > /volume/hi' +$ docker run --rm -v my-ebs-volume:/volume busybox cat /volume/hi +bye +``` + +## 6. Push plugin + +First, ensure you are logged in with `docker login`. Then you can run: +`docker plugin push tiborvass/rexray-plugin` to push it like a regular docker +image to a registry, to make it available for others to install via +`docker plugin install tiborvass/rexray-plugin EBS_ACCESSKEY=$AWS_ACCESSKEY EBS_SECRETKEY=$AWS_SECRETKEY`. diff --git a/docs/extend/index.md b/docs/extend/index.md index e7a62ba758..2687fd4311 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -154,7 +154,7 @@ This plugin is a volume driver. It requires a `host` network and the entrypoint and uses the `/run/docker/plugins/sshfs.sock` socket to communicate with Docker Engine. This plugin has no runtime parameters. -### Creating the plugin +#### Creating the plugin A new plugin can be created by running `docker plugin create ./path/to/plugin/data` where the plugin @@ -163,4 +163,4 @@ in subdirectory `rootfs`. After that the plugin `` will show up in `docker plugin ls`. Plugins can be pushed to remote registries with -`docker plugin push `. \ No newline at end of file +`docker plugin push `. From c3211e73466e878f3c1129099255f2d144e5f809 Mon Sep 17 00:00:00 2001 From: Janonymous Date: Fri, 27 Jan 2017 09:32:19 +0300 Subject: [PATCH 210/534] Fix json format of plugin Signed-off-by: Janonymous --- docs/extend/plugin_api.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index e299a0991e..693b77a2f3 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -79,7 +79,7 @@ This is the JSON format for a plugin: "InsecureSkipVerify": false, "CAFile": "/usr/shared/docker/certs/example-ca.pem", "CertFile": "/usr/shared/docker/certs/example-cert.pem", - "KeyFile": "/usr/shared/docker/certs/example-key.pem", + "KeyFile": "/usr/shared/docker/certs/example-key.pem" } } ``` From 8c8a05b5293070d411e88ac4070d4c8baeef458e Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Fri, 27 Jan 2017 14:45:06 -0800 Subject: [PATCH 211/534] Add docs for service create based on plugins. Signed-off-by: Anusha Ragunathan --- docs/extend/index.md | 56 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/docs/extend/index.md b/docs/extend/index.md index 2687fd4311..1410b205e5 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -107,6 +107,62 @@ remove it, use the `docker plugin remove` command. For other available commands and options, see the [command line reference](../reference/commandline/index.md). +## Service creation using plugins + +In swarm mode, it is possible to create a service that allows for attaching +to networks or mounting volumes. Swarm schedules services based on plugin availability +on a node. In this example, a volume plugin is installed on a swarm worker and a volume +is created using the plugin. In the manager, a service is created with the relevant +mount options. It can be observed that the service is scheduled to run on the worker +node with the said volume plugin and volume. + +In the following example, node1 is the manager and node2 is the worker. + +1. Prepare manager. In node 1: + + ```bash + $ docker swarm init + Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager. + ``` + +2. Join swarm, install plugin and create volume on worker. In node 2: + + ```bash + $ docker swarm join \ + --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ + 192.168.99.100:2377 + ``` + + ```bash + $ docker plugin install tiborvass/sample-volume-plugin + latest: Pulling from tiborvass/sample-volume-plugin + eb9c16fbdc53: Download complete + Digest: sha256:00b42de88f3a3e0342e7b35fa62394b0a9ceb54d37f4c50be5d3167899994639 + Status: Downloaded newer image for tiborvass/sample-volume-plugin:latest + Installed plugin tiborvass/sample-volume-plugin + ``` + + ```bash + $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol + ``` + +3. Create a service using the plugin and volume. In node1: + + ```bash + $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top + + $ docker service ls + z1sj8bb8jnfn my-service replicated 1/1 busybox:latest + ``` + docker service ls shows service 1 instance of service running. + +4. Observe the task getting scheduled in node 2: + + ```bash + $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' + 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" + ``` + ## Developing a plugin #### The rootfs directory From 68fae631fc125de92bb41d45fdf2fe0a7364a58b Mon Sep 17 00:00:00 2001 From: David Williamson Date: Mon, 9 Jan 2017 15:24:20 -0800 Subject: [PATCH 212/534] Provide password to volume create command * Depending on the host configuration between the docker engine and the remote host we wish to mount using sshfs, a password may be required. * Confirmed syntax for optional parameter with @vieux * Verified change works as desired with engine running on Ubuntu host and mounting a volume from a file system on a Mac. Signed-off-by: David Williamson --- docs/extend/index.md | 17 ++++++++++++++--- 1 file changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 1410b205e5..0dd672c312 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -46,6 +46,8 @@ operation, such as creating a volume. In the following example, you install the `sshfs` plugin, verify that it is enabled, and use it to create a volume. +> **Note**: This example is intended for instructional purposes only. Once the volume is created, your SSH password to the remote host will be exposed as plaintext when inspecting the volume. You should delete the volume as soon as you are done with the example. + 1. Install the `sshfs` plugin. ```bash @@ -75,13 +77,16 @@ enabled, and use it to create a volume. 3. Create a volume using the plugin. This example mounts the `/remote` directory on host `1.2.3.4` into a - volume named `sshvolume`. This volume can now be mounted into containers. + volume named `sshvolume`. + + This volume can now be mounted into containers. ```bash $ docker volume create \ -d vieux/sshfs \ --name sshvolume \ - -o sshcmd=user@1.2.3.4:/remote + -o sshcmd=user@1.2.3.4:/remote \ + -o password=$(cat file_containing_password_for_remote_host) sshvolume ``` @@ -97,11 +102,17 @@ enabled, and use it to create a volume. 5. Start a container that uses the volume `sshvolume`. ```bash - $ docker run -v sshvolume:/data busybox ls /data + $ docker run --rm -v sshvolume:/data busybox ls /data ``` +6. Remove the volume `sshvolume` + ```bash + docker volume rm sshvolume + + sshvolume + ``` To disable a plugin, use the `docker plugin disable` command. To completely remove it, use the `docker plugin remove` command. For other available commands and options, see the From 492d382760466acf55726b9837503d319cecc6e3 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Wed, 23 Nov 2016 05:27:09 -0800 Subject: [PATCH 213/534] Add `capability` filter to `docker plugin ls` This fix adds `--filter capability=[volumedriver|authz]` to `docker plugin ls`. The related docs has been updated. An integration test has been added. Signed-off-by: Yong Tang --- docs/extend/config.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index b98be592a0..eaa52fd9bc 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -51,9 +51,13 @@ Config provides the base accessible fields for working with V0 plugin format currently supported: - - **docker.volumedriver/1.0** + - **docker.volumedriver/1.0** - - **docker.authz/1.0** + - **docker.networkdriver/1.0** + + - **docker.ipamdriver/1.0** + + - **docker.authz/1.0** - **`socket`** *string* From 22e4ee858bb4a0988eddc8f2321fd63006854c04 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sun, 25 Dec 2016 01:11:12 -0800 Subject: [PATCH 214/534] Add daemon option --default-shm-size This fix fixes issue raised in 29492 where it was not possible to specify a default `--default-shm-size` in daemon configuration for each `docker run``. The flag `--default-shm-size` which is reloadable, has been added to the daemon configuation. Related docs has been updated. This fix fixes 29492. Signed-off-by: Yong Tang --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 80d776f91c..e4b5a33a87 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -37,6 +37,7 @@ Options: --default-gateway value Container default gateway IPv4 address --default-gateway-v6 value Container default gateway IPv6 address --default-runtime string Default OCI runtime for containers (default "runc") + --default-shm-size bytes Set the default shm size for containers (default 64 MiB) --default-ulimit value Default ulimits for containers (default []) --disable-legacy-registry Disable contacting legacy registries --dns value DNS server to use (default []) @@ -1161,6 +1162,7 @@ This is a full example of the allowed configuration options on Linux: "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, + "default-shm-size": "64M", "shutdown-timeout": 15, "debug": true, "hosts": [], From 334eb148da02f6619566e9b73219901e67014dff Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Thu, 2 Feb 2017 23:08:35 -0500 Subject: [PATCH 215/534] Make propagated mount persist outside rootfs This persists the "propagated mount" for plugins outside the main rootfs. This enables `docker plugin upgrade` to not remove potentially important data during upgrade rather than forcing plugin authors to hard code a host path to persist data to. Also migrates old plugins that have a propagated mount which is in the rootfs on daemon startup. Signed-off-by: Brian Goff --- docs/extend/config.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/extend/config.md b/docs/extend/config.md index eaa52fd9bc..dab755d97b 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -118,6 +118,8 @@ Config provides the base accessible fields for working with V0 plugin format - **`propagatedMount`** *string* path to be mounted as rshared, so that mounts under that path are visible to docker. This is useful for volume plugins. + This path will be bind-mounted outisde of the plugin rootfs so it's contents + are preserved on upgrade. - **`env`** *PluginEnv array* From f98d95c4316e203a2a887a6af4216b506c294c84 Mon Sep 17 00:00:00 2001 From: Lewis Daly Date: Mon, 30 Jan 2017 20:59:13 +1030 Subject: [PATCH 216/534] Updated dockerd docs with note about user namespaces Signed-off-by: Lewis Daly Updated uid/gid reference to be more canonical - and signed commit Signed-off-by: Lewis Daly Editorial suggestion I tried my hand at rewriting this a bit for readability. Can you please verify that the facts are correct, especially about the permission changes? You can feel free to squash my commit with yours so that it's signed. You'll need to pull the change from your fork to work on it locally. Editorial suggestion I tried my hand at rewriting this a bit for readability. Can you please verify that the facts are correct, especially about the permission changes? You can feel free to squash my commit with yours so that it's signed. You'll need to pull the change from your fork to work on it locally. --- docs/reference/commandline/dockerd.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e4b5a33a87..e61b5ae53b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -993,6 +993,19 @@ with user namespaces enabled or not. If the daemon is configured with user namespaces, the Security Options entry in the response will list "userns" as one of the enabled security features. +#### Behavior differences when user namespaces are enabled + +When you start the Docker daemon with `--userns-remap`, Docker segregates the graph directory +where the images are stored by adding an extra directory with a name corresponding to the +remapped UID and GID. For example, if the remapped UID and GID begin with `165536`, all +images and containers running with that remap setting are located in `/var/lib/docker/165536.165536` +instead of `/var/lib/docker/`. + +In addition, the files and directories within the new directory, which correspond to +images and container layers, are also owned by the new UID and GID. To set the ownership +correctly, you need to re-pull the images and restart the containers after starting the +daemon with `--userns-remap`. + ### Detailed information on `subuid`/`subgid` ranges Given potential advanced use of the subordinate ID ranges by power users, the From 67e0914d9373386db580e37e099d99164b6afbe5 Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Mon, 6 Feb 2017 14:44:56 -0800 Subject: [PATCH 217/534] Add plugin debug docs. Doc fix for reference 30761 Signed-off-by: Anusha Ragunathan --- docs/extend/index.md | 64 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 61 insertions(+), 3 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 0dd672c312..a2aa208d6b 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -17,10 +17,10 @@ title: Managed plugin system * [Installing and using a plugin](index.md#installing-and-using-a-plugin) * [Developing a plugin](index.md#developing-a-plugin) +* [Debugging plugins](index.md#debugging-plugins) Docker Engine's plugins system allows you to install, start, stop, and remove -plugins using Docker Engine. This mechanism is currently only available for -volume drivers, but more plugin driver types will be available in future releases. +plugins using Docker Engine. For information about the legacy plugin system available in Docker Engine 1.12 and earlier, see [Understand legacy Docker Engine plugins](legacy_plugins.md). @@ -212,7 +212,9 @@ Consider the following `config.json` file. "types": ["docker.volumedriver/1.0"], "socket": "sshfs.sock" }, - "capabilities": ["CAP_SYS_ADMIN"] + "linux": { + "capabilities": ["CAP_SYS_ADMIN"] + } } ``` @@ -231,3 +233,59 @@ in subdirectory `rootfs`. After that the plugin `` will show up in `docker plugin ls`. Plugins can be pushed to remote registries with `docker plugin push `. + + +## Debugging plugins + +Stdout of a plugin is redirected to dockerd logs. Such entries have a +`plugin=` suffix. Here are a few examples of commands for pluginID +`f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62` and their +corresponding log entries in the docker daemon logs. + +```bash +$ docker plugin install tiborvass/sample-volume-plugins + +INFO[0036] Starting... Found 0 volumes on startup plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +``` + +```bash +$ docker volume create -d tiborvass/sample-volume-plugins samplevol + +INFO[0193] Create Called... Ensuring directory /data/samplevol exists on host... plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0193] open /var/lib/docker/plugin-data/local-persist.json: no such file or directory plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0193] Created volume samplevol with mountpoint /data/samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0193] Path Called... Returned path /data/samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +``` + +```bash +$ docker run -v samplevol:/tmp busybox sh + +INFO[0421] Get Called... Found samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0421] Mount Called... Mounted samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0421] Path Called... Returned path /data/samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +INFO[0421] Unmount Called... Unmounted samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 +``` + +#### Using docker-runc to obtain logfiles and shell into the plugin. + +`docker-runc`, the default docker container runtime can be used for debugging +plugins. This is specifically useful to collect plugin logs if they are +redirected to a file. + +```bash +$ docker-runc list +ID PID STATUS BUNDLE CREATED +f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 2679 running /run/docker/libcontainerd/f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 2017-02-06T21:53:03.031537592Z +r +``` + +```bash +$ docker-runc exec f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 cat /var/log/plugin.log +``` + +If the plugin has a built-in shell, then exec into the plugin can be done as +follows: +```bash +$ docker-runc exec -t f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 sh +``` + From 8b5d4d85b55ce030089ec21a7df9cffb979fa847 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Tue, 7 Feb 2017 15:42:48 -0800 Subject: [PATCH 218/534] Standardized formatting of CLI reference commands Command name should be a H1 Only Description, Examples, and Related Commands should be H2 Changed 'Related information' heading to 'Related commands' since 99% it is only linking commands Added some examples where relevant Signed-off-by: Misty Stanley-Jones --- docs/reference/commandline/dockerd.md | 647 +++++++++++++------------- 1 file changed, 312 insertions(+), 335 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e61b5ae53b..1ebdc3b666 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -93,13 +93,17 @@ Options: Options with [] may be specified multiple times. -dockerd is the persistent process that manages containers. Docker +## Description + +`dockerd` is the persistent process that manages containers. Docker uses different binaries for the daemon and client. To run the daemon you type `dockerd`. To run the daemon with debug output, use `dockerd -D`. -## Daemon socket option +## Examples + +### Daemon socket option The Docker daemon can listen for [Docker Engine API](../api/) requests via three different types of Socket: `unix`, `tcp`, and `fd`. @@ -118,8 +122,7 @@ interface using its IP address: `-H tcp://192.168.59.103:2375`. It is conventional to use port `2375` for un-encrypted, and port `2376` for encrypted communication with the daemon. -> **Note:** -> If you're using an HTTPS encrypted socket, keep in mind that only +> **Note**: If you're using an HTTPS encrypted socket, keep in mind that only > TLS1.0 and greater are supported. Protocols SSLv3 and under are not > supported anymore for security reasons. @@ -136,18 +139,21 @@ time using multiple `-H` options: ```bash # listen using the default unix socket, and on 2 specific IP addresses on this host. + $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 ``` The Docker client will honor the `DOCKER_HOST` environment variable to set the -`-H` flag for the client. +`-H` flag for the client. Use **one** of the following commands: ```bash $ docker -H tcp://0.0.0.0:2375 ps -# or +``` + +```bash $ export DOCKER_HOST="tcp://0.0.0.0:2375" + $ docker ps -# both are equal ``` Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than @@ -165,7 +171,7 @@ The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes precedence over `HTTP_PROXY`. -### Bind Docker to another host/port or a Unix socket +#### Bind Docker to another host/port or a Unix socket > **Warning**: > Changing the default `docker` daemon binding to a @@ -231,7 +237,7 @@ $ docker pull ubuntu $ docker -H tcp://127.0.0.1:2375 pull ubuntu ``` -### Daemon storage-driver option +### Daemon storage-driver The Docker daemon has support for several different image layer storage drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. @@ -268,22 +274,20 @@ the same file can share a single page cache entry (or entries), it makes `overlay` as efficient with memory as `aufs` driver. Call `dockerd -s overlay` to use it. -> **Note:** -> As promising as `overlay` is, the feature is still quite young and should not -> be used in production. Most notably, using `overlay` can cause excessive -> inode consumption (especially as the number of images grows), as well as -> being incompatible with the use of RPMs. +> **Note**: As promising as `overlay` is, the feature is still quite young and +> should not be used in production. Most notably, using `overlay` can cause +> excessive inode consumption (especially as the number of images grows), as +> well as > being incompatible with the use of RPMs. The `overlay2` uses the same fast union filesystem but takes advantage of [additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2` to use it. -> **Note:** -> Both `overlay` and `overlay2` are currently unsupported on `btrfs` or any -> Copy on Write filesystem and should only be used over `ext4` partitions. +> **Note**: Both `overlay` and `overlay2` are currently unsupported on `btrfs` +> or any Copy on Write filesystem and should only be used over `ext4` partitions. -### Storage driver options +### Options per storage driver Particular storage-driver can be configured with options specified with `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, @@ -291,380 +295,376 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### Devicemapper options -* `dm.thinpooldev` +##### `dm.thinpooldev` - Specifies a custom block storage device to use for the thin pool. +Specifies a custom block storage device to use for the thin pool. - If using a block device for device mapper storage, it is best to use `lvm` - to create and manage the thin-pool volume. This volume is then handed to Docker - to exclusively create snapshot volumes needed for images and containers. +If using a block device for device mapper storage, it is best to use `lvm` +to create and manage the thin-pool volume. This volume is then handed to Docker +to exclusively create snapshot volumes needed for images and containers. - Managing the thin-pool outside of Engine makes for the most feature-rich - method of having Docker utilize device mapper thin provisioning as the - backing storage for Docker containers. The highlights of the lvm-based - thin-pool management feature include: automatic or interactive thin-pool - resize support, dynamically changing thin-pool features, automatic thinp - metadata checking when lvm activates the thin-pool, etc. +Managing the thin-pool outside of Engine makes for the most feature-rich +method of having Docker utilize device mapper thin provisioning as the +backing storage for Docker containers. The highlights of the lvm-based +thin-pool management feature include: automatic or interactive thin-pool +resize support, dynamically changing thin-pool features, automatic thinp +metadata checking when lvm activates the thin-pool, etc. - As a fallback if no thin pool is provided, loopback files are - created. Loopback is very slow, but can be used without any - pre-configuration of storage. It is strongly recommended that you do - not use loopback in production. Ensure your Engine daemon has a - `--storage-opt dm.thinpooldev` argument provided. +As a fallback if no thin pool is provided, loopback files are +created. Loopback is very slow, but can be used without any +pre-configuration of storage. It is strongly recommended that you do +not use loopback in production. Ensure your Engine daemon has a +`--storage-opt dm.thinpooldev` argument provided. - Example use: +###### Example: - ```bash - $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool - ``` +```bash +$ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool +``` -* `dm.basesize` +##### `dm.basesize` - Specifies the size to use when creating the base device, which limits the - size of images and containers. The default value is 10G. Note, thin devices - are inherently "sparse", so a 10G device which is mostly empty doesn't use - 10 GB of space on the pool. However, the filesystem will use more space for - the empty case the larger the device is. +Specifies the size to use when creating the base device, which limits the +size of images and containers. The default value is 10G. Note, thin devices +are inherently "sparse", so a 10G device which is mostly empty doesn't use +10 GB of space on the pool. However, the filesystem will use more space for +the empty case the larger the device is. - The base device size can be increased at daemon restart which will allow - all future images and containers (based on those new images) to be of the - new base device size. +The base device size can be increased at daemon restart which will allow +all future images and containers (based on those new images) to be of the +new base device size. - Example use: +###### Examples - ```bash - $ sudo dockerd --storage-opt dm.basesize=50G - ``` +```bash +$ sudo dockerd --storage-opt dm.basesize=50G +``` - This will increase the base device size to 50G. The Docker daemon will throw an - error if existing base device size is larger than 50G. A user can use - this option to expand the base device size however shrinking is not permitted. +This will increase the base device size to 50G. The Docker daemon will throw an +error if existing base device size is larger than 50G. A user can use +this option to expand the base device size however shrinking is not permitted. - This value affects the system-wide "base" empty filesystem - that may already be initialized and inherited by pulled images. Typically, - a change to this value requires additional steps to take effect: +This value affects the system-wide "base" empty filesystem +that may already be initialized and inherited by pulled images. Typically, +a change to this value requires additional steps to take effect: - ```bash - $ sudo service docker stop - $ sudo rm -rf /var/lib/docker - $ sudo service docker start - ``` + ```bash +$ sudo service docker stop - Example use: +$ sudo rm -rf /var/lib/docker - ```bash - $ sudo dockerd --storage-opt dm.basesize=20G - ``` +$ sudo service docker start +``` -* `dm.loopdatasize` - > **Note**: - > This option configures devicemapper loopback, which should not - > be used in production. +##### `dm.loopdatasize` - Specifies the size to use when creating the loopback file for the - "data" device which is used for the thin pool. The default size is - 100G. The file is sparse, so it will not initially take up this - much space. +> **Note**: This option configures devicemapper loopback, which should not +> be used in production. - Example use: +Specifies the size to use when creating the loopback file for the +"data" device which is used for the thin pool. The default size is +100G. The file is sparse, so it will not initially take up this +much space. - ```bash - $ sudo dockerd --storage-opt dm.loopdatasize=200G - ``` +###### Example -* `dm.loopmetadatasize` +```bash +$ sudo dockerd --storage-opt dm.loopdatasize=200G +``` - > **Note**: - > This option configures devicemapper loopback, which should not - > be used in production. +##### `dm.loopmetadatasize` - Specifies the size to use when creating the loopback file for the - "metadata" device which is used for the thin pool. The default size - is 2G. The file is sparse, so it will not initially take up - this much space. +> **Note**: This option configures devicemapper loopback, which should not +> be used in production. - Example use: +Specifies the size to use when creating the loopback file for the +"metadata" device which is used for the thin pool. The default size +is 2G. The file is sparse, so it will not initially take up +this much space. - ```bash - $ sudo dockerd --storage-opt dm.loopmetadatasize=4G - ``` +###### Example -* `dm.fs` +```bash +$ sudo dockerd --storage-opt dm.loopmetadatasize=4G +``` - Specifies the filesystem type to use for the base device. The supported - options are "ext4" and "xfs". The default is "xfs" +##### `dm.fs` - Example use: +Specifies the filesystem type to use for the base device. The supported +options are "ext4" and "xfs". The default is "xfs" - ```bash - $ sudo dockerd --storage-opt dm.fs=ext4 - ``` +###### Example -* `dm.mkfsarg` +```bash +$ sudo dockerd --storage-opt dm.fs=ext4 +``` - Specifies extra mkfs arguments to be used when creating the base device. +##### `dm.mkfsarg` - Example use: +Specifies extra mkfs arguments to be used when creating the base device. - ```bash - $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" - ``` +###### Example -* `dm.mountopt` +```bash +$ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" +``` - Specifies extra mount options used when mounting the thin devices. +##### `dm.mountopt` - Example use: +Specifies extra mount options used when mounting the thin devices. - ```bash - $ sudo dockerd --storage-opt dm.mountopt=nodiscard - ``` +###### Example -* `dm.datadev` +```bash +$ sudo dockerd --storage-opt dm.mountopt=nodiscard +``` - (Deprecated, use `dm.thinpooldev`) +##### `dm.datadev` - Specifies a custom blockdevice to use for data for the thin pool. +(Deprecated, use `dm.thinpooldev`) - If using a block device for device mapper storage, ideally both datadev and - metadatadev should be specified to completely avoid using the loopback - device. +Specifies a custom blockdevice to use for data for the thin pool. - Example use: +If using a block device for device mapper storage, ideally both `datadev` and +`metadatadev` should be specified to completely avoid using the loopback +device. - ```bash - $ sudo dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 - ``` +###### Example -* `dm.metadatadev` +```bash +$ sudo dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 +``` - (Deprecated, use `dm.thinpooldev`) +##### `dm.metadatadev` - Specifies a custom blockdevice to use for metadata for the thin pool. +(Deprecated, use `dm.thinpooldev`) - For best performance the metadata should be on a different spindle than the - data, or even better on an SSD. +Specifies a custom blockdevice to use for metadata for the thin pool. - If setting up a new metadata pool it is required to be valid. This can be - achieved by zeroing the first 4k to indicate empty metadata, like this: +For best performance the metadata should be on a different spindle than the +data, or even better on an SSD. - ```bash - $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 - ``` +If setting up a new metadata pool it is required to be valid. This can be +achieved by zeroing the first 4k to indicate empty metadata, like this: - Example use: +```bash +$ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 +``` - ```bash - $ sudo dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 - ``` +###### Example -* `dm.blocksize` +```bash +$ sudo dockerd \ + --storage-opt dm.datadev=/dev/sdb1 \ + --storage-opt dm.metadatadev=/dev/sdc1 +``` - Specifies a custom blocksize to use for the thin pool. The default - blocksize is 64K. +##### `dm.blocksize` - Example use: +Specifies a custom blocksize to use for the thin pool. The default +blocksize is 64K. - ```bash - $ sudo dockerd --storage-opt dm.blocksize=512K - ``` +###### Example -* `dm.blkdiscard` +```bash +$ sudo dockerd --storage-opt dm.blocksize=512K +``` - Enables or disables the use of blkdiscard when removing devicemapper - devices. This is enabled by default (only) if using loopback devices and is - required to resparsify the loopback file on image/container removal. +##### `dm.blkdiscard` - Disabling this on loopback can lead to *much* faster container removal - times, but will make the space used in `/var/lib/docker` directory not be - returned to the system for other use when containers are removed. +Enables or disables the use of `blkdiscard` when removing devicemapper +devices. This is enabled by default (only) if using loopback devices and is +required to resparsify the loopback file on image/container removal. - Example use: +Disabling this on loopback can lead to *much* faster container removal +times, but will make the space used in `/var/lib/docker` directory not be +returned to the system for other use when containers are removed. - ```bash - $ sudo dockerd --storage-opt dm.blkdiscard=false - ``` +###### Examples -* `dm.override_udev_sync_check` +```bash +$ sudo dockerd --storage-opt dm.blkdiscard=false +``` - Overrides the `udev` synchronization checks between `devicemapper` and `udev`. - `udev` is the device manager for the Linux kernel. +##### `dm.override_udev_sync_check` - To view the `udev` sync support of a Docker daemon that is using the - `devicemapper` driver, run: +Overrides the `udev` synchronization checks between `devicemapper` and `udev`. +`udev` is the device manager for the Linux kernel. - ```bash - $ docker info - [...] - Udev Sync Supported: true - [...] - ``` +To view the `udev` sync support of a Docker daemon that is using the +`devicemapper` driver, run: - When `udev` sync support is `true`, then `devicemapper` and udev can - coordinate the activation and deactivation of devices for containers. +```bash +$ docker info +[...] +Udev Sync Supported: true +[...] +``` - When `udev` sync support is `false`, a race condition occurs between - the`devicemapper` and `udev` during create and cleanup. The race condition - results in errors and failures. (For information on these failures, see - [docker#4036](https://github.com/docker/docker/issues/4036)) +When `udev` sync support is `true`, then `devicemapper` and udev can +coordinate the activation and deactivation of devices for containers. - To allow the `docker` daemon to start, regardless of `udev` sync not being - supported, set `dm.override_udev_sync_check` to true: +When `udev` sync support is `false`, a race condition occurs between +the`devicemapper` and `udev` during create and cleanup. The race condition +results in errors and failures. (For information on these failures, see +[docker#4036](https://github.com/docker/docker/issues/4036)) - ```bash - $ sudo dockerd --storage-opt dm.override_udev_sync_check=true - ``` +To allow the `docker` daemon to start, regardless of `udev` sync not being +supported, set `dm.override_udev_sync_check` to true: - When this value is `true`, the `devicemapper` continues and simply warns - you the errors are happening. +```bash +$ sudo dockerd --storage-opt dm.override_udev_sync_check=true +``` - > **Note:** - > The ideal is to pursue a `docker` daemon and environment that does - > support synchronizing with `udev`. For further discussion on this - > topic, see [docker#4036](https://github.com/docker/docker/issues/4036). - > Otherwise, set this flag for migrating existing Docker daemons to - > a daemon with a supported environment. +When this value is `true`, the `devicemapper` continues and simply warns +you the errors are happening. -* `dm.use_deferred_removal` +> **Note**: The ideal is to pursue a `docker` daemon and environment that does +> support synchronizing with `udev`. For further discussion on this +> topic, see [docker#4036](https://github.com/docker/docker/issues/4036). +> Otherwise, set this flag for migrating existing Docker daemons to +> a daemon with a supported environment. - Enables use of deferred device removal if `libdm` and the kernel driver - support the mechanism. +##### `dm.use_deferred_removal` - Deferred device removal means that if device is busy when devices are - being removed/deactivated, then a deferred removal is scheduled on - device. And devices automatically go away when last user of the device - exits. +Enables use of deferred device removal if `libdm` and the kernel driver +support the mechanism. - For example, when a container exits, its associated thin device is removed. - If that device has leaked into some other mount namespace and can't be - removed, the container exit still succeeds and this option causes the - system to schedule the device for deferred removal. It does not wait in a - loop trying to remove a busy device. +Deferred device removal means that if device is busy when devices are +being removed/deactivated, then a deferred removal is scheduled on +device. And devices automatically go away when last user of the device +exits. - Example use: +For example, when a container exits, its associated thin device is removed. +If that device has leaked into some other mount namespace and can't be +removed, the container exit still succeeds and this option causes the +system to schedule the device for deferred removal. It does not wait in a +loop trying to remove a busy device. - ```bash - $ sudo dockerd --storage-opt dm.use_deferred_removal=true - ``` +###### Example -* `dm.use_deferred_deletion` +```bash +$ sudo dockerd --storage-opt dm.use_deferred_removal=true +``` - Enables use of deferred device deletion for thin pool devices. By default, - thin pool device deletion is synchronous. Before a container is deleted, - the Docker daemon removes any associated devices. If the storage driver - can not remove a device, the container deletion fails and daemon returns. +##### `dm.use_deferred_deletion` - Error deleting container: Error response from daemon: Cannot destroy container +Enables use of deferred device deletion for thin pool devices. By default, +thin pool device deletion is synchronous. Before a container is deleted, +the Docker daemon removes any associated devices. If the storage driver +can not remove a device, the container deletion fails and daemon returns. - To avoid this failure, enable both deferred device deletion and deferred - device removal on the daemon. +```none +Error deleting container: Error response from daemon: Cannot destroy container +``` - ```bash - $ sudo dockerd \ - --storage-opt dm.use_deferred_deletion=true \ - --storage-opt dm.use_deferred_removal=true - ``` +To avoid this failure, enable both deferred device deletion and deferred +device removal on the daemon. - With these two options enabled, if a device is busy when the driver is - deleting a container, the driver marks the device as deleted. Later, when - the device isn't in use, the driver deletes it. +```bash +$ sudo dockerd \ + --storage-opt dm.use_deferred_deletion=true \ + --storage-opt dm.use_deferred_removal=true +``` - In general it should be safe to enable this option by default. It will help - when unintentional leaking of mount point happens across multiple mount - namespaces. +With these two options enabled, if a device is busy when the driver is +deleting a container, the driver marks the device as deleted. Later, when +the device isn't in use, the driver deletes it. -* `dm.min_free_space` +In general it should be safe to enable this option by default. It will help +when unintentional leaking of mount point happens across multiple mount +namespaces. - Specifies the min free space percent in a thin pool require for new device - creation to succeed. This check applies to both free data space as well - as free metadata space. Valid values are from 0% - 99%. Value 0% disables - free space checking logic. If user does not specify a value for this option, - the Engine uses a default value of 10%. +##### `dm.min_free_space` - Whenever a new a thin pool device is created (during `docker pull` or during - container creation), the Engine checks if the minimum free space is - available. If sufficient space is unavailable, then device creation fails - and any relevant `docker` operation fails. +Specifies the min free space percent in a thin pool require for new device +creation to succeed. This check applies to both free data space as well +as free metadata space. Valid values are from 0% - 99%. Value 0% disables +free space checking logic. If user does not specify a value for this option, +the Engine uses a default value of 10%. - To recover from this error, you must create more free space in the thin pool - to recover from the error. You can create free space by deleting some images - and containers from the thin pool. You can also add more storage to the thin - pool. +Whenever a new a thin pool device is created (during `docker pull` or during +container creation), the Engine checks if the minimum free space is +available. If sufficient space is unavailable, then device creation fails +and any relevant `docker` operation fails. - To add more space to a LVM (logical volume management) thin pool, just add - more storage to the volume group container thin pool; this should automatically - resolve any errors. If your configuration uses loop devices, then stop the - Engine daemon, grow the size of loop files and restart the daemon to resolve - the issue. +To recover from this error, you must create more free space in the thin pool +to recover from the error. You can create free space by deleting some images +and containers from the thin pool. You can also add more storage to the thin +pool. - Example use: +To add more space to a LVM (logical volume management) thin pool, just add +more storage to the volume group container thin pool; this should automatically +resolve any errors. If your configuration uses loop devices, then stop the +Engine daemon, grow the size of loop files and restart the daemon to resolve +the issue. - ```bash - $ sudo dockerd --storage-opt dm.min_free_space=10% - ``` +###### Example -* `dm.xfs_nospace_max_retries` +```bash +$ sudo dockerd --storage-opt dm.min_free_space=10% +``` - Specifies the maximum number of retries XFS should attempt to complete - IO when ENOSPC (no space) error is returned by underlying storage device. +##### `dm.xfs_nospace_max_retries` - By default XFS retries infinitely for IO to finish and this can result - in unkillable process. To change this behavior one can set - xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting - ENOSPC and will shutdown filesystem. +Specifies the maximum number of retries XFS should attempt to complete +IO when ENOSPC (no space) error is returned by underlying storage device. - Example use: +By default XFS retries infinitely for IO to finish and this can result +in unkillable process. To change this behavior one can set +xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting +ENOSPC and will shutdown filesystem. - ```bash - $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 - ``` +###### Example + +```bash +$ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 +``` #### ZFS options -* `zfs.fsname` +##### `zfs.fsname` - Set zfs filesystem under which docker will create its own datasets. - By default docker will pick up the zfs filesystem where docker graph - (`/var/lib/docker`) is located. +Set zfs filesystem under which docker will create its own datasets. +By default docker will pick up the zfs filesystem where docker graph +(`/var/lib/docker`) is located. - Example use: +###### Example - ```bash - $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker - ``` +```bash +$ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker +``` #### Btrfs options -* `btrfs.min_space` +##### `btrfs.min_space` - Specifies the minimum size to use when creating the subvolume which is used - for containers. If user uses disk quota for btrfs when creating or running - a container with **--storage-opt size** option, docker should ensure the - **size** cannot be smaller than **btrfs.min_space**. +Specifies the minimum size to use when creating the subvolume which is used +for containers. If user uses disk quota for btrfs when creating or running +a container with **--storage-opt size** option, docker should ensure the +**size** cannot be smaller than **btrfs.min_space**. - Example use: +###### Example - ```bash - $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G - ``` +```bash +$ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G +``` #### Overlay2 options -* `overlay2.override_kernel_check` +##### `overlay2.override_kernel_check` - Overrides the Linux kernel version check allowing overlay2. Support for - specifying multiple lower directories needed by overlay2 was added to the - Linux kernel in 4.0.0. However, some older kernel versions may be patched - to add multiple lower directory support for OverlayFS. This option should - only be used after verifying this support exists in the kernel. Applying - this option on a kernel without this support will cause failures on mount. +Overrides the Linux kernel version check allowing overlay2. Support for +specifying multiple lower directories needed by overlay2 was added to the +Linux kernel in 4.0.0. However, some older kernel versions may be patched +to add multiple lower directory support for OverlayFS. This option should +only be used after verifying this support exists in the kernel. Applying +this option on a kernel without this support will cause failures on mount. -## Docker runtime execution options +### Docker runtime execution options The Docker daemon relies on a [OCI](https://github.com/opencontainers/runtime-spec) compliant runtime @@ -707,9 +707,9 @@ This is the same example via the command line: $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement ``` -> **Note**: defining runtime arguments via the command line is not supported. +> **Note**: Defining runtime arguments via the command line is not supported. -## Options for the runtime +#### Options for the runtime You can configure the runtime using options specified with the `--exec-opt` flag. All the flag's options have the `native` prefix. A @@ -739,7 +739,7 @@ Will make `hyperv` the default isolation technology on Windows. If no isolation value is specified on daemon start, on Windows client, the default is `hyperv`, and on Windows server, the default is `process`. -## Daemon DNS options +#### Daemon DNS options To set the DNS server for all Docker containers, use: @@ -753,7 +753,7 @@ To set the DNS search domain for all Docker containers, use: $ sudo dockerd --dns-search example.com ``` -## Insecure registries +#### Insecure registries Docker considers a private registry either secure or insecure. In the rest of this section, *registry* is used for *private registry*, and `myregistry:5000` @@ -797,11 +797,11 @@ because its use creates security vulnerabilities it should ONLY be enabled for testing purposes. For increased security, users should add their CA to their system's list of trusted CAs instead of enabling `--insecure-registry`. -## Legacy Registries +##### Legacy Registries Enabling `--disable-legacy-registry` forces a docker daemon to only interact with registries which support the V2 protocol. Specifically, the daemon will not attempt `push`, `pull` and `login` to v1 registries. The exception to this is `search` which can still be performed on v1 registries. -## Running a Docker daemon behind an HTTPS_PROXY +#### Running a Docker daemon behind an HTTPS_PROXY When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub certificates will be replaced by the proxy's certificates. These certificates @@ -818,7 +818,7 @@ This will only add the proxy and authentication to the Docker daemon's requests your `docker build`s and running containers will need extra configuration to use the proxy -## Default Ulimits +#### Default `ulimit` settings `--default-ulimit` allows you to set the default `ulimit` options to use for all containers. It takes the same options as `--ulimit` for `docker run`. If @@ -830,7 +830,7 @@ Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linu set the maximum number of processes available to a user, not to a container. For details please check the [run](run.md) reference. -## Nodes discovery +#### Node discovery The `--cluster-advertise` option specifies the `host:port` or `interface:port` combination that this particular daemon instance should use when advertising @@ -856,39 +856,16 @@ $ sudo dockerd \ The currently supported cluster store options are: -* `discovery.heartbeat` +| Option | Description | +|-----------------------|-------------| +| `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | +| `discovery.ttl` | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. | +| `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | +| `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | +| `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | +| `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | - Specifies the heartbeat timer in seconds which is used by the daemon as a - keepalive mechanism to make sure discovery module treats the node as alive - in the cluster. If not configured, the default value is 20 seconds. - -* `discovery.ttl` - - Specifies the ttl (time-to-live) in seconds which is used by the discovery - module to timeout a node if a valid heartbeat is not received within the - configured ttl value. If not configured, the default value is 60 seconds. - -* `kv.cacertfile` - - Specifies the path to a local file with PEM encoded CA certificates to trust - -* `kv.certfile` - - Specifies the path to a local file with a PEM encoded certificate. This - certificate is used as the client cert for communication with the - Key/Value store. - -* `kv.keyfile` - - Specifies the path to a local file with a PEM encoded private key. This - private key is used as the client key for communication with the - Key/Value store. - -* `kv.path` - - Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. - -## Access authorization +#### Access authorization Docker's access authorization can be extended by authorization plugins that your organization can purchase or build themselves. You can install one or more @@ -913,7 +890,7 @@ For information about how to create an authorization plugin, see [authorization plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. -## Daemon user namespace options +#### Daemon user namespace options The Linux kernel [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling a process, and therefore a container, to have a unique range of user and @@ -946,7 +923,7 @@ and provided subordinate uid and gid ranges. This default user will be named > pull`, `docker push`, and container startup as users expect with > user namespaces disabled. -### Starting the daemon with user namespaces enabled +##### Start the daemon with user namespaces enabled To enable user namespace support, start the daemon with the `--userns-remap` flag, which accepts values in the following formats: @@ -993,7 +970,7 @@ with user namespaces enabled or not. If the daemon is configured with user namespaces, the Security Options entry in the response will list "userns" as one of the enabled security features. -#### Behavior differences when user namespaces are enabled +##### Behavior differences when user namespaces are enabled When you start the Docker daemon with `--userns-remap`, Docker segregates the graph directory where the images are stored by adding an extra directory with a name corresponding to the @@ -1006,7 +983,7 @@ images and container layers, are also owned by the new UID and GID. To set the o correctly, you need to re-pull the images and restart the containers after starting the daemon with `--userns-remap`. -### Detailed information on `subuid`/`subgid` ranges +##### Detailed information on `subuid`/`subgid` ranges Given potential advanced use of the subordinate ID ranges by power users, the following paragraphs define how the Docker daemon currently uses the range entries @@ -1029,7 +1006,7 @@ following algorithm to create the mapping ranges: 2. Map segments will be created from each range in increasing value with a length matching the length of each segment. Therefore the range segment with the lowest numeric starting value will be equal to the remapped root, and continue up through host uid/gid equal to the range segment length. As an example, if the lowest segment starts at ID 1000 and has a length of 100, then a map of 1000 -> 0 (the remapped root) up through 1100 -> 100 will be created from this segment. If the next segment starts at ID 10000, then the next map will start with mapping 10000 -> 101 up to the length of this second segment. This will continue until no more segments are found in the subordinate files for this user. 3. If more than five range segments exist for a single user, only the first five will be utilized, matching the kernel's limitation of only five entries in `/proc/self/uid_map` and `proc/self/gid_map`. -### Disable user namespace for a container +##### Disable user namespace for a container If you enable user namespaces on the daemon, all containers are started with user namespaces enabled. In some situations you might want to disable @@ -1039,7 +1016,7 @@ To enable those advanced features for a specific container use `--userns=host` in the `run/exec/create` command. This option will completely disable user namespace mapping for the container's user. -### User namespace known restrictions +##### User namespace known restrictions The following standard Docker features are currently incompatible when running a Docker daemon with user namespaces enabled: @@ -1062,7 +1039,7 @@ process. The most notable restriction that we are aware of at this time is the inability to use `mknod`. Permission will be denied for device creation even as container `root` inside a user namespace. -## Miscellaneous options +### Miscellaneous options IP masquerading uses address translation to allow containers without a public IP to talk to other machines on the Internet. This may interfere with some @@ -1077,7 +1054,7 @@ set like this: export DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 -## Default cgroup parent +#### Default cgroup parent The `--cgroup-parent` option allows you to set the default cgroup parent to use for containers. If this option is not set, it defaults to `/docker` for @@ -1104,7 +1081,7 @@ This setting can also be set per container, using the `--cgroup-parent` option on `docker create` and `docker run`, and takes precedence over the `--cgroup-parent` option on the daemon. -## Daemon Metrics +#### Daemon metrics The `--metrics-addr` option takes a tcp address to serve the metrics API. This feature is still experimental, therefore, the daemon must be running in experimental @@ -1118,7 +1095,7 @@ If you are running a prometheus server you can add this address to your scrape c to have prometheus collect metrics on Docker. For more information on prometheus you can view the website [here](https://prometheus.io/). -```yml +```none scrape_configs: - job_name: 'docker' static_configs: @@ -1129,7 +1106,7 @@ Please note that this feature is still marked as experimental as metrics and met names could change while this feature is still in experimental. Please provide feedback on what you would like to see collected in the API. -## Daemon configuration file +#### Daemon configuration file The `--config-file` option allows you to set any configuration option for the daemon in a JSON format. This file uses the same flag names as keys, @@ -1144,7 +1121,7 @@ For example, the daemon fails to start if you set daemon labels in the configuration file and also set daemon labels via the `--label` flag. Options that are not present in the file are ignored when the daemon starts. -### Linux configuration file +##### On Linux The default location of the configuration file on Linux is `/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a @@ -1229,7 +1206,7 @@ This is a full example of the allowed configuration options on Linux: } ``` -### Windows configuration file +##### On Windows The default location of the configuration file on Windows is `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be @@ -1276,7 +1253,7 @@ This is a full example of the allowed configuration options on Windows: } ``` -### Configuration reloading +#### Configuration reload behavior Some options can be reconfigured when the daemon is running without requiring to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event @@ -1313,7 +1290,7 @@ Configuration reload will log a warning message if it detects a change in previously configured cluster configurations. -## Running multiple daemons +### Run multiple daemons > **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of > unsolved problems. This solution may not work properly in some cases. Solutions are currently under development @@ -1326,7 +1303,7 @@ by providing them as flags, or by using a [daemon configuration file](#daemon-co The following daemon options must be configured for each daemon: -```bash +```none -b, --bridge= Attach containers to a network bridge --exec-root=/var/run/docker Root of the Docker execdriver -g, --graph=/var/lib/docker Root of the Docker runtime From 9058a43af2b0259fe5a7c860d101f264cf0bb928 Mon Sep 17 00:00:00 2001 From: Yong Tang Date: Sun, 12 Feb 2017 15:15:15 -0800 Subject: [PATCH 219/534] Add missing link in deprecated.md This fix adds a missing link in deprecated.md ``` Deprecated In Release: [v1.13.0]... ``` Signed-off-by: Yong Tang --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1298370ba9..a88e398079 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -23,7 +23,7 @@ see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-depreca ### Top-level network properties in NetworkSettings -**Deprecated In Release: v1.13.0** +**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Target For Removal In Release: v1.16** From 8de64caac16d3fb35bec3087134487cd19a52106 Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Fri, 10 Feb 2017 14:08:36 -0800 Subject: [PATCH 220/534] Add plugin socket related debug docs. Signed-off-by: Anusha Ragunathan --- docs/extend/index.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/extend/index.md b/docs/extend/index.md index a2aa208d6b..2b0cbeec2e 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -289,3 +289,28 @@ follows: $ docker-runc exec -t f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 sh ``` +#### Using curl to debug plugin socket issues. + +To verify if the plugin API socket that the docker daemon communicates with +is responsive, use curl. In this example, we will make API calls from the +docker host to volume and network plugins using curl 7.47.0 to ensure that +the plugin is listening on the said socket. For a well functioning plugin, +these basic requests should work. Note that plugin sockets are available on the host under `/var/run/docker/plugins/` + + +```bash +curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/e8a37ba56fc879c991f7d7921901723c64df6b42b87e6a0b055771ecf8477a6d/plugin.sock http:/VolumeDriver.List + +{"Mountpoint":"","Err":"","Volumes":[{"Name":"myvol1","Mountpoint":"/data/myvol1"},{"Name":"myvol2","Mountpoint":"/data/myvol2"}],"Volume":null} +``` + +```bash +curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/45e00a7ce6185d6e365904c8bcf62eb724b1fe307e0d4e7ecc9f6c1eb7bcdb70/plugin.sock http:/NetworkDriver.GetCapabilities + +{"Scope":"local"} +``` +When using curl 7.5 and above, the URL should be of the form +`http://hostname/APICall`, where `hostname` is the valid hostname where the +plugin is installed and `APICall` is the call to the plugin API. + +For example, `http://localhost/VolumeDriver.List` From 470920b7b2c3761a613123df35632e42ea74154b Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Fri, 27 Jan 2017 16:51:44 -0800 Subject: [PATCH 221/534] Update plugin_volume apidocs for v2 Signed-off-by: Misty Stanley-Jones --- docs/extend/plugins_volume.md | 252 ++++++++++++++++++++++------------ 1 file changed, 168 insertions(+), 84 deletions(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 0a79efa022..807ab5a486 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -16,67 +16,86 @@ keywords: "Examples, Usage, volume, docker, data, volumes, plugin, api" # Write a volume plugin Docker Engine volume plugins enable Engine deployments to be integrated with -external storage systems, such as Amazon EBS, and enable data volumes to persist -beyond the lifetime of a single Engine host. See the +external storage systems such as Amazon EBS, and enable data volumes to persist +beyond the lifetime of a single Docker host. See the [plugin documentation](legacy_plugins.md) for more information. ## Changelog ### 1.13.0 -- If used as part of the v2 plugin architecture, mountpoints that are part of paths returned by plugin have to be mounted under the directory specified by PropagatedMount in the plugin configuration [#26398](https://github.com/docker/docker/pull/26398) +- If used as part of the v2 plugin architecture, mountpoints that are part of + paths returned by the plugin must be mounted under the directory specified by + `PropagatedMount` in the plugin configuration + ([#26398](https://github.com/docker/docker/pull/26398)) ### 1.12.0 -- Add `Status` field to `VolumeDriver.Get` response ([#21006](https://github.com/docker/docker/pull/21006#)) -- Add `VolumeDriver.Capabilities` to get capabilities of the volume driver([#22077](https://github.com/docker/docker/pull/22077)) +- Add `Status` field to `VolumeDriver.Get` response + ([#21006](https://github.com/docker/docker/pull/21006#)) +- Add `VolumeDriver.Capabilities` to get capabilities of the volume driver + ([#22077](https://github.com/docker/docker/pull/22077)) ### 1.10.0 -- Add `VolumeDriver.Get` which gets the details about the volume ([#16534](https://github.com/docker/docker/pull/16534)) -- Add `VolumeDriver.List` which lists all volumes owned by the driver ([#16534](https://github.com/docker/docker/pull/16534)) +- Add `VolumeDriver.Get` which gets the details about the volume + ([#16534](https://github.com/docker/docker/pull/16534)) +- Add `VolumeDriver.List` which lists all volumes owned by the driver + ([#16534](https://github.com/docker/docker/pull/16534)) ### 1.8.0 -- Initial support for volume driver plugins ([#14659](https://github.com/docker/docker/pull/14659)) +- Initial support for volume driver plugins + ([#14659](https://github.com/docker/docker/pull/14659)) ## Command-line changes -A volume plugin makes use of the `-v`and `--volume-driver` flag on the `docker run` command. The `-v` flag accepts a volume name and the `--volume-driver` flag a driver type, for example: +To give a container access to a volume, use the `--volume` and `--volume-driver` +flags on the `docker container run` command. The `--volume` (or `-v`) flag +accepts a volume name and path on the host, and the `--volume-driver` flag +accepts a driver type. - $ docker run -ti -v volumename:/data --volume-driver=flocker busybox sh +```bash +$ docker volume create --driver=flocker volumename -This command passes the `volumename` through to the volume plugin as a -user-given name for the volume. The `volumename` must not begin with a `/`. +$ docker container run -it --volume volumename:/data busybox sh +``` -By having the user specify a `volumename`, a plugin can associate the volume -with an external volume beyond the lifetime of a single container or container -host. This can be used, for example, to move a stateful container from one -server to another. +### `--volume` -By specifying a `volumedriver` in conjunction with a `volumename`, users can use plugins such as [Flocker](https://clusterhq.com/docker-plugin/) to manage volumes external to a single host, such as those on EBS. +The `--volume` (or `-v`) flag takes a value that is in the format +`:`. The two parts of the value are +separated by a colon (`:`) character. +- The volume name is a human-readable name for the volume, and cannot begin with + a `/` character. It is referred to as `volume_name` in the rest of this topic. +- The `Mountpoint` is the path on the host (v1) or in the plugin (v2) where the + volume has been made available. + +### `volumedriver` + +Specifying a `volumedriver` in conjunction with a `volumename` allows you to +use plugins such as [Flocker](https://github.com/ScatterHQ/flocker) to manage +volumes external to a single host, such as those on EBS. ## Create a VolumeDriver The container creation endpoint (`/containers/create`) accepts a `VolumeDriver` -field of type `string` allowing to specify the name of the driver. It's default -value of `"local"` (the default driver for local volumes). +field of type `string` allowing to specify the name of the driver. If not +specified, it defaults to `"local"` (the default driver for local volumes). ## Volume plugin protocol -If a plugin registers itself as a `VolumeDriver` when activated, then it is -expected to provide writeable paths on the host filesystem for the Docker -daemon to provide to containers to consume. - -The Docker daemon handles bind-mounting the provided paths into user -containers. +If a plugin registers itself as a `VolumeDriver` when activated, it must +provide the Docker Daemon with writeable paths on the host filesystem. The Docker +daemon provides these paths to containers to consume. The Docker daemon makes +the volumes available by bind-mounting the provided paths into the containers. > **Note**: Volume plugins should *not* write data to the `/var/lib/docker/` > directory, including `/var/lib/docker/volumes`. The `/var/lib/docker/` > directory is reserved for Docker. -### /VolumeDriver.Create +### `/VolumeDriver.Create` **Request**: ```json @@ -87,9 +106,9 @@ containers. ``` Instruct the plugin that the user wants to create a volume, given a user -specified volume name. The plugin does not need to actually manifest the -volume on the filesystem yet (until Mount is called). -Opts is a map of driver specific options passed through from the user request. +specified volume name. The plugin does not need to actually manifest the +volume on the filesystem yet (until `Mount` is called). +`Opts` is a map of driver specific options passed through from the user request. **Response**: ```json @@ -100,7 +119,7 @@ Opts is a map of driver specific options passed through from the user request. Respond with a string error if an error occurred. -### /VolumeDriver.Remove +### `/VolumeDriver.Remove` **Request**: ```json @@ -109,7 +128,8 @@ Respond with a string error if an error occurred. } ``` -Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. +Delete the specified volume from disk. This request is issued when a user +invokes `docker rm -v` to remove volumes associated with a container. **Response**: ```json @@ -120,7 +140,7 @@ Delete the specified volume from disk. This request is issued when a user invoke Respond with a string error if an error occurred. -### /VolumeDriver.Mount +### `/VolumeDriver.Mount` **Request**: ```json @@ -131,47 +151,76 @@ Respond with a string error if an error occurred. ``` Docker requires the plugin to provide a volume, given a user specified volume -name. This is called once per container start. If the same volume_name is requested +name. `Mount` is called once per container start. If the same `volume_name` is requested more than once, the plugin may need to keep track of each new mount request and provision at the first mount request and deprovision at the last corresponding unmount request. `ID` is a unique ID for the caller that is requesting the mount. **Response**: -```json -{ - "Mountpoint": "/path/to/directory/on/host", - "Err": "" -} -``` -Respond with the path on the host filesystem where the volume has been made -available, and/or a string error if an error occurred. +- **v1**: -### /VolumeDriver.Path + ```json + { + "Mountpoint": "/path/to/directory/on/host", + "Err": "" + } + ``` + +- **v2**: + + ```json + { + "Mountpoint": "/path/under/PropagatedMount", + "Err": "" + } + ``` + +`Mountpoint` is the path on the host (v1) or in the plugin (v2) where the volume +has been made available. + +`Err` is either empty or contains an error string. + +### `/VolumeDriver.Path` **Request**: + ```json { "Name": "volume_name" } ``` -Docker needs reminding of the path to the volume on the host. +Request the path to the volume with the given `volume_name`. **Response**: -```json -{ - "Mountpoint": "/path/to/directory/on/host", - "Err": "" -} -``` -Respond with the path on the host filesystem where the volume has been made -available, and/or a string error if an error occurred. `Mountpoint` is optional, -however, the plugin may be queried again later if one is not provided. +- **v1**: -### /VolumeDriver.Unmount + ```json + { + "Mountpoin": "/path/to/directory/on/host", + "Err": "" + } + ``` + +- **v2**: + + ```json + { + "Mountpoint": "/path/under/PropagatedMount", + "Err": "" + } + ``` + +Respond with the path on the host (v1) or inside the plugin (v2) where the +volume has been made available, and/or a string error if an error occurred. + +`Mountpoint` is optional. However, the plugin may be queried again later if one +is not provided. + +### `/VolumeDriver.Unmount` **Request**: ```json @@ -181,8 +230,8 @@ however, the plugin may be queried again later if one is not provided. } ``` -Indication that Docker no longer is using the named volume. This is called once -per container stop. Plugin may deduce that it is safe to deprovision it at +Docker is no longer using the named volume. `Unmount` is called once per +container stop. Plugin may deduce that it is safe to deprovision the volume at this point. `ID` is a unique ID for the caller that is requesting the mount. @@ -197,7 +246,7 @@ this point. Respond with a string error if an error occurred. -### /VolumeDriver.Get +### `/VolumeDriver.Get` **Request**: ```json @@ -206,20 +255,36 @@ Respond with a string error if an error occurred. } ``` -Get the volume info. +Get info about `volume_name`. **Response**: -```json -{ - "Volume": { - "Name": "volume_name", - "Mountpoint": "/path/to/directory/on/host", - "Status": {} - }, - "Err": "" -} -``` + +- **v1**: + + ```json + { + "Volume": { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host", + "Status": {} + }, + "Err": "" + } + ``` + +- **v2**: + + ```json + { + "Volume": { + "Name": "volume_name", + "Mountpoint": "/path/under/PropagatedMount", + "Status": {} + }, + "Err": "" + } + ``` Respond with a string error if an error occurred. `Mountpoint` and `Status` are optional. @@ -235,17 +300,35 @@ optional. Get the list of volumes registered with the plugin. **Response**: -```json -{ - "Volumes": [ - { - "Name": "volume_name", - "Mountpoint": "/path/to/directory/on/host" - } - ], - "Err": "" -} -``` + +- **v1**: + + ```json + { + "Volumes": [ + { + "Name": "volume_name", + "Mountpoint": "/path/to/directory/on/host" + } + ], + "Err": "" + } + ``` + +- **v2**: + + ```json + { + "Volumes": [ + { + "Name": "volume_name", + "Mountpoint": "/path/under/PropagatedMount" + } + ], + "Err": "" + } + ``` + Respond with a string error if an error occurred. `Mountpoint` is optional. @@ -257,8 +340,9 @@ Respond with a string error if an error occurred. `Mountpoint` is optional. ``` Get the list of capabilities the driver supports. -The driver is not required to implement this endpoint, however, in such cases -the default values will be taken. + +The driver is not required to implement `Capabilities`. If it is not +implemented, the default values are used. **Response**: ```json @@ -270,7 +354,7 @@ the default values will be taken. ``` Supported scopes are `global` and `local`. Any other value in `Scope` will be -ignored and assumed to be `local`. Scope allows cluster managers to handle the -volume differently, for instance with a scope of `global`, the cluster manager -knows it only needs to create the volume once instead of on every engine. More -capabilities may be added in the future. +ignored, and `local` is used. `Scope` allows cluster managers to handle the +volume in different ways. For instance, a scope of `global`, signals to the +cluster manager that it only needs to create the volume once instead of on each +Docker host. More capabilities may be added in the future. From 83826b9fa07a8ff9d3a4bd2f801e22af385ac58a Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Tue, 14 Feb 2017 16:40:15 -0800 Subject: [PATCH 222/534] Fix some Liquid errors Signed-off-by: Misty Stanley-Jones --- docs/deprecated.md | 6 +++++- docs/extend/index.md | 26 ++++++++++++++------------ docs/extend/plugins_graphdriver.md | 4 ++++ 3 files changed, 23 insertions(+), 13 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a88e398079..9d3da96322 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -183,7 +183,11 @@ Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and `fluentd-tag` have been deprecated in favor of the generic `tag` option. - docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" +```bash +{% raw %} +docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" +{% endraw %} +``` ### LXC built-in exec driver **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** diff --git a/docs/extend/index.md b/docs/extend/index.md index 2b0cbeec2e..99e02a8bab 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -20,7 +20,7 @@ title: Managed plugin system * [Debugging plugins](index.md#debugging-plugins) Docker Engine's plugins system allows you to install, start, stop, and remove -plugins using Docker Engine. +plugins using Docker Engine. For information about the legacy plugin system available in Docker Engine 1.12 and earlier, see [Understand legacy Docker Engine plugins](legacy_plugins.md). @@ -78,7 +78,7 @@ enabled, and use it to create a volume. 3. Create a volume using the plugin. This example mounts the `/remote` directory on host `1.2.3.4` into a volume named `sshvolume`. - + This volume can now be mounted into containers. ```bash @@ -110,7 +110,7 @@ enabled, and use it to create a volume. 6. Remove the volume `sshvolume` ```bash docker volume rm sshvolume - + sshvolume ``` To disable a plugin, use the `docker plugin disable` command. To completely @@ -122,10 +122,10 @@ commands and options, see the In swarm mode, it is possible to create a service that allows for attaching to networks or mounting volumes. Swarm schedules services based on plugin availability -on a node. In this example, a volume plugin is installed on a swarm worker and a volume +on a node. In this example, a volume plugin is installed on a swarm worker and a volume is created using the plugin. In the manager, a service is created with the relevant mount options. It can be observed that the service is scheduled to run on the worker -node with the said volume plugin and volume. +node with the said volume plugin and volume. In the following example, node1 is the manager and node2 is the worker. @@ -152,7 +152,7 @@ In the following example, node1 is the manager and node2 is the worker. Status: Downloaded newer image for tiborvass/sample-volume-plugin:latest Installed plugin tiborvass/sample-volume-plugin ``` - + ```bash $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol ``` @@ -163,15 +163,17 @@ In the following example, node1 is the manager and node2 is the worker. $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top $ docker service ls - z1sj8bb8jnfn my-service replicated 1/1 busybox:latest + z1sj8bb8jnfn my-service replicated 1/1 busybox:latest ``` docker service ls shows service 1 instance of service running. 4. Observe the task getting scheduled in node 2: ```bash - $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' + {% raw %} + $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" + {% endraw %} ``` ## Developing a plugin @@ -228,7 +230,7 @@ with Docker Engine. This plugin has no runtime parameters. A new plugin can be created by running `docker plugin create ./path/to/plugin/data` where the plugin data contains a plugin configuration file `config.json` and a root filesystem -in subdirectory `rootfs`. +in subdirectory `rootfs`. After that the plugin `` will show up in `docker plugin ls`. Plugins can be pushed to remote registries with @@ -292,7 +294,7 @@ $ docker-runc exec -t f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b8 #### Using curl to debug plugin socket issues. To verify if the plugin API socket that the docker daemon communicates with -is responsive, use curl. In this example, we will make API calls from the +is responsive, use curl. In this example, we will make API calls from the docker host to volume and network plugins using curl 7.47.0 to ensure that the plugin is listening on the said socket. For a well functioning plugin, these basic requests should work. Note that plugin sockets are available on the host under `/var/run/docker/plugins/` @@ -309,8 +311,8 @@ curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/d {"Scope":"local"} ``` -When using curl 7.5 and above, the URL should be of the form -`http://hostname/APICall`, where `hostname` is the valid hostname where the +When using curl 7.5 and above, the URL should be of the form +`http://hostname/APICall`, where `hostname` is the valid hostname where the plugin is installed and `APICall` is the call to the plugin API. For example, `http://localhost/VolumeDriver.List` diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index e30632692f..74caa938f8 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -287,7 +287,9 @@ and `Parent`. `Parent` may be an empty string, in which case there is no parent. **Response**: ``` +{% raw %} {{ TAR STREAM }} +{% endraw %} ``` ### /GraphDriver.Changes @@ -330,7 +332,9 @@ Respond with a non-empty string error if an error occurred. **Request**: ``` +{% raw %} {{ TAR STREAM }} +{% endraw %} ``` Extract the changeset from the given diff into the layer with the specified `ID` From a368f2b5fb1c3a4a876ce54568c19be3ee977538 Mon Sep 17 00:00:00 2001 From: Daniel Zhang Date: Mon, 9 Jan 2017 09:22:05 +0800 Subject: [PATCH 223/534] Add daemon flag to set no_new_priv as default for unprivileged containers. Signed-off-by: Daniel Zhang --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1ebdc3b666..84f4d998ed 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -70,6 +70,7 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) --metrics-addr string Set address and port to serve the metrics api (default "") --mtu int Set the containers network MTU + --no-new-privileges Disable container processes from gaining new privileges --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring @@ -1190,6 +1191,7 @@ This is a full example of the allowed configuration options on Linux: "seccomp-profile": "", "insecure-registries": [], "disable-legacy-registry": false, + "no-new-privileges": false, "default-runtime": "runc", "oom-score-adjust": -500, "runtimes": { From 0e83bac7fef06cbf02826938f87239f590125b80 Mon Sep 17 00:00:00 2001 From: yupengzte Date: Wed, 8 Feb 2017 16:31:16 +0800 Subject: [PATCH 224/534] fix the type Signed-off-by: yupengzte --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 84f4d998ed..e42b7340f7 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -64,7 +64,7 @@ Options: --label value Set key=value labels to the daemon (default []) --live-restore Enable live restore of docker when containers are still running (Linux only) --log-driver string Default driver for container logs (default "json-file") - -l, --log-level string Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info") + -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") --log-opt value Default log driver options for containers (default map[]) --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) From d22d51d03436c8b3c29a6d7ff2539945f438c436 Mon Sep 17 00:00:00 2001 From: Harald Albers Date: Thu, 23 Feb 2017 13:49:00 +0100 Subject: [PATCH 225/534] Clarify why `hosts` in `daemon.json` does not work Signed-off-by: Harald Albers --- docs/reference/commandline/dockerd.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e42b7340f7..92b9349a41 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1208,6 +1208,13 @@ This is a full example of the allowed configuration options on Linux: } ``` +> **Note:** You cannot set options in `daemon.json` that have already been set on +> daemon startup as a flag. +> On systems that use `systemd` to start the Docker daemon, `-H` is already set, so +> you cannot use the `hosts` key in `daemon.json` to add listening addresses. +> See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how +> to accomplish this task with a systemd drop-in file. + ##### On Windows The default location of the configuration file on Windows is From 05722517770bb4a13b6dc9a1af3111d2deed4b3d Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Wed, 15 Feb 2017 10:33:28 -0800 Subject: [PATCH 226/534] Service creation using network plugin filters. Signed-off-by: Anusha Ragunathan --- docs/extend/index.md | 57 ---------- docs/extend/plugins_services.md | 184 ++++++++++++++++++++++++++++++++ 2 files changed, 184 insertions(+), 57 deletions(-) create mode 100644 docs/extend/plugins_services.md diff --git a/docs/extend/index.md b/docs/extend/index.md index 99e02a8bab..5ff8728507 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -118,63 +118,6 @@ remove it, use the `docker plugin remove` command. For other available commands and options, see the [command line reference](../reference/commandline/index.md). -## Service creation using plugins - -In swarm mode, it is possible to create a service that allows for attaching -to networks or mounting volumes. Swarm schedules services based on plugin availability -on a node. In this example, a volume plugin is installed on a swarm worker and a volume -is created using the plugin. In the manager, a service is created with the relevant -mount options. It can be observed that the service is scheduled to run on the worker -node with the said volume plugin and volume. - -In the following example, node1 is the manager and node2 is the worker. - -1. Prepare manager. In node 1: - - ```bash - $ docker swarm init - Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager. - ``` - -2. Join swarm, install plugin and create volume on worker. In node 2: - - ```bash - $ docker swarm join \ - --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ - 192.168.99.100:2377 - ``` - - ```bash - $ docker plugin install tiborvass/sample-volume-plugin - latest: Pulling from tiborvass/sample-volume-plugin - eb9c16fbdc53: Download complete - Digest: sha256:00b42de88f3a3e0342e7b35fa62394b0a9ceb54d37f4c50be5d3167899994639 - Status: Downloaded newer image for tiborvass/sample-volume-plugin:latest - Installed plugin tiborvass/sample-volume-plugin - ``` - - ```bash - $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol - ``` - -3. Create a service using the plugin and volume. In node1: - - ```bash - $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top - - $ docker service ls - z1sj8bb8jnfn my-service replicated 1/1 busybox:latest - ``` - docker service ls shows service 1 instance of service running. - -4. Observe the task getting scheduled in node 2: - - ```bash - {% raw %} - $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' - 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" - {% endraw %} - ``` ## Developing a plugin diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md new file mode 100644 index 0000000000..d68218f8f8 --- /dev/null +++ b/docs/extend/plugins_services.md @@ -0,0 +1,184 @@ +--- +description: Using services with plugins +keywords: "API, Usage, plugins, documentation, developer" +title: Plugins and Services +--- + + + +# Using Volume and Network plugins in Docker services + +In swarm mode, it is possible to create a service that allows for attaching +to networks or mounting volumes that are backed by plugins. Swarm schedules +services based on plugin availability on a node. + + +### Volume plugins + +In this example, a volume plugin is installed on a swarm worker and a volume +is created using the plugin. In the manager, a service is created with the +relevant mount options. It can be observed that the service is scheduled to +run on the worker node with the said volume plugin and volume. Note that, +node1 is the manager and node2 is the worker. + +1. Prepare manager. In node 1: + + ```bash + $ docker swarm init + Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager. + ``` + +2. Join swarm, install plugin and create volume on worker. In node 2: + + ```bash + $ docker swarm join \ + --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ + 192.168.99.100:2377 + ``` + + ```bash + $ docker plugin install tiborvass/sample-volume-plugin + latest: Pulling from tiborvass/sample-volume-plugin + eb9c16fbdc53: Download complete + Digest: sha256:00b42de88f3a3e0342e7b35fa62394b0a9ceb54d37f4c50be5d3167899994639 + Status: Downloaded newer image for tiborvass/sample-volume-plugin:latest + Installed plugin tiborvass/sample-volume-plugin + ``` + + ```bash + $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol + ``` + +3. Create a service using the plugin and volume. In node1: + + ```bash + $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top + + $ docker service ls + z1sj8bb8jnfn my-service replicated 1/1 busybox:latest + ``` + docker service ls shows service 1 instance of service running. + +4. Observe the task getting scheduled in node 2: + + ```bash + $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' + 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" + ``` + +### Network plugins + +In this example, a global scope network plugin is installed on both the +swarm manager and worker. A service is created with replicated instances +using the installed plugin. We will observe how the availability of the +plugin determines network creation and container scheduling. + +Note that node1 is the manager and node2 is the worker. + + +1. Install a global scoped network plugin on both manager and worker. On node1 + and node2: + + ```bash + $ docker plugin install bboreham/weave2 + Plugin "bboreham/weave2" is requesting the following privileges: + - network: [host] + - capabilities: [CAP_SYS_ADMIN CAP_NET_ADMIN] + Do you grant the above permissions? [y/N] y + latest: Pulling from bboreham/weave2 + 7718f575adf7: Download complete + Digest: sha256:2780330cc15644b60809637ee8bd68b4c85c893d973cb17f2981aabfadfb6d72 + Status: Downloaded newer image for bboreham/weave2:latest + Installed plugin bboreham/weave2 + ``` + +2. Create a network using plugin on manager. On node1: + + ```bash + $ docker network create --driver=bboreham/weave2:latest globalnet + + $ docker network ls + NETWORK ID NAME DRIVER SCOPE + qlj7ueteg6ly globalnet bboreham/weave2:latest swarm + ``` + +3. Create a service on the manager and have replicas set to 8. Observe that +containers get scheduled on both manager and worker. + + On node 1: + + ```bash + $ docker service create --network globalnet --name myservice --replicas=8 mrjana/simpleweb simpleweb +w90drnfzw85nygbie9kb89vpa + ``` + + ```bash + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 87520965206a mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 5 seconds ago Up 4 seconds myservice.4.ytdzpktmwor82zjxkh118uf1v + 15e24de0f7aa mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 5 seconds ago Up 4 seconds myservice.2.kh7a9n3iauq759q9mtxyfs9hp + c8c8f0144cdc mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 5 seconds ago Up 4 seconds myservice.6.sjhpj5gr3xt33e3u2jycoj195 + 2e8e4b2c5c08 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 5 seconds ago Up 4 seconds myservice.8.2z29zowsghx66u2velublwmrh + ``` + + On node 2: + + ```bash + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 53c0ae7c1dae mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 2 seconds ago Up Less than a second myservice.7.x44tvvdm3iwkt9kif35f7ykz1 + 9b56c627fee0 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 2 seconds ago Up Less than a second myservice.1.x7n1rm6lltw5gja3ueikze57q + d4f5927ba52c mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 2 seconds ago Up 1 second myservice.5.i97bfo9uc6oe42lymafs9rz6k + 478c0d395bd7 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 2 seconds ago Up Less than a second myservice.3.yr7nkffa48lff1vrl2r1m1ucs + ``` + +4. Scale down the number of instances. On node1: + + ```bash + $ docker service scale myservice=0 + myservice scaled to 0 + ``` + +5. Disable and uninstall the plugin on the worker. On node2: + + ```bash + $ docker plugin rm -f bboreham/weave2 + bboreham/weave2 + ``` + +6. Scale up the number of instances again. Observe that all containers are +scheduled on the master and not on the worker, because the plugin is not available on the worker anymore. + + On node 1: + + ```bash + $ docker service scale myservice=8 + myservice scaled to 8 + ``` + + ```bash + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + cf4b0ec2415e mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.3.r7p5o208jmlzpcbm2ytl3q6n1 + 57c64a6a2b88 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.4.dwoezsbb02ccstkhlqjy2xe7h + 3ac68cc4e7b8 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 35 seconds myservice.5.zx4ezdrm2nwxzkrwnxthv0284 + 006c3cb318fc mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.8.q0e3umt19y3h3gzo1ty336k5r + dd2ffebde435 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.7.a77y3u22prjipnrjg7vzpv3ba + a86c74d8b84b mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.6.z9nbn14bagitwol1biveeygl7 + 2846a7850ba0 mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 37 seconds myservice.2.ypufz2eh9fyhppgb89g8wtj76 + e2ec01efcd8a mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 38 seconds myservice.1.8w7c4ttzr6zcb9sjsqyhwp3yl + ``` + + On node 2: + + ```bash + $ docker ps + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + ``` From 208e64df4b9ac85cad365b044be0d80841375f32 Mon Sep 17 00:00:00 2001 From: scjane Date: Tue, 21 Feb 2017 18:38:07 +0800 Subject: [PATCH 227/534] Update index.md Signed-off-by: Wang Jie Signed-off-by: Sebastiaan van Stijn --- docs/extend/index.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 5ff8728507..b0d3404bbe 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -19,7 +19,7 @@ title: Managed plugin system * [Developing a plugin](index.md#developing-a-plugin) * [Debugging plugins](index.md#debugging-plugins) -Docker Engine's plugins system allows you to install, start, stop, and remove +Docker Engine's plugin system allows you to install, start, stop, and remove plugins using Docker Engine. For information about the legacy plugin system available in Docker Engine 1.12 @@ -34,7 +34,7 @@ Plugins are distributed as Docker images and can be hosted on Docker Hub or on a private registry. To install a plugin, use the `docker plugin install` command, which pulls the -plugin from Docker hub or your private registry, prompts you to grant +plugin from Docker Hub or your private registry, prompts you to grant permissions or capabilities if necessary, and enables the plugin. To check the status of installed plugins, use the `docker plugin ls` command. @@ -62,6 +62,7 @@ enabled, and use it to create a volume. ``` The plugin requests 2 privileges: + - It needs access to the `host` network. - It needs the `CAP_SYS_ADMIN` capability, which allows the plugin to run the `mount` command. From 1850a5efbea2170ce9ce7ac13425747c964dba47 Mon Sep 17 00:00:00 2001 From: Victor Vieux Date: Mon, 27 Feb 2017 17:16:47 -0800 Subject: [PATCH 228/534] update deprecation dates Signed-off-by: Victor Vieux --- docs/deprecated.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9d3da96322..109c3030b4 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -25,7 +25,7 @@ see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-depreca **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** When inspecting a container, `NetworkSettings` contains top-level information about the default ("bridge") network; @@ -43,28 +43,28 @@ information. ## `filter` param for `/images/json` endpoint **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** The `filter` param to filter the list of image by reference (name or name:tag) is now implemented as a regular filter, named `reference`. ### `repository:shortid` image references **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** `repository:shortid` syntax for referencing images is very little used, collides with tag references can be confused with digest references. ### `docker daemon` subcommand **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** The daemon is moved to a separate binary (`dockerd`), and should be used instead. ### Duplicate keys with conflicting values in engine labels **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** Duplicate keys with conflicting values have been deprecated. A warning is displayed in the output, and an error will be returned in the future. @@ -77,7 +77,7 @@ in the output, and an error will be returned in the future. ### API calls without a version **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** API versions should be supplied to all API calls to ensure compatibility with future Engine versions. Instead of just requesting, for example, the URL @@ -86,7 +86,7 @@ future Engine versions. Instead of just requesting, for example, the URL ### Backing filesystem without `d_type` support for overlay/overlay2 **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v1.16** +**Target For Removal In Release: v17.12** The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` @@ -106,7 +106,7 @@ The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated **Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Target For Removal In Release: v1.15** +**Target For Removal In Release: v17.09** The shorthand (`-h`) is less common than `--help` on Linux and cannot be used on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on @@ -116,14 +116,14 @@ on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v1.14** +**Target For Removal In Release: v17.06** The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v1.14** +**Target For Removal In Release: v17.06** The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. @@ -169,7 +169,7 @@ Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Target For Removal In Release: v1.15** +**Target For Removal In Release: v17.09** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. @@ -266,9 +266,9 @@ of the `--changes` flag that allows to pass `Dockerfile` commands. ### Interacting with V1 registries -**Disabled By Default In Release: v1.14** +**Disabled By Default In Release: v17.04** -**Target For Removal In Release: v1.17** +**Target For Removal In Release: v17.10** Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 From 3801121fd672eaed4a0666d9181031c60700756d Mon Sep 17 00:00:00 2001 From: Keith Hudgins Date: Mon, 30 Jan 2017 11:13:51 -0800 Subject: [PATCH 229/534] added listing for Nimble Storage's volume plugin Signed-off-by: Keith Hudgins updated nimble plugin url --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 0de2ad0633..b7c9709790 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -76,6 +76,7 @@ Plugin [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. [Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. +[Nimble Storage Volume Plugin](https://connect.nimblestorage.com/community/app-integration/docker)| A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. [Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. From 9a687a1dbc56bdd6e51ec09e7467b25ef79b8900 Mon Sep 17 00:00:00 2001 From: Harald Albers Date: Tue, 14 Mar 2017 09:54:29 +0100 Subject: [PATCH 230/534] Remove docs for `dockerd --no-new-privileges` Signed-off-by: Harald Albers --- docs/reference/commandline/dockerd.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 92b9349a41..3b40540a33 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -70,7 +70,6 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) --metrics-addr string Set address and port to serve the metrics api (default "") --mtu int Set the containers network MTU - --no-new-privileges Disable container processes from gaining new privileges --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring From f17ecc8482105648eafcbe0c3ae223bd2ca7bc90 Mon Sep 17 00:00:00 2001 From: Jimmy Song Date: Tue, 7 Mar 2017 17:06:27 +0800 Subject: [PATCH 231/534] modify docker plugin sshfs entrypoint Signed-off-by: Jimmy Song --- docs/extend/index.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index b0d3404bbe..2fe2d83687 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -150,7 +150,7 @@ Consider the following `config.json` file. { "description": "sshFS plugin for Docker", "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["/go/bin/docker-volume-sshfs"], + "entrypoint": ["/docker-volume-sshfs"], "network": { "type": "host" }, @@ -165,7 +165,7 @@ Consider the following `config.json` file. ``` This plugin is a volume driver. It requires a `host` network and the -`CAP_SYS_ADMIN` capability. It depends upon the `/go/bin/docker-volume-sshfs` +`CAP_SYS_ADMIN` capability. It depends upon the `/docker-volume-sshfs` entrypoint and uses the `/run/docker/plugins/sshfs.sock` socket to communicate with Docker Engine. This plugin has no runtime parameters. From 554fd0de4921644a455c6c9e90837b352bddfad0 Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Tue, 7 Mar 2017 18:26:09 -0800 Subject: [PATCH 232/534] Add support in plugin config for accessing host ipc namespace. Plugins might need access to host ipc namespace. A good usecase is a volume plugin running iscsi multipath commands that need access to host kernel locks. Tested with a custom plugin (aragunathan/global-net-plugin-full) that's built with `"ipchost" : true` in config.json. Observed using `readlink /proc/self/ns/ipc` that plugin and host have the same ns. Signed-off-by: Anusha Ragunathan --- docs/extend/config.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/extend/config.md b/docs/extend/config.md index dab755d97b..3fc377f76c 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -115,6 +115,9 @@ Config provides the base accessible fields for working with V0 plugin format options of the mount. +- **`ipchost`** *boolean* + Access to host ipc namespace. + - **`propagatedMount`** *string* path to be mounted as rshared, so that mounts under that path are visible to docker. This is useful for volume plugins. From 201766da0f64ef74f6712d98a134803076328d4a Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Fri, 10 Mar 2017 14:17:24 -0800 Subject: [PATCH 233/534] Add pid host support Tested using global-net-plugin-ipc which sets PidHost in config.json. Plugins might need access to host pid namespace. Add support for that. Tested using aragunathan/global-net-plugin-ipc which sets "pidhost" in config.json. Observed using `readlink /proc/self/ns/pid` that plugin and host have the same ns. Signed-off-by: Anusha Ragunathan --- docs/extend/config.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/extend/config.md b/docs/extend/config.md index 3fc377f76c..ad43e898c7 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -117,6 +117,8 @@ Config provides the base accessible fields for working with V0 plugin format - **`ipchost`** *boolean* Access to host ipc namespace. +- **`pidhost`** *boolean* + Access to host pid namespace. - **`propagatedMount`** *string* From 0aa5b9b09a8b6b968fc9e8d5efbb4989bbba1e6a Mon Sep 17 00:00:00 2001 From: Greg Pflaum Date: Wed, 22 Mar 2017 22:48:34 -0400 Subject: [PATCH 234/534] Fix indenting in table of contents Fixes reference 30841. The "filter" section had the wrong header size, which caused sections following it to have the wrong indentation in the right-side table of contents. Signed-off-by: Greg Pflaum --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 109c3030b4..3ac563b36b 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -40,7 +40,7 @@ docker 1.9, but kept around for backward compatibility. Refer to [#17538](https://github.com/docker/docker/pull/17538) for further information. -## `filter` param for `/images/json` endpoint +### `filter` param for `/images/json` endpoint **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Target For Removal In Release: v17.12** From 6955a488fb67ec4a6ac08bba55ba5c3f88614cf1 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Tue, 14 Feb 2017 16:07:50 -0800 Subject: [PATCH 235/534] Update docs for experimental features Signed-off-by: Misty Stanley-Jones Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 63 +++++++++++++++------------ 1 file changed, 35 insertions(+), 28 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3b40540a33..e9c64bd12e 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -17,33 +17,34 @@ keywords: "container, daemon, runtime" # daemon ```markdown -Usage: dockerd [OPTIONS] +Usage: dockerd COMMAND A self-sufficient runtime for containers. Options: - --add-runtime value Register an additional OCI compatible runtime (default []) + --add-runtime runtime Register an additional OCI compatible runtime (default []) --api-cors-header string Set CORS headers in the Engine API - --authorization-plugin value Authorization plugins to load (default []) + --authorization-plugin list Authorization plugins to load (default []) --bip string Specify network bridge IP -b, --bridge string Attach containers to a network bridge --cgroup-parent string Set parent cgroup for all containers --cluster-advertise string Address or interface name to advertise --cluster-store string URL of the distributed storage backend - --cluster-store-opt value Set cluster store options (default map[]) + --cluster-store-opt map Set cluster store options (default map[]) --config-file string Daemon configuration file (default "/etc/docker/daemon.json") --containerd string Path to containerd socket + --cpu-rt-period int Limit the CPU real-time period in microseconds + --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds -D, --debug Enable debug mode - --default-gateway value Container default gateway IPv4 address - --default-gateway-v6 value Container default gateway IPv6 address + --default-gateway ip Container default gateway IPv4 address + --default-gateway-v6 ip Container default gateway IPv6 address --default-runtime string Default OCI runtime for containers (default "runc") - --default-shm-size bytes Set the default shm size for containers (default 64 MiB) - --default-ulimit value Default ulimits for containers (default []) + --default-ulimit ulimit Default ulimits for containers (default []) --disable-legacy-registry Disable contacting legacy registries - --dns value DNS server to use (default []) - --dns-opt value DNS options to use (default []) - --dns-search value DNS search domains to use (default []) - --exec-opt value Runtime execution options (default []) + --dns list DNS server to use (default []) + --dns-opt list DNS options to use (default []) + --dns-search list DNS search domains to use (default []) + --exec-opt list Runtime execution options (default []) --exec-root string Root directory for execution state files (default "/var/run/docker") --experimental Enable experimental features --fixed-cidr string IPv4 subnet for fixed IPs @@ -51,39 +52,39 @@ Options: -g, --graph string Root of the Docker runtime (default "/var/lib/docker") -G, --group string Group for the unix socket (default "docker") --help Print usage - -H, --host value Daemon socket(s) to connect to (default []) + -H, --host list Daemon socket(s) to connect to (default []) --icc Enable inter-container communication (default true) --init Run an init in the container to forward signals and reap processes --init-path string Path to the docker-init binary - --insecure-registry value Enable insecure registry communication (default []) - --ip value Default IP when binding container ports (default 0.0.0.0) + --insecure-registry list Enable insecure registry communication (default []) + --ip ip Default IP when binding container ports (default 0.0.0.0) --ip-forward Enable net.ipv4.ip_forward (default true) --ip-masq Enable IP masquerading (default true) --iptables Enable addition of iptables rules (default true) --ipv6 Enable IPv6 networking - --label value Set key=value labels to the daemon (default []) - --live-restore Enable live restore of docker when containers are still running (Linux only) + --label list Set key=value labels to the daemon (default []) + --live-restore Enable live restore of docker when containers are still running --log-driver string Default driver for container logs (default "json-file") - -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") - --log-opt value Default log driver options for containers (default map[]) + -l, --log-level string Set the logging level ("debug", "info", "warn", "error", "fatal") (default "info") + --log-opt map Default log driver options for containers (default map[]) --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) - --metrics-addr string Set address and port to serve the metrics api (default "") + --metrics-addr string Set default address and port to serve the metrics api on --mtu int Set the containers network MTU --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring - --registry-mirror value Preferred Docker registry mirror (default []) - --seccomp-profile value Path to seccomp profile + --registry-mirror list Preferred Docker registry mirror (default []) + --seccomp-profile string Path to seccomp profile --selinux-enabled Enable selinux support - --shutdown-timeout=15 Set the shutdown timeout value in seconds + --shutdown-timeout int Set the default shutdown timeout (default 15) -s, --storage-driver string Storage driver to use - --storage-opt value Storage driver options (default []) + --storage-opt list Storage driver options (default []) --swarm-default-advertise-addr string Set default address or interface for swarm advertised address --tls Use TLS; implied by --tlsverify - --tlscacert string Trust certs signed only by this CA (default "/root/.docker/ca.pem") - --tlscert string Path to TLS certificate file (default "/root/.docker/cert.pem") - --tlskey string Path to TLS key file (default "/root/.docker/key.pem") + --tlscacert string Trust certs signed only by this CA (default "~/.docker/ca.pem") + --tlscert string Path to TLS certificate file (default "~/.docker/cert.pem") + --tlskey string Path to TLS key file (default ~/.docker/key.pem") --tlsverify Use TLS and verify the remote --userland-proxy Use userland proxy for loopback traffic (default true) --userland-proxy-path string Path to the userland proxy binary @@ -99,7 +100,13 @@ Options with [] may be specified multiple times. uses different binaries for the daemon and client. To run the daemon you type `dockerd`. -To run the daemon with debug output, use `dockerd -D`. +To run the daemon with debug output, use `dockerd -D` or add `debug: true` to +the `daemon.json` file. + +> **Note**: In Docker 1.13 and higher, enable experimental features by starting +> `dockerd` with the `--experimental` flag or adding `experimental: true` to the +> `daemon.json` file. In earlier Docker versions, a different build was required +> to enable experimental features. ## Examples From 40f126415f12771f427bcd7e2548e791ff00828f Mon Sep 17 00:00:00 2001 From: JimGalasyn Date: Tue, 28 Mar 2017 09:59:25 -0700 Subject: [PATCH 236/534] Delete extraneous H1 title Signed-off-by: JimGalasyn --- docs/extend/legacy_plugins.md | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index b7c9709790..3f0ea0a90b 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -15,8 +15,6 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide" will be rejected. --> -# Use Docker Engine plugins - This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker, refer to [Docker Engine plugin system](index.md). @@ -76,7 +74,7 @@ Plugin [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. [Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. -[Nimble Storage Volume Plugin](https://connect.nimblestorage.com/community/app-integration/docker)| A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. +[Nimble Storage Volume Plugin](https://connect.nimblestorage.com/community/app-integration/docker)| A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. [Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. From ec31931b6c8c81b5ba9afec90db7b544acadce66 Mon Sep 17 00:00:00 2001 From: Josh Hawn Date: Mon, 21 Nov 2016 22:17:24 -0800 Subject: [PATCH 237/534] Deprecate --graph flag; Replace with --data-root Docker-DCO-1.1-Signed-off-by: Josh Hawn (github: jlhawn) --- docs/deprecated.md | 10 ++++++++++ docs/reference/commandline/dockerd.md | 15 ++++++++------- 2 files changed, 18 insertions(+), 7 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 3ac563b36b..509f1607f8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -20,6 +20,16 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +### `-g` and `--graph` flags on `dockerd` + +**Deprecated In Release: v1.14.0** + +**Target For Removal In Release: v1.17** + +The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was +used to indicate the directory in which to store persistent data and resource +configuration and has been replaced with the more descriptive `--data-root` +flag. ### Top-level network properties in NetworkSettings diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e9c64bd12e..c9fceba7b3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -35,6 +35,7 @@ Options: --containerd string Path to containerd socket --cpu-rt-period int Limit the CPU real-time period in microseconds --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds + --data-root string Root directory of persistent Docker state (default "/var/lib/docker") -D, --debug Enable debug mode --default-gateway ip Container default gateway IPv4 address --default-gateway-v6 ip Container default gateway IPv6 address @@ -49,7 +50,6 @@ Options: --experimental Enable experimental features --fixed-cidr string IPv4 subnet for fixed IPs --fixed-cidr-v6 string IPv6 subnet for fixed IPs - -g, --graph string Root of the Docker runtime (default "/var/lib/docker") -G, --group string Group for the unix socket (default "docker") --help Print usage -H, --host list Daemon socket(s) to connect to (default []) @@ -1139,6 +1139,7 @@ This is a full example of the allowed configuration options on Linux: ```json { "authorization-plugins": [], + "data-root": "", "dns": [], "dns-opts": [], "dns-search": [], @@ -1153,7 +1154,6 @@ This is a full example of the allowed configuration options on Linux: "log-opts": {}, "mtu": 0, "pidfile": "", - "graph": "", "cluster-store": "", "cluster-store-opts": {}, "cluster-advertise": "", @@ -1232,6 +1232,7 @@ This is a full example of the allowed configuration options on Windows: ```json { "authorization-plugins": [], + "data-root": "", "dns": [], "dns-opts": [], "dns-search": [], @@ -1243,7 +1244,6 @@ This is a full example of the allowed configuration options on Windows: "log-driver": "", "mtu": 0, "pidfile": "", - "graph": "", "cluster-store": "", "cluster-advertise": "", "max-concurrent-downloads": 3, @@ -1321,7 +1321,7 @@ The following daemon options must be configured for each daemon: ```none -b, --bridge= Attach containers to a network bridge --exec-root=/var/run/docker Root of the Docker execdriver --g, --graph=/var/lib/docker Root of the Docker runtime +--data-root=/var/lib/docker Root of persisted Docker data -p, --pidfile=/var/run/docker.pid Path to use for daemon PID file -H, --host=[] Daemon socket(s) to connect to --iptables=true Enable addition of iptables rules @@ -1338,8 +1338,9 @@ It is very important to properly understand the meaning of those options and to If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none` - `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for your running daemon here. -- `--graph` is the path where images are stored. The default value is `/var/lib/docker`. To avoid any conflict with other daemons -set this parameter separately for each daemon. +- `--data-root` is the path where persisted data such as images, volumes, and +cluster state are stored. The default value is `/var/lib/docker`. To avoid any +conflict with other daemons, set this parameter separately for each daemon. - `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your pid file here. - `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. @@ -1365,6 +1366,6 @@ $ sudo dockerd \ --iptables=false \ --ip-masq=false \ --bridge=none \ - --graph=/var/lib/docker-bootstrap \ + --data-root=/var/lib/docker-bootstrap \ --exec-root=/var/run/docker-bootstrap ``` From c37f85855b410a5c45c67ab2cf9df92f55a9ce20 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 28 Mar 2017 16:19:35 +0200 Subject: [PATCH 238/534] Add conflict check for flags, and update deprecation versions Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 509f1607f8..b57bbbecfe 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -22,15 +22,16 @@ see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-depreca ### `-g` and `--graph` flags on `dockerd` -**Deprecated In Release: v1.14.0** - -**Target For Removal In Release: v1.17** +**Deprecated In Release: v17.05.0** The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource configuration and has been replaced with the more descriptive `--data-root` flag. +These flags were added before Docker 1.0, so will not be _removed_, only +_hidden_, to discourage their use. + ### Top-level network properties in NetworkSettings **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** From 86cdfc54b4a397729565df5860b7dfc98faf9246 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 31 Mar 2017 17:59:33 +0200 Subject: [PATCH 239/534] Merge docker/cli pull request 32265 from breuner/patch-1 Updated legacy_plugins.md to include BeeGFS plugin (cherry picked from commit d0057bd12dae5276b83e931e9e9ea185c58486a2) Signed-off-by: Sebastiaan van Stijn --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 3f0ea0a90b..901a40ad5d 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -58,6 +58,7 @@ Plugin Plugin | Description ----------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). +[BeeGFS Volume Plugin](https://github.com/RedCoolBeans/docker-volume-beegfs) | An open source volume plugin to create persistent volumes in a BeeGFS parallel file system. [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. [Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. From 154466d3134cd53ccd8db79736750271a4fbb621 Mon Sep 17 00:00:00 2001 From: Karthik Nayak Date: Tue, 4 Apr 2017 20:02:47 +0530 Subject: [PATCH 240/534] Docs: Add the flag `--api-enable-cors` to deprecated.md Signed-off-by: Karthik Nayak --- docs/deprecated.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b57bbbecfe..bcdfd4de1f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -299,3 +299,12 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the - DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE - DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE + +### `--api-enable-cors` flag on dockerd + +**Deprecated In Release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** + +**Target For Removal In Release: v17.09** + +The flag `--api-enable-cors` is deprecated since v1.6.0. Use the flag +`--api-cors-header` instead. From 74745c7bc2953991ce1fc2d0b61cd6dc12c4fb1a Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Tue, 4 Apr 2017 10:06:32 -0700 Subject: [PATCH 241/534] Escape some double brackets Signed-off-by: Misty Stanley-Jones --- docs/extend/plugins_services.md | 20 +++++++++++--------- 1 file changed, 11 insertions(+), 9 deletions(-) diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md index d68218f8f8..79e344f9ce 100644 --- a/docs/extend/plugins_services.md +++ b/docs/extend/plugins_services.md @@ -17,15 +17,15 @@ title: Plugins and Services In swarm mode, it is possible to create a service that allows for attaching to networks or mounting volumes that are backed by plugins. Swarm schedules -services based on plugin availability on a node. +services based on plugin availability on a node. ### Volume plugins In this example, a volume plugin is installed on a swarm worker and a volume -is created using the plugin. In the manager, a service is created with the -relevant mount options. It can be observed that the service is scheduled to -run on the worker node with the said volume plugin and volume. Note that, +is created using the plugin. In the manager, a service is created with the +relevant mount options. It can be observed that the service is scheduled to +run on the worker node with the said volume plugin and volume. Note that, node1 is the manager and node2 is the worker. 1. Prepare manager. In node 1: @@ -51,7 +51,7 @@ node1 is the manager and node2 is the worker. Status: Downloaded newer image for tiborvass/sample-volume-plugin:latest Installed plugin tiborvass/sample-volume-plugin ``` - + ```bash $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol ``` @@ -62,28 +62,30 @@ node1 is the manager and node2 is the worker. $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top $ docker service ls - z1sj8bb8jnfn my-service replicated 1/1 busybox:latest + z1sj8bb8jnfn my-service replicated 1/1 busybox:latest ``` docker service ls shows service 1 instance of service running. 4. Observe the task getting scheduled in node 2: ```bash + {% raw %} $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" + {% endraw %} ``` ### Network plugins -In this example, a global scope network plugin is installed on both the +In this example, a global scope network plugin is installed on both the swarm manager and worker. A service is created with replicated instances -using the installed plugin. We will observe how the availability of the +using the installed plugin. We will observe how the availability of the plugin determines network creation and container scheduling. Note that node1 is the manager and node2 is the worker. -1. Install a global scoped network plugin on both manager and worker. On node1 +1. Install a global scoped network plugin on both manager and worker. On node1 and node2: ```bash From 74f2a2a54de66817577eba9bbc45e616a3944d86 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Mon, 14 Nov 2016 13:53:53 -0500 Subject: [PATCH 242/534] Implement plugins for logging drivers Logging plugins use the same HTTP interface as other plugins for basic command operations meanwhile actual logging operations are handled (on Unix) via a fifo. The plugin interface looks like so: ```go type loggingPlugin interface { StartLogging(fifoPath string, loggingContext Context) error StopLogging(fifoPath) ``` This means a plugin must implement `LoggingDriver.StartLogging` and `LoggingDriver.StopLogging` endpoints and be able to consume the passed in fifo. Logs are sent via stream encoder to the fifo encoded with protobuf. Signed-off-by: Brian Goff --- docs/extend/config.md | 2 + docs/extend/plugins_logging.md | 220 +++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+) create mode 100644 docs/extend/plugins_logging.md diff --git a/docs/extend/config.md b/docs/extend/config.md index ad43e898c7..4feb6bf315 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -59,6 +59,8 @@ Config provides the base accessible fields for working with V0 plugin format - **docker.authz/1.0** + - **docker.logdriver/1.0** + - **`socket`** *string* socket is the name of the socket the engine should use to communicate with the plugins. diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md new file mode 100644 index 0000000000..fd02a6a6f7 --- /dev/null +++ b/docs/extend/plugins_logging.md @@ -0,0 +1,220 @@ +--- +title: "Docker log driver plugins" +description: "Log driver plugins." +keywords: "Examples, Usage, plugins, docker, documentation, user guide, logging" +--- + + + +# Logging driver plugins + +This document describes logging driver plugins for Docker. + +Logging drivers enables users to forward container logs to another service for +processing. Docker includes several logging drivers as built-ins, however can +never hope to support all use-cases with built-in drivers. Plugins allow Docker +to support a wide range of logging services without requiring to embed client +libraries for these services in the main Docker codebase. See the +[plugin documentation](legacy_plugins.md) for more information. + +## Create a logging plugin + +The main interface for logging plugins uses the same JSON+HTTP RPC protocol used +by other plugin types. See the +[example](https://github.com/cpuguy83/docker-log-driver-test) plugin for a +reference implementation of a logging plugin. The example wraps the built-in +`jsonfilelog` log driver. + +## LogDriver protocol + +Logging plugins must register as a `LogDriver` during plugin activation. Once +activated users can specify the plugin as a log driver. + +There are two HTTP endpoints that logging plugins must implement: + +### `/LogDriver.StartLogging` + +Signals to the plugin that a container is starting that the plugin should start +receiving logs for. + +Logs will be streamed over the defined file in the request. On Linux this file +is a FIFO. Logging plugins are not currently supported on Windows. + +**Request**: +```json +{ + "File": "/path/to/file/stream", + "Info": { + "ContainerID": "123456" + } +} +``` + +`File` is the path to the log stream that needs to be consumed. Each call to +`StartLogging` should provide a different file path, even if it's a container +that the plugin has already received logs for prior. The file is created by +docker with a randomly generated name. + +`Info` is details about the container that's being logged. This is fairly +free-form, but is defined by the following struct definition: + +```go +type Info struct { + Config map[string]string + ContainerID string + ContainerName string + ContainerEntrypoint string + ContainerArgs []string + ContainerImageID string + ContainerImageName string + ContainerCreated time.Time + ContainerEnv []string + ContainerLabels map[string]string + LogPath string + DaemonName string +} +``` + + +`ContainerID` will always be supplied with this struct, but other fields may be +empty or missing. + +**Response** +```json +{ + "Err": "" +} +``` + +If an error occurred during this request, add an error message to the `Err` field +in the response. If no error then you can either send an empty response (`{}`) +or an empty value for the `Err` field. + +The driver should at this point be consuming log messages from the passed in file. +If messages are unconsumed, it may cause the contaier to block while trying to +write to its stdio streams. + +Log stream messages are encoded as protocol buffers. The protobuf definitions are +in the +[docker repository](https://github.com/docker/docker/blob/master/api/types/plugins/logdriver/entry.proto). + +Since protocol buffers are not self-delimited you must decode them from the stream +using the following stream format: + +``` +[size][message] +``` + +Where `size` is a 4-byte big endian binary encoded uint32. `size` in this case +defines the size of the next message. `message` is the actual log entry. + +A reference golang implementation of a stream encoder/decoder can be found +[here](https://github.com/docker/docker/blob/master/api/types/plugins/logdriver/io.go) + +### `/LogDriver.StopLogging` + +Signals to the plugin to stop collecting logs from the defined file. +Once a response is received, the file will be removed by Docker. You must make +sure to collect all logs on the stream before responding to this request or risk +losing log data. + +Requests on this endpoint does not mean that the container has been removed +only that it has stopped. + +**Request**: +```json +{ + "File": "/path/to/file/stream" +} +``` + +**Response**: +```json +{ + "Err": "" +} +``` + +If an error occurred during this request, add an error message to the `Err` field +in the response. If no error then you can either send an empty response (`{}`) +or an empty value for the `Err` field. + +## Optional endpoints + +Logging plugins can implement two extra logging endpoints: + +### `/LogDriver.Capabilities` + +Defines the capabilities of the log driver. You must implement this endpoint for +Docker to be able to take advantage of any of the defined capabilities. + +**Request**: +```json +{} +``` + +**Response**: +```json +{ + "ReadLogs": true +} +``` + +Supported capabilities: + +- `ReadLogs` - this tells Docker that the plugin is capable of reading back logs +to clients. Plugins that report that they support `ReadLogs` must implement the +`/LogDriver.ReadLogs` endpoint + +### `/LogDriver.ReadLogs` + +Reads back logs to the client. This is used when `docker logs ` is +called. + +In order for Docker to use this endpoint, the plugin must specify as much when +`/LogDriver.Capabilities` is called. + + +**Request**: +```json +{ + "ReadConfig": {}, + "Info": { + "ContainerID": "123456" + } +} +``` + +`ReadConfig` is the list of options for reading, it is defined with the following +golang struct: + +```go +type ReadConfig struct { + Since time.Time + Tail int + Follow bool +} +``` + +- `Since` defines the oldest log that should be sent. +- `Tail` defines the number of lines to read (e.g. like the command `tail -n 10`) +- `Follow` signals that the client wants to stay attached to receive new log messages +as they come in once the existing logs have been read. + +`Info` is the same type defined in `/LogDriver.StartLogging`. It should be used +to determine what set of logs to read. + +**Response**: +``` +{{ log stream }} +``` + +The response should be the encoded log message using the same format as the +messages that the plugin consumed from Docker. From 46079bf8ec2831c6ea82f85d725fd9c99a1af2f2 Mon Sep 17 00:00:00 2001 From: Vincent Demeester Date: Thu, 16 Mar 2017 10:08:27 +0100 Subject: [PATCH 243/534] Merge docker/cli pull request 31848 from thaJeztah/17.04.0-changelog-updates [17.04.x] deprecation and changelog updates (cherry picked from commit f15f5937514c386ace40a43441dddc28337ca12d) Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index bcdfd4de1f..8f6070a810 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -277,9 +277,9 @@ of the `--changes` flag that allows to pass `Dockerfile` commands. ### Interacting with V1 registries -**Disabled By Default In Release: v17.04** +**Disabled By Default In Release: v17.06** -**Target For Removal In Release: v17.10** +**Target For Removal In Release: v17.12** Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 From 96684ab1fc407cd8213c3fea4c5ad3523935c890 Mon Sep 17 00:00:00 2001 From: Alfred Landrum Date: Mon, 20 Mar 2017 11:38:17 -0700 Subject: [PATCH 244/534] Let graphdrivers declare diff stream fidelity This allows graphdrivers to declare that they can reproduce the original diff stream for a layer. If they do so, the layer store will not use tar-split processing, but will still verify the digest on layer export. This makes it easier to experiment with non-default diff formats. Signed-off-by: Alfred Landrum --- docs/extend/plugins_graphdriver.md | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index 74caa938f8..c134b1ebc4 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -84,6 +84,29 @@ The request also includes a list of UID and GID mappings, structed as follows: Respond with a non-empty string error if an error occurred. +### /GraphDriver.Capabilities + +**Request**: +```json +{} +``` + +Get behavioral characteristics of the graph driver. If a plugin does not handle +this request, the engine will use default values for all capabilities. + +**Response**: +```json +{ + "ReproducesExactDiffs": false, +} +``` + +Respond with values of capabilities: + +* **ReproducesExactDiffs** Defaults to false. Flags that this driver is capable +of reproducing exactly equivalent diffs for read-only filesystem layers. + + ### /GraphDriver.Create **Request**: From 5cda8f90e3eec57107ca5a3f29ba9cc03f11aacb Mon Sep 17 00:00:00 2001 From: Alvin Deng Date: Wed, 19 Apr 2017 16:35:04 -0500 Subject: [PATCH 245/534] Add examples of storage-opts and log-opts for the daemon Signed-off-by: Alvin Deng --- docs/reference/commandline/dockerd.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index c9fceba7b3..b523a1aaa8 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -302,6 +302,24 @@ options for `zfs` start with `zfs` and options for `btrfs` start with `btrfs`. #### Devicemapper options +This is an example of the configuration file for devicemapper on Linux: + +```json +{ + "storage-driver": "devicemapper", + "storage-opts": [ + "dm.thinpooldev=/dev/mapper/thin-pool", + "dm.use_deferred_deletion=true", + "dm.use_deferred_removal=true" + ], + "log-driver": "json-file", + "log-opts": { + "max-size": "10m", + "max-file": "10" + } +} +``` + ##### `dm.thinpooldev` Specifies a custom block storage device to use for the thin pool. From 361640b54f66c0e5ea9c587dacf4fbd3ba549305 Mon Sep 17 00:00:00 2001 From: Alvin Deng Date: Thu, 20 Apr 2017 21:26:12 -0500 Subject: [PATCH 246/534] Update dockerd.md Signed-off-by: Alvin Deng --- docs/reference/commandline/dockerd.md | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b523a1aaa8..1fc3ee196f 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -311,12 +311,7 @@ This is an example of the configuration file for devicemapper on Linux: "dm.thinpooldev=/dev/mapper/thin-pool", "dm.use_deferred_deletion=true", "dm.use_deferred_removal=true" - ], - "log-driver": "json-file", - "log-opts": { - "max-size": "10m", - "max-file": "10" - } + ] } ``` From 1087ec74edd70205ba83858cccd8f006d6de09ab Mon Sep 17 00:00:00 2001 From: Julien Kassar Date: Mon, 24 Apr 2017 19:45:59 -0400 Subject: [PATCH 247/534] Update legacy_plugins.md to include HBM authz plugin Signed-off-by: Julien Kassar --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 901a40ad5d..68bba59f46 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -88,6 +88,7 @@ Plugin Plugin | Description ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). + [HBM plugin](https://github.com/kassisol/hbm) | An authorization plugin that prevents from executing commands with certains parameters. ## Troubleshooting a plugin From e592b4b5b84b64ccd1607b19e1e567f90cfaca0b Mon Sep 17 00:00:00 2001 From: yuexiao-wang Date: Fri, 28 Apr 2017 01:59:39 +0800 Subject: [PATCH 248/534] fix errors in config Signed-off-by: yuexiao-wang --- docs/extend/config.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 4feb6bf315..c7444525fc 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -164,27 +164,27 @@ Config provides the base accessible fields for working with V0 plugin format - **`capabilities`** *string array* - capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) + capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) - **`allowAllDevices`** *boolean* - If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. + If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. - **`devices`** *PluginDevice array* - device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) + device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) - - **`name`** *string* + - **`name`** *string* - name of the device. + name of the device. - - **`description`** *string* + - **`description`** *string* - description of the device. + description of the device. - - **`path`** *string* + - **`path`** *string* - path of the device. + path of the device. ## Example Config From 7708e0cb9795d2859140785a0e2d434b4e82a79f Mon Sep 17 00:00:00 2001 From: Michael Friis Date: Thu, 27 Apr 2017 21:29:36 -0700 Subject: [PATCH 249/534] fix typo in plugins_logging.md Signed-off-by: Michael Friis --- docs/extend/plugins_logging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md index fd02a6a6f7..8f687b4a01 100644 --- a/docs/extend/plugins_logging.md +++ b/docs/extend/plugins_logging.md @@ -98,7 +98,7 @@ in the response. If no error then you can either send an empty response (`{}`) or an empty value for the `Err` field. The driver should at this point be consuming log messages from the passed in file. -If messages are unconsumed, it may cause the contaier to block while trying to +If messages are unconsumed, it may cause the container to block while trying to write to its stdio streams. Log stream messages are encoded as protocol buffers. The protobuf definitions are From 329893c51414d38f25aeb452075276b19701fd6f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 11 Apr 2017 12:15:18 +0200 Subject: [PATCH 250/534] Deprecate "asynchronous" service create and service update Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 8f6070a810..7e0bfc0a60 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -20,6 +20,17 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +### Asynchronous `service create` and `service update` + +**Deprecated In Release: v17.05.0** + +**Disabled by default in release: v17.09** + +Docker 17.05.0 added an optional `--detach=false` option to make the +`docker service create` and `docker service update` work synchronously. This +option will be enable by default in Docker 17.09, at which point the `--detach` +flag can be used to use the previous (asynchronous) behavior. + ### `-g` and `--graph` flags on `dockerd` **Deprecated In Release: v17.05.0** From 6b43237b4667dc8260910d91318bc75937a40d95 Mon Sep 17 00:00:00 2001 From: David Sheets Date: Mon, 1 May 2017 14:27:42 +0100 Subject: [PATCH 251/534] docs/dockerd: correct authz plugin chain semantics Signed-off-by: David Sheets --- docs/reference/commandline/dockerd.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1fc3ee196f..b7381079b0 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -901,10 +901,10 @@ file. The plugin's implementation determines whether you can specify a name or path. Consult with your Docker administrator to get information about the plugins available to you. -Once a plugin is installed, requests made to the `daemon` through the command -line or Docker's Engine API are allowed or denied by the plugin. If you have -multiple plugins installed, at least one must allow the request for it to -complete. +Once a plugin is installed, requests made to the `daemon` through the +command line or Docker's Engine API are allowed or denied by the plugin. +If you have multiple plugins installed, each plugin, in order, must +allow the request for it to complete. For information about how to create an authorization plugin, see [authorization plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. From 617ff3f0acd65220fc7839b491eccbea7eb3dec7 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Mon, 1 May 2017 10:15:03 -0400 Subject: [PATCH 252/534] Add no-new-privileg flag The daemon config for defaulting to no-new-privileges for containers was added in d7fda019bb7e24f42f8ae1ddecb3fd52df3c48bf, but somehow we managed to omit the flag itself, but also documented the flag. This just adds the actual flag. Signed-off-by: Brian Goff --- docs/reference/commandline/dockerd.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b7381079b0..22cda151f3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -71,6 +71,7 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) --metrics-addr string Set default address and port to serve the metrics api on --mtu int Set the containers network MTU + --no-new-privileges Set no-new-privileges by default for new containers --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring From 2c5c84ed83c6888801450302df73c91e74db2055 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Thu, 16 Feb 2017 15:33:03 -0500 Subject: [PATCH 253/534] Add option to auto-configure blkdev for devmapper Instead of forcing users to manually configure a block device to use with devmapper, this gives the user the option to let the devmapper driver configure a device for them. Adds several new options to the devmapper storage-opts: - dm.directlvm_device="" - path to the block device to configure for direct-lvm - dm.thinp_percent=95 - sets the percentage of space to use for storage from the passed in block device - dm.thinp_metapercent=1 - sets the percentage of space to for metadata storage from the passed in block device - dm.thinp_autoextend_threshold=80 - sets the threshold for when `lvm` should automatically extend the thin pool as a percentage of the total storage space - dm.thinp_autoextend_percent=20 - sets the percentage to increase the thin pool by when an autoextend is triggered. Defaults are taken from [here](https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#/configure-direct-lvm-mode-for-production) The only option that is required is `dm.directlvm_device` for docker to set everything up. Changes to these settings are not currently supported and will error out. Future work could support allowing changes to these values. Signed-off-by: Brian Goff --- docs/reference/commandline/dockerd.md | 54 +++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 22cda151f3..5f28f6f21c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -343,6 +343,60 @@ not use loopback in production. Ensure your Engine daemon has a $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool ``` +##### `dm.directlvm_device` + +As an alternative to providing a thin pool as above, Docker can setup a block +device for you. + +###### Example: + +```bash +$ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf +``` + +##### `dm.thinp_percent` + +Sets the percentage of passed in block device to use for storage. + +###### Example: + +```bash +$ sudo dockerd --storage-opt dm.thinp_percent=95 +``` + +##### `dm.thinp_metapercent` + +Sets the percentage of the passed in block device to use for metadata storage. + +###### Example: + +```bash +$ sudo dockerd --storage-opt dm.thinp_metapercent=1 +``` + +##### `dm.thinp_autoextend_threshold` + +Sets the value of the percentage of space used before `lvm` attempts to +autoextend the available space [100 = disabled] + +###### Example: + +```bash +$ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 +``` + +##### `dm.thinp_autoextend_percent` + +Sets the value percentage value to increase the thin pool by when when `lvm` +attempts to autoextend the available space [100 = disabled] + +###### Example: + +```bash +$ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20 +``` + + ##### `dm.basesize` Specifies the size to use when creating the base device, which limits the From 02c3f2e388cc756ba6e4a8d5e28661848a29018e Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Thu, 13 Apr 2017 21:56:50 -0400 Subject: [PATCH 254/534] Add support for metrics plugins Allows for a plugin type that can be used to scrape metrics. This is useful because metrics are not neccessarily at a standard location... `--metrics-addr` must be set, and must currently be a TCP socket. Even if metrics are done via a unix socket, there's no guarentee where the socket may be located on the system, making bind-mounting such a socket into a container difficult (and racey, failure-prone on daemon restart). Metrics plugins side-step this issue by always listening on a unix socket and then bind-mounting that into a known path in the plugin container. Note there has been similar work in the past (and ultimately punted at the time) for consistent access to the Docker API from within a container. Why not add metrics to the Docker API and just provide a plugin with access to the Docker API? Certainly this can be useful, but gives a lot of control/access to a plugin that may only need the metrics. We can look at supporting API plugins separately for this reason. Signed-off-by: Brian Goff --- docs/extend/config.md | 2 + docs/extend/plugins_metrics.md | 85 ++++++++++++++++++++++++++++++++++ 2 files changed, 87 insertions(+) create mode 100644 docs/extend/plugins_metrics.md diff --git a/docs/extend/config.md b/docs/extend/config.md index c7444525fc..bb6c7f2ceb 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -61,6 +61,8 @@ Config provides the base accessible fields for working with V0 plugin format - **docker.logdriver/1.0** + - **docker.metricscollector/1.0** + - **`socket`** *string* socket is the name of the socket the engine should use to communicate with the plugins. diff --git a/docs/extend/plugins_metrics.md b/docs/extend/plugins_metrics.md new file mode 100644 index 0000000000..a86c7f22d2 --- /dev/null +++ b/docs/extend/plugins_metrics.md @@ -0,0 +1,85 @@ +--- +title: "Docker metrics collector plugins" +description: "Metrics plugins." +keywords: "Examples, Usage, plugins, docker, documentation, user guide, metrics" +--- + + + +# Metrics Collector Plugins + +Docker exposes internal metrics based on the prometheus format. Metrics plugins +enable accessing these metrics in a consistent way by providing a Unix +socket at a predefined path where the plugin can scrape the metrics. + +> **Note**: that while the plugin interface for metrics is non-experimental, the naming +of the metrics and metric labels is still considered experimental and may change +in a future version. + +## Creating a metrics plugin + +You must currently set `PropagatedMount` in the plugin `config.json` to +`/run/docker`. This allows the plugin to receive updated mounts +(the bind-mounted socket) from Docker after the plugin is already configured. + +## MetricsCollector protocol + +Metrics plugins must register as implementing the`MetricsCollector` interface +in `config.json`. + +On Unix platforms, the socket is located at `/run/docker/metrics.sock` in the +plugin's rootfs. + +`MetricsCollector` must implement two endpoints: + +### `MetricsCollector.StartMetrics` + +Signals to the plugin that the metrics socket is now available for scraping + +**Request** +```json +{} +``` + +The request has no playload. + +**Response** +```json +{ + "Err": "" +} +``` + +If an error occurred during this request, add an error message to the `Err` field +in the response. If no error then you can either send an empty response (`{}`) +or an empty value for the `Err` field. Errors will only be logged. + +### `MetricsCollector.StopMetrics` + +Signals to the plugin that the metrics socket is no longer available. +This may happen when the daemon is shutting down. + +**Request** +```json +{} +``` + +The request has no playload. + +**Response** +```json +{ + "Err": "" +} +``` + +If an error occurred during this request, add an error message to the `Err` field +in the response. If no error then you can either send an empty response (`{}`) +or an empty value for the `Err` field. Errors will only be logged. From 86448c3645415690977b0fab6e9d9867c85776fb Mon Sep 17 00:00:00 2001 From: Noah Treuhaft Date: Tue, 9 May 2017 14:00:31 -0700 Subject: [PATCH 255/534] Add daemon option to push foreign layers The --allow-nondistributable-artifacts daemon option specifies registries to which foreign layers should be pushed. (By default, foreign layers are not pushed to registries.) Additionally, to make this option effective, foreign layers are now pulled from the registry if possible, falling back to the URLs in the image manifest otherwise. This option is useful when pushing images containing foreign layers to a registry on an air-gapped network so hosts on that network can pull the images without connecting to another server. Signed-off-by: Noah Treuhaft --- docs/reference/commandline/dockerd.md | 30 +++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5f28f6f21c..93774c841b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -23,6 +23,7 @@ A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) + --allow-nondistributable-artifacts list Push nondistributable artifacts to specified registries (default []) --api-cors-header string Set CORS headers in the Engine API --authorization-plugin list Authorization plugins to load (default []) --bip string Specify network bridge IP @@ -828,6 +829,32 @@ To set the DNS search domain for all Docker containers, use: $ sudo dockerd --dns-search example.com ``` +#### Allow push of nondistributable artifacts + +Some images (e.g., Windows base images) contain artifacts whose distribution is +restricted by license. When these images are pushed to a registry, restricted +artifacts are not included. + +To override this behavior for specific registries, use the +`--allow-nondistributable-artifacts` option in one of the following forms: + +* `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon + to push nondistributable artifacts to myregistry:5000. +* `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to + push nondistributable artifacts to all registries whose resolved IP address + is within the subnet described by the CIDR syntax. + +This option can be used multiple times. + +This option is useful when pushing images containing nondistributable artifacts +to a registry on an air-gapped network so hosts on that network can pull the +images without connecting to another server. + +> **Warning**: Nondistributable artifacts typically have restrictions on how +> and where they can be distributed and shared. Only use this feature to push +> artifacts to private registries and ensure that you are in compliance with +> any terms that cover redistributing nondistributable artifacts. + #### Insecure registries Docker considers a private registry either secure or insecure. In the rest of @@ -1261,6 +1288,7 @@ This is a full example of the allowed configuration options on Linux: "default-gateway-v6": "", "icc": false, "raw-logs": false, + "allow-nondistributable-artifacts": [], "registry-mirrors": [], "seccomp-profile": "", "insecure-registries": [], @@ -1330,6 +1358,7 @@ This is a full example of the allowed configuration options on Windows: "bridge": "", "fixed-cidr": "", "raw-logs": false, + "allow-nondistributable-artifacts": [], "registry-mirrors": [], "insecure-registries": [], "disable-legacy-registry": false @@ -1361,6 +1390,7 @@ The list of currently supported options that can be reconfigured is this: - `runtimes`: it updates the list of available OCI runtimes that can be used to run containers - `authorization-plugin`: specifies the authorization plugins to use. +- `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. From bfdb9bc14d869150faa8e7e6213d666e1154f69b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 1 Jun 2017 17:05:42 +0200 Subject: [PATCH 256/534] Update deprecated.md for removal of --email flag Signed-off-by: Sebastiaan van Stijn (cherry picked from commit 43239f62bedc4721d27744d21c122622988bb3ae) Signed-off-by: Tibor Vass --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7e0bfc0a60..e1e7e12645 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -138,7 +138,7 @@ on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on ### `-e` and `--email` flags on `docker login` **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v17.06** +**Removed In Release: [v17.06](https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce)** The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. From 6dff63c3ef9f3e19ee6f5425f29888745e2e1579 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 8 Jun 2017 01:09:45 +0200 Subject: [PATCH 257/534] Merge docker/cli pull request 33572 from hsluoyz/patch-1 Add Casbin plugin to the list of Authorization plugins in docs. (cherry picked from commit 220831d541bfe9bf566c1038773198d431560dd3) Signed-off-by: Sebastiaan van Stijn (cherry picked from commit 0ad3e3294e74f443130b5e1fb1ef6b31f4f92366) Signed-off-by: Tibor Vass --- docs/extend/legacy_plugins.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 68bba59f46..dc77743b86 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -87,8 +87,9 @@ Plugin Plugin | Description ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ - [Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). - [HBM plugin](https://github.com/kassisol/hbm) | An authorization plugin that prevents from executing commands with certains parameters. +[Casbin AuthZ Plugin](https://github.com/casbin/casbin-authz-plugin) | An authorization plugin based on [Casbin](https://github.com/casbin/casbin), which supports access control models like ACL, RBAC, ABAC. The access control model can be customized. The policy can be persisted into file or DB. +[HBM plugin](https://github.com/kassisol/hbm) | An authorization plugin that prevents from executing commands with certains parameters. +[Twistlock AuthZ Broker](https://github.com/twistlock/authz) | A basic extendable authorization plugin that runs directly on the host or inside a container. This plugin allows you to define user policies that it evaluates during authorization. Basic authorization is provided if Docker daemon is started with the --tlsverify flag (username is extracted from the certificate common name). ## Troubleshooting a plugin From 703cb68d5c4780a813a90de3e112ba3deeb125c9 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sun, 11 Jun 2017 14:47:23 +0200 Subject: [PATCH 258/534] Update docs, completion scripts for disable-legacy-registry Signed-off-by: Sebastiaan van Stijn (cherry picked from commit 2b8f0eef7338f37104464154ba65aef7db3b9703) Signed-off-by: Tibor Vass --- docs/deprecated.md | 2 +- docs/reference/commandline/dockerd.md | 15 +++++++++++++-- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index e1e7e12645..9905994b73 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -292,7 +292,7 @@ of the `--changes` flag that allows to pass `Dockerfile` commands. **Target For Removal In Release: v17.12** -Version 1.9 adds a flag (`--disable-legacy-registry=false`) which prevents the +Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though enabled by default, this signals the intent to deprecate the v1 protocol. diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 93774c841b..ef6bcd3831 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -42,7 +42,7 @@ Options: --default-gateway-v6 ip Container default gateway IPv6 address --default-runtime string Default OCI runtime for containers (default "runc") --default-ulimit ulimit Default ulimits for containers (default []) - --disable-legacy-registry Disable contacting legacy registries + --disable-legacy-registry Disable contacting legacy registries (default true) --dns list DNS server to use (default []) --dns-opt list DNS options to use (default []) --dns-search list DNS search domains to use (default []) @@ -901,7 +901,18 @@ system's list of trusted CAs instead of enabling `--insecure-registry`. ##### Legacy Registries -Enabling `--disable-legacy-registry` forces a docker daemon to only interact with registries which support the V2 protocol. Specifically, the daemon will not attempt `push`, `pull` and `login` to v1 registries. The exception to this is `search` which can still be performed on v1 registries. +Operations against registries supporting only the legacy v1 protocol are +disabled by default. Specifically, the daemon will not attempt `push`, +`pull` and `login` to v1 registries. The exception to this is `search` +which can still be performed on v1 registries. + +Add `"disable-legacy-registry":false` to the [daemon configuration +file](#daemon-configuration-file), or set the +`--disable-legacy-registry=false` flag, if you need to interact with +registries that have not yet migrated to the v2 protocol. + +Interaction v1 registries will no longer be supported in Docker v17.12, +and the `disable-legacy-registry` configuration option will be removed. #### Running a Docker daemon behind an HTTPS_PROXY From b66da7036e92e5cecd39ecb5cef69e450f18ded3 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Wed, 21 Jun 2017 11:01:54 -0700 Subject: [PATCH 259/534] Fix small typo Signed-off-by: Misty Stanley-Jones --- docs/extend/plugins_authorization.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index ac1837f754..5b49d657ef 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -27,7 +27,7 @@ same is true for callers using Docker's Engine API to contact the daemon. If you require greater access control, you can create authorization plugins and add them to your Docker daemon configuration. Using an authorization plugin, a Docker administrator can configure granular access policies for managing access -to Docker daemon. +to the Docker daemon. Anyone with the appropriate skills can develop an authorization plugin. These skills, at their most basic, are knowledge of Docker, understanding of REST, and From 623a78c9b599d14f8f53a2b3e9844df3405898ea Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Thu, 22 Jun 2017 15:09:14 -0700 Subject: [PATCH 260/534] Remove specific config info about userns-remap Signed-off-by: Misty Stanley-Jones --- docs/reference/commandline/dockerd.md | 170 +++----------------------- 1 file changed, 18 insertions(+), 152 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ef6bcd3831..95a953f1c4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -969,14 +969,14 @@ $ sudo dockerd \ The currently supported cluster store options are: -| Option | Description | -|-----------------------|-------------| -| `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | +| Option | Description | +|:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | | `discovery.ttl` | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. | -| `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | -| `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | -| `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | -| `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | +| `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | +| `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | +| `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | +| `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | #### Access authorization @@ -1005,152 +1005,18 @@ plugin](../../extend/plugins_authorization.md) section in the Docker extend sect #### Daemon user namespace options -The Linux kernel [user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling -a process, and therefore a container, to have a unique range of user and -group IDs which are outside the traditional user and group range utilized by -the host system. Potentially the most important security improvement is that, -by default, container processes running as the `root` user will have expected -administrative privilege (with some restrictions) inside the container but will -effectively be mapped to an unprivileged `uid` on the host. +The Linux kernel +[user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) +provides additional security by enabling a process, and therefore a container, +to have a unique range of user and group IDs which are outside the traditional +user and group range utilized by the host system. Potentially the most important +security improvement is that, by default, container processes running as the +`root` user will have expected administrative privilege (with some restrictions) +inside the container but will effectively be mapped to an unprivileged `uid` on +the host. -When user namespace support is enabled, Docker creates a single daemon-wide mapping -for all containers running on the same engine instance. The mappings will -utilize the existing subordinate user and group ID feature available on all modern -Linux distributions. -The [`/etc/subuid`](http://man7.org/linux/man-pages/man5/subuid.5.html) and -[`/etc/subgid`](http://man7.org/linux/man-pages/man5/subgid.5.html) files will be -read for the user, and optional group, specified to the `--userns-remap` -parameter. If you do not wish to specify your own user and/or group, you can -provide `default` as the value to this flag, and a user will be created on your behalf -and provided subordinate uid and gid ranges. This default user will be named -`dockremap`, and entries will be created for it in `/etc/passwd` and -`/etc/group` using your distro's standard user and group creation tools. - -> **Note**: The single mapping per-daemon restriction is in place for now -> because Docker shares image layers from its local cache across all -> containers running on the engine instance. Since file ownership must be -> the same for all containers sharing the same layer content, the decision -> was made to map the file ownership on `docker pull` to the daemon's user and -> group mappings so that there is no delay for running containers once the -> content is downloaded. This design preserves the same performance for `docker -> pull`, `docker push`, and container startup as users expect with -> user namespaces disabled. - -##### Start the daemon with user namespaces enabled - -To enable user namespace support, start the daemon with the -`--userns-remap` flag, which accepts values in the following formats: - - - uid - - uid:gid - - username - - username:groupname - -If numeric IDs are provided, translation back to valid user or group names -will occur so that the subordinate uid and gid information can be read, given -these resources are name-based, not id-based. If the numeric ID information -provided does not exist as entries in `/etc/passwd` or `/etc/group`, daemon -startup will fail with an error message. - -**Example: starting with default Docker user management:** - -```bash -$ sudo dockerd --userns-remap=default -``` - -When `default` is provided, Docker will create - or find the existing - user and group -named `dockremap`. If the user is created, and the Linux distribution has -appropriate support, the `/etc/subuid` and `/etc/subgid` files will be populated -with a contiguous 65536 length range of subordinate user and group IDs, starting -at an offset based on prior entries in those files. For example, Ubuntu will -create the following range, based on an existing user named `user1` already owning -the first 65536 range: - -```bash -$ cat /etc/subuid -user1:100000:65536 -dockremap:165536:65536 -``` - -If you have a preferred/self-managed user with subordinate ID mappings already -configured, you can provide that username or uid to the `--userns-remap` flag. -If you have a group that doesn't match the username, you may provide the `gid` -or group name as well; otherwise the username will be used as the group name -when querying the system for the subordinate group ID range. - -The output of `docker info` can be used to determine if the daemon is running -with user namespaces enabled or not. If the daemon is configured with user -namespaces, the Security Options entry in the response will list "userns" as -one of the enabled security features. - -##### Behavior differences when user namespaces are enabled - -When you start the Docker daemon with `--userns-remap`, Docker segregates the graph directory -where the images are stored by adding an extra directory with a name corresponding to the -remapped UID and GID. For example, if the remapped UID and GID begin with `165536`, all -images and containers running with that remap setting are located in `/var/lib/docker/165536.165536` -instead of `/var/lib/docker/`. - -In addition, the files and directories within the new directory, which correspond to -images and container layers, are also owned by the new UID and GID. To set the ownership -correctly, you need to re-pull the images and restart the containers after starting the -daemon with `--userns-remap`. - -##### Detailed information on `subuid`/`subgid` ranges - -Given potential advanced use of the subordinate ID ranges by power users, the -following paragraphs define how the Docker daemon currently uses the range entries -found within the subordinate range files. - -The simplest case is that only one contiguous range is defined for the -provided user or group. In this case, Docker will use that entire contiguous -range for the mapping of host uids and gids to the container process. This -means that the first ID in the range will be the remapped root user, and the -IDs above that initial ID will map host ID 1 through the end of the range. - -From the example `/etc/subuid` content shown above, the remapped root -user would be uid 165536. - -If the system administrator has set up multiple ranges for a single user or -group, the Docker daemon will read all the available ranges and use the -following algorithm to create the mapping ranges: - -1. The range segments found for the particular user will be sorted by *start ID* ascending. -2. Map segments will be created from each range in increasing value with a length matching the length of each segment. Therefore the range segment with the lowest numeric starting value will be equal to the remapped root, and continue up through host uid/gid equal to the range segment length. As an example, if the lowest segment starts at ID 1000 and has a length of 100, then a map of 1000 -> 0 (the remapped root) up through 1100 -> 100 will be created from this segment. If the next segment starts at ID 10000, then the next map will start with mapping 10000 -> 101 up to the length of this second segment. This will continue until no more segments are found in the subordinate files for this user. -3. If more than five range segments exist for a single user, only the first five will be utilized, matching the kernel's limitation of only five entries in `/proc/self/uid_map` and `proc/self/gid_map`. - -##### Disable user namespace for a container - -If you enable user namespaces on the daemon, all containers are started -with user namespaces enabled. In some situations you might want to disable -this feature for a container, for example, to start a privileged container (see -[user namespace known restrictions](#user-namespace-known-restrictions)). -To enable those advanced features for a specific container use `--userns=host` -in the `run/exec/create` command. -This option will completely disable user namespace mapping for the container's user. - -##### User namespace known restrictions - -The following standard Docker features are currently incompatible when -running a Docker daemon with user namespaces enabled: - - - sharing PID or NET namespaces with the host (`--pid=host` or `--net=host`) - - Using `--privileged` mode flag on `docker run` (unless also specifying `--userns=host`) - -In general, user namespaces are an advanced feature and will require -coordination with other capabilities. For example, if volumes are mounted from -the host, file ownership will have to be pre-arranged if the user or -administrator wishes the containers to have expected access to the volume -contents. Note that when using external volume or graph driver plugins, those -external software programs must be made aware of user and group mapping ranges -if they are to work seamlessly with user namespace support. - -Finally, while the `root` user inside a user namespaced container process has -many of the expected admin privileges that go along with being the superuser, the -Linux kernel has restrictions based on internal knowledge that this is a user namespaced -process. The most notable restriction that we are aware of at this time is the -inability to use `mknod`. Permission will be denied for device creation even as -container `root` inside a user namespace. +For details about how to use this feature, as well as limitations, see +[Isolate containers with a user namespace](https://docs.docker.com/engine/security/userns-remap/). ### Miscellaneous options From e2b8415a5495e9e47d82629ca38cfa3c36828e75 Mon Sep 17 00:00:00 2001 From: Dhawal Yogesh Bhanushali Date: Tue, 20 Jun 2017 13:53:09 -0700 Subject: [PATCH 261/534] Adding docs for "overlay2.size" storage-opt Signed-off-by: Dhawal Yogesh Bhanushali --- docs/reference/commandline/dockerd.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 95a953f1c4..3e2d434974 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -740,6 +740,18 @@ to add multiple lower directory support for OverlayFS. This option should only be used after verifying this support exists in the kernel. Applying this option on a kernel without this support will cause failures on mount. +##### `overlay2.size` + +Sets the default max size of the container. It is supported only when the +backing fs is `xfs` and mounted with `pquota` mount option. Under these +conditions the user can pass any size less then the backing fs size. + +###### Example + +```bash +$ sudo dockerd -s overlay2 --storage-opt overlay2.size=1G +``` + ### Docker runtime execution options The Docker daemon relies on a From 55c735ed8a26e8f86ab8f71ce733f0a4a3790da8 Mon Sep 17 00:00:00 2001 From: Aleksa Sarai Date: Thu, 13 Jul 2017 00:38:47 +1000 Subject: [PATCH 262/534] docs: add documentation for dm.libdm_log_level This is a new option added specifically to allow for debugging of bugs in Docker's storage drivers or libdm itself. Signed-off-by: Aleksa Sarai --- docs/reference/commandline/dockerd.md | 28 +++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3e2d434974..676da28df8 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -700,6 +700,34 @@ ENOSPC and will shutdown filesystem. $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 ``` +##### `dm.libdm_log_level` + +Specifies the maxmimum `libdm` log level that will be forwarded to the +`dockerd` log (as specified by `--log-level`). This option is primarily +intended for debugging problems involving `libdm`. Using values other than the +defaults may cause false-positive warnings to be logged. + +Values specified must fall within the range of valid `libdm` log levels. At the +time of writing, the following is the list of `libdm` log levels as well as +their corresponding levels when output by `dockerd`. + +| `libdm` Level | Value | `--log-level` | +| ------------- | -----:| ------------- | +| `_LOG_FATAL` | 2 | error | +| `_LOG_ERR` | 3 | error | +| `_LOG_WARN` | 4 | warn | +| `_LOG_NOTICE` | 5 | info | +| `_LOG_INFO` | 6 | info | +| `_LOG_DEBUG` | 7 | debug | + +###### Example + +```bash +$ sudo dockerd \ + --log-level debug \ + --storage-opt dm.libdm_log_level=7 +``` + #### ZFS options ##### `zfs.fsname` From 104b630763af5aaeab77ad5deade025243560e9a Mon Sep 17 00:00:00 2001 From: Kir Kolyshkin Date: Fri, 28 Jul 2017 10:28:23 -0700 Subject: [PATCH 263/534] Fix repo references in docs Since CLI was moved to a separate repo, these references are incorrect. Fixed with the help of sed script, verified manually. Signed-off-by: Kir Kolyshkin --- docs/deprecated.md | 4 ++-- docs/extend/EBS_volume.md | 4 ++-- docs/extend/config.md | 4 ++-- docs/extend/index.md | 4 ++-- docs/extend/legacy_plugins.md | 4 ++-- docs/extend/plugin_api.md | 4 ++-- docs/extend/plugins_authorization.md | 4 ++-- docs/extend/plugins_graphdriver.md | 4 ++-- docs/extend/plugins_logging.md | 4 ++-- docs/extend/plugins_metrics.md | 4 ++-- docs/extend/plugins_network.md | 4 ++-- docs/extend/plugins_services.md | 4 ++-- docs/extend/plugins_volume.md | 4 ++-- docs/reference/commandline/dockerd.md | 4 ++-- 14 files changed, 28 insertions(+), 28 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9905994b73..e6714fee13 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -5,8 +5,8 @@ description: "Deprecated Features." keywords: "docker, documentation, about, technology, deprecate" --- - -# A proof-of-concept Rexray plugin +# Volume plugin for Amazon EBS + +## A proof-of-concept Rexray plugin In this example, a simple Rexray plugin will be created for the purposes of using it on an Amazon EC2 instance with EBS. It is not meant to be a complete Rexray plugin. diff --git a/docs/extend/config.md b/docs/extend/config.md index d7ef8d83a4..912349c0cc 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -1,5 +1,4 @@ --- -title: "Plugin config" description: "How develop and use a plugin with the managed plugin system" keywords: "API, Usage, plugins, documentation, developer" --- diff --git a/docs/extend/index.md b/docs/extend/index.md index 37edcc0f42..6817ea8f57 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,7 +1,6 @@ --- description: Develop and use a plugin with the managed plugin system keywords: "API, Usage, plugins, documentation, developer" -title: Managed plugin system --- +# Use Docker Engine plugins + This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker, refer to [Docker Engine plugin system](index.md). diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index fcba621b02..2a317f953a 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -1,5 +1,4 @@ --- -title: "Plugins API" description: "How to write Docker plugins extensions " keywords: "API, Usage, plugins, documentation, developer" --- diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index edba6028a9..d7aa66cd80 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -1,5 +1,4 @@ --- -title: "Access authorization plugin" description: "How to create authorization plugins to manage access control to your Docker daemon." keywords: "security, authorization, authentication, docker, documentation, plugin, extend" redirect_from: @@ -15,7 +14,7 @@ redirect_from: will be rejected. --> -# Create an authorization plugin +# Access authorization plugin This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker Engine, diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index 9854fc1dca..d8d4531d8c 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -1,5 +1,4 @@ --- -title: "Graphdriver plugins" description: "How to manage image and container filesystems with external plugins" keywords: "Examples, Usage, storage, image, docker, data, graph, plugin, api" advisory: experimental @@ -14,6 +13,7 @@ advisory: experimental will be rejected. --> +# Graphdriver plugins ## Changelog diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md index add6699a02..fbfd2197c1 100644 --- a/docs/extend/plugins_logging.md +++ b/docs/extend/plugins_logging.md @@ -1,5 +1,4 @@ --- -title: "Docker log driver plugins" description: "Log driver plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide, logging" --- @@ -13,7 +12,7 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide, logging" will be rejected. --> -# Logging driver plugins +# Docker log driver plugins This document describes logging driver plugins for Docker. diff --git a/docs/extend/plugins_metrics.md b/docs/extend/plugins_metrics.md index f7a8825dcd..c29aeef932 100644 --- a/docs/extend/plugins_metrics.md +++ b/docs/extend/plugins_metrics.md @@ -1,5 +1,4 @@ --- -title: "Docker metrics collector plugins" description: "Metrics plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide, metrics" --- @@ -13,7 +12,7 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide, metrics" will be rejected. --> -# Metrics Collector Plugins +# Docker metrics collector plugins Docker exposes internal metrics based on the prometheus format. Metrics plugins enable accessing these metrics in a consistent way by providing a Unix diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 143f17a7f4..dcebca6095 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -1,5 +1,4 @@ --- -title: "Docker network driver plugins" description: "Network driver plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide" --- @@ -13,7 +12,7 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide" will be rejected. --> -# Engine network driver plugins +# Docker network driver plugins This document describes Docker Engine network driver plugins generally available in Docker Engine. To view information on plugins diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md index 48da8143fc..cd94d11cb5 100644 --- a/docs/extend/plugins_services.md +++ b/docs/extend/plugins_services.md @@ -1,5 +1,4 @@ --- -description: Using services with plugins keywords: "API, Usage, plugins, documentation, developer" title: Plugins and Services --- @@ -70,7 +69,7 @@ node1 is the manager and node2 is the worker. ```bash {% raw %} - $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' + $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" {% endraw %} ``` diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 294124d067..6f0a12d730 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -1,5 +1,4 @@ --- -title: "Volume plugins" description: "How to manage data with external volume plugins" keywords: "Examples, Usage, volume, docker, data, volumes, plugin, api" --- @@ -13,7 +12,7 @@ keywords: "Examples, Usage, volume, docker, data, volumes, plugin, api" will be rejected. --> -# Write a volume plugin +# Docker volume plugins Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems such as Amazon EBS, and enable data volumes to persist From ea95a033eb8477c269932c6af194166da13cca88 Mon Sep 17 00:00:00 2001 From: BastianHofmann Date: Tue, 24 Oct 2017 11:36:23 +0200 Subject: [PATCH 276/534] Fix json formatting in dockerd docs Fix another json formatiing error. Signed-off-by: Bastian Hofmann --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 7ea8f8133e..6e83be2d01 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -102,11 +102,11 @@ Options with [] may be specified multiple times. uses different binaries for the daemon and client. To run the daemon you type `dockerd`. -To run the daemon with debug output, use `dockerd -D` or add `debug: true` to +To run the daemon with debug output, use `dockerd -D` or add `"debug": true` to the `daemon.json` file. > **Note**: In Docker 1.13 and higher, enable experimental features by starting -> `dockerd` with the `--experimental` flag or adding `experimental: true` to the +> `dockerd` with the `--experimental` flag or adding `"experimental": true` to the > `daemon.json` file. In earlier Docker versions, a different build was required > to enable experimental features. From 520d4a11a82b8c9c1cd0bd6c2bdf58fdcbf1aaab Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 26 Oct 2017 02:15:49 +0200 Subject: [PATCH 277/534] Fix daemon.json runtime example The example in the documentation used "runc", which is a reserved runtime name (as it's the default). This patch updates the example, and uses a different name. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6e83be2d01..a94d01ddd6 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1326,8 +1326,8 @@ This is a full example of the allowed configuration options on Linux: "default-runtime": "runc", "oom-score-adjust": -500, "runtimes": { - "runc": { - "path": "runc" + "cc-runtime": { + "path": "/usr/bin/cc-runtime" }, "custom": { "path": "/usr/local/bin/my-runc-replacement", From 2bfbc4d2ad0a360b40b3f9b17eded3a34eebb9f0 Mon Sep 17 00:00:00 2001 From: Abdur Rehman Date: Tue, 31 Oct 2017 15:21:09 +0500 Subject: [PATCH 278/534] fix a number of minor typos Fix 19 typos, grammatical errors and duplicated words. These fixes have minimal impact on the code as these are either in the doc files or in comments inside the code files. Signed-off-by: Abdur Rehman --- docs/extend/config.md | 2 +- docs/reference/commandline/dockerd.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 912349c0cc..e68af3bf6a 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -126,7 +126,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`propagatedMount`** *string* path to be mounted as rshared, so that mounts under that path are visible to docker. This is useful for volume plugins. - This path will be bind-mounted outisde of the plugin rootfs so it's contents + This path will be bind-mounted outside of the plugin rootfs so it's contents are preserved on upgrade. - **`env`** *PluginEnv array* diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a94d01ddd6..f8573910f1 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -393,7 +393,7 @@ $ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 ##### `dm.thinp_autoextend_percent` -Sets the value percentage value to increase the thin pool by when when `lvm` +Sets the value percentage value to increase the thin pool by when `lvm` attempts to autoextend the available space [100 = disabled] ###### Example: From e95b6d5cbce2085a4959ed386642efb7304ca916 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 13 Nov 2017 01:26:09 +0100 Subject: [PATCH 279/534] Remove deprecated "daemon" subcommand The `docker daemon` subcommand was only present for backward compatibility, but deprecated in v1.13, and scheduled for removal in v17.12 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 6a728d50dc..7ac8848fea 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -81,7 +81,7 @@ The `filter` param to filter the list of image by reference (name or name:tag) i ### `docker daemon` subcommand **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Removed In Release: v17.12** The daemon is moved to a separate binary (`dockerd`), and should be used instead. From d0980a4a02114b92a8f193f129510abaf2568343 Mon Sep 17 00:00:00 2001 From: Skip Date: Mon, 20 Nov 2017 15:25:29 +0100 Subject: [PATCH 280/534] Docs - Typo in volume plugin doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit There is a typo in the `plugins_volume.md#volumedriverpath` section. The `/VolumeDriver.Path` response (v1) should be `Mountpoint`and not `Mountpoin`. Signed-off-by: scipio3000 Signed-off-by: Günther Jungbluth --- docs/extend/plugins_volume.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 6f0a12d730..b9a7139099 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -199,7 +199,7 @@ Request the path to the volume with the given `volume_name`. ```json { - "Mountpoin": "/path/to/directory/on/host", + "Mountpoint": "/path/to/directory/on/host", "Err": "" } ``` From a9ab6a426d60604c8696a087438587ba94d7d30c Mon Sep 17 00:00:00 2001 From: doncicuto <30386061+doncicuto@users.noreply.github.com> Date: Tue, 28 Nov 2017 13:36:30 +0100 Subject: [PATCH 281/534] Update Flocker plugin link ClusterHQ shutted down at the end of 2016. It seems that ScatterHQ is a group of ClusterHQ former employees that have forked Flocker. It seems that no code has been commited in the last months but if somebody comes to this Docker page it should be referenced to ScatterHQ. Signed-off-by: Miguel Angel Alvarez Cabrerizo --- docs/extend/legacy_plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index b2a6589815..0fd6d11655 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -65,7 +65,7 @@ The sections below provide an inexhaustive overview of available plugins. | [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. | | [DigitalOcean Block Storage plugin](https://github.com/omallo/docker-volume-plugin-dostorage) | Integrates DigitalOcean's [block storage solution](https://www.digitalocean.com/products/storage/) into the Docker ecosystem by automatically attaching a given block storage volume to a DigitalOcean droplet and making the contents of the volume available to Docker containers running on that droplet. | | [DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. | -| [Flocker plugin](https://clusterhq.com/docker-plugin/) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. | +| [Flocker plugin](https://github.com/ScatterHQ/flocker) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. | | [Fuxi Volume Plugin](https://github.com/openstack/fuxi) | A volume plugin that is developed as part of the OpenStack Kuryr project and implements the Docker volume plugin API by utilizing Cinder, the OpenStack block storage service. | | [gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). | | [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. | From 4133903ea824aa74b768c662f72b9710dd787c88 Mon Sep 17 00:00:00 2001 From: Renaud Gaubert Date: Wed, 8 Nov 2017 18:13:11 +0100 Subject: [PATCH 282/534] Added docs for dockerd Signed-off-by: Renaud Gaubert --- docs/reference/commandline/dockerd.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index f8573910f1..e95016bd6b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -72,6 +72,7 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) --metrics-addr string Set default address and port to serve the metrics api on --mtu int Set the containers network MTU + --node-generic-resources list Advertise user-defined resource --no-new-privileges Set no-new-privileges by default for new containers --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") @@ -1237,6 +1238,23 @@ Please note that this feature is still marked as experimental as metrics and met names could change while this feature is still in experimental. Please provide feedback on what you would like to see collected in the API. +#### Node Generic Resources + +The `--node-generic-resources` option takes a list of key-value +pair (`key=value`) that allows you to advertise user defined resources +in a swarm cluster. + +The current expected use case is to advertise NVIDIA GPUs so that services +requesting `NVIDIA-GPU=[0-16]` can land on a node that has enough GPUs for +the task to run. + +Example of usage: +```json +{ + "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"] +} +``` + ### Daemon configuration file The `--config-file` option allows you to set any configuration option @@ -1325,6 +1343,7 @@ This is a full example of the allowed configuration options on Linux: "no-new-privileges": false, "default-runtime": "runc", "oom-score-adjust": -500, + "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"], "runtimes": { "cc-runtime": { "path": "/usr/bin/cc-runtime" From c39b3f883b2cbda0beb3b8bb91c0cc1f4bda584e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 12 Dec 2017 17:09:38 -0800 Subject: [PATCH 283/534] Update docs and completion-scripts for deprecated features - the `--disable-legacy-registry` daemon flag was removed - duplicate keys with conflicting values for engine labels now produce an error instead of a warning. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 29 ++++++++++++++++++++++----- docs/reference/commandline/dockerd.md | 21 +++++++------------ 2 files changed, 31 insertions(+), 19 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7ac8848fea..78df5347d0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -88,10 +88,10 @@ The daemon is moved to a separate binary (`dockerd`), and should be used instead ### Duplicate keys with conflicting values in engine labels **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Removed In Release: v17.12** -Duplicate keys with conflicting values have been deprecated. A warning is displayed -in the output, and an error will be returned in the future. +When setting duplicate keys with conflicting values, an error will be produced, and the daemon +will fail to start. ### `MAINTAINER` in Dockerfile **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** @@ -110,12 +110,16 @@ future Engine versions. Instead of just requesting, for example, the URL ### Backing filesystem without `d_type` support for overlay/overlay2 **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Removed In Release: v17.12** The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` if it is formatted with the `ftype=0` option. +Starting with Docker 17.12, new installations will not support running overlay2 on +a backing filesystem without `d_type` support. For existing installations that upgrade +to 17.12, a warning will be printed. + Please also refer to [#27358](https://github.com/docker/docker/issues/27358) for further information. @@ -292,7 +296,7 @@ of the `--changes` flag that allows to pass `Dockerfile` commands. **Disabled By Default In Release: v17.06** -**Target For Removal In Release: v17.12** +**Removed In Release: v17.12** Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the docker daemon from `pull`, `push`, and `login` operations against v1 @@ -303,6 +307,21 @@ Support for the v1 protocol to the public registry was removed in 1.13. Any mirror configurations using v1 should be updated to use a [v2 registry mirror](https://docs.docker.com/registry/recipes/mirror/). +Starting with Docker 17.12, support for V1 registries has been removed, and the +`--disable-legacy-registry` flag can no longer be used, and `dockerd` will fail to +start when set. + +### `--disable-legacy-registry` override daemon option + +**Disabled In Release: v17.12** + +**Target For Removal In Release: v18.03** + +The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print +an error when used. For this error to be printed, the flag itself is still present, +but hidden. The flag will be removed in Docker 18.03. + + ### Docker Content Trust ENV passphrase variables name change **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e95016bd6b..4ce22bde67 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -42,7 +42,6 @@ Options: --default-gateway-v6 ip Container default gateway IPv6 address --default-runtime string Default OCI runtime for containers (default "runc") --default-ulimit ulimit Default ulimits for containers (default []) - --disable-legacy-registry Disable contacting legacy registries (default true) --dns list DNS server to use (default []) --dns-opt list DNS options to use (default []) --dns-search list DNS search domains to use (default []) @@ -1054,18 +1053,14 @@ system's list of trusted CAs instead of enabling `--insecure-registry`. #### Legacy Registries -Operations against registries supporting only the legacy v1 protocol are -disabled by default. Specifically, the daemon will not attempt `push`, -`pull` and `login` to v1 registries. The exception to this is `search` -which can still be performed on v1 registries. +Starting with Docker 17.12, operations against registries supporting only the +legacy v1 protocol are no longer supported. Specifically, the daemon will not +attempt `push`, `pull` and `login` to v1 registries. The exception to this is +`search` which can still be performed on v1 registries. -Add `"disable-legacy-registry":false` to the [daemon configuration -file](#daemon-configuration-file), or set the -`--disable-legacy-registry=false` flag, if you need to interact with -registries that have not yet migrated to the v2 protocol. +The `disable-legacy-registry` configuration option has been removed and, when +used, will produce an error on daemon startup. -Interaction v1 registries will no longer be supported in Docker v17.12, -and the `disable-legacy-registry` configuration option will be removed. ### Running a Docker daemon behind an HTTPS_PROXY @@ -1339,7 +1334,6 @@ This is a full example of the allowed configuration options on Linux: "registry-mirrors": [], "seccomp-profile": "", "insecure-registries": [], - "disable-legacy-registry": false, "no-new-privileges": false, "default-runtime": "runc", "oom-score-adjust": -500, @@ -1408,8 +1402,7 @@ This is a full example of the allowed configuration options on Windows: "raw-logs": false, "allow-nondistributable-artifacts": [], "registry-mirrors": [], - "insecure-registries": [], - "disable-legacy-registry": false + "insecure-registries": [] } ``` From eb2697dce8381cb0282a4c656e11a271a6427b67 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 12 Dec 2017 17:27:05 -0800 Subject: [PATCH 284/534] Fix anchors to "Storage driver options" Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e95016bd6b..f046ba1eeb 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -263,7 +263,7 @@ snapshots. For each devicemapper graph location – typically `/var/lib/docker/devicemapper` – a thin pool is created based on two block devices, one for data and one for metadata. By default, these block devices are created automatically by using loopback mounts of automatically created -sparse files. Refer to [Storage driver options](#storage-driver-options) below +sparse files. Refer to [Devicemapper options](#devicemapper-options) below for a way how to customize this setup. [~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) article explains how to tune your existing setup without the use of options. @@ -275,7 +275,7 @@ does not share executable memory between devices. Use The `zfs` driver is probably not as fast as `btrfs` but has a longer track record on stability. Thanks to `Single Copy ARC` shared blocks between clones will be cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem -set `zfs.fsname` option as described in [Storage driver options](#storage-driver-options). +set `zfs.fsname` option as described in [ZFS options](#zfs-options). The `overlay` is a very fast union filesystem. It is now merged in the main Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay` From 72979af3053a7548a6b68e97e07e9e5336dd946b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 13 Dec 2017 14:16:46 -0800 Subject: [PATCH 285/534] Updated deprecation status for "repository:shortid" The `repository:shortid` syntax for referencing images is very little used, collides with with tag references can be confused with digest references. The `repository:shortid` notation was deprecated in Docker 1.13, and scheduled for removal in Docker 17.12. This patch updates the deprecation status for this feature. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 78df5347d0..9ac2f08c16 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -74,9 +74,13 @@ The `filter` param to filter the list of image by reference (name or name:tag) i ### `repository:shortid` image references **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Removed In Release: v17.12** -`repository:shortid` syntax for referencing images is very little used, collides with tag references can be confused with digest references. +The `repository:shortid` syntax for referencing images is very little used, +collides with tag references, and can be confused with digest references. + +Support for the `repository:shortid` notation to reference images was removed +in Docker 17.12. ### `docker daemon` subcommand **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** From f125fc339eef5276e89f42be3a7caadec0911ce4 Mon Sep 17 00:00:00 2001 From: Misty Stanley-Jones Date: Thu, 8 Feb 2018 14:08:31 -0800 Subject: [PATCH 286/534] Clarify network plugins and swarm mode Signed-off-by: Misty Stanley-Jones --- docs/extend/plugins_network.md | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index dcebca6095..66302c5cf2 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -26,18 +26,13 @@ LibNetwork, which shares plugin infrastructure with Engine. Effectively, network driver plugins are activated in the same way as other plugins, and use the same kind of protocol. -## Network driver plugins and swarm mode +## Network plugins and swarm mode -Docker 1.12 adds support for cluster management and orchestration called -[swarm mode](https://docs.docker.com/engine/swarm/). Docker Engine running in swarm mode currently -only supports the built-in overlay driver for networking. Therefore existing -networking plugins will not work in swarm mode. +[Legacy plugins](legacy_plugins.md) do not work in swarm mode. However, +plugins written using the [v2 plugin system](index.md) do work in swarm mode, as +long as they are installed on each swarm worker node. -When you run Docker Engine outside of swarm mode, all networking plugins that -worked in Docker 1.11 will continue to function normally. They do not require -any modification. - -## Using network driver plugins +## Use network driver plugins The means of installing and running a network driver plugin depend on the particular plugin. So, be sure to install your plugin according to the @@ -57,6 +52,13 @@ referring to that network will be sent to the plugin, $ docker run --network=mynet busybox top +## Find network plugins + +Network plugins are written by third parties, and are published by those +third parties, either on +[Docker Store](https://store.docker.com/search?category=network&q=&type=plugin) +or on the third party's site. + ## Write a network plugin Network plugins implement the [Docker plugin @@ -68,7 +70,7 @@ The network driver protocol, in addition to the plugin activation call, is documented as part of libnetwork: [https://github.com/docker/libnetwork/blob/master/docs/remote.md](https://github.com/docker/libnetwork/blob/master/docs/remote.md). -# Related Information +## Related Information To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. From 266d92c4e53c71bfbbbec44ebd107ea746ee933e Mon Sep 17 00:00:00 2001 From: Anusha Ragunathan Date: Thu, 5 Apr 2018 12:14:59 -0700 Subject: [PATCH 287/534] Update examples to reflect docker-runc's runtime root for plugins. Signed-off-by: Anusha Ragunathan --- docs/extend/index.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 6817ea8f57..4e8cae9855 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -218,20 +218,22 @@ plugins. This is specifically useful to collect plugin logs if they are redirected to a file. ```bash -$ docker-runc list -ID PID STATUS BUNDLE CREATED -f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 2679 running /run/docker/libcontainerd/f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 2017-02-06T21:53:03.031537592Z -r +$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins list + +ID PID STATUS BUNDLE CREATED OWNER +93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 15806 running /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 2018-02-08T21:40:08.621358213Z root +9b4606d84e06b56df84fadf054a21374b247941c94ce405b0a261499d689d9c9 14992 running /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/9b4606d84e06b56df84fadf054a21374b247941c94ce405b0a261499d689d9c9 2018-02-08T21:35:12.321325872Z root +c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d 14984 running /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d 2018-02-08T21:35:12.321288966Z root ``` ```bash -$ docker-runc exec f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 cat /var/log/plugin.log +$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 cat /var/log/plugin.log ``` If the plugin has a built-in shell, then exec into the plugin can be done as follows: ```bash -$ docker-runc exec -t f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 sh +$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec -t 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 sh ``` #### Using curl to debug plugin socket issues. From 5bea8fe8cbbf444aaa2adbe292e8d6760d3e2f2f Mon Sep 17 00:00:00 2001 From: selansen Date: Thu, 18 Jan 2018 16:59:39 -0800 Subject: [PATCH 288/534] Allow user to specify default address pools for docker networks This is separate commit for CLI files to address PR 36054 Signed-off-by: selansen --- docs/reference/commandline/dockerd.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index be214e15f5..44cb8c4b28 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -40,6 +40,7 @@ Options: -D, --debug Enable debug mode --default-gateway ip Container default gateway IPv4 address --default-gateway-v6 ip Container default gateway IPv6 address + --default-address-pool Set the default address pool for local node networks --default-runtime string Default OCI runtime for containers (default "runc") --default-ulimit ulimit Default ulimits for containers (default []) --dns list DNS server to use (default []) @@ -1348,7 +1349,9 @@ This is a full example of the allowed configuration options on Linux: "--debug" ] } - } + }, + "default-address-pools":[{"base":"172.80.0.0/16","size":24}, + {"base":"172.90.0.0/16","size":24}] } ``` From 07efeb4cb2f29648fa675b10e64b1100647d091b Mon Sep 17 00:00:00 2001 From: wanglei Date: Wed, 16 May 2018 20:16:08 +0800 Subject: [PATCH 289/534] Developer Certificate of Origin Version 1.1 Copyright (C) 2004, 2006 The Linux Foundation and its contributors. 1 Letterman Drive Suite D4700 San Francisco, CA, 94129 Everyone is permitted to copy and distribute verbatim copies of this license document, but changing it is not allowed. Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. Signed-off-by: nkwangleiGIT --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 44cb8c4b28..3dc4fd5aed 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1425,7 +1425,7 @@ The list of currently supported options that can be reconfigured is this: - `cluster-store-opts`: it uses the new options to reload the discovery store. - `cluster-advertise`: it modifies the address advertised after reloading. - `labels`: it replaces the daemon labels with a new set of labels. -- `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/engine/admin/live-restore/). +- `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. - `max-concurrent-uploads`: it updates the max concurrent uploads for each push. - `default-runtime`: it updates the runtime to be used if not is From 5dd2325bcb1db0abf377352b232c70811e68f995 Mon Sep 17 00:00:00 2001 From: Ying Li Date: Mon, 7 May 2018 11:10:13 -0700 Subject: [PATCH 290/534] Document that reserved namespaces org.docker.*, io.docker.*, and org.dockerproject.* in engine labels are now deprecated. Signed-off-by: Ying Li --- docs/deprecated.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9ac2f08c16..75514a0ff9 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -19,6 +19,16 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +### Reserved namespaces in engine labels + +**Deprecated in Release: v18.06.0** + +The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engine labels +were always documented to be reserved, but there was never any enforcement. + +Usage of these namespaces will now cause a warning in the engine logs to discourage their +use, and will error instead in 18.12 and above. + ### Asynchronous `service create` and `service update` **Deprecated In Release: v17.05.0** From 55733bd2aa3094956b344fa66823ef24e0a482f6 Mon Sep 17 00:00:00 2001 From: Vimal Raghubir Date: Mon, 25 Jun 2018 14:54:33 -0400 Subject: [PATCH 291/534] Add: Add missing option Signed-off-by: Vimal-Raghubir --- docs/reference/commandline/dockerd.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3dc4fd5aed..1afc27e8b1 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1432,11 +1432,12 @@ The list of currently supported options that can be reconfigured is this: specified at container creation. It defaults to "default" which is the runtime shipped with the official docker packages. - `runtimes`: it updates the list of available OCI runtimes that can - be used to run containers + be used to run containers. - `authorization-plugin`: specifies the authorization plugins to use. - `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. +- `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From 17908e0c197fd1b96febe0df795340530c211dd0 Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Mon, 19 Feb 2018 18:31:29 +0900 Subject: [PATCH 292/534] support SSH connection e.g. docker -H ssh://me@server The `docker` CLI also needs to be installed on the remote host to provide `docker system dial-stdio`, which proxies the daemon socket to stdio. Please refer to docs/reference/commandline/dockerd.md . Signed-off-by: Akihiro Suda --- docs/reference/commandline/dockerd.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1afc27e8b1..8f8aa722a3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -181,6 +181,19 @@ The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes precedence over `HTTP_PROXY`. +Starting with Docker 18.09, the Docker client supports connecting to a remote +daemon via SSH: + +``` +$ docker -H ssh://me@example.com:22 ps +$ docker -H ssh://me@example.com ps +$ docker -H ssh://example.com ps +``` + +To use SSH connection, you need to set up `ssh` so that it can reach the +remote host with public key authentication. +Also, you need to have `docker` binary 18.09 or later on the daemon host. + #### Bind Docker to another host/port or a Unix socket > **Warning**: From f677e6226e5f21a01fbaf5994c96ac259f014e30 Mon Sep 17 00:00:00 2001 From: Anda Xu Date: Thu, 23 Aug 2018 13:46:29 -0700 Subject: [PATCH 293/534] update docs with the new features option in daemon.json Signed-off-by: Anda Xu --- docs/reference/commandline/dockerd.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8f8aa722a3..d211fce5a6 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1297,6 +1297,7 @@ This is a full example of the allowed configuration options on Linux: "exec-opts": [], "exec-root": "", "experimental": false, + "features": {}, "storage-driver": "", "storage-opts": [], "labels": [], @@ -1392,6 +1393,7 @@ This is a full example of the allowed configuration options on Windows: "dns-search": [], "exec-opts": [], "experimental": false, + "features":{}, "storage-driver": "", "storage-opts": [], "labels": [], @@ -1446,11 +1448,12 @@ The list of currently supported options that can be reconfigured is this: the runtime shipped with the official docker packages. - `runtimes`: it updates the list of available OCI runtimes that can be used to run containers. -- `authorization-plugin`: specifies the authorization plugins to use. +- `authorization-plugin`: it specifies the authorization plugins to use. - `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. - `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. - `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. +- `features`: it explicitly enables or disables specific features. Updating and reloading the cluster configurations such as `--cluster-store`, `--cluster-advertise` and `--cluster-store-opts` will take effect only if From 634b2e17d672df702b8797cb9845e0607a4d611d Mon Sep 17 00:00:00 2001 From: Anda Xu Date: Mon, 27 Aug 2018 15:52:50 -0700 Subject: [PATCH 294/534] update docs with current supported features options Signed-off-by: Anda Xu --- docs/reference/commandline/dockerd.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index d211fce5a6..3ce72bb44b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1424,6 +1424,16 @@ This is a full example of the allowed configuration options on Windows: } ``` +#### Feature options +The optional field `features` in `daemon.json` allows users to enable or disable specific +daemon features. For example, `{"features":{"buildkit": true}}` enables `buildkit` as the +default docker image builder. + +The list of currently supported feature options: +- `buildkit`: It enables `buildkit` as default builder when set to `true` or disables it by +`false`. Note that if this option is not explicitly set in the daemon config file, then it +is up to the cli to determine which builder to invoke. + #### Configuration reload behavior Some options can be reconfigured when the daemon is running without requiring From fccfd499a16645df71b082baecf1bcd3e932abff Mon Sep 17 00:00:00 2001 From: Alex Mayer Date: Fri, 28 Sep 2018 17:40:22 -0400 Subject: [PATCH 295/534] Docs: Add Spaces Around Parenthesis Where Needed Signed-off-by: Alex Mayer --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 75514a0ff9..200669391d 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -167,7 +167,7 @@ The docker login command is removing the ability to automatically register for a **Target For Removal In Release: v17.06** -The flag `--security-opt` doesn't use the colon separator(`:`) anymore to divide keys and values, it uses the equal symbol(`=`) for consistency with other similar flags, like `--storage-opt`. +The flag `--security-opt` doesn't use the colon separator (`:`) anymore to divide keys and values, it uses the equal symbol (`=`) for consistency with other similar flags, like `--storage-opt`. ### `/containers/(id or name)/copy` endpoint From f4a68cbf89853a1477879ad26ede2883d08dd8e0 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 9 Oct 2018 18:27:42 +0200 Subject: [PATCH 296/534] Deprecate legacy overlay storage driver The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive inode consumption). The legacy `overlay` storage driver will be removed in a future release. Users of the `overlay` storage driver should migrate to the `overlay2` storage driver. The legacy `overlay` storage driver allowed using overlayFS-backed filesystems on pre 4.x kernels. Now that all supported distributions are able to run `overlay2` (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 200669391d..b577cc5411 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -19,6 +19,22 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +### Legacy "overlay" storage driver + +**Deprecated in Release: v18.09.0** + +The `overlay` storage driver is deprecated in favor of the `overlay2` storage +driver, which has all the benefits of `overlay`, without its limitations (excessive +inode consumption). The legacy `overlay` storage driver will be removed in a future +release. Users of the `overlay` storage driver should migrate to the `overlay2` +storage driver. + +The legacy `overlay` storage driver allowed using overlayFS-backed filesystems +on pre 4.x kernels. Now that all supported distributions are able to run `overlay2` +(as they are either on kernel 4.x, or have support for multiple lowerdirs +backported), there is no reason to keep maintaining the `overlay` storage driver. + + ### Reserved namespaces in engine labels **Deprecated in Release: v18.06.0** From 71ad3e90fe4370e5c2f4a453e5ab3e5d32ffb568 Mon Sep 17 00:00:00 2001 From: Brandon Mitchell Date: Sat, 13 Oct 2018 13:53:34 -0400 Subject: [PATCH 297/534] Documenting example default-ulimit in daemon.json Signed-off-by: Brandon Mitchell --- docs/reference/commandline/dockerd.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3ce72bb44b..9feb4e8afa 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1327,7 +1327,13 @@ This is a full example of the allowed configuration options on Linux: "userns-remap": "", "group": "", "cgroup-parent": "", - "default-ulimits": {}, + "default-ulimits": { + "nofile": { + "Name": "nofile", + "Hard": 64000, + "Soft": 64000 + } + }, "init": false, "init-path": "/usr/libexec/docker-init", "ipv6": false, From 0832aafb7c551564997f5f03eaf51b811c42b5de Mon Sep 17 00:00:00 2001 From: Brandon Mitchell Date: Sat, 13 Oct 2018 14:07:47 -0400 Subject: [PATCH 298/534] Adjust formatting on daemon.json documentation Signed-off-by: Brandon Mitchell --- docs/reference/commandline/dockerd.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3ce72bb44b..5d4e9baf17 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1364,8 +1364,10 @@ This is a full example of the allowed configuration options on Linux: ] } }, - "default-address-pools":[{"base":"172.80.0.0/16","size":24}, - {"base":"172.90.0.0/16","size":24}] + "default-address-pools":[ + {"base":"172.80.0.0/16","size":24}, + {"base":"172.90.0.0/16","size":24} + ] } ``` From 2937b702a91cdfb1bf09d7d2a30a92e8901afa08 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 9 Oct 2018 18:51:13 +0200 Subject: [PATCH 299/534] Deprecate "devicemapper" storage driver. The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are recommended to migrate to a different storage driver, such as `overlay2`, which is now the default storage driver. The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels that have no support for other storage drivers (such as overlay2, or AUFS). Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b577cc5411..1ca390a897 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -34,6 +34,21 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. +### device mapper storage driver + +**Deprecated in Release: v18.09.0** + +The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will +be removed in a future release. Users of the `devicemapper` storage driver are +recommended to migrate to a different storage driver, such as `overlay2`, which +is now the default storage driver. + +The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels +that have no support for other storage drivers (such as overlay2, or AUFS). + +Now that support for `overlay2` is added to all supported distros (as they are +either on kernel 4.x, or have support for multiple lowerdirs backported), there +is no reason to continue maintenance of the `devicemapper` storage driver. ### Reserved namespaces in engine labels From e047dfc880fb817f9fa29b3ce8613d4b8e80ac9f Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Thu, 25 Oct 2018 16:02:26 +0900 Subject: [PATCH 300/534] docs, ssh: unsupport password auth explicitly The issue with password auth is tracked in reference 1476 and reference 1477 . Signed-off-by: Akihiro Suda --- docs/reference/commandline/dockerd.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e530a4478e..213f38758c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -191,7 +191,10 @@ $ docker -H ssh://example.com ps ``` To use SSH connection, you need to set up `ssh` so that it can reach the -remote host with public key authentication. +remote host with public key authentication. Password authentication is not +supported. If your key is protected with passphrase, you need to set up +`ssh-agent`. + Also, you need to have `docker` binary 18.09 or later on the daemon host. #### Bind Docker to another host/port or a Unix socket From 60615177d405eb882173e57cbdac2a894dd5b410 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 21 Nov 2018 13:26:08 +0100 Subject: [PATCH 301/534] Update daemon.json example to show that log-opts must be a string log-opts are passed to logging-drivers as-is, so the daemon is not aware what value-type each option takes. For this reason, all options must be provided as a string, even if they are used as numeric values by the logging driver. For example, to pass the "max-file" option to the default (json-file) logging driver, this value has to be passed as a string; ```json { "log-driver": "json-file", "log-opts": { "max-size": "10m", "max-file": "3" } } ``` When passed as a _number_ (`"max-file": 3`), the daemon will invalidate the configuration file, and fail to start; unable to configure the Docker daemon with file /etc/docker/daemon.json: json: cannot unmarshal number into Go value of type string This patch adds an example to the daemon.json to show these values have to be passed as strings. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 213f38758c..5a7f7145b2 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1305,8 +1305,13 @@ This is a full example of the allowed configuration options on Linux: "storage-opts": [], "labels": [], "live-restore": true, - "log-driver": "", - "log-opts": {}, + "log-driver": "json-file", + "log-opts": { + "max-size": "10m", + "max-files":"5", + "labels": "somelabel", + "env": "os,customer" + }, "mtu": 0, "pidfile": "", "cluster-store": "", From e391713e4c23c74199ff16db3971b19eeb3a8169 Mon Sep 17 00:00:00 2001 From: Anne Henmi Date: Wed, 12 Dec 2018 08:41:11 -0700 Subject: [PATCH 302/534] Fixed typo. Signed-off-by: Anne Henmi --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5a7f7145b2..ee12fc5a96 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -303,7 +303,7 @@ the same file can share a single page cache entry (or entries), it makes > **Note**: As promising as `overlay` is, the feature is still quite young and > should not be used in production. Most notably, using `overlay` can cause > excessive inode consumption (especially as the number of images grows), as -> well as > being incompatible with the use of RPMs. +> well as being incompatible with the use of RPMs. The `overlay2` uses the same fast union filesystem but takes advantage of [additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux From 295e4c98b749fada341115e15e20a207be560684 Mon Sep 17 00:00:00 2001 From: Frederic Hemberger Date: Fri, 21 Dec 2018 09:04:21 +0100 Subject: [PATCH 303/534] docs(metrics-addr): Use port 9323, allocated for Docker in prometheus Signed-off-by: Frederic Hemberger --- docs/reference/commandline/dockerd.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ee12fc5a96..d31757f2a0 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1231,10 +1231,14 @@ The `--metrics-addr` option takes a tcp address to serve the metrics API. This feature is still experimental, therefore, the daemon must be running in experimental mode for this feature to work. -To serve the metrics API on localhost:1337 you would specify `--metrics-addr 127.0.0.1:1337` -allowing you to make requests on the API at `127.0.0.1:1337/metrics` to receive metrics in the +To serve the metrics API on `localhost:9323` you would specify `--metrics-addr 127.0.0.1:9323`, +allowing you to make requests on the API at `127.0.0.1:9323/metrics` to receive metrics in the [prometheus](https://prometheus.io/docs/instrumenting/exposition_formats/) format. +Port `9323` is the [default port associated with Docker +metrics](https://github.com/prometheus/prometheus/wiki/Default-port-allocations) +to avoid collisions with other prometheus exporters and services. + If you are running a prometheus server you can add this address to your scrape configs to have prometheus collect metrics on Docker. For more information on prometheus you can view the website [here](https://prometheus.io/). @@ -1243,7 +1247,7 @@ on prometheus you can view the website [here](https://prometheus.io/). scrape_configs: - job_name: 'docker' static_configs: - - targets: ['127.0.0.1:1337'] + - targets: ['127.0.0.1:9323'] ``` Please note that this feature is still marked as experimental as metrics and metric From 780cc1866546fdaa4e8c5390fa6806b06acb921d Mon Sep 17 00:00:00 2001 From: Ian Campbell Date: Wed, 9 Jan 2019 11:18:25 +0000 Subject: [PATCH 304/534] Documentation on writing a plugin Signed-off-by: Ian Campbell --- docs/extend/cli_plugins.md | 98 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 98 insertions(+) create mode 100644 docs/extend/cli_plugins.md diff --git a/docs/extend/cli_plugins.md b/docs/extend/cli_plugins.md new file mode 100644 index 0000000000..a99204ca58 --- /dev/null +++ b/docs/extend/cli_plugins.md @@ -0,0 +1,98 @@ +--- +description: "Writing Docker CLI Plugins" +keywords: "docker, cli plugin" +--- + + + +# Docker CLI Plugin Spec + +The `docker` CLI supports adding additional top-level subcommands as +additional out-of-process commands which can be installed +independently. These plugins run on the client side and should not be +confused with "plugins" which run on the server. + +This document contains information for authors of such plugins. + +## Requirements for CLI Plugins + +### Naming + +A valid CLI plugin name consists only of lower case letters `a-z` +and the digits `0-9`. The leading character must be a letter. A valid +name therefore would match the regex `^[a-z][a-z0-9]*$`. + +The binary implementing a plugin must be named `docker-$name` where +`$name` is the name of the plugin. On Windows a `.exe` suffix is +mandatory. + +## Required sub-commands + +A CLI plugin must support being invoked in at least these two ways: + +* `docker-$name docker-cli-plugin-metadata` -- outputs metadata about + the plugin. +* `docker-$name [GLOBAL OPTIONS] $name [OPTIONS AND FURTHER SUB + COMMANDS]` -- the primary entry point to the plugin's functionality. + +A plugin may implement other subcommands but these will never be +invoked by the current Docker CLI. However doing so is strongly +discouraged: new subcommands may be added in the future without +consideration for additional non-specified subcommands which may be +used by plugins in the field. + +### The `docker-cli-plugin-metadata` subcommand + +When invoked in this manner the plugin must produce a JSON object +(and nothing else) on its standard output and exit success (0). + +The JSON object has the following defined keys: +* `SchemaVersion` (_string_) mandatory: must contain precisely "0.1.0". +* `Vendor` (_string_) mandatory: contains the name of the plugin vendor/author. May be truncated to 11 characters in some display contexts. +* `ShortDescription` (_string_) optional: a short description of the plugin, suitable for a single line help message. +* `Version` (_string_) optional: the version of the plugin, this is considered to be an opaque string by the core and therefore has no restrictions on its syntax. +* `URL` (_string_) optional: a pointer to the plugin's web page. + +A binary which does not correctly output the metadata +(e.g. syntactically invalid, missing mandatory keys etc) is not +considered a valid CLI plugin and will not be run. + +### The primary entry point subcommand + +This is the entry point for actually running the plugin. It maybe have +options or further subcommands. + +#### Required global options + +A plugin is required to support all of the global options of the +top-level CLI, i.e. those listed by `man docker 1` with the exception +of `-v`. + +## Installation + +Plugins distributed in packages for system wide installation on +Unix(-like) systems should be installed in either +`/usr/lib/docker/cli-plugins` or `/usr/libexec/docker/cli-plugins` +depending on which of `/usr/lib` and `/usr/libexec` is usual on that +system. System Administrators may also choose to manually install into +the `/usr/local/lib` or `/usr/local/libexec` equivalents but packages +should not do so. + +Plugins distributed on Windows for system wide installation should be +installed in `%PROGRAMDATA%\Docker\cli-plugins`. + +User's may on all systems install plugins into `~/.docker/cli-plugins`. + +## Implementing a plugin in Go + +When writing a plugin in Go the easiest way to meet the above +requirements is to simply call the +`github.com/docker/cli/cli-plugins/plugin.Run` method from your `main` +function to instantiate the plugin. From 96f4952fe592c2a7247248124ec0453f82f7b870 Mon Sep 17 00:00:00 2001 From: stevejr <> Date: Wed, 13 Feb 2019 11:38:48 +0000 Subject: [PATCH 305/534] Corrected max-file option - was incorrectly spelt as max-files Signed-off-by: Steve Richards --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index d31757f2a0..825b838c41 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1312,7 +1312,7 @@ This is a full example of the allowed configuration options on Linux: "log-driver": "json-file", "log-opts": { "max-size": "10m", - "max-files":"5", + "max-file":"5", "labels": "somelabel", "env": "os,customer" }, From 7d7103b8f705437dfa4b6edfec8575529ca46741 Mon Sep 17 00:00:00 2001 From: Ian Campbell Date: Thu, 31 Jan 2019 17:50:58 +0000 Subject: [PATCH 306/534] cli-plugins: use `docker system dial-stdio` to call the daemon This means that plugins can use whatever methods the monolithic CLI supports, which is good for consistency. This relies on `os.Args[0]` being something which can be executed again to reach the same binary, since it is propagated (via an envvar) to the plugin for this purpose. This essentially requires that the current working directory and path are not modified by the monolithic CLI before it launches the plugin nor by the plugin before it initializes the client. This should be the case. Previously the fake apiclient used by `TestExperimentalCLI` was not being used, since `cli.Initialize` was unconditionally overwriting it with a real one (talking to a real daemon during unit testing, it seems). This wasn't expected nor desirable and no longer happens with the new arrangements, exposing the fact that no `pingFunc` is provided, leading to a panic. Add a `pingFunc` to the fake client to avoid this. Signed-off-by: Ian Campbell --- docs/extend/cli_plugins.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/extend/cli_plugins.md b/docs/extend/cli_plugins.md index a99204ca58..c47d93809e 100644 --- a/docs/extend/cli_plugins.md +++ b/docs/extend/cli_plugins.md @@ -75,6 +75,19 @@ A plugin is required to support all of the global options of the top-level CLI, i.e. those listed by `man docker 1` with the exception of `-v`. +## Connecting to the docker engine + +For consistency plugins should prefer to dial the engine by using the +`system dial-stdio` subcommand of the main Docker CLI binary. + +To facilitate this plugins will be executed with the +`$DOCKER_CLI_PLUGIN_ORIGINAL_CLI_COMMAND` environment variable +pointing back to the main Docker CLI binary. + +All global options (everything from after the binary name up to, but +not including, the primary entry point subcommand name) should be +passed back to the CLI. + ## Installation Plugins distributed in packages for system wide installation on From 027deb2a7c22d1caaec9d9903a0578d952899a50 Mon Sep 17 00:00:00 2001 From: Ian Campbell Date: Thu, 31 Jan 2019 15:32:25 +0000 Subject: [PATCH 307/534] Add a field to the config file for plugin use. This is a bit manual (as the unit test attests) so we may find we want to add some helpers/accessors, but this is enough to let plugins use it and to preserve the information through round-trips. Signed-off-by: Ian Campbell --- docs/extend/cli_plugins.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/extend/cli_plugins.md b/docs/extend/cli_plugins.md index c47d93809e..4721bafcd1 100644 --- a/docs/extend/cli_plugins.md +++ b/docs/extend/cli_plugins.md @@ -75,6 +75,18 @@ A plugin is required to support all of the global options of the top-level CLI, i.e. those listed by `man docker 1` with the exception of `-v`. +## Configuration + +Plugins are expected to make use of existing global configuration +where it makes sense and likewise to consider extending the global +configuration (by patching `docker/cli` to add new fields) where that +is sensible. + +Where plugins unavoidably require specific configuration the +`.plugins.«name»` key in the global `config.json` is reserved for +their use. However the preference should be for shared/global +configuration whenever that makes sense. + ## Connecting to the docker engine For consistency plugins should prefer to dial the engine by using the From 87f11bbc4d05929b6dc0031c2dd1d501e79369cd Mon Sep 17 00:00:00 2001 From: Stefan Scherer Date: Tue, 19 Mar 2019 09:30:14 -0700 Subject: [PATCH 308/534] Search Windows CLI plugins also in ProgramFiles Signed-off-by: Stefan Scherer --- docs/extend/cli_plugins.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extend/cli_plugins.md b/docs/extend/cli_plugins.md index 4721bafcd1..766e54ff3a 100644 --- a/docs/extend/cli_plugins.md +++ b/docs/extend/cli_plugins.md @@ -111,7 +111,8 @@ the `/usr/local/lib` or `/usr/local/libexec` equivalents but packages should not do so. Plugins distributed on Windows for system wide installation should be -installed in `%PROGRAMDATA%\Docker\cli-plugins`. +installed in either `%ProgramData%\Docker\cli-plugins` or +`%ProgramFiles%\Docker\cli-plugins`. User's may on all systems install plugins into `~/.docker/cli-plugins`. From 5511f958713e78478e0247011ff806f4adddaf98 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 13 May 2019 16:37:30 -0700 Subject: [PATCH 309/534] Fix PowerShell codehint for rouge Rouge is case-sensitive, and only works with powershell all lowercase. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 825b838c41..29427c4149 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -812,7 +812,7 @@ Defaults to 20G. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt size=40G ``` @@ -827,7 +827,7 @@ deployments). ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.globalmode=false ``` @@ -838,7 +838,7 @@ used for booting a utility VM. Defaults to `%ProgramFiles%\Linux Containers`. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.kirdpath=c:\path\to\files ``` @@ -849,7 +849,7 @@ Defaults to `bootx64.efi`. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.kernel=kernel.efi ``` @@ -860,7 +860,7 @@ Defaults to `initrd.img`. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.initrd=myinitrd.img ``` @@ -872,7 +872,7 @@ are kernel specific. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt "lcow.bootparameters='option=value'" ``` @@ -883,7 +883,7 @@ and initrd booting. Defaults to `uvm.vhdx` under `lcow.kirdpath`. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.vhdx=custom.vhdx ``` @@ -894,7 +894,7 @@ to 300. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.timeout=240 ``` @@ -905,7 +905,7 @@ containers. Defaults to 20. Cannot be less than 20. ###### Example -```PowerShell +```powershell C:\> dockerd --storage-opt lcow.sandboxsize=40 ``` From 30e49ba9fe54cb6339d8b5a9ba78d2ef23efdfd1 Mon Sep 17 00:00:00 2001 From: Andrew Hsu Date: Tue, 18 Jun 2019 18:51:19 +0000 Subject: [PATCH 310/534] deprecate registry v2 schema 1 Co-Authored-By: Sebastiaan van Stijn Signed-off-by: Andrew Hsu Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1ca390a897..58ac15b0d0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -19,6 +19,19 @@ The following list of features are deprecated in Engine. To learn more about Docker Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +### Pushing and pulling with image manifest v2 schema 1 + +**Deprecated in Release: v19.03.0** + +**Target For Removal In Release: v19.09.0** + +The image manifest +[v2 schema 1](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-1.md) +format is deprecated in favor of the +[v2 schema 2](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-2.md) format. + +If the registry you are using still supports v2 schema 1, urge their administrators to move to v2 schema 2. + ### Legacy "overlay" storage driver **Deprecated in Release: v18.09.0** From b5af6523da69c11bd75048da9bb27f86cdb4b520 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 26 Oct 2018 18:31:53 +0200 Subject: [PATCH 311/534] Deprecate AuFS storage driver The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are recommended to migrate to a different storage driver, such as `overlay2`, which is now the default storage driver. The `aufs` storage driver facilitates running Docker on distros that have no support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with a 3.14 kernel. Now that Ubuntu 14.04 is no longer a supported distro for Docker, and `overlay2` is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 58ac15b0d0..9e269b9aa0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -63,6 +63,24 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. +### AuFS storage driver + +**Deprecated in Release: v19.03.0** + +The `aufs` storage driver is deprecated in favor of `overlay2`, and will +be removed in a future release. Users of the `aufs` storage driver are +recommended to migrate to a different storage driver, such as `overlay2`, which +is now the default storage driver. + +The `aufs` storage driver facilitates running Docker on distros that have no +support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with +a 3.14 kernel. + +Now that Ubuntu 14.04 is no longer a supported distro for Docker, and `overlay2` +is available to all supported distros (as they are either on kernel 4.x, or have +support for multiple lowerdirs backported), there is no reason to continue +maintenance of the `aufs` storage driver. + ### Reserved namespaces in engine labels **Deprecated in Release: v18.06.0** From 1679ee297ef8be702c4ecec8176e54d0e8fcdd2f Mon Sep 17 00:00:00 2001 From: Lukas Heeren Date: Mon, 14 Oct 2019 13:33:37 +0200 Subject: [PATCH 312/534] daemon: document --max-download-attempts option update docs based on PR 39949 Signed-off-by: Lukas Heeren Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 29427c4149..b4a71dd466 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -70,6 +70,7 @@ Options: --log-opt map Default log driver options for containers (default map[]) --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) + --max-download-attempts int Set the max download attempts for each pull (default 5) --metrics-addr string Set default address and port to serve the metrics api on --mtu int Set the containers network MTU --node-generic-resources list Advertise user-defined resource @@ -1323,6 +1324,7 @@ This is a full example of the allowed configuration options on Linux: "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, + "max-download-attempts": 5, "default-shm-size": "64M", "shutdown-timeout": 15, "debug": true, @@ -1424,6 +1426,7 @@ This is a full example of the allowed configuration options on Windows: "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, + "max-download-attempts": 5, "shutdown-timeout": 15, "debug": true, "hosts": [], @@ -1473,6 +1476,7 @@ The list of currently supported options that can be reconfigured is this: - `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. - `max-concurrent-uploads`: it updates the max concurrent uploads for each push. +- `max-download-attempts`: it updates the max download attempts for each pull. - `default-runtime`: it updates the runtime to be used if not is specified at container creation. It defaults to "default" which is the runtime shipped with the official docker packages. From 439f79c3058971d99c41255034752f72d2722a67 Mon Sep 17 00:00:00 2001 From: Rob Gulewich Date: Tue, 30 Jul 2019 10:10:03 -0700 Subject: [PATCH 313/534] docker run: specify cgroup namespace mode with --cgroupns Signed-off-by: Rob Gulewich --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b4a71dd466..4206f1cf46 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -38,6 +38,7 @@ Options: --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds --data-root string Root directory of persistent Docker state (default "/var/lib/docker") -D, --debug Enable debug mode + --default-cgroupns-mode string Container default cgroup namespace mode (default "host") --default-gateway ip Container default gateway IPv4 address --default-gateway-v6 ip Container default gateway IPv6 address --default-address-pool Set the default address pool for local node networks @@ -1299,6 +1300,7 @@ This is a full example of the allowed configuration options on Linux: { "authorization-plugins": [], "data-root": "", + "default-cgroupns-mode": "private", "dns": [], "dns-opts": [], "dns-search": [], From c88d21783152b80217b1e8e6429656f69a365ef5 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 14:27:22 +0100 Subject: [PATCH 314/534] docs: minor touch-ups in "deprecated.md" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9e269b9aa0..f745b96e59 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -47,7 +47,7 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. -### device mapper storage driver +### Device mapper storage driver **Deprecated in Release: v18.09.0** @@ -91,7 +91,7 @@ were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their use, and will error instead in 18.12 and above. -### Asynchronous `service create` and `service update` +### Asynchronous `service create` and `service update` as default **Deprecated In Release: v17.05.0** @@ -278,7 +278,7 @@ Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. -### Driver Specific Log Tags +### Driver-specific log tags **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** From a0237c68038d619f89af4422aea2161820248009 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 14:28:35 +0100 Subject: [PATCH 315/534] docs: update some release-versions in "deprecated.md" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index f745b96e59..73c5443728 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -23,7 +23,7 @@ see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-depreca **Deprecated in Release: v19.03.0** -**Target For Removal In Release: v19.09.0** +**Target For Removal In Release: v20.03.0** The image manifest [v2 schema 1](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-1.md) @@ -89,7 +89,7 @@ The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engin were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their -use, and will error instead in 18.12 and above. +use, and will error instead in v20.03.0 and above. ### Asynchronous `service create` and `service update` as default From d959aefadc792f955bf2b3dc29ed2f357bf97376 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 15:03:42 +0100 Subject: [PATCH 316/534] docs: sort "deprecated.md" by deprecation release Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 176 ++++++++++++++++++++++++--------------------- 1 file changed, 95 insertions(+), 81 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 73c5443728..25392157ad 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -32,6 +32,26 @@ format is deprecated in favor of the If the registry you are using still supports v2 schema 1, urge their administrators to move to v2 schema 2. + +### AuFS storage driver + +**Deprecated in Release: v19.03.0** + +The `aufs` storage driver is deprecated in favor of `overlay2`, and will +be removed in a future release. Users of the `aufs` storage driver are +recommended to migrate to a different storage driver, such as `overlay2`, which +is now the default storage driver. + +The `aufs` storage driver facilitates running Docker on distros that have no +support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with +a 3.14 kernel. + +Now that Ubuntu 14.04 is no longer a supported distro for Docker, and `overlay2` +is available to all supported distros (as they are either on kernel 4.x, or have +support for multiple lowerdirs backported), there is no reason to continue +maintenance of the `aufs` storage driver. + + ### Legacy "overlay" storage driver **Deprecated in Release: v18.09.0** @@ -63,23 +83,6 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -### AuFS storage driver - -**Deprecated in Release: v19.03.0** - -The `aufs` storage driver is deprecated in favor of `overlay2`, and will -be removed in a future release. Users of the `aufs` storage driver are -recommended to migrate to a different storage driver, such as `overlay2`, which -is now the default storage driver. - -The `aufs` storage driver facilitates running Docker on distros that have no -support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with -a 3.14 kernel. - -Now that Ubuntu 14.04 is no longer a supported distro for Docker, and `overlay2` -is available to all supported distros (as they are either on kernel 4.x, or have -support for multiple lowerdirs backported), there is no reason to continue -maintenance of the `aufs` storage driver. ### Reserved namespaces in engine labels @@ -91,6 +94,38 @@ were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their use, and will error instead in v20.03.0 and above. + +### `--disable-legacy-registry` override daemon option + +**Disabled In Release: v17.12** + +**Target For Removal In Release: v18.03** + +The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print +an error when used. For this error to be printed, the flag itself is still present, +but hidden. The flag will be removed in Docker 18.03. + + +### Interacting with V1 registries + +**Disabled By Default In Release: v17.06** + +**Removed In Release: v17.12** + +Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the +docker daemon from `pull`, `push`, and `login` operations against v1 +registries. Though enabled by default, this signals the intent to deprecate +the v1 protocol. + +Support for the v1 protocol to the public registry was removed in 1.13. Any +mirror configurations using v1 should be updated to use a +[v2 registry mirror](https://docs.docker.com/registry/recipes/mirror/). + +Starting with Docker 17.12, support for V1 registries has been removed, and the +`--disable-legacy-registry` flag can no longer be used, and `dockerd` will fail to +start when set. + + ### Asynchronous `service create` and `service update` as default **Deprecated In Release: v17.05.0** @@ -199,12 +234,16 @@ to 17.12, a warning will be printed. Please also refer to [#27358](https://github.com/docker/docker/issues/27358) for further information. -### Three arguments form in `docker import` -**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +### `--automated` and `--stars` flags on `docker search` + +**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +**Target For Removal In Release: v17.09** + +The `docker search --automated` and `docker search --stars` options are deprecated. +Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. -The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated since November 2013. It's no more supported. ### `-h` shorthand for `--help` @@ -231,14 +270,6 @@ The docker login command is removing the ability to automatically register for a The flag `--security-opt` doesn't use the colon separator (`:`) anymore to divide keys and values, it uses the equal symbol (`=`) for consistency with other similar flags, like `--storage-opt`. -### `/containers/(id or name)/copy` endpoint - -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** - -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. - ### Ambiguous event fields in API **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** @@ -269,14 +300,6 @@ defining it at container creation (`POST /containers/create`). The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. -### `--automated` and `--stars` flags on `docker search` - -**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** - -**Target For Removal In Release: v17.09** - -The `docker search --automated` and `docker search --stars` options are deprecated. -Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. ### Driver-specific log tags **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** @@ -293,6 +316,27 @@ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" {% endraw %} ``` + +### Docker Content Trust ENV passphrase variables name change +**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables + +- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE +- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE + + +### `/containers/(id or name)/copy` endpoint + +**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** + +**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** + +The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. + + ### LXC built-in exec driver **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** @@ -300,6 +344,7 @@ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. + ### Old Command Line Options **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** @@ -358,6 +403,16 @@ The single-dash (`-help`) was removed, in favor of the double-dash `--help` docker -help docker [COMMAND] -help + +### `--api-enable-cors` flag on dockerd + +**Deprecated In Release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** + +**Removed In Release: [v17.09](https://github.com/docker/docker-ce/releases/tag/v17.09.0-ce)** + +The flag `--api-enable-cors` is deprecated since v1.6.0. Use the flag +`--api-cors-header` instead. + ### `--run` flag on docker commit **Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** @@ -368,51 +423,10 @@ The flag `--run` of the docker commit (and its short version `-run`) were deprec of the `--changes` flag that allows to pass `Dockerfile` commands. -### Interacting with V1 registries - -**Disabled By Default In Release: v17.06** - -**Removed In Release: v17.12** - -Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the -docker daemon from `pull`, `push`, and `login` operations against v1 -registries. Though enabled by default, this signals the intent to deprecate -the v1 protocol. - -Support for the v1 protocol to the public registry was removed in 1.13. Any -mirror configurations using v1 should be updated to use a -[v2 registry mirror](https://docs.docker.com/registry/recipes/mirror/). - -Starting with Docker 17.12, support for V1 registries has been removed, and the -`--disable-legacy-registry` flag can no longer be used, and `dockerd` will fail to -start when set. - -### `--disable-legacy-registry` override daemon option - -**Disabled In Release: v17.12** - -**Target For Removal In Release: v18.03** - -The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print -an error when used. For this error to be printed, the flag itself is still present, -but hidden. The flag will be removed in Docker 18.03. - - -### Docker Content Trust ENV passphrase variables name change -**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** +### Three arguments form in `docker import` +**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables +The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated since November 2013. It's no more supported. -- DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE -- DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE - -### `--api-enable-cors` flag on dockerd - -**Deprecated In Release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** - -**Removed In Release: [v17.09](https://github.com/docker/docker-ce/releases/tag/v17.09.0-ce)** - -The flag `--api-enable-cors` is deprecated since v1.6.0. Use the flag -`--api-cors-header` instead. From c0efb2dbe35e43f536946015bb3a2e4211b7e116 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 14:30:24 +0100 Subject: [PATCH 317/534] docs: mark deprecation of "dab" files and top-level "deploy" subcommand Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 25392157ad..78c709fcff 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -33,6 +33,30 @@ format is deprecated in favor of the If the registry you are using still supports v2 schema 1, urge their administrators to move to v2 schema 2. +### Top-level `docker deploy` subcommand (experimental) + +**Deprecated in Release: v19.03.0** + +**Target For Removal In Release: v20.03.0** + +The top-level `docker deploy` command (using the "Docker Application Bundle" +(.dab) file format was introduced as an experimental feature in Docker 1.13 / +17.03, but superseded by support for Docker Compose files using the `docker stack deploy` +subcommand. + + +### `docker stack deploy` using "dab" files (experimental) + +**Deprecated in Release: v19.03.0** + +**Target For Removal In Release: v20.03.0** + +With no development being done on this feature, and no active use of the file +format, support for the DAB file format and the top-level docker deploy command +(hidden by default in 19.03), will be removed, in favour of `docker stack deploy` +using compose files. + + ### AuFS storage driver **Deprecated in Release: v19.03.0** From caf7cc9f63aad0ddd3f749a97f5c2fd698948f71 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 14:31:17 +0100 Subject: [PATCH 318/534] docs: mark deprecation of "docker engine" subcommands Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 78c709fcff..a78a238993 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -33,6 +33,22 @@ format is deprecated in favor of the If the registry you are using still supports v2 schema 1, urge their administrators to move to v2 schema 2. +### `docker engine` subcommands + +**Deprecated in Release: v19.03.0** + +**Target For Removal In Release: v20.03.0** + +The `docker engine activate`, `docker engine check`, and `docker engine update` +provided an alternative installation method to upgrade Docker Community engines +to Docker Enterprise, using an image-based distribution of the Docker Engine. + +This feature was only available on Linux, and only when executed on a local node. +Given the limitations of this feature, and the feature not getting widely adopted, +the `docker engine` subcommands will be removed, in favor of installation through +standard package managers. + + ### Top-level `docker deploy` subcommand (experimental) **Deprecated in Release: v19.03.0** From 3c001dd80a825cd0698b267b2631f12be475c1e6 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 14:41:38 +0100 Subject: [PATCH 319/534] docs: add status table to "deprecated.md" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 59 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a78a238993..7e694adb8a 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -15,9 +15,62 @@ keywords: "docker, documentation, about, technology, deprecate" # Deprecated Engine Features -The following list of features are deprecated in Engine. -To learn more about Docker Engine's deprecation policy, -see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). +The following list of features are deprecated in Engine. To learn more about Docker +Engine's deprecation policy, see [Feature Deprecation Policy](https://docs.docker.com/engine/#feature-deprecation-policy). + +The table below provides an overview of the current status of deprecated features: + +- **Deprecated**: the feature is marked "deprecated" and should no longer be used. + The feature may be removed, disabled, or change behavior in a future release. + The _"Deprecated"_ column contains the release in which the feature was marked + deprecated, whereas the _"Remove"_ column contains a tentative release in which + the feature is to be removed. If no release is included in the _"Remove"_ column, + the release is yet to be decided on. +- **Removed**: the feature was removed, disabled, or hidden. Refer to the linked + section for details. Some features are "soft" deprecated, which means that they + remain functional for backward compatibility, and to allow users to migrate to + alternatives. In such cases, a warning may be printed, and users should not rely + on this feature. + +Status | Feature | Deprecated | Remove +-----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 +Deprecated | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 +Deprecated | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 +Deprecated | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03.0 | v20.03.0 +Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03.0 | - +Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09.0 | - +Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09.0 | - +Deprecated | [Reserved namespaces in engine labels](#reserved-namespaces-in-engine-labels) | v18.06.0 | v20.03.0 +Deprecated | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12.0 | v18.03.0 +Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06.0 | v17.12.0 +Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05.0 | v17.10.0 +Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05.0 | - +Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13.0 | v17.12.0 +Deprecated | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13.0 | v17.12.0 +Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13.0 | v17.12.0 +Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13.0 | v17.12.0 +Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13.0 | v17.12.0 +Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13.0 | - +Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13.0 | v17.12.0 +Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13.0 | v17.12.0 +Deprecated | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12.0 | v17.09.0 +Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12.0 | v17.09.0 +Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11.0 | v17.06.0 +Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11.0 | v17.06.0 +Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10.0 | - +Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10.0 | v1.12.0 +Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10.0 | v1.12.0 +Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10.0 | v1.12.0 +Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9.0 | v1.12.0 +Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9.0 | v1.12.0 +Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8.0 | v1.12.0 +Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8.0 | v1.10.0 +Removed | [Old Command Line Options](#old-command-line-options) | v1.8.0 | v1.10.0 +Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6.0 | v17.09.0 +Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10.0 | v1.13.0 +Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12.0 + ### Pushing and pulling with image manifest v2 schema 1 From 795a3cd5c7fcf71994d2d88a98822aacc9fd67ab Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 16:40:11 +0100 Subject: [PATCH 320/534] docs: update deprecation status of "engine *" commands, and dab files Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7e694adb8a..da37d95dfb 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -35,9 +35,9 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 -Deprecated | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 -Deprecated | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 -Deprecated | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03.0 | v20.03.0 +Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 +Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 +Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03.0 | v20.03.0 Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03.0 | - Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09.0 | - Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09.0 | - @@ -90,7 +90,7 @@ If the registry you are using still supports v2 schema 1, urge their administrat **Deprecated in Release: v19.03.0** -**Target For Removal In Release: v20.03.0** +**Removed in Release: v20.03.0** The `docker engine activate`, `docker engine check`, and `docker engine update` provided an alternative installation method to upgrade Docker Community engines @@ -106,7 +106,7 @@ standard package managers. **Deprecated in Release: v19.03.0** -**Target For Removal In Release: v20.03.0** +**Removed in Release: v20.03.0** The top-level `docker deploy` command (using the "Docker Application Bundle" (.dab) file format was introduced as an experimental feature in Docker 1.13 / @@ -118,7 +118,7 @@ subcommand. **Deprecated in Release: v19.03.0** -**Target For Removal In Release: v20.03.0** +**Removed in Release: v20.03.0** With no development being done on this feature, and no active use of the file format, support for the DAB file format and the top-level docker deploy command From 220957633adce83b5fd1949b683935b8e1a92fbd Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 16:52:21 +0100 Subject: [PATCH 321/534] docs: update deprecated status for --disable-legacy-registry Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7e694adb8a..b17ae068dc 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -42,7 +42,7 @@ Deprecated | [AuFS storage driver](#aufs-storage-driver) Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09.0 | - Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09.0 | - Deprecated | [Reserved namespaces in engine labels](#reserved-namespaces-in-engine-labels) | v18.06.0 | v20.03.0 -Deprecated | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12.0 | v18.03.0 +Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12.0 | v19.03.0 Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06.0 | v17.12.0 Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05.0 | v17.10.0 Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05.0 | - @@ -192,11 +192,11 @@ use, and will error instead in v20.03.0 and above. **Disabled In Release: v17.12** -**Target For Removal In Release: v18.03** +**Removed In Release: v19.03** The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print an error when used. For this error to be printed, the flag itself is still present, -but hidden. The flag will be removed in Docker 18.03. +but hidden. The flag has been removed in Docker 19.03. ### Interacting with V1 registries From ee9e8f510c0e4dfdb6d213447eabadbe95f30f20 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Feb 2020 17:24:52 +0100 Subject: [PATCH 322/534] docs: update deprecation status of "filter" option Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7e694adb8a..04c6c5730c 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -47,7 +47,7 @@ Removed | [Interacting with V1 registries](#interacting-with-v1-registries) Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05.0 | v17.10.0 Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05.0 | - Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13.0 | v17.12.0 -Deprecated | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13.0 | v17.12.0 +Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13.0 | v20.03.0 Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13.0 | v17.12.0 Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13.0 | v17.12.0 Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13.0 | v17.12.0 @@ -267,9 +267,10 @@ information. ### `filter` param for `/images/json` endpoint **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Removed In Release: v20.03** -The `filter` param to filter the list of image by reference (name or name:tag) is now implemented as a regular filter, named `reference`. +The `filter` param to filter the list of image by reference (name or name:tag) +is now implemented as a regular filter, named `reference`. ### `repository:shortid` image references **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** From 0c43972b5fc7e90eb3466753a308fe06b86b9c8e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 11 Feb 2020 17:06:55 +0100 Subject: [PATCH 323/534] deprecated: use of reserved namespaces for engine labels Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index ace1ce6d79..f676403654 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -41,7 +41,7 @@ Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-st Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03.0 | - Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09.0 | - Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09.0 | - -Deprecated | [Reserved namespaces in engine labels](#reserved-namespaces-in-engine-labels) | v18.06.0 | v20.03.0 +Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06.0 | v20.03.0 Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12.0 | v19.03.0 Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06.0 | v17.12.0 Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05.0 | v17.10.0 @@ -177,10 +177,12 @@ either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -### Reserved namespaces in engine labels +### Use of reserved namespaces in engine labels **Deprecated in Release: v18.06.0** +**Removed In Release: v20.03** + The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engine labels were always documented to be reserved, but there was never any enforcement. From 287862324b160bcef88edd1f33bc55a04d70f02b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 13 Feb 2020 17:18:10 +0100 Subject: [PATCH 324/534] Remove deprecated docker search --automated and --stars flags The `docker search --automated` and `docker search --stars` options were deprecated in release v1.12.0, and scheduled for removal in v17.09. This patch removes the deprecated flags, in favor of their equivalent `--filter` options (`docker search --filter=is-automated=` and `docker search --filter=stars=...`). Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index f676403654..548bdd539f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -54,7 +54,7 @@ Removed | [Duplicate keys with conflicting values in engine labels](#duplicat Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13.0 | - Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13.0 | v17.12.0 Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13.0 | v17.12.0 -Deprecated | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12.0 | v17.09.0 +Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12.0 | v20.03.0 Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12.0 | v17.09.0 Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11.0 | v17.06.0 Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11.0 | v17.06.0 @@ -335,10 +335,10 @@ further information. **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Target For Removal In Release: v17.09** +**Removed In Release: v20.03** The `docker search --automated` and `docker search --stars` options are deprecated. -Use `docker search --filter=is-automated=...` and `docker search --filter=stars=...` instead. +Use `docker search --filter=is-automated=` and `docker search --filter=stars=...` instead. ### `-h` shorthand for `--help` From 09dd0e3543381f492ef0728a540700ea4a1ab686 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 20 Feb 2020 00:19:06 +0100 Subject: [PATCH 325/534] docs: add redirect for old location of daemon reference Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 4206f1cf46..492cdac849 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1,8 +1,9 @@ --- title: "dockerd" -aliases: ["/engine/reference/commandline/daemon/"] description: "The daemon command description and usage" keywords: "container, daemon, runtime" +redirect_from: +- /engine/reference/commandline/daemon/ --- - -# Docker CLI Plugin Spec - -The `docker` CLI supports adding additional top-level subcommands as -additional out-of-process commands which can be installed -independently. These plugins run on the client side and should not be -confused with "plugins" which run on the server. - -This document contains information for authors of such plugins. - -## Requirements for CLI Plugins - -### Naming - -A valid CLI plugin name consists only of lower case letters `a-z` -and the digits `0-9`. The leading character must be a letter. A valid -name therefore would match the regex `^[a-z][a-z0-9]*$`. - -The binary implementing a plugin must be named `docker-$name` where -`$name` is the name of the plugin. On Windows a `.exe` suffix is -mandatory. - -## Required sub-commands - -A CLI plugin must support being invoked in at least these two ways: - -* `docker-$name docker-cli-plugin-metadata` -- outputs metadata about - the plugin. -* `docker-$name [GLOBAL OPTIONS] $name [OPTIONS AND FURTHER SUB - COMMANDS]` -- the primary entry point to the plugin's functionality. - -A plugin may implement other subcommands but these will never be -invoked by the current Docker CLI. However doing so is strongly -discouraged: new subcommands may be added in the future without -consideration for additional non-specified subcommands which may be -used by plugins in the field. - -### The `docker-cli-plugin-metadata` subcommand - -When invoked in this manner the plugin must produce a JSON object -(and nothing else) on its standard output and exit success (0). - -The JSON object has the following defined keys: -* `SchemaVersion` (_string_) mandatory: must contain precisely "0.1.0". -* `Vendor` (_string_) mandatory: contains the name of the plugin vendor/author. May be truncated to 11 characters in some display contexts. -* `ShortDescription` (_string_) optional: a short description of the plugin, suitable for a single line help message. -* `Version` (_string_) optional: the version of the plugin, this is considered to be an opaque string by the core and therefore has no restrictions on its syntax. -* `URL` (_string_) optional: a pointer to the plugin's web page. - -A binary which does not correctly output the metadata -(e.g. syntactically invalid, missing mandatory keys etc) is not -considered a valid CLI plugin and will not be run. - -### The primary entry point subcommand - -This is the entry point for actually running the plugin. It maybe have -options or further subcommands. - -#### Required global options - -A plugin is required to support all of the global options of the -top-level CLI, i.e. those listed by `man docker 1` with the exception -of `-v`. - -## Configuration - -Plugins are expected to make use of existing global configuration -where it makes sense and likewise to consider extending the global -configuration (by patching `docker/cli` to add new fields) where that -is sensible. - -Where plugins unavoidably require specific configuration the -`.plugins.«name»` key in the global `config.json` is reserved for -their use. However the preference should be for shared/global -configuration whenever that makes sense. - -## Connecting to the docker engine - -For consistency plugins should prefer to dial the engine by using the -`system dial-stdio` subcommand of the main Docker CLI binary. - -To facilitate this plugins will be executed with the -`$DOCKER_CLI_PLUGIN_ORIGINAL_CLI_COMMAND` environment variable -pointing back to the main Docker CLI binary. - -All global options (everything from after the binary name up to, but -not including, the primary entry point subcommand name) should be -passed back to the CLI. - -## Installation - -Plugins distributed in packages for system wide installation on -Unix(-like) systems should be installed in either -`/usr/lib/docker/cli-plugins` or `/usr/libexec/docker/cli-plugins` -depending on which of `/usr/lib` and `/usr/libexec` is usual on that -system. System Administrators may also choose to manually install into -the `/usr/local/lib` or `/usr/local/libexec` equivalents but packages -should not do so. - -Plugins distributed on Windows for system wide installation should be -installed in either `%ProgramData%\Docker\cli-plugins` or -`%ProgramFiles%\Docker\cli-plugins`. - -User's may on all systems install plugins into `~/.docker/cli-plugins`. - -## Implementing a plugin in Go - -When writing a plugin in Go the easiest way to meet the above -requirements is to simply call the -`github.com/docker/cli/cli-plugins/plugin.Run` method from your `main` -function to instantiate the plugin. From 78f435c2a23340b4bf44c13d71e57d02b2c7e25f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 25 May 2020 11:07:10 +0200 Subject: [PATCH 337/534] docs: fix broken link in dockerd.md The Engine API docs are not available in this GitHub repository, so linking to the docs.docker.com website instead. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 63db3bc801..d285b81334 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -124,7 +124,7 @@ the `daemon.json` file. ### Daemon socket option -The Docker daemon can listen for [Docker Engine API](../api/) +The Docker daemon can listen for [Docker Engine API](https://docs.docker.com/engine/api/) requests via three different types of Socket: `unix`, `tcp`, and `fd`. By default, a `unix` domain socket (or IPC socket) is created at From 2ef9d2ca1295d75a8a1ca72b078b4bb87bce3685 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 28 May 2020 17:28:43 +0200 Subject: [PATCH 338/534] docs/extend: fix broken link and some markdown touch-ups Signed-off-by: Sebastiaan van Stijn --- docs/extend/config.md | 104 ++++++++++++++++----------------- docs/extend/plugins_network.md | 16 +++-- 2 files changed, 62 insertions(+), 58 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index e68af3bf6a..0d5ae9ae4f 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -34,11 +34,11 @@ Config provides the base accessible fields for working with V0 plugin format - **`description`** *string* - description of the plugin + description of the plugin - **`documentation`** *string* - link to the documentation about the plugin + link to the documentation about the plugin - **`interface`** *PluginInterface* @@ -96,7 +96,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`name`** *string* - name of the mount. + name of the mount. - **`description`** *string* @@ -104,11 +104,11 @@ Config provides the base accessible fields for working with V0 plugin format - **`source`** *string* - source of the mount. + source of the mount. - **`destination`** *string* - destination of the mount. + destination of the mount. - **`type`** *string* @@ -116,7 +116,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`options`** *string array* - options of the mount. + options of the mount. - **`ipchost`** *boolean* Access to host ipc namespace. @@ -135,7 +135,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`name`** *string* - name of the env. + name of the env. - **`description`** *string* @@ -143,7 +143,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`value`** *string* - value of the env. + value of the env. - **`args`** *PluginArgs* @@ -151,7 +151,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`name`** *string* - name of the args. + name of the args. - **`description`** *string* @@ -159,7 +159,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`value`** *string array* - values of the args. + values of the args. - **`linux`** *PluginLinux* @@ -169,7 +169,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`allowAllDevices`** *boolean* - If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. + If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. - **`devices`** *PluginDevice array* @@ -177,7 +177,7 @@ Config provides the base accessible fields for working with V0 plugin format - **`name`** *string* - name of the device. + name of the device. - **`description`** *string* @@ -193,45 +193,45 @@ Config provides the base accessible fields for working with V0 plugin format ```json { - "Args": { - "Description": "", - "Name": "", - "Settable": null, - "Value": null - }, - "Description": "A sample volume plugin for Docker", - "Documentation": "https://docs.docker.com/engine/extend/plugins/", - "Entrypoint": [ - "/usr/bin/sample-volume-plugin", - "/data" - ], - "Env": [ - { - "Description": "", - "Name": "DEBUG", - "Settable": [ - "value" - ], - "Value": "0" - } - ], - "Interface": { - "Socket": "plugin.sock", - "Types": [ - "docker.volumedriver/1.0" - ] - }, - "Linux": { - "Capabilities": null, - "AllowAllDevices": false, - "Devices": null - }, - "Mounts": null, - "Network": { - "Type": "" - }, - "PropagatedMount": "/data", - "User": {}, - "Workdir": "" + "Args": { + "Description": "", + "Name": "", + "Settable": null, + "Value": null + }, + "Description": "A sample volume plugin for Docker", + "Documentation": "https://docs.docker.com/engine/extend/plugins/", + "Entrypoint": [ + "/usr/bin/sample-volume-plugin", + "/data" + ], + "Env": [ + { + "Description": "", + "Name": "DEBUG", + "Settable": [ + "value" + ], + "Value": "0" + } + ], + "Interface": { + "Socket": "plugin.sock", + "Types": [ + "docker.volumedriver/1.0" + ] + }, + "Linux": { + "Capabilities": null, + "AllowAllDevices": false, + "Devices": null + }, + "Mounts": null, + "Network": { + "Type": "" + }, + "PropagatedMount": "/data", + "User": {}, + "Workdir": "" } ``` diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 66302c5cf2..93aea4bd0f 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -42,14 +42,18 @@ Once running however, network driver plugins are used just like the built-in network drivers: by being mentioned as a driver in network-oriented Docker commands. For example, - $ docker network create --driver weave mynet +```bash +$ docker network create --driver weave mynet +``` Some network driver plugins are listed in [plugins](legacy_plugins.md) The `mynet` network is now owned by `weave`, so subsequent commands referring to that network will be sent to the plugin, - $ docker run --network=mynet busybox top +```bash +$ docker run --network=mynet busybox top +``` ## Find network plugins @@ -61,8 +65,8 @@ or on the third party's site. ## Write a network plugin -Network plugins implement the [Docker plugin -API](plugin_api.md) and the network plugin protocol +Network plugins implement the [Docker plugin API](plugin_api.md) and the network +plugin protocol ## Network plugin protocol @@ -74,5 +78,5 @@ documented as part of libnetwork: To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. -- [Docker networks feature overview](https://docs.docker.com/engine/userguide/networking/) -- The [LibNetwork](https://github.com/docker/libnetwork) project +- [Docker networks feature overview](https://docs.docker.com/engine/userguide/networking/) +- The [LibNetwork](https://github.com/docker/libnetwork) project From bbd8d22cb97b1ce42c575050c83f04c8396ae38c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 16 Apr 2020 15:00:40 +0200 Subject: [PATCH 339/534] Deprecate classic Swarm and overlay networks using external k/v stores Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b55f029bc3..147911e9b8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 @@ -88,6 +89,15 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10.0 | v1.13.0 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12.0 +### Classic Swarm and overlay networks using cluster store + +**Deprecated in Release: v20.03.0** + +Standalone ("classic") Swarm has been deprecated, and with that the use of overlay +networks using an external key/value store. The corresponding`--cluster-advertise`, +`--cluster-store`, and `--cluster-store-opt` daemon options have been marked +deprecated, and will be disabled or removed in a future release. + ### CLI plugins support **Deprecated in Release: v20.03.0** From e2dd62d05ee8b90d9d3e276ba91b1c5f603727d9 Mon Sep 17 00:00:00 2001 From: Sargun Dhillon Date: Fri, 12 Jan 2018 09:30:31 -0800 Subject: [PATCH 340/534] Split up environment variable documentation between cli and dockerd This creates a new section of environment variables in the CLI docs which documents environment variables that can both be used on dockerd and on docker cli. In addition, it moves some of the environment variable documentation from the docker cli documentation to the dockerd documentation, as these environment variables are dockerd-specific. Signed-off-by: Sargun Dhillon Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 63db3bc801..2ac35680ea 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -120,6 +120,17 @@ the `daemon.json` file. > Enable experimental features by starting `dockerd` with the `--experimental` > flag or adding `"experimental": true` to the `daemon.json` file. +### Environment variables + +For easy reference, the following list of environment variables are supported +by the `dockerd` command line: + +* `DOCKER_DRIVER` The graph driver to use. +* `DOCKER_NOWARN_KERNEL_VERSION` Prevent warnings that your Linux kernel is + unsuitable for Docker. +* `DOCKER_RAMDISK` If set this will disable 'pivot_root'. +* `DOCKER_TMPDIR` Location for temporary Docker files. + ## Examples ### Daemon socket option From 8396c73a7cac3933f89b358209c7f15df961fa32 Mon Sep 17 00:00:00 2001 From: Sargun Dhillon Date: Fri, 12 Jan 2018 09:32:45 -0800 Subject: [PATCH 341/534] Add MOBY_DISABLE_PIGZ to dockerd environment variable docs This reflects a Moby change to add documentation around disabling a new feature -- to use pgzip to decompress layers, rather than the built-in go gzip. Signed-off-by: Sargun Dhillon Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2ac35680ea..580c2209fe 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -130,6 +130,8 @@ by the `dockerd` command line: unsuitable for Docker. * `DOCKER_RAMDISK` If set this will disable 'pivot_root'. * `DOCKER_TMPDIR` Location for temporary Docker files. +* `MOBY_DISABLE_PIGZ` Do not use [`unpigz`](https://linux.die.net/man/1/pigz) to + decompress layers in parallel when pulling images, even if it is installed. ## Examples From 13f21dc154e9c2e00a0149b449289f478c6e208b Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Wed, 24 Jun 2020 19:09:15 +0900 Subject: [PATCH 342/534] docs: update for cgroup v2 and rootless Signed-off-by: Akihiro Suda --- docs/reference/commandline/dockerd.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ae6ab6af60..eaa94ded46 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -290,7 +290,7 @@ $ docker -H tcp://127.0.0.1:2375 pull ubuntu ### Daemon storage-driver On Linux, the Docker daemon has support for several different image layer storage -drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay` and `overlay2`. +drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay`, `overlay2`, and `fuse-overlayfs`. The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that is unlikely to be merged into the main kernel. These are also known to cause @@ -341,6 +341,9 @@ to use it. > Both `overlay` and `overlay2` are currently unsupported on `btrfs` > or any Copy on Write filesystem and should only be used over `ext4` partitions. +The `fuse-overlayfs` driver is similar to `overlay2` but works in userspace. +The `fuse-overlayfs` driver is expected to be used for [Rootless mode](https://docs.docker.com/engine/security/rootless/). + On Windows, the Docker daemon supports a single image layer storage driver depending on the image platform: `windowsfilter` for Windows images, and `lcow` for Linux containers on Windows. @@ -999,7 +1002,8 @@ single `native.cgroupdriver` option is available. The `native.cgroupdriver` option specifies the management of the container's cgroups. You can only specify `cgroupfs` or `systemd`. If you specify `systemd` and it is not available, the system errors out. If you omit the -`native.cgroupdriver` option,` cgroupfs` is used. +`native.cgroupdriver` option,` cgroupfs` is used on cgroup v1 hosts, `systemd` +is used on cgroup v2 hosts with systemd available. This example sets the `cgroupdriver` to `systemd`: From aab695f5023e624fa4491558650cc98868248540 Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Fri, 24 Jul 2020 20:49:00 +0900 Subject: [PATCH 343/534] deprecate `docker run --kernel-memory` `docker run --kernel-memory` is being deprecated in https://github.com/moby/moby/pull/41254 Signed-off-by: Akihiro Suda --- docs/deprecated.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 147911e9b8..76883a4f4a 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.03.0 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 @@ -89,6 +90,13 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10.0 | v1.13.0 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12.0 +### Kernel memory limit + +**Deprecated in Release: v20.03.0** + +Specifying kernel memory limit (`docker run --kernel-memory`) is now marked as deprecated, +as [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). + ### Classic Swarm and overlay networks using cluster store **Deprecated in Release: v20.03.0** From e0509054a0ad33739c3df3de03fad17be497173e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 14 May 2020 01:27:58 +0200 Subject: [PATCH 344/534] document deprecation of legacy `~/.dockercfg` config-file The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this file with a new CLI configuration file, located in `~/.docker/config.json`. When implementing the new configuration file, the old file (and file-format) was kept as a fall-back, to assist existing users with migrating to the new file. Given that the old file format encourages insecure storage of credentials (credentials are stored unencrypted), and that no version of the CLI since Docker v1.7.0 has created this file, the file is marked deprecated, and support for this file will be removed in a future release. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 76883a4f4a..2db7073134 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,6 +52,7 @@ Status | Feature -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.03.0 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - +Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03.0 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 @@ -106,6 +107,22 @@ networks using an external key/value store. The corresponding`--cluster-advertis `--cluster-store`, and `--cluster-store-opt` daemon options have been marked deprecated, and will be disabled or removed in a future release. + +### Support for legacy `~/.dockercfg` configuration files + +**Deprecated in Release: v20.03.0** + +The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials +after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this +file with a new CLI configuration file, located in `~/.docker/config.json`. When +implementing the new configuration file, the old file (and file-format) was kept +as a fall-back, to assist existing users with migrating to the new file. + +Given that the old file format encourages insecure storage of credentials +(credentials are stored unencrypted), and that no version of the CLI since +Docker v1.7.0 has created this file, the file is marked deprecated, and support +for this file will be removed in a future release. + ### CLI plugins support **Deprecated in Release: v20.03.0** From e06c7252420d28eb9fb725c832d969c7fd7d5aa8 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 23 Sep 2020 20:48:26 +0200 Subject: [PATCH 345/534] deprecate Dockerfile legacy 'ENV name value' syntax The Dockerfile `ENV` instruction allows values to be set using either `ENV name=value` or `ENV name value`. The latter (`ENV name value`) form can be ambiguous, for example, the following defines a single env-variable (`ONE`) with value `"TWO= THREE=world"`, but may have intended to be setting three env-vars: ENV ONE TWO= THREE=world This format also does not allow setting multiple environment-variables in a single `ENV` line in the Dockerfile. Use of the `ENV name value` syntax is discouraged, and may be removed in a future release. Users are encouraged to update their Dockerfiles to use the `ENV name=value` syntax, for example: ENV ONE="" TWO="" THREE="world" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2db7073134..8abc7fad85 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -54,6 +54,7 @@ Deprecated | [Kernel memory limit](#kernel-memory-limit) Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03.0 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - +Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03.0 | - Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 @@ -129,6 +130,30 @@ for this file will be removed in a future release. CLI Plugin API is now marked as deprecated. +### Dockerfile legacy `ENV name value` syntax + +**Deprecated in Release: v20.03.0** + +The Dockerfile `ENV` instruction allows values to be set using either `ENV name=value` +or `ENV name value`. The latter (`ENV name value`) form can be ambiguous, for example, +the following defines a single env-variable (`ONE`) with value `"TWO= THREE=world"`, +but may have intended to be setting three env-vars: + +```dockerfile +ENV ONE TWO= THREE=world +``` + +This format also does not allow setting multiple environment-variables in a single +`ENV` line in the Dockerfile. + +Use of the `ENV name value` syntax is discouraged, and may be removed in a future +release. Users are encouraged to update their Dockerfiles to use the `ENV name=value` +syntax, for example: + +```dockerfile +ENV ONE="" TWO="" THREE="world" +``` + ### Pushing and pulling with image manifest v2 schema 1 **Deprecated in Release: v19.03.0** From 96e8a962bf452552e6b5fea402e1556815938996 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 2 Oct 2020 14:10:27 +0200 Subject: [PATCH 346/534] docs: deprecate CLI options for experimental CLI features Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 8abc7fad85..bf52a85d6e 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -54,7 +54,8 @@ Deprecated | [Kernel memory limit](#kernel-memory-limit) Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03.0 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - -Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03.0 | - +Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03.0 | - +Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03.0 | v20.03.0 Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 @@ -124,6 +125,12 @@ Given that the old file format encourages insecure storage of credentials Docker v1.7.0 has created this file, the file is marked deprecated, and support for this file will be removed in a future release. +### Configuration options for experimental CLI features + +The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` +field in the CLI configuration file are deprecated. Experimental features will be +enabled by default, and these configuration options will no longer be functional. + ### CLI plugins support **Deprecated in Release: v20.03.0** From 214ab7bf9d5d3b84e15d4e6520533710277aaf8e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 2 Oct 2020 14:15:42 +0200 Subject: [PATCH 347/534] docs/deprecated: remove minor versions Some deprecations are ammended during a major (YY.MM) release, to inform users as early as possible about deprecations. Removing the minor version from this overview clarifies that features are marked deprecated during which major release's lifecycle. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 126 ++++++++++++++++++++++----------------------- 1 file changed, 63 insertions(+), 63 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index bf52a85d6e..d960e23094 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,59 +50,59 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.03.0 | - -Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03.0 | - -Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03.0 | - -Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03.0 | - -Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03.0 | - -Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03.0 | v20.03.0 -Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03.0 | v20.03.0 -Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03.0 | v20.03.0 -Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03.0 | v20.03.0 -Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03.0 | v20.03.0 -Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03.0 | - -Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09.0 | - -Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09.0 | - -Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06.0 | v20.03.0 -Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12.0 | v19.03.0 -Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06.0 | v17.12.0 -Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05.0 | v17.10.0 -Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05.0 | - -Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13.0 | v17.12.0 -Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13.0 | v20.03.0 -Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13.0 | v17.12.0 -Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13.0 | v17.12.0 -Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13.0 | v17.12.0 -Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13.0 | - -Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13.0 | v17.12.0 -Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13.0 | v17.12.0 -Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12.0 | v20.03.0 -Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12.0 | v17.09.0 -Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11.0 | v17.06.0 -Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11.0 | v17.06.0 -Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10.0 | - -Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10.0 | v1.12.0 -Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10.0 | v1.12.0 -Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10.0 | v1.12.0 -Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9.0 | v1.12.0 -Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9.0 | v1.12.0 -Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8.0 | v1.12.0 -Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8.0 | v1.10.0 -Removed | [Old Command Line Options](#old-command-line-options) | v1.8.0 | v1.10.0 -Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6.0 | v17.09.0 -Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10.0 | v1.13.0 -Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12.0 +Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.03 | - +Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03 | - +Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03 | - +Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03 | - +Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03 | - +Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.03 +Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.03 +Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.03 +Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.03 +Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.03 +Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - +Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - +Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - +Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.03 +Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 +Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 +Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 +Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - +Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 +Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.03 +Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 +Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 +Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 +Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - +Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 +Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 +Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.03 +Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 +Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 +Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 +Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - +Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 +Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 +Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 +Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 +Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 +Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 +Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 +Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 +Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 +Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 +Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 ### Kernel memory limit -**Deprecated in Release: v20.03.0** +**Deprecated in Release: v20.03** Specifying kernel memory limit (`docker run --kernel-memory`) is now marked as deprecated, as [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). ### Classic Swarm and overlay networks using cluster store -**Deprecated in Release: v20.03.0** +**Deprecated in Release: v20.03** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -112,7 +112,7 @@ deprecated, and will be disabled or removed in a future release. ### Support for legacy `~/.dockercfg` configuration files -**Deprecated in Release: v20.03.0** +**Deprecated in Release: v20.03** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -133,13 +133,13 @@ enabled by default, and these configuration options will no longer be functional ### CLI plugins support -**Deprecated in Release: v20.03.0** +**Deprecated in Release: v20.03** CLI Plugin API is now marked as deprecated. ### Dockerfile legacy `ENV name value` syntax -**Deprecated in Release: v20.03.0** +**Deprecated in Release: v20.03** The Dockerfile `ENV` instruction allows values to be set using either `ENV name=value` or `ENV name value`. The latter (`ENV name value`) form can be ambiguous, for example, @@ -163,9 +163,9 @@ ENV ONE="" TWO="" THREE="world" ### Pushing and pulling with image manifest v2 schema 1 -**Deprecated in Release: v19.03.0** +**Deprecated in Release: v19.03** -**Target For Removal In Release: v20.03.0** +**Target For Removal In Release: v20.03** The image manifest [v2 schema 1](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-1.md) @@ -177,9 +177,9 @@ If the registry you are using still supports v2 schema 1, urge their administrat ### `docker engine` subcommands -**Deprecated in Release: v19.03.0** +**Deprecated in Release: v19.03** -**Removed in Release: v20.03.0** +**Removed in Release: v20.03** The `docker engine activate`, `docker engine check`, and `docker engine update` provided an alternative installation method to upgrade Docker Community engines @@ -193,9 +193,9 @@ standard package managers. ### Top-level `docker deploy` subcommand (experimental) -**Deprecated in Release: v19.03.0** +**Deprecated in Release: v19.03** -**Removed in Release: v20.03.0** +**Removed in Release: v20.03** The top-level `docker deploy` command (using the "Docker Application Bundle" (.dab) file format was introduced as an experimental feature in Docker 1.13 / @@ -205,9 +205,9 @@ subcommand. ### `docker stack deploy` using "dab" files (experimental) -**Deprecated in Release: v19.03.0** +**Deprecated in Release: v19.03** -**Removed in Release: v20.03.0** +**Removed in Release: v20.03** With no development being done on this feature, and no active use of the file format, support for the DAB file format and the top-level docker deploy command @@ -217,7 +217,7 @@ using compose files. ### AuFS storage driver -**Deprecated in Release: v19.03.0** +**Deprecated in Release: v19.03** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -236,7 +236,7 @@ maintenance of the `aufs` storage driver. ### Legacy "overlay" storage driver -**Deprecated in Release: v18.09.0** +**Deprecated in Release: v18.09** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -251,7 +251,7 @@ backported), there is no reason to keep maintaining the `overlay` storage driver ### Device mapper storage driver -**Deprecated in Release: v18.09.0** +**Deprecated in Release: v18.09** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are @@ -268,7 +268,7 @@ is no reason to continue maintenance of the `devicemapper` storage driver. ### Use of reserved namespaces in engine labels -**Deprecated in Release: v18.06.0** +**Deprecated in Release: v18.06** **Removed In Release: v20.03** @@ -276,7 +276,7 @@ The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engin were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their -use, and will error instead in v20.03.0 and above. +use, and will error instead in v20.03 and above. ### `--disable-legacy-registry` override daemon option @@ -312,11 +312,11 @@ start when set. ### Asynchronous `service create` and `service update` as default -**Deprecated In Release: v17.05.0** +**Deprecated In Release: v17.05** **Disabled by default in release: [v17.10](https://github.com/docker/docker-ce/releases/tag/v17.10.0-ce)** -Docker 17.05.0 added an optional `--detach=false` option to make the +Docker 17.05 added an optional `--detach=false` option to make the `docker service create` and `docker service update` work synchronously. This option will be enabled by default in Docker 17.10, at which point the `--detach` flag can be used to use the previous (asynchronous) behavior. @@ -326,7 +326,7 @@ and `docker service scale` in Docker 17.10. ### `-g` and `--graph` flags on `dockerd` -**Deprecated In Release: v17.05.0** +**Deprecated In Release: v17.05** The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource From b869863399e648df99f2d2922c9700abe1aa389f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 13 Oct 2020 09:38:48 +0200 Subject: [PATCH 348/534] docs: update 20.03 -> 20.10 in deprecated.md Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 52 +++++++++++++++++++++++----------------------- 1 file changed, 26 insertions(+), 26 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index d960e23094..348e1f6e93 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,33 +50,33 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.03 | - -Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.03 | - -Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.03 | - -Deprecated | [CLI plugins support](#cli-plugins-support) | v20.03 | - -Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.03 | - -Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.03 -Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.03 -Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.03 -Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.03 -Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.03 +Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - +Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - +Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - +Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - +Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - +Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.10 +Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 +Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 +Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 +Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - -Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.03 +Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 -Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.03 +Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 -Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.03 +Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 @@ -95,14 +95,14 @@ Removed | [Three arguments form in `docker import`](#three-arguments-form-in- ### Kernel memory limit -**Deprecated in Release: v20.03** +**Deprecated in Release: v20.10** Specifying kernel memory limit (`docker run --kernel-memory`) is now marked as deprecated, as [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). ### Classic Swarm and overlay networks using cluster store -**Deprecated in Release: v20.03** +**Deprecated in Release: v20.10** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -112,7 +112,7 @@ deprecated, and will be disabled or removed in a future release. ### Support for legacy `~/.dockercfg` configuration files -**Deprecated in Release: v20.03** +**Deprecated in Release: v20.10** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -133,13 +133,13 @@ enabled by default, and these configuration options will no longer be functional ### CLI plugins support -**Deprecated in Release: v20.03** +**Deprecated in Release: v20.10** CLI Plugin API is now marked as deprecated. ### Dockerfile legacy `ENV name value` syntax -**Deprecated in Release: v20.03** +**Deprecated in Release: v20.10** The Dockerfile `ENV` instruction allows values to be set using either `ENV name=value` or `ENV name value`. The latter (`ENV name value`) form can be ambiguous, for example, @@ -165,7 +165,7 @@ ENV ONE="" TWO="" THREE="world" **Deprecated in Release: v19.03** -**Target For Removal In Release: v20.03** +**Target For Removal In Release: v20.10** The image manifest [v2 schema 1](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-1.md) @@ -179,7 +179,7 @@ If the registry you are using still supports v2 schema 1, urge their administrat **Deprecated in Release: v19.03** -**Removed in Release: v20.03** +**Removed in Release: v20.10** The `docker engine activate`, `docker engine check`, and `docker engine update` provided an alternative installation method to upgrade Docker Community engines @@ -195,7 +195,7 @@ standard package managers. **Deprecated in Release: v19.03** -**Removed in Release: v20.03** +**Removed in Release: v20.10** The top-level `docker deploy` command (using the "Docker Application Bundle" (.dab) file format was introduced as an experimental feature in Docker 1.13 / @@ -207,7 +207,7 @@ subcommand. **Deprecated in Release: v19.03** -**Removed in Release: v20.03** +**Removed in Release: v20.10** With no development being done on this feature, and no active use of the file format, support for the DAB file format and the top-level docker deploy command @@ -270,13 +270,13 @@ is no reason to continue maintenance of the `devicemapper` storage driver. **Deprecated in Release: v18.06** -**Removed In Release: v20.03** +**Removed In Release: v20.10** The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engine labels were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their -use, and will error instead in v20.03 and above. +use, and will error instead in v20.10 and above. ### `--disable-legacy-registry` override daemon option @@ -358,7 +358,7 @@ information. ### `filter` param for `/images/json` endpoint **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v20.03** +**Removed In Release: v20.10** The `filter` param to filter the list of image by reference (name or name:tag) is now implemented as a regular filter, named `reference`. @@ -424,7 +424,7 @@ further information. **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Removed In Release: v20.03** +**Removed In Release: v20.10** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=` and `docker search --filter=stars=...` instead. From 7e5de1d1d83c8a715fddc08347bb9a7cf2a7d976 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 16 Oct 2020 10:58:49 +0200 Subject: [PATCH 349/534] Deprecate Linux containers on Windows (LCOW) (experimental) The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after its introduction, the feature never reached completeness, and development has now stopped in favor of running docker natively on Linux in WSL2. Developers that need to run Linux workloads on a Windows host are encouraged to use Docker Desktop with WSL2 instead. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 348e1f6e93..d4f7e37e89 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - @@ -93,6 +94,18 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Linux containers on Windows (LCOW) (experimental) + +**Deprecated in Release: v20.10** + +The experimental feature to run Linux containers on Windows (LCOW) was introduced +as a technical preview in Docker 17.09. While many enhancements were made after +its introduction, the feature never reached completeness, and development has +now stopped in favor of running docker natively on Linux in WSL2. + +Developers who want to run Linux workloads on a Windows host are encouraged to use +[Docker Desktop with WSL2](https://docs.docker.com/docker-for-windows/wsl/) instead. + ### Kernel memory limit **Deprecated in Release: v20.10** From 8af6003656c78a24ea19969773868d7f4e4e4534 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 24 Oct 2020 01:03:47 +0200 Subject: [PATCH 350/534] Deprecation: add experimental docker build --stream option Docker v17.07 introduced an experimental `--stream` flag on `docker build` which allowed the build-context to be incrementally sent to the daemon, instead of unconditionally sending the whole build-context. This functionality has been reimplemented as part of BuildKit, which uses streaming by default and the `--stream` option will be ignored when using the classic builder, printing a deprecation warning instead. Users that want to use this feature are encouraged to enable BuildKit by setting the `DOCKER_BUILDKIT=1` environment variable or through the daemon or CLI configuration files. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index d4f7e37e89..91ae713f65 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -56,6 +56,7 @@ Deprecated | [Classic Swarm and overlay networks using external key/value stores Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - +Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.10 Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 @@ -174,6 +175,23 @@ syntax, for example: ENV ONE="" TWO="" THREE="world" ``` +### `docker build --stream` flag (experimental) + +**Deprecated in Release: v20.10** +**Removed in Release: v20.10** + +Docker v17.07 introduced an experimental `--stream` flag on `docker build` which +allowed the build-context to be incrementally sent to the daemon, instead of +unconditionally sending the whole build-context. + +This functionality has been reimplemented as part of BuildKit, which uses streaming +by default and the `--stream` option will be ignored when using the classic builder, +printing a deprecation warning instead. + +Users that want to use this feature are encouraged to enable BuildKit by setting +the `DOCKER_BUILDKIT=1` environment variable or through the daemon or CLI configuration +files. + ### Pushing and pulling with image manifest v2 schema 1 **Deprecated in Release: v19.03** From 9da676d555455c2ecf4efad240abc5d3de177119 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 26 Oct 2020 18:30:01 +0100 Subject: [PATCH 351/534] docs: remove some references to obsolete docker versions Signed-off-by: Sebastiaan van Stijn --- docs/extend/index.md | 4 ++-- docs/reference/commandline/dockerd.md | 20 ++++++++------------ 2 files changed, 10 insertions(+), 14 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index bd0e9e1204..ecdf00b8f8 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -21,8 +21,8 @@ keywords: "API, Usage, plugins, documentation, developer" Docker Engine's plugin system allows you to install, start, stop, and remove plugins using Docker Engine. -For information about the legacy plugin system available in Docker Engine 1.12 -and earlier, see [Understand legacy Docker Engine plugins](legacy_plugins.md). +For information about legacy (non-managed) plugins, refer to +[Understand legacy Docker Engine plugins](legacy_plugins.md). > **Note** > diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index eaa94ded46..d0706d4966 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -205,8 +205,7 @@ The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes precedence over `HTTP_PROXY`. -Starting with Docker 18.09, the Docker client supports connecting to a remote -daemon via SSH: +The Docker client supports connecting to a remote daemon via SSH: ``` $ docker -H ssh://me@example.com:22 ps @@ -219,17 +218,17 @@ remote host with public key authentication. Password authentication is not supported. If your key is protected with passphrase, you need to set up `ssh-agent`. -Also, you need to have `docker` binary 18.09 or later on the daemon host. - #### Bind Docker to another host/port or a Unix socket -> **Warning**: +> **Warning** +> > Changing the default `docker` daemon binding to a > TCP port or Unix *docker* user group will increase your security risks > by allowing non-root users to gain *root* access on the host. Make sure > you control access to `docker`. If you are binding > to a TCP port, anyone with access to that port has full Docker access; > so it is not advisable on an open network. +{: .warning :} With `-H` it is possible to make the Docker daemon to listen on a specific IP and port. By default, it will listen on @@ -1110,13 +1109,10 @@ system's list of trusted CAs instead of enabling `--insecure-registry`. #### Legacy Registries -Starting with Docker 17.12, operations against registries supporting only the -legacy v1 protocol are no longer supported. Specifically, the daemon will not -attempt `push`, `pull` and `login` to v1 registries. The exception to this is -`search` which can still be performed on v1 registries. - -The `disable-legacy-registry` configuration option has been removed and, when -used, will produce an error on daemon startup. +Operations against registries supporting only the legacy v1 protocol are no longer +supported. Specifically, the daemon will not attempt `push`, `pull` and `login` +to v1 registries. The exception to this is `search` which can still be performed +on v1 registries. ### Running a Docker daemon behind an HTTPS_PROXY From 0a20e1c48af6506268270006dedf294dc9f0e11c Mon Sep 17 00:00:00 2001 From: Tianon Gravi Date: Fri, 6 Nov 2020 11:52:06 -0800 Subject: [PATCH 352/534] Use consistent formatting and sorted keys in all JSON examples for dockerd reference To create this, I ran every JSON document through `jq -S` (which sorts the keys and consistently pretty-prints the result in a format which matches the majority of documents in this file). Signed-off-by: Tianon Gravi --- docs/reference/commandline/dockerd.md | 198 ++++++++++++++------------ 1 file changed, 105 insertions(+), 93 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index d0706d4966..f1b2c104dc 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -967,18 +967,18 @@ The following is an example adding 2 runtimes via the configuration: ```json { - "default-runtime": "runc", - "runtimes": { - "runc": { - "path": "runc" - }, - "custom": { - "path": "/usr/local/bin/my-runc-replacement", - "runtimeArgs": [ - "--debug" - ] - } - } + "default-runtime": "runc", + "runtimes": { + "custom": { + "path": "/usr/local/bin/my-runc-replacement", + "runtimeArgs": [ + "--debug" + ] + }, + "runc": { + "path": "runc" + } + } } ``` @@ -1303,7 +1303,10 @@ the task to run. Example of usage: ```json { - "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"] + "node-generic-resources": [ + "NVIDIA-GPU=UUID1", + "NVIDIA-GPU=UUID2" + ] } ``` @@ -1332,9 +1335,39 @@ This is a full example of the allowed configuration options on Linux: ```json { + "allow-nondistributable-artifacts": [], + "api-cors-header": "", "authorization-plugins": [], + "bip": "", + "bridge": "", + "cgroup-parent": "", + "cluster-advertise": "", + "cluster-store": "", + "cluster-store-opts": {}, "data-root": "", + "debug": true, + "default-address-pools": [ + { + "base": "172.80.0.0/16", + "size": 24 + }, + { + "base": "172.90.0.0/16", + "size": 24 + } + ], "default-cgroupns-mode": "private", + "default-gateway": "", + "default-gateway-v6": "", + "default-runtime": "runc", + "default-shm-size": "64M", + "default-ulimits": { + "nofile": { + "Hard": 64000, + "Name": "nofile", + "Soft": 64000 + } + }, "dns": [], "dns-opts": [], "dns-search": [], @@ -1342,73 +1375,42 @@ This is a full example of the allowed configuration options on Linux: "exec-root": "", "experimental": false, "features": {}, - "storage-driver": "", - "storage-opts": [], + "fixed-cidr": "", + "fixed-cidr-v6": "", + "group": "", + "hosts": [], + "icc": false, + "init": false, + "init-path": "/usr/libexec/docker-init", + "insecure-registries": [], + "ip": "0.0.0.0", + "ip-forward": false, + "ip-masq": false, + "iptables": false, + "ipv6": false, "labels": [], "live-restore": true, "log-driver": "json-file", + "log-level": "", "log-opts": { - "max-size": "10m", - "max-file":"5", + "env": "os,customer", "labels": "somelabel", - "env": "os,customer" + "max-file": "5", + "max-size": "10m" }, - "mtu": 0, - "pidfile": "", - "cluster-store": "", - "cluster-store-opts": {}, - "cluster-advertise": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, "max-download-attempts": 5, - "default-shm-size": "64M", - "shutdown-timeout": 15, - "debug": true, - "hosts": [], - "log-level": "", - "tls": true, - "tlsverify": true, - "tlscacert": "", - "tlscert": "", - "tlskey": "", - "swarm-default-advertise-addr": "", - "api-cors-header": "", - "selinux-enabled": false, - "userns-remap": "", - "group": "", - "cgroup-parent": "", - "default-ulimits": { - "nofile": { - "Name": "nofile", - "Hard": 64000, - "Soft": 64000 - } - }, - "init": false, - "init-path": "/usr/libexec/docker-init", - "ipv6": false, - "iptables": false, - "ip-forward": false, - "ip-masq": false, - "userland-proxy": false, - "userland-proxy-path": "/usr/libexec/docker-proxy", - "ip": "0.0.0.0", - "bridge": "", - "bip": "", - "fixed-cidr": "", - "fixed-cidr-v6": "", - "default-gateway": "", - "default-gateway-v6": "", - "icc": false, - "raw-logs": false, - "allow-nondistributable-artifacts": [], - "registry-mirrors": [], - "seccomp-profile": "", - "insecure-registries": [], + "mtu": 0, "no-new-privileges": false, - "default-runtime": "runc", + "node-generic-resources": [ + "NVIDIA-GPU=UUID1", + "NVIDIA-GPU=UUID2" + ], "oom-score-adjust": -500, - "node-generic-resources": ["NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2"], + "pidfile": "", + "raw-logs": false, + "registry-mirrors": [], "runtimes": { "cc-runtime": { "path": "/usr/bin/cc-runtime" @@ -1420,10 +1422,20 @@ This is a full example of the allowed configuration options on Linux: ] } }, - "default-address-pools":[ - {"base":"172.80.0.0/16","size":24}, - {"base":"172.90.0.0/16","size":24} - ] + "seccomp-profile": "", + "selinux-enabled": false, + "shutdown-timeout": 15, + "storage-driver": "", + "storage-opts": [], + "swarm-default-advertise-addr": "", + "tls": true, + "tlscacert": "", + "tlscert": "", + "tlskey": "", + "tlsverify": true, + "userland-proxy": false, + "userland-proxy-path": "/usr/libexec/docker-proxy", + "userns-remap": "" } ``` @@ -1446,42 +1458,42 @@ This is a full example of the allowed configuration options on Windows: ```json { + "allow-nondistributable-artifacts": [], "authorization-plugins": [], + "bridge": "", + "cluster-advertise": "", + "cluster-store": "", "data-root": "", + "debug": true, + "default-ulimits": {}, "dns": [], "dns-opts": [], "dns-search": [], "exec-opts": [], "experimental": false, - "features":{}, - "storage-driver": "", - "storage-opts": [], + "features": {}, + "fixed-cidr": "", + "group": "", + "hosts": [], + "insecure-registries": [], "labels": [], "log-driver": "", - "mtu": 0, - "pidfile": "", - "cluster-store": "", - "cluster-advertise": "", + "log-level": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, "max-download-attempts": 5, + "mtu": 0, + "pidfile": "", + "raw-logs": false, + "registry-mirrors": [], "shutdown-timeout": 15, - "debug": true, - "hosts": [], - "log-level": "", - "tlsverify": true, + "storage-driver": "", + "storage-opts": [], + "swarm-default-advertise-addr": "", "tlscacert": "", "tlscert": "", "tlskey": "", - "swarm-default-advertise-addr": "", - "group": "", - "default-ulimits": {}, - "bridge": "", - "fixed-cidr": "", - "raw-logs": false, - "allow-nondistributable-artifacts": [], - "registry-mirrors": [], - "insecure-registries": [] + "tlsverify": true } ``` From 6d43c8b028fe69c2dd5ada7f5a506e1dccd3e8cf Mon Sep 17 00:00:00 2001 From: Tianon Gravi Date: Fri, 6 Nov 2020 12:34:10 -0800 Subject: [PATCH 353/534] Add "--containerd-namespace" daemon flags in completion and docs This also adds the missing `containerd` key in the example daemon configuration files. Signed-off-by: Tianon Gravi --- docs/reference/commandline/dockerd.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index f1b2c104dc..ae586c7437 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -32,6 +32,8 @@ Options: --cgroup-parent string Set parent cgroup for all containers --config-file string Daemon configuration file (default "/etc/docker/daemon.json") --containerd string containerd grpc address + --containerd-namespace string Containerd namespace to use (default "moby") + --containerd-plugins-namespace string Containerd namespace to use for plugins (default "plugins.moby") --cpu-rt-period int Limit the CPU real-time period in microseconds for the parent cgroup for all containers --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds for the @@ -1344,6 +1346,9 @@ This is a full example of the allowed configuration options on Linux: "cluster-advertise": "", "cluster-store": "", "cluster-store-opts": {}, + "containerd": "/run/containerd/containerd.sock", + "containerd-namespace": "docker", + "containerd-plugin-namespace": "docker-plugins", "data-root": "", "debug": true, "default-address-pools": [ @@ -1463,6 +1468,9 @@ This is a full example of the allowed configuration options on Windows: "bridge": "", "cluster-advertise": "", "cluster-store": "", + "containerd": "\\\\.\\pipe\\containerd-containerd", + "containerd-namespace": "docker", + "containerd-plugin-namespace": "docker-plugins", "data-root": "", "debug": true, "default-ulimits": {}, From 909530f98fbe766f225a5d6020cbcf711b53866b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Benjamin=20B=C3=B6hmke?= Date: Wed, 18 Nov 2020 21:59:16 +0100 Subject: [PATCH 354/534] added ip6tables to daemon CLI and config file doc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Benjamin Böhmke --- docs/reference/commandline/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ae586c7437..a482577fb7 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -70,6 +70,7 @@ Options: --ip-forward Enable net.ipv4.ip_forward (default true) --ip-masq Enable IP masquerading (default true) --iptables Enable addition of iptables rules (default true) + --ip6tables Enable addition of ip6tables rules (default false) --ipv6 Enable IPv6 networking --label list Set key=value labels to the daemon --live-restore Enable live restore of docker when containers are still running @@ -1392,6 +1393,7 @@ This is a full example of the allowed configuration options on Linux: "ip-forward": false, "ip-masq": false, "iptables": false, + "ip6tables": false, "ipv6": false, "labels": [], "live-restore": true, From 3b481f45315cf29271f3b2b510c7a9c37c38a859 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 7 Dec 2020 13:34:50 +0100 Subject: [PATCH 355/534] Add deprecation note for non-compliant registries Docker Engine v20.10 and up includes optimizations to verify if images in the local image cache need updating before pulling, preventing the Docker Engine from making unnecessary API requests. These optimizations require the container image registry to conform to the Open Container Initiative Distribution Specification (https://github.com/opencontainers/distribution-spec). While most registries conform to the specification, we encountered some registries to be non-compliant, resulting in `docker pull` to fail. As a temporary solution, Docker Engine v20.10 includes a fallback mechanism to allow `docker pull` to be functional when using a non-compliant registry. A warning message is printed in this situation: WARNING Failed to pull manifest by the resolved digest. This registry does not appear to conform to the distribution registry specification; falling back to pull by tag. This fallback is DEPRECATED, and will be removed in a future release. The fallback is added to allow users to either migrate their images to a compliant registry, or for these registries to become compliant. Note that this fallback only addresses failures on `docker pull`. Other commands, such as `docker stack deploy`, or pulling images with `containerd` will continue to fail. Given that other functionality is still broken with these registries, we consider this fallback a _temporary_ solution, and will remove the fallback in an upcoming major release. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 91ae713f65..b21091edc3 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -95,6 +95,38 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Pulling images from non-compliant image registries + +**Deprecated in Release: v20.10** + +Docker Engine v20.10 and up includes optimizations to verify if images in the +local image cache need updating before pulling, preventing the Docker Engine +from making unnecessary API requests. These optimizations require the container +image registry to conform to the [Open Container Initiative Distribution Specification](https://github.com/opencontainers/distribution-spec). + +While most registries conform to the specification, we encountered some registries +to be non-compliant, resulting in `docker pull` to fail. + +As a temporary solution, Docker Engine v20.10 includes a fallback mechanism to +allow `docker pull` to be functional when using a non-compliant registry. A +warning message is printed in this situation: + + WARNING Failed to pull manifest by the resolved digest. This registry does not + appear to conform to the distribution registry specification; falling back to + pull by tag. This fallback is DEPRECATED, and will be removed in a future + release. + +The fallback is added to allow users to either migrate their images to a compliant +registry, or for these registries to become compliant. + +Note that this fallback only addresses failures on `docker pull`. Other commands, +such as `docker stack deploy`, or pulling images with `containerd` will continue +to fail. + +Given that other functionality is still broken with these registries, we consider +this fallback a _temporary_ solution, and will remove the fallback in an upcoming +major release. + ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** From c14597262446453453b00008a06fed326d7e243d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 9 Dec 2020 02:36:30 +0100 Subject: [PATCH 356/534] Deprecation: add pulling from non-compliant registries to table Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b21091edc3..267f8eef87 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - From c1f595174461af6ed26ef0d95081c0d5197626cb Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 4 Jan 2021 14:36:25 +0100 Subject: [PATCH 357/534] deprecate blkio-weight options with cgroups v1 These options were deprecated and removed in the Linux kernel v5.0 and up in; - https://github.com/torvalds/linux/commit/f382fb0bcef4c37dc049e9f6963e3baf204d815c - https://github.com/torvalds/linux/commit/fb5772cbfe48575711bf789767d561582376f7f1 - https://github.com/torvalds/linux/commit/23aa16489c06e6739c7c99e9fdccf723d2691a5d Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 267f8eef87..9af6c5bb54 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,6 +52,7 @@ Status | Feature -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - +Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options–with-cgroups-v1) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - @@ -140,6 +141,16 @@ now stopped in favor of running docker natively on Linux in WSL2. Developers who want to run Linux workloads on a Windows host are encouraged to use [Docker Desktop with WSL2](https://docs.docker.com/docker-for-windows/wsl/) instead. +### BLKIO weight options with cgroups v1 + +**Deprecated in Release: v20.10** + +Specifying blkio weight (`docker run --blkio-weight` and `docker run --blkio-weight-device`) +is now marked as deprecated when using cgrous v1 because the corresponding features +were [removed in Linux kernel v5.0 and up](https://github.com/torvalds/linux/commit/f382fb0bcef4c37dc049e9f6963e3baf204d815c). +When using cgroups v2, the `--blkio-weight` options are implemented using +[`io.weight](https://github.com/torvalds/linux/blob/v5.0/Documentation/admin-guide/cgroup-v2.rst#io). + ### Kernel memory limit **Deprecated in Release: v20.10** From 606f148a3ebb4bd5f9a13040d48cdc25b89e607c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 8 Jan 2021 16:03:51 +0100 Subject: [PATCH 358/534] docs: fix typo in deprecated.md Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9af6c5bb54..4cff064feb 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -146,7 +146,7 @@ Developers who want to run Linux workloads on a Windows host are encouraged to u **Deprecated in Release: v20.10** Specifying blkio weight (`docker run --blkio-weight` and `docker run --blkio-weight-device`) -is now marked as deprecated when using cgrous v1 because the corresponding features +is now marked as deprecated when using cgroups v1 because the corresponding features were [removed in Linux kernel v5.0 and up](https://github.com/torvalds/linux/commit/f382fb0bcef4c37dc049e9f6963e3baf204d815c). When using cgroups v2, the `--blkio-weight` options are implemented using [`io.weight](https://github.com/torvalds/linux/blob/v5.0/Documentation/admin-guide/cgroup-v2.rst#io). From 166b160c1496eb49ad7794e25d8082d3a02fcf5b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 17 Apr 2021 15:13:20 +0200 Subject: [PATCH 359/534] docs: document log-opts for "dual logging" cache These options are available in Docker 20.10 and up, but were previously only available in Docker EE, and not documented. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a482577fb7..0ac52c826a 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1400,6 +1400,10 @@ This is a full example of the allowed configuration options on Linux: "log-driver": "json-file", "log-level": "", "log-opts": { + "cache-disabled": "false", + "cache-max-file": "5", + "cache-max-size": "20m", + "cache-compress": "true", "env": "os,customer", "labels": "somelabel", "max-file": "5", From 4fb3c19175344e6f31e5a3d41f5bcada4b757444 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 May 2021 09:31:50 +0200 Subject: [PATCH 360/534] docs: dockerd: fix broken link and markdown touch-ups Jekyll doesn't work well with markdown links that are wrapped, so changing the link to be on a single line. While at it, also added/changed some code-hints. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 0ac52c826a..e3414168e0 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1203,8 +1203,8 @@ command line or Docker's Engine API are allowed or denied by the plugin. If you have multiple plugins installed, each plugin, in order, must allow the request for it to complete. -For information about how to create an authorization plugin, see [authorization -plugin](../../extend/plugins_authorization.md) section in the Docker extend section of this documentation. +For information about how to create an authorization plugin, refer to the +[authorization plugin](../../extend/plugins_authorization.md) section. ### Daemon user namespace options @@ -1232,10 +1232,16 @@ Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be set like this: - DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 - # or - export DOCKER_TMPDIR=/mnt/disk2/tmp - /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +```console +$ DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +``` + +or + +```console +$ export DOCKER_TMPDIR=/mnt/disk2/tmp +$ /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +```` #### Default cgroup parent @@ -1564,7 +1570,9 @@ previously configured cluster configurations. ### Run multiple daemons -> **Note:** Running multiple daemons on a single host is considered as "experimental". The user should be aware of +> **Note:** +> +> Running multiple daemons on a single host is considered as "experimental". The user should be aware of > unsolved problems. This solution may not work properly in some cases. Solutions are currently under development > and will be delivered in the near future. @@ -1616,7 +1624,7 @@ The `--tls*` options enable use of specific certificates for individual daemons. Example script for a separate “bootstrap” instance of the Docker daemon without network: -```bash +```console $ sudo dockerd \ -H unix:///var/run/docker-bootstrap.sock \ -p /var/run/docker-bootstrap.pid \ From 3e4839149e6483782796e6f67c720eb0ca5337b0 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 26 May 2021 14:14:02 +0200 Subject: [PATCH 361/534] docs: fix link to command-line reference This link worked on GitHub, but was broken on docs.docker.com, so replacing with a regular link directly to the docs instead. Signed-off-by: Sebastiaan van Stijn --- docs/extend/index.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index ecdf00b8f8..dd885d535f 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -122,7 +122,7 @@ enabled, and use it to create a volume. To disable a plugin, use the `docker plugin disable` command. To completely remove it, use the `docker plugin remove` command. For other available commands and options, see the -[command line reference](../reference/commandline/index.md). +[command line reference](https://docs.docker.com/engine/reference/commandline/cli/). ## Developing a plugin From b9d23617383df08f93184b014f2d570964aec7fc Mon Sep 17 00:00:00 2001 From: Anca Iordache Date: Tue, 25 May 2021 16:26:41 +0200 Subject: [PATCH 362/534] Document `--validate` daemon option Signed-off-by: Anca Iordache --- docs/reference/commandline/dockerd.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e3414168e0..0b984fe56b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -104,6 +104,7 @@ Options: --userland-proxy Use userland proxy for loopback traffic (default true) --userland-proxy-path string Path to the userland proxy binary --userns-remap string User/Group setting for user namespaces + --validate Validate daemon configuration and exit -v, --version Print version information and quit ``` @@ -1334,6 +1335,25 @@ For example, the daemon fails to start if you set daemon labels in the configuration file and also set daemon labels via the `--label` flag. Options that are not present in the file are ignored when the daemon starts. +The `--validate` option allows to validate a configuration file without +starting the Docker daemon. A non-zero exit code is returned for invalid +configuration files. + +```console +$ dockerd --validate --config-file=/tmp/valid-config.json +configuration OK + +$ echo $? +0 + +$ dockerd --validate --config-file /tmp/invalid-config.json +unable to configure the Docker daemon with file /tmp/invalid-config.json: the following directives don't match any configuration option: unknown-option + +$ echo $? +1 +``` + + ##### On Linux The default location of the configuration file on Linux is From 6a94727d360b402520d5426b04a4d2c15627cdf4 Mon Sep 17 00:00:00 2001 From: Mathieu Champlon Date: Mon, 28 Jun 2021 14:57:15 +0200 Subject: [PATCH 363/534] Deprecate Kubernetes stack support Signed-off-by: Mathieu Champlon --- docs/deprecated.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 4cff064feb..97d36e7751 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Kubernetes stack support](#kubernetes-stack-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options–with-cgroups-v1) | v20.10 | - @@ -97,6 +98,13 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Kubernetes stack support + +**Deprecated in Release: v20.10** + +Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for +Kubernetes in the `stack` command in the docker CLI is now marked as deprecated as well. + ### Pulling images from non-compliant image registries **Deprecated in Release: v20.10** From 3f1d2cacf61870561ea145264055f5b34c84479e Mon Sep 17 00:00:00 2001 From: Mathieu Champlon Date: Thu, 1 Jul 2021 08:59:10 +0200 Subject: [PATCH 364/534] Deprecate Kubernetes context support Signed-off-by: Mathieu Champlon --- docs/deprecated.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 97d36e7751..5d18481ad0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,7 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Kubernetes stack support](#kubernetes-stack-support) | v20.10 | - +Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options–with-cgroups-v1) | v20.10 | - @@ -98,12 +98,12 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 -### Kubernetes stack support +### Kubernetes stack and context support **Deprecated in Release: v20.10** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for -Kubernetes in the `stack` command in the docker CLI is now marked as deprecated as well. +Kubernetes in the `stack` and `context` commands in the docker CLI is now marked as deprecated as well. ### Pulling images from non-compliant image registries From f31c962fe410590aab6b46a3142a12457d014132 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 28 Jul 2021 12:45:09 +0200 Subject: [PATCH 365/534] context: deprecate support for encrypted TLS private keys > Legacy PEM encryption as specified in RFC 1423 is insecure by design. Since > it does not authenticate the ciphertext, it is vulnerable to padding oracle > attacks that can let an attacker recover the plaintext From https://go-review.googlesource.com/c/go/+/264159 > It's unfortunate that we don't implement PKCS#8 encryption so we can't > recommend an alternative but PEM encryption is so broken that it's worth > deprecating outright. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 5d18481ad0..34cd3ec195 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | - Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - @@ -98,6 +99,15 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Support for encrypted TLS private keys + +**Deprecated in Release: v20.10** + +Use of encrypted TLS private keys has been deprecated, and will be removed in a +future release. Golang has deprecated support for legacy PEM encryption (as +specified in [RFC 1423](https://datatracker.ietf.org/doc/html/rfc1423)), as it +is insecure by design (see [https://go-review.googlesource.com/c/go/+/264159](https://go-review.googlesource.com/c/go/+/264159)). + ### Kubernetes stack and context support **Deprecated in Release: v20.10** From 5788ae275ab5cfce5c37d55dd22239944ed8f985 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 29 Jul 2021 09:21:42 +0200 Subject: [PATCH 366/534] docs: update deprecation status of LCOW Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 34cd3ec195..cca68493bc 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,7 +53,7 @@ Status | Feature Deprecated | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | - Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - -Deprecated | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | - +Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v21.xx Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options–with-cgroups-v1) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - @@ -150,6 +150,7 @@ major release. ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** +**Removed in Release: v21.xx** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after From 6fde812e5819ebd269446509a4c9f49ca0a95e22 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 21 Aug 2021 14:54:14 +0200 Subject: [PATCH 367/534] docs: use "console" code-hint for shell examples This replaces the use of bash where suitable, to allow easier copy/pasting of shell examples without copying the prompt or process output. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 +- docs/extend/index.md | 40 ++++++----- docs/extend/plugin_api.md | 8 ++- docs/extend/plugins_authorization.md | 17 ++--- docs/extend/plugins_graphdriver.md | 4 +- docs/extend/plugins_logging.md | 1 + docs/extend/plugins_network.md | 4 +- docs/extend/plugins_services.md | 39 +++++------ docs/extend/plugins_volume.md | 2 +- docs/reference/commandline/dockerd.md | 96 +++++++++++++-------------- 10 files changed, 113 insertions(+), 102 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index cca68493bc..e4975a967d 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -589,9 +589,9 @@ Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and `fluentd-tag` have been deprecated in favor of the generic `tag` option. -```bash +```console {% raw %} -docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" +$ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" {% endraw %} ``` diff --git a/docs/extend/index.md b/docs/extend/index.md index dd885d535f..2608d0590b 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -55,7 +55,7 @@ enabled, and use it to create a volume. 1. Install the `sshfs` plugin. - ```bash + ```console $ docker plugin install vieux/sshfs Plugin "vieux/sshfs" is requesting the following privileges: @@ -74,7 +74,7 @@ enabled, and use it to create a volume. 2. Check that the plugin is enabled in the output of `docker plugin ls`. - ```bash + ```console $ docker plugin ls ID NAME TAG DESCRIPTION ENABLED @@ -87,7 +87,7 @@ enabled, and use it to create a volume. This volume can now be mounted into containers. - ```bash + ```console $ docker volume create \ -d vieux/sshfs \ --name sshvolume \ @@ -96,9 +96,10 @@ enabled, and use it to create a volume. sshvolume ``` + 4. Verify that the volume was created successfully. - ```bash + ```console $ docker volume ls DRIVER NAME @@ -107,18 +108,19 @@ enabled, and use it to create a volume. 5. Start a container that uses the volume `sshvolume`. - ```bash + ```console $ docker run --rm -v sshvolume:/data busybox ls /data ``` 6. Remove the volume `sshvolume` - ```bash - docker volume rm sshvolume + ```console + $ docker volume rm sshvolume sshvolume ``` + To disable a plugin, use the `docker plugin disable` command. To completely remove it, use the `docker plugin remove` command. For other available commands and options, see the @@ -134,7 +136,7 @@ example, it was created from a Dockerfile: >**Note:** The `/run/docker/plugins` directory is mandatory inside of the plugin's filesystem for docker to communicate with the plugin. -```bash +```console $ git clone https://github.com/vieux/docker-volume-sshfs $ cd docker-volume-sshfs $ docker build -t rootfsimage . @@ -193,13 +195,13 @@ Stdout of a plugin is redirected to dockerd logs. Such entries have a `f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62` and their corresponding log entries in the docker daemon logs. -```bash +```console $ docker plugin install tiborvass/sample-volume-plugin INFO[0036] Starting... Found 0 volumes on startup plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 ``` -```bash +```console $ docker volume create -d tiborvass/sample-volume-plugin samplevol INFO[0193] Create Called... Ensuring directory /data/samplevol exists on host... plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 @@ -208,7 +210,7 @@ INFO[0193] Created volume samplevol with mountpoint /data/samp INFO[0193] Path Called... Returned path /data/samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 ``` -```bash +```console $ docker run -v samplevol:/tmp busybox sh INFO[0421] Get Called... Found samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 @@ -223,7 +225,7 @@ INFO[0421] Unmount Called... Unmounted samplevol plugin=f52a3df433b9a plugins. This is specifically useful to collect plugin logs if they are redirected to a file. -```bash +```console $ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins list ID PID STATUS BUNDLE CREATED OWNER @@ -232,13 +234,14 @@ ID PID S c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d 14984 running /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d 2018-02-08T21:35:12.321288966Z root ``` -```bash +```console $ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 cat /var/log/plugin.log ``` If the plugin has a built-in shell, then exec into the plugin can be done as follows: -```bash + +```console $ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec -t 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 sh ``` @@ -251,17 +254,18 @@ the plugin is listening on the said socket. For a well functioning plugin, these basic requests should work. Note that plugin sockets are available on the host under `/var/run/docker/plugins/` -```bash -curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/e8a37ba56fc879c991f7d7921901723c64df6b42b87e6a0b055771ecf8477a6d/plugin.sock http:/VolumeDriver.List +```console +$ curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/e8a37ba56fc879c991f7d7921901723c64df6b42b87e6a0b055771ecf8477a6d/plugin.sock http:/VolumeDriver.List {"Mountpoint":"","Err":"","Volumes":[{"Name":"myvol1","Mountpoint":"/data/myvol1"},{"Name":"myvol2","Mountpoint":"/data/myvol2"}],"Volume":null} ``` -```bash -curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/45e00a7ce6185d6e365904c8bcf62eb724b1fe307e0d4e7ecc9f6c1eb7bcdb70/plugin.sock http:/NetworkDriver.GetCapabilities +```console +$ curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/45e00a7ce6185d6e365904c8bcf62eb724b1fe307e0d4e7ecc9f6c1eb7bcdb70/plugin.sock http:/NetworkDriver.GetCapabilities {"Scope":"local"} ``` + When using curl 7.5 and above, the URL should be of the form `http://hostname/APICall`, where `hostname` is the valid hostname where the plugin is installed and `APICall` is the call to the plugin API. diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 2a317f953a..d1c415af9c 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -114,7 +114,7 @@ a `service` file and a `socket` file. The `service` file (for example `/lib/systemd/system/your-plugin.service`): -``` +```systemd [Unit] Description=Your plugin Before=docker.service @@ -127,9 +127,10 @@ ExecStart=/usr/lib/docker/your-plugin [Install] WantedBy=multi-user.target ``` + The `socket` file (for example `/lib/systemd/system/your-plugin.socket`): -``` +```systemd [Unit] Description=Your plugin @@ -166,7 +167,8 @@ Plugins are activated via the following "handshake" API call. **Request:** empty body **Response:** -``` + +```json { "Implements": ["VolumeDriver"] } diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index d62c795862..4a9d0a2001 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -116,7 +116,7 @@ value. This value can be the plugin’s socket or a path to a specification file Authorization plugins can be loaded without restarting the daemon. Refer to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. -```bash +```console $ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... ``` @@ -124,26 +124,26 @@ Docker's authorization subsystem supports multiple `--authorization-plugin` para ### Calling authorized command (allow) -```bash +```console $ docker pull centos -... +<...> f1b10cd84249: Pull complete -... +<...> ``` ### Calling unauthorized command (deny) -```bash +```console $ docker pull centos -... +<...> docker: Error response from daemon: authorization denied by plugin PLUGIN_NAME: volumes are not allowed. ``` ### Error from plugins -```bash +```console $ docker pull centos -... +<...> docker: Error response from daemon: plugin PLUGIN_NAME failed with error: AuthZPlugin.AuthZReq: Cannot connect to the Docker daemon. Is the docker daemon running on this host?. ``` @@ -180,6 +180,7 @@ should implement the following two methods: "Err": "The error message if things go wrong" } ``` + #### /AuthZPlugin.AuthZRes **Request**: diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index d8d4531d8c..da7ab68ea2 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -31,7 +31,7 @@ You need to install and enable the plugin and then restart the Docker daemon before using the plugin. See the following example for the correct ordering of steps. -``` +```console $ docker plugin install cpuguy83/docker-overlay2-graphdriver-plugin # this command also enables the driver $ pkill dockerd @@ -309,6 +309,7 @@ Get an archive of the changes between the filesystem layers specified by the `ID and `Parent`. `Parent` may be an empty string, in which case there is no parent. **Response**: + ``` {% raw %} {{ TAR STREAM }} @@ -354,6 +355,7 @@ Respond with a non-empty string error if an error occurred. ### /GraphDriver.ApplyDiff **Request**: + ``` {% raw %} {{ TAR STREAM }} diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md index fbfd2197c1..6405dd26cf 100644 --- a/docs/extend/plugins_logging.md +++ b/docs/extend/plugins_logging.md @@ -211,6 +211,7 @@ as they come in once the existing logs have been read. to determine what set of logs to read. **Response**: + ``` {% raw %}{{ log stream }}{% endraw %} ``` diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 93aea4bd0f..1e218bb4d4 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -42,7 +42,7 @@ Once running however, network driver plugins are used just like the built-in network drivers: by being mentioned as a driver in network-oriented Docker commands. For example, -```bash +```console $ docker network create --driver weave mynet ``` @@ -51,7 +51,7 @@ Some network driver plugins are listed in [plugins](legacy_plugins.md) The `mynet` network is now owned by `weave`, so subsequent commands referring to that network will be sent to the plugin, -```bash +```console $ docker run --network=mynet busybox top ``` diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md index cd94d11cb5..20889dd976 100644 --- a/docs/extend/plugins_services.md +++ b/docs/extend/plugins_services.md @@ -29,20 +29,20 @@ node1 is the manager and node2 is the worker. 1. Prepare manager. In node 1: - ```bash + ```console $ docker swarm init Swarm initialized: current node (dxn1zf6l61qsb1josjja83ngz) is now a manager. ``` 2. Join swarm, install plugin and create volume on worker. In node 2: - ```bash + ```console $ docker swarm join \ - --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ - 192.168.99.100:2377 + --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ + 192.168.99.100:2377 ``` - ```bash + ```console $ docker plugin install tiborvass/sample-volume-plugin latest: Pulling from tiborvass/sample-volume-plugin eb9c16fbdc53: Download complete @@ -51,23 +51,24 @@ node1 is the manager and node2 is the worker. Installed plugin tiborvass/sample-volume-plugin ``` - ```bash + ```console $ docker volume create -d tiborvass/sample-volume-plugin --name pluginVol ``` 3. Create a service using the plugin and volume. In node1: - ```bash + ```console $ docker service create --name my-service --mount type=volume,volume-driver=tiborvass/sample-volume-plugin,source=pluginVol,destination=/tmp busybox top $ docker service ls z1sj8bb8jnfn my-service replicated 1/1 busybox:latest ``` - docker service ls shows service 1 instance of service running. + + `docker service ls` shows service 1 instance of service running. 4. Observe the task getting scheduled in node 2: - ```bash + ```console {% raw %} $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" @@ -87,7 +88,7 @@ Note that node1 is the manager and node2 is the worker. 1. Install a global scoped network plugin on both manager and worker. On node1 and node2: - ```bash + ```console $ docker plugin install bboreham/weave2 Plugin "bboreham/weave2" is requesting the following privileges: - network: [host] @@ -102,7 +103,7 @@ Note that node1 is the manager and node2 is the worker. 2. Create a network using plugin on manager. On node1: - ```bash + ```console $ docker network create --driver=bboreham/weave2:latest globalnet $ docker network ls @@ -115,12 +116,12 @@ containers get scheduled on both manager and worker. On node 1: - ```bash + ```console $ docker service create --network globalnet --name myservice --replicas=8 mrjana/simpleweb simpleweb w90drnfzw85nygbie9kb89vpa ``` - ```bash + ```console $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 87520965206a mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 5 seconds ago Up 4 seconds myservice.4.ytdzpktmwor82zjxkh118uf1v @@ -131,7 +132,7 @@ w90drnfzw85nygbie9kb89vpa On node 2: - ```bash + ```console $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 53c0ae7c1dae mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 2 seconds ago Up Less than a second myservice.7.x44tvvdm3iwkt9kif35f7ykz1 @@ -142,14 +143,14 @@ w90drnfzw85nygbie9kb89vpa 4. Scale down the number of instances. On node1: - ```bash + ```console $ docker service scale myservice=0 myservice scaled to 0 ``` 5. Disable and uninstall the plugin on the worker. On node2: - ```bash + ```console $ docker plugin rm -f bboreham/weave2 bboreham/weave2 ``` @@ -159,12 +160,12 @@ scheduled on the master and not on the worker, because the plugin is not availab On node 1: - ```bash + ```console $ docker service scale myservice=8 myservice scaled to 8 ``` - ```bash + ```console $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES cf4b0ec2415e mrjana/simpleweb@sha256:317d7f221d68c86d503119b0ea12c29de42af0a22ca087d522646ad1069a47a4 "simpleweb" 39 seconds ago Up 36 seconds myservice.3.r7p5o208jmlzpcbm2ytl3q6n1 @@ -179,7 +180,7 @@ scheduled on the master and not on the worker, because the plugin is not availab On node 2: - ```bash + ```console $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES ``` diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 74abfd1bff..23e7cddf9e 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -54,7 +54,7 @@ flags on the `docker container run` command. The `--volume` (or `-v`) flag accepts a volume name and path on the host, and the `--volume-driver` flag accepts a driver type. -```bash +```console $ docker volume create --driver=flocker volumename $ docker container run -it --volume volumename:/data busybox sh diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 0b984fe56b..a870c3079d 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -175,20 +175,21 @@ find examples of using Systemd socket activation with Docker and Systemd in the You can configure the Docker daemon to listen to multiple sockets at the same time using multiple `-H` options: -```bash -# listen using the default unix socket, and on 2 specific IP addresses on this host. +The example below runs the daemon listenin on the default unix socket, and +on 2 specific IP addresses on this host: +```console $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 ``` The Docker client will honor the `DOCKER_HOST` environment variable to set the `-H` flag for the client. Use **one** of the following commands: -```bash +```console $ docker -H tcp://0.0.0.0:2375 ps ``` -```bash +```console $ export DOCKER_HOST="tcp://0.0.0.0:2375" $ docker ps @@ -198,7 +199,7 @@ Setting the `DOCKER_TLS_VERIFY` environment variable to any value other than the empty string is equivalent to setting the `--tlsverify` flag. The following are equivalent: -```bash +```console $ docker --tlsverify ps # or $ export DOCKER_TLS_VERIFY=1 @@ -211,7 +212,7 @@ precedence over `HTTP_PROXY`. The Docker client supports connecting to a remote daemon via SSH: -``` +```console $ docker -H ssh://me@example.com:22 ps $ docker -H ssh://me@example.com ps $ docker -H ssh://example.com ps @@ -268,22 +269,21 @@ when no `-H` was passed in. Run Docker in daemon mode: -```bash +```console $ sudo /dockerd -H 0.0.0.0:5555 & ``` Download an `ubuntu` image: -```bash +```console $ docker -H :5555 pull ubuntu ``` You can use multiple `-H`, for example, if you want to listen on both TCP and a Unix socket -```bash -# Run docker in daemon mode -$ sudo /dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & +```console +$ sudo dockerd -H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock & # Download an ubuntu image, use default Unix socket $ docker pull ubuntu # OR use the TCP port @@ -396,7 +396,7 @@ not use loopback in production. Ensure your Engine daemon has a ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool ``` @@ -407,7 +407,7 @@ device for you. ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf ``` @@ -417,7 +417,7 @@ Sets the percentage of passed in block device to use for storage. ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.thinp_percent=95 ``` @@ -427,7 +427,7 @@ Sets the percentage of the passed in block device to use for metadata storage. ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.thinp_metapercent=1 ``` @@ -438,7 +438,7 @@ autoextend the available space [100 = disabled] ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 ``` @@ -449,7 +449,7 @@ attempts to autoextend the available space [100 = disabled] ###### Example: -```bash +```console $ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20 ``` @@ -468,7 +468,7 @@ new base device size. ###### Examples -```bash +```console $ sudo dockerd --storage-opt dm.basesize=50G ``` @@ -480,7 +480,7 @@ This value affects the system-wide "base" empty filesystem that may already be initialized and inherited by pulled images. Typically, a change to this value requires additional steps to take effect: - ```bash +```console $ sudo service docker stop $ sudo rm -rf /var/lib/docker @@ -503,7 +503,7 @@ much space. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.loopdatasize=200G ``` @@ -521,7 +521,7 @@ this much space. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.loopmetadatasize=4G ``` @@ -532,7 +532,7 @@ options are "ext4" and "xfs". The default is "xfs" ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.fs=ext4 ``` @@ -542,7 +542,7 @@ Specifies extra mkfs arguments to be used when creating the base device. ###### Example -```bash +```console $ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" ``` @@ -552,7 +552,7 @@ Specifies extra mount options used when mounting the thin devices. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.mountopt=nodiscard ``` @@ -568,7 +568,7 @@ device. ###### Example -```bash +```console $ sudo dockerd \ --storage-opt dm.datadev=/dev/sdb1 \ --storage-opt dm.metadatadev=/dev/sdc1 @@ -586,13 +586,13 @@ data, or even better on an SSD. If setting up a new metadata pool it is required to be valid. This can be achieved by zeroing the first 4k to indicate empty metadata, like this: -```bash +```console $ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 ``` ###### Example -```bash +```console $ sudo dockerd \ --storage-opt dm.datadev=/dev/sdb1 \ --storage-opt dm.metadatadev=/dev/sdc1 @@ -605,7 +605,7 @@ blocksize is 64K. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.blocksize=512K ``` @@ -621,7 +621,7 @@ returned to the system for other use when containers are removed. ###### Examples -```bash +```console $ sudo dockerd --storage-opt dm.blkdiscard=false ``` @@ -633,11 +633,11 @@ Overrides the `udev` synchronization checks between `devicemapper` and `udev`. To view the `udev` sync support of a Docker daemon that is using the `devicemapper` driver, run: -```bash +```console $ docker info -[...] +<...> Udev Sync Supported: true -[...] +<...> ``` When `udev` sync support is `true`, then `devicemapper` and udev can @@ -651,7 +651,7 @@ results in errors and failures. (For information on these failures, see To allow the `docker` daemon to start, regardless of `udev` sync not being supported, set `dm.override_udev_sync_check` to true: -```bash +```console $ sudo dockerd --storage-opt dm.override_udev_sync_check=true ``` @@ -684,7 +684,7 @@ loop trying to remove a busy device. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.use_deferred_removal=true ``` @@ -702,7 +702,7 @@ Error deleting container: Error response from daemon: Cannot destroy container To avoid this failure, enable both deferred device deletion and deferred device removal on the daemon. -```bash +```console $ sudo dockerd \ --storage-opt dm.use_deferred_deletion=true \ --storage-opt dm.use_deferred_removal=true @@ -742,7 +742,7 @@ the issue. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.min_free_space=10% ``` @@ -758,7 +758,7 @@ ENOSPC and will shutdown filesystem. ###### Example -```bash +```console $ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 ``` @@ -784,7 +784,7 @@ their corresponding levels when output by `dockerd`. ###### Example -```bash +```console $ sudo dockerd \ --log-level debug \ --storage-opt dm.libdm_log_level=7 @@ -800,7 +800,7 @@ By default docker will pick up the zfs filesystem where docker graph ###### Example -```bash +```console $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker ``` @@ -815,7 +815,7 @@ a container with **--storage-opt size** option, docker should ensure the ###### Example -```bash +```console $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G ``` @@ -838,7 +838,7 @@ conditions the user can pass any size less then the backing fs size. ###### Example -```bash +```console $ sudo dockerd -s overlay2 --storage-opt overlay2.size=1G ``` @@ -960,7 +960,7 @@ By default, the Docker daemon automatically starts `containerd`. If you want to control `containerd` startup, manually start `containerd` and pass the path to the `containerd` socket using the `--containerd` flag. For example: -```bash +```console $ sudo dockerd --containerd /var/run/dev/docker-containerd.sock ``` @@ -988,7 +988,7 @@ The following is an example adding 2 runtimes via the configuration: This is the same example via the command line: -```bash +```console $ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement ``` @@ -1010,7 +1010,7 @@ is used on cgroup v2 hosts with systemd available. This example sets the `cgroupdriver` to `systemd`: -```bash +```console $ sudo dockerd --exec-opt native.cgroupdriver=systemd ``` @@ -1031,13 +1031,13 @@ value is specified on daemon start, on Windows client, the default is To set the DNS server for all Docker containers, use: -```bash +```console $ sudo dockerd --dns 8.8.8.8 ``` To set the DNS search domain for all Docker containers, use: -```bash +```console $ sudo dockerd --dns-search example.com ``` @@ -1163,7 +1163,7 @@ TLS. To configure the client TLS settings used by the daemon can be configured using the `--cluster-store-opt` flag, specifying the paths to PEM encoded files. For example: -```bash +```console $ sudo dockerd \ --cluster-advertise 192.168.1.2:2376 \ --cluster-store etcd://192.168.1.2:2379 \ @@ -1190,7 +1190,7 @@ organization can purchase or build themselves. You can install one or more authorization plugins when you start the Docker `daemon` using the `--authorization-plugin=PLUGIN_ID` option. -```bash +```console $ sudo dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... ``` From 9eea889bbb43e303543480f404e430fecba9f107 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 13 Sep 2021 16:20:09 +0200 Subject: [PATCH 368/534] docs: fix some broken anchors Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- docs/extend/plugins_authorization.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index e4975a967d..a43bed1daf 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -54,7 +54,7 @@ Deprecated | [Support for encrypted TLS private keys](#support-for-encrypted-tls Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v21.xx -Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options–with-cgroups-v1) | v20.10 | - +Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 4a9d0a2001..2fc444e101 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -114,7 +114,7 @@ Enable the authorization plugin with a dedicated command line flag in the `--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. This value can be the plugin’s socket or a path to a specification file. Authorization plugins can be loaded without restarting the daemon. Refer -to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reloading) for more information. +to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reload-behavior) for more information. ```console $ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... From 9544d670c2271a1174882b9be5de71975d71d03b Mon Sep 17 00:00:00 2001 From: Pawel Date: Sat, 11 Sep 2021 22:52:44 +0200 Subject: [PATCH 369/534] Use private network address for default-address-pools setting in daemon.json example Signed-off-by: Pawel --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a870c3079d..e9c6011f96 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1380,11 +1380,11 @@ This is a full example of the allowed configuration options on Linux: "debug": true, "default-address-pools": [ { - "base": "172.80.0.0/16", + "base": "172.30.0.0/16", "size": 24 }, { - "base": "172.90.0.0/16", + "base": "172.31.0.0/16", "size": 24 } ], From d230074054ff3abb24bf8c3ac4db2673eab8cb61 Mon Sep 17 00:00:00 2001 From: Brian Goff Date: Thu, 23 Sep 2021 17:53:46 +0000 Subject: [PATCH 370/534] doc: Add `default-runtime` opt to daemon.json This is a new flag for the Windows daemon. Signed-off-by: Brian Goff --- docs/reference/commandline/dockerd.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e9c6011f96..eedbda931f 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1505,6 +1505,7 @@ This is a full example of the allowed configuration options on Windows: "containerd-plugin-namespace": "docker-plugins", "data-root": "", "debug": true, + "default-runtime": "", "default-ulimits": {}, "dns": [], "dns-opts": [], @@ -1537,6 +1538,13 @@ This is a full example of the allowed configuration options on Windows: } ``` +The `default-runtime` option is by default unset, in which case dockerd will auto-detect the runtime. This detection is currently based on if the `containerd` flag is set. + +Accepted values: + +- `com.docker.hcsshim.v1` - This is the built-in runtime that Docker has used since Windows supported was first added and uses the v1 HCS API's in Windows. +- `io.containerd.runhcs.v1` - This is uses the containerd `runhcs` shim to run the container and uses the v2 HCS API's in Windows. + #### Feature options The optional field `features` in `daemon.json` allows users to enable or disable specific daemon features. For example, `{"features":{"buildkit": true}}` enables `buildkit` as the From 33dfb195c13e5d8b028d188de570096b3b389857 Mon Sep 17 00:00:00 2001 From: Dmitriy Fishman Date: Thu, 30 Sep 2021 18:12:46 +0300 Subject: [PATCH 371/534] Update deprecated.md Signed-off-by: Dmitriy Fishman --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a43bed1daf..7752fa081d 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -560,7 +560,7 @@ See the events API documentation for the new format. **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is not longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. +To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is no longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** From a241d92ac9b35b45f298949b22cd89708224ff03 Mon Sep 17 00:00:00 2001 From: Peter Dave Hello Date: Fri, 15 Oct 2021 06:04:36 +0800 Subject: [PATCH 372/534] Update most links in docs to use https by default cc @thaJeztah docker/docker.github.io#13680 Signed-off-by: Peter Dave Hello --- docs/extend/legacy_plugins.md | 4 ++-- docs/extend/plugin_api.md | 2 +- docs/reference/commandline/dockerd.md | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 0fd6d11655..586c4d6082 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -72,7 +72,7 @@ The sections below provide an inexhaustive overview of available plugins. | [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. | | [HPE 3Par Volume Plugin](https://github.com/hpe-storage/python-hpedockerplugin/) | A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage arrays. | | [Infinit volume plugin](https://infinit.sh/documentation/docker/volume-plugin) | A volume plugin that makes it easy to mount and manage Infinit volumes using Docker. | -| [IPFS Volume Plugin](http://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. | +| [IPFS Volume Plugin](https://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. | | [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. | | [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. | | [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. | @@ -80,7 +80,7 @@ The sections below provide an inexhaustive overview of available plugins. | [Nimble Storage Volume Plugin](https://connect.nimblestorage.com/community/app-integration/docker) | A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. | | [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. | | [Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. | -| [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](http://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. | +| [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](https://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. | | [REX-Ray plugin](https://github.com/emccode/rexray) | A volume plugin which is written in Go and provides advanced storage functionality for many platforms including VirtualBox, EC2, Google Compute Engine, OpenStack, and EMC. | | [Virtuozzo Storage and Ploop plugin](https://github.com/virtuozzo/docker-volume-ploop) | A volume plugin with support for Virtuozzo Storage distributed cloud file system as well as ploop devices. | | [VMware vSphere Storage Plugin](https://github.com/vmware/docker-volume-vsphere) | Docker Volume Driver for vSphere enables customers to address persistent storage requirements for Docker containers in vSphere environments. | diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index d1c415af9c..812b465085 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -90,7 +90,7 @@ The `TLSConfig` field is optional and TLS will only be verified if this configur Plugins should be started before Docker, and stopped after Docker. For example, when packaging a plugin for a platform which supports `systemd`, you might use [`systemd` dependencies]( -http://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to +https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to manage startup and shutdown order. When upgrading a plugin, you should first stop the Docker daemon, upgrade the diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e9c6011f96..3973bbe443 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -165,7 +165,7 @@ communication with the daemon. > supported anymore for security reasons. On Systemd based systems, you can communicate with the daemon via -[Systemd socket activation](http://0pointer.de/blog/projects/socket-activation.html), +[Systemd socket activation](https://0pointer.de/blog/projects/socket-activation.html), use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but you can also specify individual sockets: `dockerd -H fd://3`. If the specified socket activated files aren't found, then Docker will exit. You can @@ -308,7 +308,7 @@ devices, one for data and one for metadata. By default, these block devices are created automatically by using loopback mounts of automatically created sparse files. Refer to [Devicemapper options](#devicemapper-options) below for a way how to customize this setup. -[~jpetazzo/Resizing Docker containers with the Device Mapper plugin](http://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) +[~jpetazzo/Resizing Docker containers with the Device Mapper plugin](https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) article explains how to tune your existing setup without the use of options. The `btrfs` driver is very fast for `docker build` - but like `devicemapper` @@ -1211,7 +1211,7 @@ For information about how to create an authorization plugin, refer to the ### Daemon user namespace options The Linux kernel -[user namespace support](http://man7.org/linux/man-pages/man7/user_namespaces.7.html) +[user namespace support](https://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling a process, and therefore a container, to have a unique range of user and group IDs which are outside the traditional user and group range utilized by the host system. Potentially the most important From e571cd601d9c99cf1dd9d0d0f666c449e8173402 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 11 Jan 2022 12:21:11 +0100 Subject: [PATCH 373/534] docs/daemon: replace deprecated '-g' option for '--data-root' Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 3973bbe443..e331a73348 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -116,8 +116,8 @@ Options with [] may be specified multiple times. uses different binaries for the daemon and client. To run the daemon you type `dockerd`. -To run the daemon with debug output, use `dockerd -D` or add `"debug": true` to -the `daemon.json` file. +To run the daemon with debug output, use `dockerd --debug` or add `"debug": true` +to [the `daemon.json` file](#daemon-configuration-file). > **Enabling experimental features** > @@ -313,7 +313,7 @@ article explains how to tune your existing setup without the use of options. The `btrfs` driver is very fast for `docker build` - but like `devicemapper` does not share executable memory between devices. Use -`dockerd -s btrfs -g /mnt/btrfs_partition`. +`dockerd --storage-driver btrfs --data-root /mnt/btrfs_partition`. The `zfs` driver is probably not as fast as `btrfs` but has a longer track record on stability. Thanks to `Single Copy ARC` shared blocks between clones will be @@ -1234,14 +1234,14 @@ for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be set like this: ```console -$ DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +$ DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd --data-root /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 ``` or ```console $ export DOCKER_TMPDIR=/mnt/disk2/tmp -$ /usr/local/bin/dockerd -D -g /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +$ /usr/local/bin/dockerd --data-root /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 ```` #### Default cgroup parent From 0efaad4e98de6b7acb978711fe6f2225b4c2f002 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 28 Jul 2021 16:08:09 +0200 Subject: [PATCH 374/534] Remove support for encrypted TLS private keys > Legacy PEM encryption as specified in RFC 1423 is insecure by design. Since > it does not authenticate the ciphertext, it is vulnerable to padding oracle > attacks that can let an attacker recover the plaintext From https://go-review.googlesource.com/c/go/+/264159 > It's unfortunate that we don't implement PKCS#8 encryption so we can't > recommend an alternative but PEM encryption is so broken that it's worth > deprecating outright. This feature allowed using an encrypted private key with a supplied password, but did not provide additional security as the encryption is known to be broken, and the key is sitting next to the password in the filesystem. Users are recommended to decrypt the private key, and store it un-encrypted to continue using it. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7752fa081d..53ee3cc04e 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,7 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | - +Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v21.xx Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v21.xx @@ -103,10 +103,17 @@ Removed | [Three arguments form in `docker import`](#three-arguments-form-in- **Deprecated in Release: v20.10** -Use of encrypted TLS private keys has been deprecated, and will be removed in a -future release. Golang has deprecated support for legacy PEM encryption (as -specified in [RFC 1423](https://datatracker.ietf.org/doc/html/rfc1423)), as it -is insecure by design (see [https://go-review.googlesource.com/c/go/+/264159](https://go-review.googlesource.com/c/go/+/264159)). +**Removed in Release: v21.xx** + +Use of encrypted TLS private keys has been deprecated, and has been removed. +Golang has deprecated support for legacy PEM encryption (as specified in +[RFC 1423](https://datatracker.ietf.org/doc/html/rfc1423)), as it is insecure by +design (see [https://go-review.googlesource.com/c/go/+/264159](https://go-review.googlesource.com/c/go/+/264159)). + +This feature allowed using an encrypted private key with a supplied password, +but did not provide additional security as the encryption is known to be broken, +and the key is sitting next to the password in the filesystem. Users are recommended +to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support From da939ba82d011e756bbebf15a86157f10435eefe Mon Sep 17 00:00:00 2001 From: Leonid Skorospelov Date: Thu, 29 Jul 2021 13:55:41 +0000 Subject: [PATCH 375/534] Update dockerd.md Simple typo Signed-off-by: Leonid Skorospelov --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 0b984fe56b..88ee498487 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1574,7 +1574,7 @@ The list of currently supported options that can be reconfigured is this: be used to run containers. - `authorization-plugin`: it specifies the authorization plugins to use. - `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. -- `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure resgitries, these existing ones will be removed from daemon's config. +- `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure registries, these existing ones will be removed from daemon's config. - `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. - `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. - `features`: it explicitly enables or disables specific features. From 8e80c4697b75213ae2c587d05f92f1ff4a8426c6 Mon Sep 17 00:00:00 2001 From: CrazyMax Date: Thu, 27 Jan 2022 19:30:06 +0100 Subject: [PATCH 376/534] builder: fallback to legacy Signed-off-by: CrazyMax --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 53ee3cc04e..8594ba4276 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v21.xx | - Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v21.xx Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - @@ -99,6 +100,21 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Legacy builder fallback + +**Deprecated in Release: v21.xx** + +BuildKit implementation code and linked build flags have been removed in the CLI +and are now forwarded to [Buildx](https://docs.docker.com/buildx/working-with-buildx/) +builder component. + +If Buildx plugin is missing or broken, build command will fall back to the +legacy builder. This fallback mechanism will be removed in a future release and +user prompted to install the Buildx plugin. + +User can always fall back to the legacy builder with `DOCKER_BUILDKIT=0` but +`DOCKER_BUILDKIT=1` will always require Buildx. + ### Support for encrypted TLS private keys **Deprecated in Release: v20.10** From 14d8f4d9f0cefeb2170ae4e2d5ed2f683450b5f3 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 27 Jan 2022 18:54:06 +0100 Subject: [PATCH 377/534] Update deprecation docs - expand a bit on what's happening - clarify start of deprecation of the classic builder - show examples of error and warning Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 79 +++++++++++++++++++++++++++++++++++++++++----- 1 file changed, 71 insertions(+), 8 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 8594ba4276..a97a210a46 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ +Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v21.xx | - Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v21.xx | - Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v21.xx Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - @@ -100,20 +101,82 @@ Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on- Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +### Legacy builder for Linux images + +**Deprecated in Release: v21.xx** + +Docker v21.xx now uses BuildKit by default to build Linux images, and uses the +[Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for +`docker build`. With this change, `docker build` now exposes all advanced features +that BuildKit provides and which were previously only available through the +`docker buildx` subcommands. + +The Buildx component is installed automatically when installing the `docker` CLI +using our `.deb` or `.rpm` packages, and statically linked binaries are provided +both on `download.docker.com`, and through the [`docker/buildx-bin` image](https://hub.docker.com/r/docker/buildx-bin) +on Docker Hub. Refer the [Buildx section](http://docs.docker.com/go/buildx/) for +detailed instructions on installing the Buildx component. + +This release marks the beginning of the deprecation cycle of the classic ("legacy") +builder for Linux images. No active development will happen on the classic builder +(except for bugfixes). BuildKit development started five Years ago, left the +"experimental" phase since Docker 18.09, and is already the default builder for +[Docker Desktop](https://docs.docker.com/desktop/mac/release-notes/3.x/#docker-desktop-320). +While we're comfortable that BuildKit is stable for general use, there may be +some changes in behavior. If you encounter issues with BuildKit, we encourage +you to report issues in the [BuildKit issue tracker on GitHub](https://github.com/moby/buildkit/){:target="_blank" rel="noopener" class="_"} + +> Classic builder for building Windows images +> +> BuildKit does not (yet) provide support for building Windows images, and +> `docker build` continues to use the classic builder to build native Windows +> images on Windows daemons. + ### Legacy builder fallback **Deprecated in Release: v21.xx** -BuildKit implementation code and linked build flags have been removed in the CLI -and are now forwarded to [Buildx](https://docs.docker.com/buildx/working-with-buildx/) -builder component. +[Docker v21.xx now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), +which requires the Buildx component to build images with BuildKit. There may be +situations where the Buildx component is not available, and BuildKit cannot be +used. -If Buildx plugin is missing or broken, build command will fall back to the -legacy builder. This fallback mechanism will be removed in a future release and -user prompted to install the Buildx plugin. +To provide a smooth transition to BuildKit as the default builder, Docker v21.xx +has an automatic fallback for some situations, or produces an error to assist +users to resolve the problem. -User can always fall back to the legacy builder with `DOCKER_BUILDKIT=0` but -`DOCKER_BUILDKIT=1` will always require Buildx. +In situations where the user did not explicitly opt-in to use BuildKit (i.e., +`DOCKER_BUILDKIT=1` is not set), the CLI automatically falls back to the classic +builder, but prints a deprecation warning: + +``` +DEPRECATED: The legacy builder is deprecated and will be removed in a future release. + Install the buildx component to build images with BuildKit: + https://docs.docker.com/go/buildx/ +``` + +This situation may occur if the `docker` CLI is installed using the static binaries, +and the Buildx component is not installed or not installed correctly. This fallback +will be removed in a future release, therefore we recommend to [install the Buildx component](https://docs.docker.com/go/buildx/) +and use BuildKit for your builds, or opt-out of using BuildKit with `DOCKER_BUILDKIT=0`. + +If you opted-in to use BuildKit (`DOCKER_BUILDKIT=1`), but the Buildx component +is missing, an error is printed instead, and the `docker build` command fails: + +``` +ERROR: BuildKit is enabled but the buildx component is missing or broken. + Install the buildx component to build images with BuildKit: + https://docs.docker.com/go/buildx/ +``` + +We recommend to [install the Buildx component](https://docs.docker.com/go/buildx/) +to continue using BuildKit for your builds, but alternatively, users can either +unset the `DOCKER_BUILDKIT` environment variable to fall back to the legacy builder, +or opt-out of using BuildKit with `DOCKER_BUILDKIT=0`. + +Be aware that the [classic builder is deprecated](#legacy-builder-for-linux-images) +so both the automatic fallback and opting-out of using BuildKit will no longer +be possible in a future release. ### Support for encrypted TLS private keys From 190c7d2321a8189f73ac529d3605c72422f4f695 Mon Sep 17 00:00:00 2001 From: Nicolas De Loof Date: Thu, 24 Feb 2022 10:24:30 +0100 Subject: [PATCH 378/534] document kubernetes feature as "removed" in 21.xx Signed-off-by: Nicolas De Loof --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index a97a210a46..bf2aa33293 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,7 +53,7 @@ Status | Feature Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v21.xx | - Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v21.xx | - Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v21.xx -Deprecated | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | - +Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v21.xx Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v21.xx Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - From 759205520bc260b6d2e12937f8451b6dadaeed6c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 24 Feb 2022 11:13:50 +0100 Subject: [PATCH 379/534] docs: add more details to stack kubernetes deprecation Explicitly mention flags and environment variables that were removed, to make the deprecation more discoverable. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 25 +++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index bf2aa33293..db89710655 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -197,9 +197,30 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support **Deprecated in Release: v20.10** +**Removed in Release: v21.xx** -Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for -Kubernetes in the `stack` and `context` commands in the docker CLI is now marked as deprecated as well. +Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), +support for Kubernetes in the `stack` and `context` commands has been removed from +the cli, and options related to this functionality are now either ignored, or may +produce an error. + +The following command-line flags are removed from the `docker context` subcommands: + +- `--default-stack-orchestrator` - swarm is now the only (and default) orchestrator for stacks. +- `--kubernetes` - the kubernetes endpoint can no longer be stored in `docker context`. +- `--kubeconfig` - exporting a context as a kubeconfig file is no longer supported. + +The output produced by the `docker context inspect` subcommand no longer contains +information about `StackOrchestrator` and `Kubernetes` endpoints for new contexts. + +The following command-line flags are removed from the `docker stack` subcommands: + +- `--kubeconfig` - using a kubeconfig file as context is no longer supported. +- `--namespace` - configuring the kubernetes namespace for stacks is no longer supported. +- `--orchestrator` - swarm is now the only (and default) orchestrator for stacks. + +The `DOCKER_STACK_ORCHESTRATOR`, `DOCKER_ORCHESTRATOR`, and `KUBECONFIG` environment +variables are no longer used, and ignored. ### Pulling images from non-compliant image registries From 5df3055d3722561a8d34b7ba52c4abdf70dca20f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 24 Feb 2022 13:35:03 +0100 Subject: [PATCH 380/534] config: mark stackOrchestrator option as deprecated Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index db89710655..b7142d8740 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -220,7 +220,8 @@ The following command-line flags are removed from the `docker stack` subcommands - `--orchestrator` - swarm is now the only (and default) orchestrator for stacks. The `DOCKER_STACK_ORCHESTRATOR`, `DOCKER_ORCHESTRATOR`, and `KUBECONFIG` environment -variables are no longer used, and ignored. +variables, as well as the `stackOrchestrator` option in the `~/.docker/config.json` +cli configuration file are no longer used, and ignored. ### Pulling images from non-compliant image registries From 977529a6ef68ab453d010e4148640004e825f7dc Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 1 Mar 2022 12:46:49 +0100 Subject: [PATCH 381/534] docs: daemon: remove LCOW configuration options Support for LCOW has been removed from the daemon, so removing the corresponding daemon configuration options. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 100 +------------------------- 1 file changed, 2 insertions(+), 98 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2177df9298..8622a9e4b6 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -347,16 +347,13 @@ to use it. The `fuse-overlayfs` driver is similar to `overlay2` but works in userspace. The `fuse-overlayfs` driver is expected to be used for [Rootless mode](https://docs.docker.com/engine/security/rootless/). -On Windows, the Docker daemon supports a single image layer storage driver -depending on the image platform: `windowsfilter` for Windows images, and -`lcow` for Linux containers on Windows. +On Windows, the Docker daemon only supports the `windowsfilter` storage driver. ### Options per storage driver Particular storage-driver can be configured with options specified with `--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, -options for `zfs` start with `zfs`, options for `btrfs` start with `btrfs` -and options for `lcow` start with `lcow`. +options for `zfs` start with `zfs`, and options for `btrfs` start with `btrfs`. #### Devicemapper options @@ -856,99 +853,6 @@ Defaults to 20G. C:\> dockerd --storage-opt size=40G ``` -#### LCOW (Linux Containers on Windows) options - -##### `lcow.globalmode` - -Specifies whether the daemon instantiates utility VM instances as required -(recommended and default if omitted), or uses single global utility VM (better -performance, but has security implications and not recommended for production -deployments). - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.globalmode=false -``` - -##### `lcow.kirdpath` - -Specifies the folder path to the location of a pair of kernel and initrd files -used for booting a utility VM. Defaults to `%ProgramFiles%\Linux Containers`. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.kirdpath=c:\path\to\files -``` - -##### `lcow.kernel` - -Specifies the filename of a kernel file located in the `lcow.kirdpath` path. -Defaults to `bootx64.efi`. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.kernel=kernel.efi -``` - -##### `lcow.initrd` - -Specifies the filename of an initrd file located in the `lcow.kirdpath` path. -Defaults to `initrd.img`. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.initrd=myinitrd.img -``` - -##### `lcow.bootparameters` - -Specifies additional boot parameters for booting utility VMs when in kernel/ -initrd mode. Ignored if the utility VM is booting from VHD. These settings -are kernel specific. - -###### Example - -```powershell -C:\> dockerd --storage-opt "lcow.bootparameters='option=value'" -``` - -##### `lcow.vhdx` - -Specifies a custom VHDX to boot a utility VM, as an alternate to kernel -and initrd booting. Defaults to `uvm.vhdx` under `lcow.kirdpath`. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.vhdx=custom.vhdx -``` - -##### `lcow.timeout` - -Specifies the timeout for utility VM operations in seconds. Defaults -to 300. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.timeout=240 -``` - -##### `lcow.sandboxsize` - -Specifies the size in GB to use when creating the sandbox which is used for -containers. Defaults to 20. Cannot be less than 20. - -###### Example - -```powershell -C:\> dockerd --storage-opt lcow.sandboxsize=40 -``` - ### Docker runtime execution options The Docker daemon relies on a From 7c14421800bca46d02896fc2049825e07cbdd8e6 Mon Sep 17 00:00:00 2001 From: Mozi <29089388+pzhlkj6612@users.noreply.github.com> Date: Sun, 13 Mar 2022 20:35:31 +0800 Subject: [PATCH 382/534] docs: dockerd: fix broken link in blockquote area Signed-off-by: Mozi <29089388+pzhlkj6612@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 8622a9e4b6..0d9ad3280a 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1386,7 +1386,7 @@ This is a full example of the allowed configuration options on Linux: > daemon startup as a flag. > On systems that use `systemd` to start the Docker daemon, `-H` is already set, so > you cannot use the `hosts` key in `daemon.json` to add listening addresses. -> See https://docs.docker.com/engine/admin/systemd/#custom-docker-daemon-options for how +> See ["custom Docker daemon options"](https://docs.docker.com/config/daemon/systemd/#custom-docker-daemon-options) for how > to accomplish this task with a systemd drop-in file. ##### On Windows From 90e8911b0fe3ad05d298da34e4d5848f898a361a Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 30 Mar 2022 14:33:44 +0200 Subject: [PATCH 383/534] docs: fix (table) formatting, fix some broken links Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 95c5d31c78..ffb8a0cefa 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -771,7 +771,7 @@ time of writing, the following is the list of `libdm` log levels as well as their corresponding levels when output by `dockerd`. | `libdm` Level | Value | `--log-level` | -| ------------- | -----:| ------------- | +|---------------|------:|---------------| | `_LOG_FATAL` | 2 | error | | `_LOG_ERR` | 3 | error | | `_LOG_WARN` | 4 | warn | From 8421e00876654209a5caf99878998f6fe841974f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 1 Apr 2022 15:32:55 +0200 Subject: [PATCH 384/534] tweak description of login/logout Remove "Docker" from registry, as the registry specification is no longer docker-specific, but part of the OCI distribution spec. Also removed "Register" from one of the docs pages, as the login command hasn't supported creating a new acccount on Docker Hub for a long time. I'm wondering if we should be more explicit about what log in / out does (effectively; authenticate, and on success store the credentials or token, and on log out; remove credentials/token). Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ffb8a0cefa..5261266ae2 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -88,7 +88,7 @@ Options: --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring - --registry-mirror list Preferred Docker registry mirror + --registry-mirror list Preferred registry mirror --rootless Enable rootless mode; typically used with RootlessKit --seccomp-profile string Path to seccomp profile --selinux-enabled Enable selinux support From 2ccfdfa231bbf5194d222e8d76e829568f51ab99 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 May 2020 19:37:16 +0200 Subject: [PATCH 385/534] Deprecation: config: remove support for old ~/.dockercfg The `~/.dockercfg` file was replaced by `~/.docker/config.json` in 2015 (github.com/docker/docker/commit/18c9b6c6455f116ae59cde8544413b3d7d294a5e), but the CLI still falls back to checking if this file exists if no current (`~/.docker/config.json`) file was found. Given that no version of the CLI since Docker v1.7.0 has created this file, and if such a file exists, it means someone hasn't re-authenticated for 5 years, it's probably safe to remove this fallback. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index b7142d8740..59b5b303bb 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -59,7 +59,7 @@ Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-l Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - -Deprecated | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | - +Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v21.xx Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 @@ -298,6 +298,7 @@ deprecated, and will be disabled or removed in a future release. ### Support for legacy `~/.dockercfg` configuration files **Deprecated in Release: v20.10** +**Removed in Release: v21.xx** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -307,8 +308,11 @@ as a fall-back, to assist existing users with migrating to the new file. Given that the old file format encourages insecure storage of credentials (credentials are stored unencrypted), and that no version of the CLI since -Docker v1.7.0 has created this file, the file is marked deprecated, and support -for this file will be removed in a future release. +Docker v1.7.0 has created this file, support for this file, and its format has +been removed. + +A warning is printed in situations where the CLI would fall back to the old file, +notifying the user that the legacy file is present, but ignored. ### Configuration options for experimental CLI features From fb92768fbdc53221919ea182dce09d5f0d6a456a Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 16:32:42 +0200 Subject: [PATCH 386/534] Update v21.xx to v22.04 We plan to do a release this month, so let's update these references to reflect that version. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 59b5b303bb..2808035d85 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,16 +50,16 @@ The table below provides an overview of the current status of deprecated feature Status | Feature | Deprecated | Remove -----------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v21.xx | - -Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v21.xx | - -Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v21.xx -Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v21.xx +Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.04 | - +Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.04 | - +Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.04 +Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.04 Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - -Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v21.xx +Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - -Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v21.xx +Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 @@ -103,9 +103,9 @@ Removed | [Three arguments form in `docker import`](#three-arguments-form-in- ### Legacy builder for Linux images -**Deprecated in Release: v21.xx** +**Deprecated in Release: v22.04** -Docker v21.xx now uses BuildKit by default to build Linux images, and uses the +Docker v22.04 now uses BuildKit by default to build Linux images, and uses the [Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for `docker build`. With this change, `docker build` now exposes all advanced features that BuildKit provides and which were previously only available through the @@ -134,14 +134,14 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co ### Legacy builder fallback -**Deprecated in Release: v21.xx** +**Deprecated in Release: v22.04** -[Docker v21.xx now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), +[Docker v22.04 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), which requires the Buildx component to build images with BuildKit. There may be situations where the Buildx component is not available, and BuildKit cannot be used. -To provide a smooth transition to BuildKit as the default builder, Docker v21.xx +To provide a smooth transition to BuildKit as the default builder, Docker v22.04 has an automatic fallback for some situations, or produces an error to assist users to resolve the problem. @@ -182,7 +182,7 @@ be possible in a future release. **Deprecated in Release: v20.10** -**Removed in Release: v21.xx** +**Removed in Release: v22.04** Use of encrypted TLS private keys has been deprecated, and has been removed. Golang has deprecated support for legacy PEM encryption (as specified in @@ -197,7 +197,7 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support **Deprecated in Release: v20.10** -**Removed in Release: v21.xx** +**Removed in Release: v22.04** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from @@ -258,7 +258,7 @@ major release. ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** -**Removed in Release: v21.xx** +**Removed in Release: v22.04** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after @@ -298,7 +298,7 @@ deprecated, and will be disabled or removed in a future release. ### Support for legacy `~/.dockercfg` configuration files **Deprecated in Release: v20.10** -**Removed in Release: v21.xx** +**Removed in Release: v22.04** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this From faa9b363569e578c1328cc15445315096e6c90a2 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 14:22:30 +0200 Subject: [PATCH 387/534] docs: reformat table for compatibility My IDE's linter kept complaining: > For compatibility reasons all table rows should have borders (pipe > symbols) at the start and at the end. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 104 ++++++++++++++++++++++----------------------- 1 file changed, 52 insertions(+), 52 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2808035d85..8c0d574e01 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -48,58 +48,58 @@ The table below provides an overview of the current status of deprecated feature alternatives. In such cases, a warning may be printed, and users should not rely on this feature. -Status | Feature | Deprecated | Remove ------------|------------------------------------------------------------------------------------------------------------------------------------|------------|------------ -Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.04 | - -Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.04 | - -Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.04 -Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.04 -Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - -Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 -Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - -Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - -Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - -Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 -Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - -Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - -Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 -Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.10 -Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 -Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 -Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 -Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 -Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - -Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - -Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - -Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 -Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 -Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 -Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 -Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - -Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 -Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 -Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 -Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 -Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 -Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - -Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 -Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 -Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 -Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 -Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 -Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 -Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - -Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 -Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 -Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 -Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 -Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 -Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 -Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 -Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 -Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 -Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 -Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 +| Status | Feature | Deprecated | Remove | +|------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.04 | - | +| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.04 | - | +| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.04 | +| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.04 | +| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | +| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 | +| Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | +| Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - | +| Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - | +| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 | +| Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | +| Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | +| Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | +| Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.10 | +| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | +| Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | +| Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | +| Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | +| Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | +| Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | +| Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | +| Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | +| Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | +| Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | +| Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | +| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | +| Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | +| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | +| Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | +| Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | +| Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | +| Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - | +| Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 | +| Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 | +| Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 | +| Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | +| Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | +| Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | +| Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | +| Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | +| Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | +| Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 | +| Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 | +| Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 | +| Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 | +| Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 | +| Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 | +| Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 | +| Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | +| Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | ### Legacy builder for Linux images From 77640743fe54fec5e3819b507d0dc39acb81bdb3 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 14:36:11 +0200 Subject: [PATCH 388/534] update deprecation status of --kernel-memory(-tcp) Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 27 ++++++++++++++++++++++++--- 1 file changed, 24 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 8c0d574e01..08cf139b2b 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -57,7 +57,7 @@ The table below provides an overview of the current status of deprecated feature | Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | | Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 | | Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | -| Deprecated | [Kernel memory limit](#kernel-memory-limit) | v20.10 | - | +| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v22.04 | | Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - | | Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 | | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | @@ -281,9 +281,30 @@ When using cgroups v2, the `--blkio-weight` options are implemented using ### Kernel memory limit **Deprecated in Release: v20.10** +**Removed in Release: v22.04** -Specifying kernel memory limit (`docker run --kernel-memory`) is now marked as deprecated, -as [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). +Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported +because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). +The OCI runtime specification now marks this option (as well as `--kernel-memory-tcp`) +as ["NOT RECOMMENDED"](https://github.com/opencontainers/runtime-spec/pull/1093), +and OCI runtimes such as `runc` no longer support this option. + +Docker API v1.42 and up now ignores this option when set. Older versions of the +API continue to accept the option, but depending on the OCI runtime used, may +take no effect. + +> **Note** +> +> While not deprecated (yet) in Docker, the OCI runtime specification also +> deprecated the `memory.kmem.tcp.limit_in_bytes` option. When using `runc` as +> runtime, this option takes no effect. The linux kernel did not explicitly +> deprecate this feature, and there is a tracking ticket in the `runc` issue +> tracker to determine if this option should be reinstated or if this was an +> oversight of the Linux kernel maintainers (see [opencontainers/runc#3174](https://github.com/opencontainers/runc/issues/3174)). +> +> The `memory.kmem.tcp.limit_in_bytes` option is only supported with cgroups v1, +> and not available on installations running with cgroups v2. This option is +> only supported by the API, and not exposed on the `docker` command-line. ### Classic Swarm and overlay networks using cluster store From 9374109dc9e99c56f395f628891bc8fc5b6518ad Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 15:12:23 +0200 Subject: [PATCH 389/534] docs: update deprecation status of aufs, overlay, and devicemapper Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 61 +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 58 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 08cf139b2b..d06211f036 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -68,9 +68,9 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | -| Deprecated | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | -| Deprecated | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | -| Deprecated | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | +| Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | +| Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | +| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | | Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | | Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | | Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | @@ -445,6 +445,7 @@ using compose files. ### AuFS storage driver **Deprecated in Release: v19.03** +**Disabled by default in Release: v22.04** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -460,10 +461,28 @@ is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. +#### Disabled by default in v22.04 + +Docker already prevented deprecated storage drivers from being automatically +selected on new installations, but continued to use these drivers when upgrading +existing installations. Starting with the v22.04 release, the Docker Engine will +fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): + +```console +failed to start daemon: error initializing graphdriver: prior storage driver +aufs is deprecated and will be removed in a future release; update the the daemon +configuration and explicitly choose this storage driver to continue using it; +visit https://docs.docker.com/go/storage-driver/ for more information. +``` + +To continue using the storage driver, update the daemon configuration to use +explicitly use the given storage driver. Users are encouraged to migrate to +different storage driver. ### Legacy "overlay" storage driver **Deprecated in Release: v18.09** +**Disabled by default in Release: v22.04** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -476,9 +495,28 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. +#### Disabled by default in v22.04 + +Docker already prevented deprecated storage drivers from being automatically +selected on new installations, but continued to use these drivers when upgrading +existing installations. Starting with the v22.04 release, the Docker Engine will +fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): + +```console +failed to start daemon: error initializing graphdriver: prior storage driver +overlay is deprecated and will be removed in a future release; update the the daemon +configuration and explicitly choose this storage driver to continue using it; +visit https://docs.docker.com/go/storage-driver/ for more information. +``` + +To continue using the storage driver, update the daemon configuration to use +explicitly use the given storage driver. Users are encouraged to migrate to +different storage driver. + ### Device mapper storage driver **Deprecated in Release: v18.09** +**Disabled by default in Release: v22.04** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are @@ -492,6 +530,23 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. +#### Disabled by default in v22.04 + +Docker already prevented deprecated storage drivers from being automatically +selected on new installations, but continued to use these drivers when upgrading +existing installations. Starting with the v22.04 release, the Docker Engine will +fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): + +```console +failed to start daemon: error initializing graphdriver: prior storage driver +devicemapper is deprecated and will be removed in a future release; update the the daemon +configuration and explicitly choose this storage driver to continue using it; +visit https://docs.docker.com/go/storage-driver/ for more information. +``` + +To continue using the storage driver, update the daemon configuration to use +explicitly use the given storage driver. Users are encouraged to migrate to +different storage driver. ### Use of reserved namespaces in engine labels From 3fd7d17195226c6d0b165582f154f9f037077528 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 15:14:44 +0200 Subject: [PATCH 390/534] docs: update deprecation status of legacy overlay networks Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index d06211f036..513edbf86f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -58,7 +58,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 | | Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | | Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v22.04 | -| Deprecated | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | - | +| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v22.04 | | Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 | | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | @@ -309,12 +309,11 @@ take no effect. ### Classic Swarm and overlay networks using cluster store **Deprecated in Release: v20.10** +**Removed in Release: v22.04** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, -`--cluster-store`, and `--cluster-store-opt` daemon options have been marked -deprecated, and will be disabled or removed in a future release. - +`--cluster-store`, and `--cluster-store-opt` daemon options have been removed. ### Support for legacy `~/.dockercfg` configuration files From 7b96f6cf1429c306aa02de8a59caff9e9d1a372c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 17:44:46 +0200 Subject: [PATCH 391/534] docs: update deprecation status for overlay(2) on backingfs without d_type Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 513edbf86f..47b8ab94c0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -693,12 +693,11 @@ The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` if it is formatted with the `ftype=0` option. -Starting with Docker 17.12, new installations will not support running overlay2 on -a backing filesystem without `d_type` support. For existing installations that upgrade -to 17.12, a warning will be printed. +Support for these setups has been removed, and Docker v22.04 and up now fails to +start when attempting to use the `overlay2` or `overlay` storage driver on a +backing filesystem without `d_type` support. -Please also refer to [#27358](https://github.com/docker/docker/issues/27358) for -further information. +Refer to [#27358](https://github.com/docker/docker/issues/27358) for details. ### `--automated` and `--stars` flags on `docker search` From 1543c7f5527645bc4f0f94abc88e6cf9702a4dd9 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 22:10:24 +0200 Subject: [PATCH 392/534] docs: update deprecation status of "experimental" cli Change future tense to current tense. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 47b8ab94c0..b09d2426d0 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -63,7 +63,7 @@ The table below provides an overview of the current status of deprecated feature | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | -| Deprecated | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v20.10 | +| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v22.04 | | Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | @@ -336,9 +336,13 @@ notifying the user that the legacy file is present, but ignored. ### Configuration options for experimental CLI features +**Deprecated in Release: v19.03** + +**Removed in Release: v22.04** + The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` -field in the CLI configuration file are deprecated. Experimental features will be -enabled by default, and these configuration options will no longer be functional. +field in the CLI configuration file are deprecated. Experimental features are +enabled by default, and these configuration options are no longer functional. ### CLI plugins support From 134a7e26e6385ebed76ac96cffbd5c0787f731fb Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Apr 2022 22:14:25 +0200 Subject: [PATCH 393/534] remove "Experimental" client field from "docker version" Configuration (enabling/disabling) of Experimental client features was deprecated in Docker 19.03, and removed in 20.10. Experimental CLI features are now always enabled. In Docker 20.10, the Experimental field in `docker version` was kept (but always true). This patch removes the field from the output (both "pretty" output and the JSON struct). Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b09d2426d0..f40b76562f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -344,6 +344,10 @@ The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experi field in the CLI configuration file are deprecated. Experimental features are enabled by default, and these configuration options are no longer functional. +Starting with v22.04, the Docker CLI no longer prints `Experimental` for the client +in the output of `docker version`, and the field has been removed from the JSON +format. + ### CLI plugins support **Deprecated in Release: v20.10** From f97e7d6f534b1b1e68ee91757bb0ef4847165fe3 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sun, 10 Apr 2022 11:55:30 +0200 Subject: [PATCH 394/534] docs: deprecated: add entry for "fluent-async-connect" log-opt This option was deperecated in the upstream fluentd logging driver v1.4.0, and while we documented it as deprecated in the API changelog, there was no mention yet in the deprecated docs. relates to: - https://github.com/fluent/fluent-logger-golang/pull/56 - https://github.com/moby/moby/pull/39086 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b09d2426d0..5edc60bf9a 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -63,6 +63,7 @@ The table below provides an overview of the current status of deprecated feature | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | +| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | | Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v22.04 | | Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | @@ -391,6 +392,21 @@ Users that want to use this feature are encouraged to enable BuildKit by setting the `DOCKER_BUILDKIT=1` environment variable or through the daemon or CLI configuration files. +### `fluentd-async-connect` log opt + +**Deprecated in Release: v20.10** + +The `--log-opt fluentd-async-connect` option for the fluentd logging driver is +[deprecated in favor of `--log-opt fluentd-async`](https://github.com/moby/moby/pull/39086). +A deprecation message is logged in the daemon logs if the old option is used: + +```console +fluent#New: AsyncConnect is now deprecated, please use Async instead +``` + +Users are encouraged to use the `fluentd-async` option going forward, as support +for the old option will be removed in a future release. + ### Pushing and pulling with image manifest v2 schema 1 **Deprecated in Release: v19.03** From e523db5c1c0095ca0c099efadd53365ce22affef Mon Sep 17 00:00:00 2001 From: Phong Tran Date: Thu, 5 May 2022 19:10:15 -0700 Subject: [PATCH 395/534] Fix dead external link Signed-off-by: Phong Tran --- docs/extend/legacy_plugins.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 586c4d6082..d3981bd076 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -77,7 +77,7 @@ The sections below provide an inexhaustive overview of available plugins. | [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. | | [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. | | [Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. | -| [Nimble Storage Volume Plugin](https://connect.nimblestorage.com/community/app-integration/docker) | A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. | +| [Nimble Storage Volume Plugin](https://scod.hpedev.io/docker_volume_plugins/hpe_nimble_storage/index.html) | A volume plug-in that integrates with Nimble Storage Unified Flash Fabric arrays. The plug-in abstracts array volume capabilities to the Docker administrator to allow self-provisioning of secure multi-tenant volumes and clones. | | [OpenStorage Plugin](https://github.com/libopenstorage/openstorage) | A cluster-aware volume plugin that provides volume management for file and block storage solutions. It implements a vendor neutral specification for implementing extensions such as CoS, encryption, and snapshots. It has example drivers based on FUSE, NFS, NBD and EBS to name a few. | | [Portworx Volume Plugin](https://github.com/portworx/px-dev) | A volume plugin that turns any server into a scale-out converged compute/storage node, providing container granular storage and highly available volumes across any node, using a shared-nothing storage backend that works with any docker scheduler. | | [Quobyte Volume Plugin](https://github.com/quobyte/docker-volume) | A volume plugin that connects Docker to [Quobyte](https://www.quobyte.com/containers)'s data center file system, a general-purpose scalable and fault-tolerant storage platform. | From 4dd2c88b2bedc98028b1b461414548ec99692fa3 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 17 May 2022 11:00:51 +0200 Subject: [PATCH 396/534] docs: remove documentation about deprecated cluster-store This removes documentation related to legacy overlay networks using an external k/v store. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 52 --------------------------- 1 file changed, 52 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 5261266ae2..145eca7393 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1052,41 +1052,6 @@ Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linu set the maximum number of processes available to a user, not to a container. For details please check the [run](run.md) reference. -### Node discovery - -The `--cluster-advertise` option specifies the `host:port` or `interface:port` -combination that this particular daemon instance should use when advertising -itself to the cluster. The daemon is reached by remote hosts through this value. -If you specify an interface, make sure it includes the IP address of the actual -Docker host. For Engine installation created through `docker-machine`, the -interface is typically `eth1`. - -The daemon uses [libkv](https://github.com/docker/libkv/) to advertise -the node within the cluster. Some key-value backends support mutual -TLS. To configure the client TLS settings used by the daemon can be configured -using the `--cluster-store-opt` flag, specifying the paths to PEM encoded -files. For example: - -```console -$ sudo dockerd \ - --cluster-advertise 192.168.1.2:2376 \ - --cluster-store etcd://192.168.1.2:2379 \ - --cluster-store-opt kv.cacertfile=/path/to/ca.pem \ - --cluster-store-opt kv.certfile=/path/to/cert.pem \ - --cluster-store-opt kv.keyfile=/path/to/key.pem -``` - -The currently supported cluster store options are: - -| Option | Description | -|:----------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `discovery.heartbeat` | Specifies the heartbeat timer in seconds which is used by the daemon as a `keepalive` mechanism to make sure discovery module treats the node as alive in the cluster. If not configured, the default value is 20 seconds. | -| `discovery.ttl` | Specifies the TTL (time-to-live) in seconds which is used by the discovery module to timeout a node if a valid heartbeat is not received within the configured ttl value. If not configured, the default value is 60 seconds. | -| `kv.cacertfile` | Specifies the path to a local file with PEM encoded CA certificates to trust. | -| `kv.certfile` | Specifies the path to a local file with a PEM encoded certificate. This certificate is used as the client cert for communication with the Key/Value store. | -| `kv.keyfile` | Specifies the path to a local file with a PEM encoded private key. This private key is used as the client key for communication with the Key/Value store. | -| `kv.path` | Specifies the path in the Key/Value store. If not configured, the default value is 'docker/nodes'. | - ### Access authorization Docker's access authorization can be extended by authorization plugins that your @@ -1274,9 +1239,6 @@ This is a full example of the allowed configuration options on Linux: "bip": "", "bridge": "", "cgroup-parent": "", - "cluster-advertise": "", - "cluster-store": "", - "cluster-store-opts": {}, "containerd": "/run/containerd/containerd.sock", "containerd-namespace": "docker", "containerd-plugin-namespace": "docker-plugins", @@ -1402,8 +1364,6 @@ This is a full example of the allowed configuration options on Windows: "allow-nondistributable-artifacts": [], "authorization-plugins": [], "bridge": "", - "cluster-advertise": "", - "cluster-store": "", "containerd": "\\\\.\\pipe\\containerd-containerd", "containerd-namespace": "docker", "containerd-plugin-namespace": "docker-plugins", @@ -1471,9 +1431,6 @@ if there are conflicts, but it won't stop execution. The list of currently supported options that can be reconfigured is this: - `debug`: it changes the daemon to debug mode when set to true. -- `cluster-store`: it reloads the discovery store with the new address. -- `cluster-store-opts`: it uses the new options to reload the discovery store. -- `cluster-advertise`: it modifies the address advertised after reloading. - `labels`: it replaces the daemon labels with a new set of labels. - `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). - `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. @@ -1491,15 +1448,6 @@ The list of currently supported options that can be reconfigured is this: - `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. - `features`: it explicitly enables or disables specific features. -Updating and reloading the cluster configurations such as `--cluster-store`, -`--cluster-advertise` and `--cluster-store-opts` will take effect only if -these configurations were not previously configured. If `--cluster-store` -has been provided in flags and `cluster-advertise` not, `cluster-advertise` -can be added in the configuration file without accompanied by `--cluster-store`. -Configuration reload will log a warning message if it detects a change in -previously configured cluster configurations. - - ### Run multiple daemons > **Note:** From 5bd75afd2b5bd23f6321660b5620482ea298ade1 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 17 May 2022 11:41:21 +0200 Subject: [PATCH 397/534] docs/deprecated: update version to 22.06 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 64 +++++++++++++++++++++++----------------------- 1 file changed, 32 insertions(+), 32 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2cfdfe66f2..2d7602d5cd 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,21 +50,21 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| -| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.04 | - | -| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.04 | - | -| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.04 | -| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.04 | +| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.06 | - | +| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.06 | - | +| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.06 | +| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.06 | | Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | -| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.04 | +| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.06 | | Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | -| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v22.04 | -| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v22.04 | -| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.04 | +| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v22.06 | +| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v22.06 | +| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.06 | | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | | Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | -| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v22.04 | +| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v22.06 | | Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | @@ -104,9 +104,9 @@ The table below provides an overview of the current status of deprecated feature ### Legacy builder for Linux images -**Deprecated in Release: v22.04** +**Deprecated in Release: v22.06** -Docker v22.04 now uses BuildKit by default to build Linux images, and uses the +Docker v22.06 now uses BuildKit by default to build Linux images, and uses the [Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for `docker build`. With this change, `docker build` now exposes all advanced features that BuildKit provides and which were previously only available through the @@ -135,14 +135,14 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co ### Legacy builder fallback -**Deprecated in Release: v22.04** +**Deprecated in Release: v22.06** -[Docker v22.04 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), +[Docker v22.06 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), which requires the Buildx component to build images with BuildKit. There may be situations where the Buildx component is not available, and BuildKit cannot be used. -To provide a smooth transition to BuildKit as the default builder, Docker v22.04 +To provide a smooth transition to BuildKit as the default builder, Docker v22.06 has an automatic fallback for some situations, or produces an error to assist users to resolve the problem. @@ -183,7 +183,7 @@ be possible in a future release. **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** Use of encrypted TLS private keys has been deprecated, and has been removed. Golang has deprecated support for legacy PEM encryption (as specified in @@ -198,7 +198,7 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from @@ -259,7 +259,7 @@ major release. ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after @@ -282,7 +282,7 @@ When using cgroups v2, the `--blkio-weight` options are implemented using ### Kernel memory limit **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). @@ -310,7 +310,7 @@ take no effect. ### Classic Swarm and overlay networks using cluster store **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -319,7 +319,7 @@ networks using an external key/value store. The corresponding`--cluster-advertis ### Support for legacy `~/.dockercfg` configuration files **Deprecated in Release: v20.10** -**Removed in Release: v22.04** +**Removed in Release: v22.06** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -339,13 +339,13 @@ notifying the user that the legacy file is present, but ignored. **Deprecated in Release: v19.03** -**Removed in Release: v22.04** +**Removed in Release: v22.06** The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` field in the CLI configuration file are deprecated. Experimental features are enabled by default, and these configuration options are no longer functional. -Starting with v22.04, the Docker CLI no longer prints `Experimental` for the client +Starting with v22.06, the Docker CLI no longer prints `Experimental` for the client in the output of `docker version`, and the field has been removed from the JSON format. @@ -468,7 +468,7 @@ using compose files. ### AuFS storage driver **Deprecated in Release: v19.03** -**Disabled by default in Release: v22.04** +**Disabled by default in Release: v22.06** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -484,11 +484,11 @@ is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. -#### Disabled by default in v22.04 +#### Disabled by default in v22.06 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.04 release, the Docker Engine will +existing installations. Starting with the v22.06 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -505,7 +505,7 @@ different storage driver. ### Legacy "overlay" storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v22.04** +**Disabled by default in Release: v22.06** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -518,11 +518,11 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. -#### Disabled by default in v22.04 +#### Disabled by default in v22.06 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.04 release, the Docker Engine will +existing installations. Starting with the v22.06 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -539,7 +539,7 @@ different storage driver. ### Device mapper storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v22.04** +**Disabled by default in Release: v22.06** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are @@ -553,11 +553,11 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -#### Disabled by default in v22.04 +#### Disabled by default in v22.06 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.04 release, the Docker Engine will +existing installations. Starting with the v22.06 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -717,7 +717,7 @@ The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` if it is formatted with the `ftype=0` option. -Support for these setups has been removed, and Docker v22.04 and up now fails to +Support for these setups has been removed, and Docker v22.06 and up now fails to start when attempting to use the `overlay2` or `overlay` storage driver on a backing filesystem without `d_type` support. From 3132c4a8ab3e7056dbbd8a531e8ff1119a1cdf6a Mon Sep 17 00:00:00 2001 From: CrazyMax Date: Mon, 27 Jun 2022 11:51:41 +0200 Subject: [PATCH 398/534] docs: dockerfile reference moved to buildkit repo Signed-off-by: CrazyMax --- docs/extend/config.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 0d5ae9ae4f..a8b07b749f 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -70,11 +70,11 @@ Config provides the base accessible fields for working with V0 plugin format - **`entrypoint`** *string array* - entrypoint of the plugin, see [`ENTRYPOINT`](../reference/builder.md#entrypoint) + entrypoint of the plugin, see [`ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) - **`workdir`** *string* - workdir of the plugin, see [`WORKDIR`](../reference/builder.md#workdir) + workdir of the plugin, see [`WORKDIR`](https://docs.docker.com/engine/reference/builder/#workdir) - **`network`** *PluginNetwork* From cd30d7d74e9f22fe57c4df4880172327f92ef868 Mon Sep 17 00:00:00 2001 From: Gio d'Amelio Date: Sat, 6 Aug 2022 19:08:29 -0500 Subject: [PATCH 399/534] Update link to libnetwork docs Signed-off-by: Gio d'Amelio --- docs/extend/plugins_network.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 1e218bb4d4..9d92a96400 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -72,7 +72,7 @@ plugin protocol The network driver protocol, in addition to the plugin activation call, is documented as part of libnetwork: -[https://github.com/docker/libnetwork/blob/master/docs/remote.md](https://github.com/docker/libnetwork/blob/master/docs/remote.md). +[https://github.com/moby/moby/blob/master/libnetwork/docs/remote.md](https://github.com/moby/moby/blob/master/libnetwork/docs/remote.md). ## Related Information From b482413fb6e45866504f2bf7ae56a36d240df870 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 18 Aug 2022 10:44:00 +0200 Subject: [PATCH 400/534] complete the "-g" / "--graph" daemon option deprecation These options were soft-deprecated in Docker 17.05, and at the time considered to not be removed. However, with the move towards containerd snapshotters, having these options around adds additional complexity to handle fallbacks for deprecated (and hidden) flags, so completing the deprecation. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2d7602d5cd..72eaa896e1 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -633,13 +633,12 @@ and `docker service scale` in Docker 17.10. **Deprecated In Release: v17.05** +**Removed In Release: v22.06** + The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource configuration and has been replaced with the more descriptive `--data-root` -flag. - -These flags were added before Docker 1.0, so will not be _removed_, only -_hidden_, to discourage their use. +flag. These flags were deprecated and hidden in v17.05, and removed in v22.06. ### Top-level network properties in NetworkSettings From 22f3ec877634ecba990ed4fdb529365ce672c93d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Oct 2022 14:28:12 +0200 Subject: [PATCH 401/534] docs: update deprecation status for "overlay2.override_kernel_check" Commit https://github.com/docker/docker/commit/955c1f881ac94af19c99f0f7d5635e6a574789f2 (v17.12.0) replaced detection of support for multiple lowerdirs (as required by overlay2) to not depend on the kernel version. The `overlay2.override_kernel_check` was still used to print a warning that older kernel versions may not have full support. After this, https://github.com/docker/docker/commit/e226aea280efdc18f887e853b5eefd763b602937 (v20.10, but backported to v19.03.7) removed uses of the option altogether. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 14 ++++++++++++++ docs/reference/commandline/dockerd.md | 11 +---------- 2 files changed, 15 insertions(+), 10 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 72eaa896e1..b96b3deffa 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -69,6 +69,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | +| Disabled | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | - | | Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | | Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | | Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | @@ -464,6 +465,19 @@ format, support for the DAB file format and the top-level docker deploy command (hidden by default in 19.03), will be removed, in favour of `docker stack deploy` using compose files. +### Support for the `overlay2.override_kernel_check` storage option + +**Deprecated in Release: v19.03** +**Disabled in Release: v19.03** + +This daemon configuration option disabled the Linux kernel version check used +to detect if the kernel supported OverlayFS with multiple lower dirs, which is +required for the overlay2 storage driver. Starting with Docker v19.03.7, the +detection was improved to no longer depend on the kernel _version_, so this +option was no longer used. + +Docker v22.06 logs a warning in the daemon logs if this option is set, and +users should remove this option from their daemon configuration. ### AuFS storage driver diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 145eca7393..a989945682 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -818,20 +818,11 @@ $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G #### Overlay2 options -##### `overlay2.override_kernel_check` - -Overrides the Linux kernel version check allowing overlay2. Support for -specifying multiple lower directories needed by overlay2 was added to the -Linux kernel in 4.0.0. However, some older kernel versions may be patched -to add multiple lower directory support for OverlayFS. This option should -only be used after verifying this support exists in the kernel. Applying -this option on a kernel without this support will cause failures on mount. - ##### `overlay2.size` Sets the default max size of the container. It is supported only when the backing fs is `xfs` and mounted with `pquota` mount option. Under these -conditions the user can pass any size less then the backing fs size. +conditions the user can pass any size less than the backing fs size. ###### Example From 04ea76b8f95b7332f4ae059b9e1dd0f499eb48b0 Mon Sep 17 00:00:00 2001 From: Bishal Das Date: Sat, 15 Oct 2022 00:14:16 +0530 Subject: [PATCH 402/534] fixed the plugin command docker-runc Signed-off-by: Bishal Das --- docs/extend/index.md | 49 ++++++++++++++++++++++---------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index 2608d0590b..b0c2f29589 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -14,9 +14,9 @@ keywords: "API, Usage, plugins, documentation, developer" # Docker Engine managed plugin system -* [Installing and using a plugin](index.md#installing-and-using-a-plugin) -* [Developing a plugin](index.md#developing-a-plugin) -* [Debugging plugins](index.md#debugging-plugins) +- [Installing and using a plugin](index.md#installing-and-using-a-plugin) +- [Developing a plugin](index.md#developing-a-plugin) +- [Debugging plugins](index.md#debugging-plugins) Docker Engine's plugin system allows you to install, start, stop, and remove plugins using Docker Engine. @@ -70,7 +70,7 @@ enabled, and use it to create a volume. - It needs access to the `host` network. - It needs the `CAP_SYS_ADMIN` capability, which allows the plugin to run - the `mount` command. + the `mount` command. 2. Check that the plugin is enabled in the output of `docker plugin ls`. @@ -115,6 +115,7 @@ enabled, and use it to create a volume. ``` 6. Remove the volume `sshvolume` + ```console $ docker volume rm sshvolume @@ -126,15 +127,15 @@ remove it, use the `docker plugin remove` command. For other available commands and options, see the [command line reference](https://docs.docker.com/engine/reference/commandline/cli/). - ## Developing a plugin #### The rootfs directory + The `rootfs` directory represents the root filesystem of the plugin. In this example, it was created from a Dockerfile: ->**Note:** The `/run/docker/plugins` directory is mandatory inside of the -plugin's filesystem for docker to communicate with the plugin. +> **Note:** The `/run/docker/plugins` directory is mandatory inside of the +> plugin's filesystem for docker to communicate with the plugin. ```console $ git clone https://github.com/vieux/docker-volume-sshfs @@ -155,19 +156,19 @@ Consider the following `config.json` file. ```json { - "description": "sshFS plugin for Docker", - "documentation": "https://docs.docker.com/engine/extend/plugins/", - "entrypoint": ["/docker-volume-sshfs"], - "network": { - "type": "host" - }, - "interface" : { - "types": ["docker.volumedriver/1.0"], - "socket": "sshfs.sock" - }, - "linux": { - "capabilities": ["CAP_SYS_ADMIN"] - } + "description": "sshFS plugin for Docker", + "documentation": "https://docs.docker.com/engine/extend/plugins/", + "entrypoint": ["/docker-volume-sshfs"], + "network": { + "type": "host" + }, + "interface": { + "types": ["docker.volumedriver/1.0"], + "socket": "sshfs.sock" + }, + "linux": { + "capabilities": ["CAP_SYS_ADMIN"] + } } ``` @@ -187,7 +188,6 @@ After that the plugin `` will show up in `docker plugin ls`. Plugins can be pushed to remote registries with `docker plugin push `. - ## Debugging plugins Stdout of a plugin is redirected to dockerd logs. Such entries have a @@ -226,7 +226,7 @@ plugins. This is specifically useful to collect plugin logs if they are redirected to a file. ```console -$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins list +$ sudo runc --root /run/docker/runtime-runc/plugins.moby list ID PID STATUS BUNDLE CREATED OWNER 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 15806 running /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby-plugins/93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 2018-02-08T21:40:08.621358213Z root @@ -235,14 +235,14 @@ c5bb4b90941efcaccca999439ed06d6a6affdde7081bb34dc84126b57b3e793d 14984 r ``` ```console -$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 cat /var/log/plugin.log +$ sudo runc --root /run/docker/runtime-runc/plugins.moby exec 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 cat /var/log/plugin.log ``` If the plugin has a built-in shell, then exec into the plugin can be done as follows: ```console -$ sudo docker-runc --root /var/run/docker/plugins/runtime-root/moby-plugins exec -t 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 sh +$ sudo runc --root /run/docker/runtime-runc/plugins.moby exec -t 93f1e7dbfe11c938782c2993628c895cf28e2274072c4a346a6002446c949b25 sh ``` #### Using curl to debug plugin socket issues. @@ -253,7 +253,6 @@ docker host to volume and network plugins using curl 7.47.0 to ensure that the plugin is listening on the said socket. For a well functioning plugin, these basic requests should work. Note that plugin sockets are available on the host under `/var/run/docker/plugins/` - ```console $ curl -H "Content-Type: application/json" -XPOST -d '{}' --unix-socket /var/run/docker/plugins/e8a37ba56fc879c991f7d7921901723c64df6b42b87e6a0b055771ecf8477a6d/plugin.sock http:/VolumeDriver.List From c629e4211d02c4b9536d9c8f5a6ed4771f263825 Mon Sep 17 00:00:00 2001 From: Luis Henrique Mulinari Date: Wed, 26 Oct 2022 11:03:07 +0100 Subject: [PATCH 403/534] Fix the max-concurrent-downloads and max-concurrent-uploads configs documentation This fix tries to address issues raised in moby/moby#44346. The max-concurrent-downloads and max-concurrent-uploads limits are applied for the whole engine and not for each pull/push command. Signed-off-by: Luis Henrique Mulinari --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a989945682..bd6b610eda 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -77,8 +77,8 @@ Options: --log-driver string Default driver for container logs (default "json-file") -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") --log-opt map Default log driver options for containers (default map[]) - --max-concurrent-downloads int Set the max concurrent downloads for each pull (default 3) - --max-concurrent-uploads int Set the max concurrent uploads for each push (default 5) + --max-concurrent-downloads int Set the max concurrent downloads (default 3) + --max-concurrent-uploads int Set the max concurrent uploads (default 5) --max-download-attempts int Set the max download attempts for each pull (default 5) --metrics-addr string Set default address and port to serve the metrics api on --mtu int Set the containers network MTU From 9ea458830c7f42a75aaec122714e9d8ff125b891 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 17 Nov 2022 18:33:08 +0100 Subject: [PATCH 404/534] docs: update version to v23.0.0-dev As a stepping-stone towards switching to go modules, we're switching the versioning scheme to be SemVer(ish) instead of CalVer. Note that this currently still is `+incompatible` version, as changing to a version > 1 requires the module to be renamed, but changes in that area will arrive in a future release. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 160 ++++++++++++++++++++++----------------------- 1 file changed, 80 insertions(+), 80 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index b96b3deffa..8031ea91a8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -48,66 +48,66 @@ The table below provides an overview of the current status of deprecated feature alternatives. In such cases, a warning may be printed, and users should not rely on this feature. -| Status | Feature | Deprecated | Remove | -|------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| -| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v22.06 | - | -| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v22.06 | - | -| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v22.06 | -| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v22.06 | -| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | -| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v22.06 | -| Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | -| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v22.06 | -| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v22.06 | -| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v22.06 | -| Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | -| Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | -| Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | -| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | -| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v22.06 | -| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | -| Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | -| Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | -| Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | -| Disabled | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | - | -| Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | -| Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | -| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | -| Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | -| Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | -| Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | -| Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | -| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | -| Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | -| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | -| Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | -| Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | -| Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | -| Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - | -| Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 | -| Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 | -| Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 | -| Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | -| Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | -| Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | -| Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | -| Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | -| Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | -| Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 | -| Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 | -| Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 | -| Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 | -| Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 | -| Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 | -| Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 | -| Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | -| Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +| Status | Feature | Deprecated | Remove | +|------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| +| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | +| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | +| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0.0 | +| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0.0 | +| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | +| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v23.0.0 | +| Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | +| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v23.0.0 | +| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v23.0.0 | +| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v23.0.0 | +| Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | +| Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | +| Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | +| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | +| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0.0 | +| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | +| Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | +| Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | +| Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | +| Disabled | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | - | +| Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | +| Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | +| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | +| Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | +| Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | +| Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | +| Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | +| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | +| Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | +| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | +| Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | +| Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | +| Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | +| Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - | +| Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 | +| Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 | +| Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 | +| Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | +| Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | +| Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | +| Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | +| Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | +| Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | +| Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 | +| Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 | +| Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 | +| Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 | +| Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 | +| Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 | +| Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 | +| Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | +| Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | ### Legacy builder for Linux images -**Deprecated in Release: v22.06** +**Deprecated in Release: v23.0.0** -Docker v22.06 now uses BuildKit by default to build Linux images, and uses the +Docker v23.0.0 now uses BuildKit by default to build Linux images, and uses the [Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for `docker build`. With this change, `docker build` now exposes all advanced features that BuildKit provides and which were previously only available through the @@ -136,14 +136,14 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co ### Legacy builder fallback -**Deprecated in Release: v22.06** +**Deprecated in Release: v23.0.0** -[Docker v22.06 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), +[Docker v23.0.0 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), which requires the Buildx component to build images with BuildKit. There may be situations where the Buildx component is not available, and BuildKit cannot be used. -To provide a smooth transition to BuildKit as the default builder, Docker v22.06 +To provide a smooth transition to BuildKit as the default builder, Docker v23.0.0 has an automatic fallback for some situations, or produces an error to assist users to resolve the problem. @@ -184,7 +184,7 @@ be possible in a future release. **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** Use of encrypted TLS private keys has been deprecated, and has been removed. Golang has deprecated support for legacy PEM encryption (as specified in @@ -199,7 +199,7 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from @@ -260,7 +260,7 @@ major release. ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after @@ -283,7 +283,7 @@ When using cgroups v2, the `--blkio-weight` options are implemented using ### Kernel memory limit **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). @@ -311,7 +311,7 @@ take no effect. ### Classic Swarm and overlay networks using cluster store **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -320,7 +320,7 @@ networks using an external key/value store. The corresponding`--cluster-advertis ### Support for legacy `~/.dockercfg` configuration files **Deprecated in Release: v20.10** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -340,13 +340,13 @@ notifying the user that the legacy file is present, but ignored. **Deprecated in Release: v19.03** -**Removed in Release: v22.06** +**Removed in Release: v23.0.0** The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` field in the CLI configuration file are deprecated. Experimental features are enabled by default, and these configuration options are no longer functional. -Starting with v22.06, the Docker CLI no longer prints `Experimental` for the client +Starting with v23.0.0, the Docker CLI no longer prints `Experimental` for the client in the output of `docker version`, and the field has been removed from the JSON format. @@ -476,13 +476,13 @@ required for the overlay2 storage driver. Starting with Docker v19.03.7, the detection was improved to no longer depend on the kernel _version_, so this option was no longer used. -Docker v22.06 logs a warning in the daemon logs if this option is set, and +Docker v23.0.0 logs a warning in the daemon logs if this option is set, and users should remove this option from their daemon configuration. ### AuFS storage driver **Deprecated in Release: v19.03** -**Disabled by default in Release: v22.06** +**Disabled by default in Release: v23.0.0** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -498,11 +498,11 @@ is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. -#### Disabled by default in v22.06 +#### Disabled by default in v23.0.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.06 release, the Docker Engine will +existing installations. Starting with the v23.0.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -519,7 +519,7 @@ different storage driver. ### Legacy "overlay" storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v22.06** +**Disabled by default in Release: v23.0.0** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -532,11 +532,11 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. -#### Disabled by default in v22.06 +#### Disabled by default in v23.0.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.06 release, the Docker Engine will +existing installations. Starting with the v23.0.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -553,7 +553,7 @@ different storage driver. ### Device mapper storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v22.06** +**Disabled by default in Release: v23.0.0** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are @@ -567,11 +567,11 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -#### Disabled by default in v22.06 +#### Disabled by default in v23.0.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v22.06 release, the Docker Engine will +existing installations. Starting with the v23.0.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -647,12 +647,12 @@ and `docker service scale` in Docker 17.10. **Deprecated In Release: v17.05** -**Removed In Release: v22.06** +**Removed In Release: v23.0.0** The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource configuration and has been replaced with the more descriptive `--data-root` -flag. These flags were deprecated and hidden in v17.05, and removed in v22.06. +flag. These flags were deprecated and hidden in v17.05, and removed in v23.0.0. ### Top-level network properties in NetworkSettings @@ -730,7 +730,7 @@ The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` if it is formatted with the `ftype=0` option. -Support for these setups has been removed, and Docker v22.06 and up now fails to +Support for these setups has been removed, and Docker v23.0.0 and up now fails to start when attempting to use the `overlay2` or `overlay` storage driver on a backing filesystem without `d_type` support. From 80bd876cec8ea9d3458d2cbf5c1f0bd790cbe978 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 6 Jun 2022 14:22:38 +0200 Subject: [PATCH 405/534] docs/extend: remove note about first supported version Docker v1.12 is really old, so no need to continue including this in the docs. Also reformatted a markdown table. Signed-off-by: Sebastiaan van Stijn --- docs/extend/config.md | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index a8b07b749f..6af92c55b4 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -1,5 +1,5 @@ --- -description: "How develop and use a plugin with the managed plugin system" +description: "How to develop and use a plugin with the managed plugin system" keywords: "API, Usage, plugins, documentation, developer" --- @@ -15,17 +15,14 @@ keywords: "API, Usage, plugins, documentation, developer" # Plugin Config Version 1 of Plugin V2 -This document outlines the format of the V0 plugin configuration. The plugin -config described herein was introduced in the Docker daemon in the [v1.12.0 -release](https://github.com/docker/docker/commit/f37117045c5398fd3dca8016ea8ca0cb47e7312b). +This document outlines the format of the V0 plugin configuration. Plugin configs describe the various constituents of a docker plugin. Plugin configs can be serialized to JSON format with the following media types: -Config Type | Media Type -------------- | ------------- -config | "application/vnd.docker.plugin.v1+json" - +| Config Type | Media Type | +|-------------|-----------------------------------------| +| config | "application/vnd.docker.plugin.v1+json" | ## *Config* Field Descriptions From 1da5378601657fb0d6d524ee539ee98785889623 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 6 Jan 2023 16:55:50 +0100 Subject: [PATCH 406/534] docs: deprecated: fix link to Docker Desktop 3.2.0 release notes Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 8031ea91a8..e9af6ce5a6 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -123,7 +123,7 @@ This release marks the beginning of the deprecation cycle of the classic ("legac builder for Linux images. No active development will happen on the classic builder (except for bugfixes). BuildKit development started five Years ago, left the "experimental" phase since Docker 18.09, and is already the default builder for -[Docker Desktop](https://docs.docker.com/desktop/mac/release-notes/3.x/#docker-desktop-320). +[Docker Desktop](https://docs.docker.com/desktop/previous-versions/3.x-mac/#docker-desktop-320). While we're comfortable that BuildKit is stable for general use, there may be some changes in behavior. If you encounter issues with BuildKit, we encourage you to report issues in the [BuildKit issue tracker on GitHub](https://github.com/moby/buildkit/){:target="_blank" rel="noopener" class="_"} From 0d3f4962f48ee42dc9a1a7204110aff1b2a7e29d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 9 Jan 2023 14:31:33 +0100 Subject: [PATCH 407/534] deprecation: mark btrfs driver as deprecated for CentOS 7 and RHEL7 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index e9af6ce5a6..57ebaa923f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,6 +52,7 @@ The table below provides an overview of the current status of deprecated feature |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | +| Deprecated | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v23.0.0 | - | | Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0.0 | | Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0.0 | | Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | @@ -180,6 +181,18 @@ Be aware that the [classic builder is deprecated](#legacy-builder-for-linux-imag so both the automatic fallback and opting-out of using BuildKit will no longer be possible in a future release. +### Btrfs storage driver on CentOS 7 and RHEL 7 + +**Deprecated in Release: v23.0.0** + +The `btrfs` storage driver on CentOS and RHEL was provided as a technology preview +by CentOS and RHEL, but has been deprecated since the [Red Hat Enterprise Linux 7.4 release](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-btrfs), +and removed in CentOS 8 and RHEL 8. Users of the `btrfs` storage driver on CentOS +are recommended to migrate to a different storage driver, such as `overlay2`, which +is now the default storage driver. Docker 23.0 continues to provide the `btrfs` +storage driver to allow users to migrate to an alternative driver. The next release +of Docker will no longer provide this driver. + ### Support for encrypted TLS private keys **Deprecated in Release: v20.10** From d41bff29285e634c43eb879c8452fa4a772b2e8f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 9 Jan 2023 19:00:35 +0100 Subject: [PATCH 408/534] deprecated: update deprecation for btrfs on CentOS/RHEL 7 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 57ebaa923f..5b81f0a994 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,7 +52,7 @@ The table below provides an overview of the current status of deprecated feature |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | -| Deprecated | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v23.0.0 | - | +| Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0.0 | | Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0.0 | | Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0.0 | | Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | @@ -183,7 +183,7 @@ be possible in a future release. ### Btrfs storage driver on CentOS 7 and RHEL 7 -**Deprecated in Release: v23.0.0** +**Removed in Release: v23.0.0** The `btrfs` storage driver on CentOS and RHEL was provided as a technology preview by CentOS and RHEL, but has been deprecated since the [Red Hat Enterprise Linux 7.4 release](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-btrfs), From ae983cefd7d557f0365fb535add3746b5a6cf388 Mon Sep 17 00:00:00 2001 From: Bjorn Neergaard Date: Thu, 9 Feb 2023 23:26:02 -0700 Subject: [PATCH 409/534] docs: drop dated comments about graphdrivers Signed-off-by: Bjorn Neergaard --- docs/reference/commandline/dockerd.md | 54 ++------------------------- 1 file changed, 4 insertions(+), 50 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index bd6b610eda..ba7d9be8ab 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -295,57 +295,11 @@ $ docker -H tcp://127.0.0.1:2375 pull ubuntu On Linux, the Docker daemon has support for several different image layer storage drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay`, `overlay2`, and `fuse-overlayfs`. -The `aufs` driver is the oldest, but is based on a Linux kernel patch-set that -is unlikely to be merged into the main kernel. These are also known to cause -some serious kernel crashes. However `aufs` allows containers to share -executable and shared library memory, so is a useful choice when running -thousands of containers with the same program or libraries. +`overlay2` is the preferred storage driver for all currently supported Linux distributions, +and is selected by default. Unless users have a strong reason to prefer another storage driver, +`overlay2` should be used. -The `devicemapper` driver uses thin provisioning and Copy on Write (CoW) -snapshots. For each devicemapper graph location – typically -`/var/lib/docker/devicemapper` – a thin pool is created based on two block -devices, one for data and one for metadata. By default, these block devices -are created automatically by using loopback mounts of automatically created -sparse files. Refer to [Devicemapper options](#devicemapper-options) below -for a way how to customize this setup. -[~jpetazzo/Resizing Docker containers with the Device Mapper plugin](https://jpetazzo.github.io/2014/01/29/docker-device-mapper-resize/) -article explains how to tune your existing setup without the use of options. - -The `btrfs` driver is very fast for `docker build` - but like `devicemapper` -does not share executable memory between devices. Use -`dockerd --storage-driver btrfs --data-root /mnt/btrfs_partition`. - -The `zfs` driver is probably not as fast as `btrfs` but has a longer track record -on stability. Thanks to `Single Copy ARC` shared blocks between clones will be -cached only once. Use `dockerd -s zfs`. To select a different zfs filesystem -set `zfs.fsname` option as described in [ZFS options](#zfs-options). - -The `overlay` is a very fast union filesystem. It is now merged in the main -Linux kernel as of [3.18.0](https://lkml.org/lkml/2014/10/26/137). `overlay` -also supports page cache sharing, this means multiple containers accessing -the same file can share a single page cache entry (or entries), it makes -`overlay` as efficient with memory as `aufs` driver. Call `dockerd -s overlay` -to use it. - -The `overlay2` uses the same fast union filesystem but takes advantage of -[additional features](https://lkml.org/lkml/2015/2/11/106) added in Linux -kernel 4.0 to avoid excessive inode consumption. Call `dockerd -s overlay2` -to use it. - -> **Note** -> -> The `overlay` storage driver can cause excessive inode consumption (especially -> as the number of images grows). We recommend using the `overlay2` storage -> driver instead. - - -> **Note** -> -> Both `overlay` and `overlay2` are currently unsupported on `btrfs` -> or any Copy on Write filesystem and should only be used over `ext4` partitions. - -The `fuse-overlayfs` driver is similar to `overlay2` but works in userspace. -The `fuse-overlayfs` driver is expected to be used for [Rootless mode](https://docs.docker.com/engine/security/rootless/). +You can find out more about storage drivers and how to select one in [Select a storage driver](https://docs.docker.com/storage/storagedriver/select-storage-driver/). On Windows, the Docker daemon only supports the `windowsfilter` storage driver. From 8da9b7408b5aae870ad719583f66b1a0a1d948e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Thu, 2 Mar 2023 17:27:05 +0100 Subject: [PATCH 410/534] docs: Deprecate buildkit's build information MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Paweł Gronowski --- docs/deprecated.md | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 5b81f0a994..affa938dd4 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| +| Deprecated | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v23.1.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | | Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0.0 | @@ -104,6 +105,17 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Buildkit build information + +**Deprecated in Release: v23.0.0** + +[Build information](https://github.com/moby/buildkit/blob/v0.11/docs/buildinfo.md) +structures have been introduced in [BuildKit v0.10.0](https://github.com/moby/buildkit/releases/tag/v0.10.0) +and are generated with build metadata that allows you to see all the sources +(images, git repositories) that were used by the build with their exact +versions and also the configuration that was passed to the build. This +information is also embedded into the image configuration if one is generated. + ### Legacy builder for Linux images **Deprecated in Release: v23.0.0** From 36897ec6f530f4b2edd3cf0d385c5a64d8bcec11 Mon Sep 17 00:00:00 2001 From: Jakub Panek Date: Sat, 4 Mar 2023 13:53:25 +0100 Subject: [PATCH 411/534] connhelper: Allow socket path when using SSH Signed-off-by: Jakub Panek --- docs/reference/commandline/dockerd.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ba7d9be8ab..12be4b9e3d 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -213,6 +213,7 @@ precedence over `HTTP_PROXY`. The Docker client supports connecting to a remote daemon via SSH: ```console +$ docker -H ssh://me@example.com:22/var/run/docker.sock ps $ docker -H ssh://me@example.com:22 ps $ docker -H ssh://me@example.com ps $ docker -H ssh://example.com ps From 7f5b999257d0e0bdbdd93a45741e5485a0dbf34a Mon Sep 17 00:00:00 2001 From: Jake Stokes Date: Sat, 25 Mar 2023 18:14:50 +0000 Subject: [PATCH 412/534] Fix grammar in docs by adding omitted word Edited second paragraph under ### Daemon configuration file to change "regardless their value" to "regardless of their value" Signed-off-by: Jake Stokes --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ba7d9be8ab..a8a3cf2dee 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1143,7 +1143,7 @@ of the flag name, e.g., `labels` for the `label` flag. The options set in the configuration file must not conflict with options set via flags. The docker daemon fails to start if an option is duplicated between -the file and the flags, regardless their value. We do this to avoid +the file and the flags, regardless of their value. We do this to avoid silently ignore changes introduced in configuration reloads. For example, the daemon fails to start if you set daemon labels in the configuration file and also set daemon labels via the `--label` flag. From 9aaa3d732365c7e96f8c7c5bb2a10d9e13b8208b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Apr 2023 12:11:05 +0200 Subject: [PATCH 413/534] docs: deprecated: fix version (23.1.0 -> 24.0.0) The next release will be "v24.0.0", not "v23.1.0" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index affa938dd4..1efe015a69 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,7 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| -| Deprecated | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v23.1.0 | +| Deprecated | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v24.0.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | | Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0.0 | From 4913aebf8ceb325e0ada7c1f90d1c248543249c6 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Apr 2023 18:12:08 +0200 Subject: [PATCH 414/534] docs/reference: remove mention of DOCKER_NOWARN_KERNEL_VERSION Support for this environment variable was removed in docker 23.0 in https://github.com/moby/moby/commit/1240f8b41d34abe9252b882bddd42e140b62b83b From that patch: > All regular, non-EOL Linux distros now come with more recent kernels > out of the box. There may still be users trying to run on kernel 3.10 > or older (some embedded systems, e.g.), but those should be a rare > exception, which we don't have to take into account. > > This patch removes the kernel version check on Linux, and the corresponding > DOCKER_NOWARN_KERNEL_VERSION environment that was there to skip this > check. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a8a3cf2dee..b8d86360a8 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -130,8 +130,6 @@ For easy reference, the following list of environment variables are supported by the `dockerd` command line: * `DOCKER_DRIVER` The graph driver to use. -* `DOCKER_NOWARN_KERNEL_VERSION` Prevent warnings that your Linux kernel is - unsuitable for Docker. * `DOCKER_RAMDISK` If set this will disable 'pivot_root'. * `DOCKER_TMPDIR` Location for temporary Docker files. * `MOBY_DISABLE_PIGZ` Do not use [`unpigz`](https://linux.die.net/man/1/pigz) to From 8c0352e5ce9fcf40e9243364b49e8e4913bbe063 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Apr 2023 18:27:40 +0200 Subject: [PATCH 415/534] docs/reference: dockerd: use table for env-vars Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b8d86360a8..a5e4f0617c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -129,11 +129,17 @@ to [the `daemon.json` file](#daemon-configuration-file). For easy reference, the following list of environment variables are supported by the `dockerd` command line: -* `DOCKER_DRIVER` The graph driver to use. -* `DOCKER_RAMDISK` If set this will disable 'pivot_root'. -* `DOCKER_TMPDIR` Location for temporary Docker files. -* `MOBY_DISABLE_PIGZ` Do not use [`unpigz`](https://linux.die.net/man/1/pigz) to - decompress layers in parallel when pulling images, even if it is installed. +| Variable | Description | +|:--------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | +| `DOCKER_DRIVER` | The storage driver to use. | +| `DOCKER_RAMDISK` | If set this disables 'pivot_root'. | +| `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | +| `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | +| `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | +| `HTTPS_PROXY` | Proxy URL for HTTPS requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | +| `MOBY_DISABLE_PIGZ` | Disables the use of [`unpigz`](https://linux.die.net/man/1/pigz) to decompress layers in parallel when pulling images, even if it is installed. | | +| `NO_PROXY` | Comma-separated values specifying hosts that should be excluded from proxying. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | ## Examples From 65c268138154463952238ec0799aef2cf1020e16 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Apr 2023 18:41:27 +0200 Subject: [PATCH 416/534] docs/reference: dockerd: update seccomp flag description Update the description for the changes made in; https://github.com/moby/moby/commit/68e96f88ee1598563a66a1f53b8844291423fc88 Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index a5e4f0617c..1358469405 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -90,7 +90,7 @@ Options: --raw-logs Full timestamps without ANSI coloring --registry-mirror list Preferred registry mirror --rootless Enable rootless mode; typically used with RootlessKit - --seccomp-profile string Path to seccomp profile + --seccomp-profile string Path to seccomp profile. Use "unconfined" to disable the default seccomp profile (default "builtin") --selinux-enabled Enable selinux support --shutdown-timeout int Set the default shutdown timeout (default 15) -s, --storage-driver string Storage driver to use From a709a5aec270ac66c7a26fc52d783f2a51e988af Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Apr 2023 18:45:20 +0200 Subject: [PATCH 417/534] docs/reference: dockerd: remove default oom-score-adjust The daemon no longer adjusts its oom-score by default; see - https://github.com/moby/moby/commit/2b8e68ef06ca3daf2b0e17e51ba72999773b6270 - https://github.com/moby/moby/commit/cf7a5be0f2ee8bb4a3f7961682e13b89bc625299 Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1358469405..79a78539d5 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -85,7 +85,7 @@ Options: --network-control-plane-mtu int Network Control plane MTU (default 1500) --no-new-privileges Set no-new-privileges by default for new containers --node-generic-resource list Advertise user-defined resource - --oom-score-adjust int Set the oom_score_adj for the daemon (default -500) + --oom-score-adjust int Set the oom_score_adj for the daemon -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring --registry-mirror list Preferred registry mirror @@ -1259,7 +1259,7 @@ This is a full example of the allowed configuration options on Linux: "NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2" ], - "oom-score-adjust": -500, + "oom-score-adjust": 0, "pidfile": "", "raw-logs": false, "registry-mirrors": [], From 392b34fcec63d36ae32cc58c15d4fd8e5a28cfc5 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 2 Apr 2022 14:07:51 +0200 Subject: [PATCH 418/534] docs: update dockerd usage output for new proxy-options Adds documentation for the options that were added in https://github.com/moby/moby/commit/427c7cc5f86364466c7173e8ca59b97c3876471d Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 40 +++++++++++++++++++++++---- 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 79a78539d5..2413dd1bfc 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -18,7 +18,7 @@ redirect_from: # daemon ```markdown -Usage: dockerd COMMAND +Usage: dockerd [OPTIONS] A self-sufficient runtime for containers. @@ -35,14 +35,14 @@ Options: --containerd-namespace string Containerd namespace to use (default "moby") --containerd-plugins-namespace string Containerd namespace to use for plugins (default "plugins.moby") --cpu-rt-period int Limit the CPU real-time period in microseconds for the - parent cgroup for all containers + parent cgroup for all containers (not supported with cgroups v2) --cpu-rt-runtime int Limit the CPU real-time runtime in microseconds for the - parent cgroup for all containers + parent cgroup for all containers (not supported with cgroups v2) --cri-containerd start containerd with cri --data-root string Root directory of persistent Docker state (default "/var/lib/docker") -D, --debug Enable debug mode --default-address-pool pool-options Default address pools for node specific local networks - --default-cgroupns-mode string Default mode for containers cgroup namespace ("host" | "private") (default "host") + --default-cgroupns-mode string Default mode for containers cgroup namespace ("host" | "private") (default "private") --default-gateway ip Container default gateway IPv4 address --default-gateway-v6 ip Container default gateway IPv6 address --default-ipc-mode string Default mode for containers ipc ("shareable" | "private") (default "private") @@ -62,6 +62,8 @@ Options: -H, --host list Daemon socket(s) to connect to --host-gateway-ip ip IP address that the special 'host-gateway' string in --add-host resolves to. Defaults to the IP address of the default bridge + --http-proxy string HTTP proxy URL to use for outgoing traffic + --https-proxy string HTTPS proxy URL to use for outgoing traffic --icc Enable inter-container communication (default true) --init Run an init in the container to forward signals and reap processes --init-path string Path to the docker-init binary @@ -69,8 +71,8 @@ Options: --ip ip Default IP when binding container ports (default 0.0.0.0) --ip-forward Enable net.ipv4.ip_forward (default true) --ip-masq Enable IP masquerading (default true) + --ip6tables Enable addition of ip6tables rules (experimental) --iptables Enable addition of iptables rules (default true) - --ip6tables Enable addition of ip6tables rules (default false) --ipv6 Enable IPv6 networking --label list Set key=value labels to the daemon --live-restore Enable live restore of docker when containers are still running @@ -81,9 +83,10 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads (default 5) --max-download-attempts int Set the max download attempts for each pull (default 5) --metrics-addr string Set default address and port to serve the metrics api on - --mtu int Set the containers network MTU + --mtu int Set the containers network MTU (default 1500) --network-control-plane-mtu int Network Control plane MTU (default 1500) --no-new-privileges Set no-new-privileges by default for new containers + --no-proxy string Comma-separated list of hosts or IP addresses for which the proxy is skipped --node-generic-resource list Advertise user-defined resource --oom-score-adjust int Set the oom_score_adj for the daemon -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") @@ -143,6 +146,28 @@ by the `dockerd` command line: ## Examples +### Proxy configuration + +> **Note** +> +> Refer to the [Docker Desktop manual](https://docs.docker.com/desktop/networking/#httphttps-proxy-support) +> if you are running [Docker Desktop](https://docs.docker.com/desktop/). + +If you are behind an HTTP proxy server, for example in corporate settings, +you may have to configure the Docker daemon to use the proxy server for +operations such as pulling and pushing images. The daemon can be configured +in three ways: + +1. Using environment variables (`HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`). +2. Using the "http-proxy", "https-proxy", and "no-proxy" fields in the + [daemon configuration file](#daemon-configuration-file) (Docker Engine 23.0 or newer). +3. Using the `--http-proxy`, `--https-proxy`, and `--no-proxy` command-line + options. (Docker Engine 23.0 or newer). + +The command-line and configuration file options take precedence over environment +variables. Refer to [control and configure Docker with systemd](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) +to set these environment variables on a host using `systemd`. + ### Daemon socket option The Docker daemon can listen for [Docker Engine API](https://docs.docker.com/engine/api/) @@ -1226,6 +1251,9 @@ This is a full example of the allowed configuration options on Linux: "fixed-cidr-v6": "", "group": "", "hosts": [], + "http-proxy": "http://proxy.example.com:80", + "https-proxy": "https://proxy.example.com:443", + "no-proxy": "*.test.example.com,.example.org", "icc": false, "init": false, "init-path": "/usr/libexec/docker-init", From c9cf6d52b40d17edb85c381114e365c8ffd03a5d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 14 Apr 2023 03:03:39 +0200 Subject: [PATCH 419/534] docs: update deprecation status of BuildKit "buildInfo" for v24.0.0 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1efe015a69..a27c6f7a0f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,7 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| -| Deprecated | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v24.0.0 | +| Removed | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v24.0.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | | Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0.0 | @@ -108,6 +108,7 @@ The table below provides an overview of the current status of deprecated feature ### Buildkit build information **Deprecated in Release: v23.0.0** +**Removed in Release: v24.0.0** [Build information](https://github.com/moby/buildkit/blob/v0.11/docs/buildinfo.md) structures have been introduced in [BuildKit v0.10.0](https://github.com/moby/buildkit/releases/tag/v0.10.0) From 9d4f988787ea897dca241d292d4b5b7637f94722 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 15 Apr 2023 14:43:35 +0200 Subject: [PATCH 420/534] docs: deprecation: OOM-score adjust for the daemon The `oom-score-adjust` option was added to prevent the daemon from being OOM-killed before other processes. This option was mostly added as a convenience, as running the daemon as a systemd unit was not yet common. Having the daemon set its own limits is not best-practice, and something better handled by the process-manager starting the daemon. Docker v20.10 and newer no longer adjust the daemon's OOM score by default, instead setting the OOM-score to the systemd unit (OOMScoreAdjust) that's shipped with the packages. Users currently depending on this feature are recommended to adjust the daemon's OOM score using systemd or through other means, when starting the daemon. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index a27c6f7a0f..3248312275 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| +| Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0.0 | v25.0.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v24.0.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | @@ -105,6 +106,26 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### OOM-score adjust for the daemon + +**Deprecated in Release: v24.0.0** +**Target For Removal In Release: v25.0.0** + +The `oom-score-adjust` option was added to prevent the daemon from being +OOM-killed before other processes. This option was mostly added as a +convenience, as running the daemon as a systemd unit was not yet common. + +Having the daemon set its own limits is not best-practice, and something +better handled by the process-manager starting the daemon. + +Docker v20.10 and newer no longer adjust the daemon's OOM score by default, +instead setting the OOM-score to the systemd unit (OOMScoreAdjust) that's +shipped with the packages. + +Users currently depending on this feature are recommended to adjust the +daemon's OOM score using systemd or through other means, when starting +the daemon. + ### Buildkit build information **Deprecated in Release: v23.0.0** From 4077cf62394aa0dc1d86f17b2c2d46e0a6bcd7fe Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 21 Apr 2023 00:06:41 +0200 Subject: [PATCH 421/534] docs: update deprecation status for overlay2.override_kernel_check This opton has been removed in Docker Engine v24.0.0 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 3248312275..7884ccbd01 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -72,7 +72,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | -| Disabled | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | - | +| Removed | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | v24.0.0 | | Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | | Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | | Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | @@ -515,7 +515,7 @@ using compose files. ### Support for the `overlay2.override_kernel_check` storage option **Deprecated in Release: v19.03** -**Disabled in Release: v19.03** +**Removed in Release: v24.0.0** This daemon configuration option disabled the Linux kernel version check used to detect if the kernel supported OverlayFS with multiple lower dirs, which is @@ -523,9 +523,6 @@ required for the overlay2 storage driver. Starting with Docker v19.03.7, the detection was improved to no longer depend on the kernel _version_, so this option was no longer used. -Docker v23.0.0 logs a warning in the daemon logs if this option is set, and -users should remove this option from their daemon configuration. - ### AuFS storage driver **Deprecated in Release: v19.03** From 7cbb19b6d6fd7fe5716edbe93a1a2a13a833f6f3 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 20 Apr 2023 23:47:37 +0200 Subject: [PATCH 422/534] docs: update deprecation status of "overlay" and "AuFS" These drivers have been removed in Docker Engine 24.0.0 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7884ccbd01..dd69b278a8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -73,8 +73,8 @@ The table below provides an overview of the current status of deprecated feature | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | | Removed | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | v24.0.0 | -| Disabled | [AuFS storage driver](#aufs-storage-driver) | v19.03 | - | -| Disabled | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | - | +| Removed | [AuFS storage driver](#aufs-storage-driver) | v19.03 | v24.0.0 | +| Removed | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | v24.0.0 | | Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | | Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | | Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | @@ -526,7 +526,7 @@ option was no longer used. ### AuFS storage driver **Deprecated in Release: v19.03** -**Disabled by default in Release: v23.0.0** +**Removed in Release: v24.0.0** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -560,10 +560,10 @@ To continue using the storage driver, update the daemon configuration to use explicitly use the given storage driver. Users are encouraged to migrate to different storage driver. -### Legacy "overlay" storage driver +### Legacy overlay storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v23.0.0** +**Removed in Release: v24.0.0** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive From 8cde8ac9094c638448e6b2246c57a22b2e3a301f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 10 May 2023 10:46:48 +0200 Subject: [PATCH 423/534] cli/config: remove warning for deprecated ~/.dockercfg file The `~/.dockercfg` file was replaced by `~/.docker/config.json` in 2015 (github.com/docker/docker/commit/18c9b6c6455f116ae59cde8544413b3d7d294a5e). Commit b83bc67136e1edce2bdc225863af83d45d5472c9 (v23.0.0, but backported to v20.10) added a warning if no "current" config file was found but a legacy file was, and if the CLI would fall back to using the deprecated file. Commit 2ccfdfa231bbf5194d222e8d76e829568f51ab99 removed support for the legacy file, but kept a warning in place if a legacy file was in place, and now ignored. This patch removes the warning as well, fully deprecating the legacy `~/.dockercfg` file. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index dd69b278a8..cadb37bbc8 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -380,9 +380,6 @@ Given that the old file format encourages insecure storage of credentials Docker v1.7.0 has created this file, support for this file, and its format has been removed. -A warning is printed in situations where the CLI would fall back to the old file, -notifying the user that the legacy file is present, but ignored. - ### Configuration options for experimental CLI features **Deprecated in Release: v19.03** From 676d7662edb9a34ef976fc3b574f9f16f80b6310 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 18 May 2023 22:40:45 +0200 Subject: [PATCH 424/534] docs/deprecated: remove .patch release from deprecation status commit 214ab7bf9d5d3b84e15d4e6520533710277aaf8e removed the patch releases from the deprecation doc, but when we switched to the SemVer(ish) format for v23.0, we accidentally added them back. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 176 ++++++++++++++++++++++----------------------- 1 file changed, 88 insertions(+), 88 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index cadb37bbc8..6a4e77e653 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -48,68 +48,68 @@ The table below provides an overview of the current status of deprecated feature alternatives. In such cases, a warning may be printed, and users should not rely on this feature. -| Status | Feature | Deprecated | Remove | -|------------|------------------------------------------------------------------------------------------------------------------------------------|------------|---------| -| Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0.0 | v25.0.0 | -| Removed | [Buildkit build information](#buildkit-build-information) | v23.0.0 | v24.0.0 | -| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0.0 | - | -| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0.0 | - | -| Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0.0 | -| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0.0 | -| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0.0 | -| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | -| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v23.0.0 | -| Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | -| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v23.0.0 | -| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v23.0.0 | -| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v23.0.0 | -| Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | -| Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | -| Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | -| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | -| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0.0 | -| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | -| Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | -| Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | -| Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | -| Removed | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | v24.0.0 | -| Removed | [AuFS storage driver](#aufs-storage-driver) | v19.03 | v24.0.0 | -| Removed | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | v24.0.0 | -| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | -| Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | -| Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | -| Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | -| Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | -| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | -| Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | -| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | -| Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | -| Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | -| Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | -| Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - | -| Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 | -| Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 | -| Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 | -| Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | -| Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | -| Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | -| Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | -| Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | -| Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | -| Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 | -| Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 | -| Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 | -| Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 | -| Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 | -| Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 | -| Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 | -| Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | -| Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +| Status | Feature | Deprecated | Remove | +|------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | +| Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | +| Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | +| Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0 | - | +| Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0 | +| Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0 | +| Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0 | +| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | +| Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v23.0 | +| Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | +| Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v23.0 | +| Removed | [Classic Swarm and overlay networks using external key/value stores](#classic-swarm-and-overlay-networks-using-cluster-store) | v20.10 | v23.0 | +| Removed | [Support for the legacy `~/.dockercfg` configuration file for authentication](#support-for-legacy-dockercfg-configuration-files) | v20.10 | v23.0 | +| Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | +| Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | +| Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | +| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | +| Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0 | +| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | +| Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | +| Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | +| Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | +| Removed | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | v24.0 | +| Removed | [AuFS storage driver](#aufs-storage-driver) | v19.03 | v24.0 | +| Removed | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | v24.0 | +| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | +| Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | +| Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | +| Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | +| Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | +| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | +| Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | +| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | +| Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | +| Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | +| Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | +| Deprecated | [`MAINTAINER` in Dockerfile](#maintainer-in-dockerfile) | v1.13 | - | +| Deprecated | [API calls without a version](#api-calls-without-a-version) | v1.13 | v17.12 | +| Removed | [Backing filesystem without `d_type` support for overlay/overlay2](#backing-filesystem-without-d_type-support-for-overlayoverlay2) | v1.13 | v17.12 | +| Removed | [`--automated` and `--stars` flags on `docker search`](#--automated-and---stars-flags-on-docker-search) | v1.12 | v20.10 | +| Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | +| Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | +| Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | +| Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | +| Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | +| Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | +| Removed | [`--before` and `--since` flags on `docker ps`](#--before-and---since-flags-on-docker-ps) | v1.10 | v1.12 | +| Removed | [Driver-specific log tags](#driver-specific-log-tags) | v1.9 | v1.12 | +| Removed | [Docker Content Trust `ENV` passphrase variables name change](#docker-content-trust-env-passphrase-variables-name-change) | v1.9 | v1.12 | +| Removed | [`/containers/(id or name)/copy` endpoint](#containersid-or-namecopy-endpoint) | v1.8 | v1.12 | +| Removed | [LXC built-in exec driver](#lxc-built-in-exec-driver) | v1.8 | v1.10 | +| Removed | [Old Command Line Options](#old-command-line-options) | v1.8 | v1.10 | +| Removed | [`--api-enable-cors` flag on `dockerd`](#--api-enable-cors-flag-on-dockerd) | v1.6 | v17.09 | +| Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | +| Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | ### OOM-score adjust for the daemon -**Deprecated in Release: v24.0.0** -**Target For Removal In Release: v25.0.0** +**Deprecated in Release: v24.0** +**Target For Removal In Release: v25.0** The `oom-score-adjust` option was added to prevent the daemon from being OOM-killed before other processes. This option was mostly added as a @@ -128,8 +128,8 @@ the daemon. ### Buildkit build information -**Deprecated in Release: v23.0.0** -**Removed in Release: v24.0.0** +**Deprecated in Release: v23.0** +**Removed in Release: v24.0** [Build information](https://github.com/moby/buildkit/blob/v0.11/docs/buildinfo.md) structures have been introduced in [BuildKit v0.10.0](https://github.com/moby/buildkit/releases/tag/v0.10.0) @@ -140,9 +140,9 @@ information is also embedded into the image configuration if one is generated. ### Legacy builder for Linux images -**Deprecated in Release: v23.0.0** +**Deprecated in Release: v23.0** -Docker v23.0.0 now uses BuildKit by default to build Linux images, and uses the +Docker v23.0 now uses BuildKit by default to build Linux images, and uses the [Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for `docker build`. With this change, `docker build` now exposes all advanced features that BuildKit provides and which were previously only available through the @@ -171,14 +171,14 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co ### Legacy builder fallback -**Deprecated in Release: v23.0.0** +**Deprecated in Release: v23.0** -[Docker v23.0.0 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), +[Docker v23.0 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), which requires the Buildx component to build images with BuildKit. There may be situations where the Buildx component is not available, and BuildKit cannot be used. -To provide a smooth transition to BuildKit as the default builder, Docker v23.0.0 +To provide a smooth transition to BuildKit as the default builder, Docker v23.0 has an automatic fallback for some situations, or produces an error to assist users to resolve the problem. @@ -217,7 +217,7 @@ be possible in a future release. ### Btrfs storage driver on CentOS 7 and RHEL 7 -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** The `btrfs` storage driver on CentOS and RHEL was provided as a technology preview by CentOS and RHEL, but has been deprecated since the [Red Hat Enterprise Linux 7.4 release](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-btrfs), @@ -231,7 +231,7 @@ of Docker will no longer provide this driver. **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** Use of encrypted TLS private keys has been deprecated, and has been removed. Golang has deprecated support for legacy PEM encryption (as specified in @@ -246,7 +246,7 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from @@ -307,7 +307,7 @@ major release. ### Linux containers on Windows (LCOW) (experimental) **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after @@ -330,7 +330,7 @@ When using cgroups v2, the `--blkio-weight` options are implemented using ### Kernel memory limit **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). @@ -358,7 +358,7 @@ take no effect. ### Classic Swarm and overlay networks using cluster store **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -367,7 +367,7 @@ networks using an external key/value store. The corresponding`--cluster-advertis ### Support for legacy `~/.dockercfg` configuration files **Deprecated in Release: v20.10** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -384,13 +384,13 @@ been removed. **Deprecated in Release: v19.03** -**Removed in Release: v23.0.0** +**Removed in Release: v23.0** The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` field in the CLI configuration file are deprecated. Experimental features are enabled by default, and these configuration options are no longer functional. -Starting with v23.0.0, the Docker CLI no longer prints `Experimental` for the client +Starting with v23.0, the Docker CLI no longer prints `Experimental` for the client in the output of `docker version`, and the field has been removed from the JSON format. @@ -512,7 +512,7 @@ using compose files. ### Support for the `overlay2.override_kernel_check` storage option **Deprecated in Release: v19.03** -**Removed in Release: v24.0.0** +**Removed in Release: v24.0** This daemon configuration option disabled the Linux kernel version check used to detect if the kernel supported OverlayFS with multiple lower dirs, which is @@ -523,7 +523,7 @@ option was no longer used. ### AuFS storage driver **Deprecated in Release: v19.03** -**Removed in Release: v24.0.0** +**Removed in Release: v24.0** The `aufs` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `aufs` storage driver are @@ -539,11 +539,11 @@ is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. -#### Disabled by default in v23.0.0 +#### Disabled by default in v23.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0.0 release, the Docker Engine will +existing installations. Starting with the v23.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -560,7 +560,7 @@ different storage driver. ### Legacy overlay storage driver **Deprecated in Release: v18.09** -**Removed in Release: v24.0.0** +**Removed in Release: v24.0** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -573,11 +573,11 @@ on pre 4.x kernels. Now that all supported distributions are able to run `overla (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. -#### Disabled by default in v23.0.0 +#### Disabled by default in v23.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0.0 release, the Docker Engine will +existing installations. Starting with the v23.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -594,7 +594,7 @@ different storage driver. ### Device mapper storage driver **Deprecated in Release: v18.09** -**Disabled by default in Release: v23.0.0** +**Disabled by default in Release: v23.0** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will be removed in a future release. Users of the `devicemapper` storage driver are @@ -608,11 +608,11 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -#### Disabled by default in v23.0.0 +#### Disabled by default in v23.0 Docker already prevented deprecated storage drivers from being automatically selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0.0 release, the Docker Engine will +existing installations. Starting with the v23.0 release, the Docker Engine will fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): ```console @@ -688,12 +688,12 @@ and `docker service scale` in Docker 17.10. **Deprecated In Release: v17.05** -**Removed In Release: v23.0.0** +**Removed In Release: v23.0** The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource configuration and has been replaced with the more descriptive `--data-root` -flag. These flags were deprecated and hidden in v17.05, and removed in v23.0.0. +flag. These flags were deprecated and hidden in v17.05, and removed in v23.0. ### Top-level network properties in NetworkSettings @@ -771,7 +771,7 @@ The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` if it is formatted with the `ftype=0` option. -Support for these setups has been removed, and Docker v23.0.0 and up now fails to +Support for these setups has been removed, and Docker v23.0 and up now fails to start when attempting to use the `overlay2` or `overlay` storage driver on a backing filesystem without `d_type` support. From d21e34005be4064c2c2cf60fe16ea94c3d4c2374 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 19 May 2023 01:06:37 +0200 Subject: [PATCH 425/534] docs: remove aufs and legacy overlay Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins_graphdriver.md | 2 +- docs/reference/commandline/dockerd.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index da7ab68ea2..f0c88f734f 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -25,7 +25,7 @@ advisory: experimental Docker graph driver plugins enable admins to use an external/out-of-process graph driver for use with Docker engine. This is an alternative to using the -built-in storage drivers, such as aufs/overlay/devicemapper/btrfs. +built-in storage drivers, such as overlay2. You need to install and enable the plugin and then restart the Docker daemon before using the plugin. See the following example for the correct ordering diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6c14afccc2..2a69c2e466 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -323,7 +323,7 @@ $ docker -H tcp://127.0.0.1:2375 pull ubuntu ### Daemon storage-driver On Linux, the Docker daemon has support for several different image layer storage -drivers: `aufs`, `devicemapper`, `btrfs`, `zfs`, `overlay`, `overlay2`, and `fuse-overlayfs`. +drivers: `overlay2`, `fuse-overlayfs`, `btrfs`, `zfs`, and `devicemapper`. `overlay2` is the preferred storage driver for all currently supported Linux distributions, and is selected by default. Unless users have a strong reason to prefer another storage driver, From f5d598c19e257eb1d77b474e1c69f9df48e02d98 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 19 May 2023 01:10:17 +0200 Subject: [PATCH 426/534] docs/deprecated: remove "disabled by default" for AuFS, overlay These drivers have been removed in docker 24.0, so it's no longer possible to enable them. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 54 ++++++++-------------------------------------- 1 file changed, 9 insertions(+), 45 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 6a4e77e653..9a96be8d2e 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -525,12 +525,12 @@ option was no longer used. **Deprecated in Release: v19.03** **Removed in Release: v24.0** -The `aufs` storage driver is deprecated in favor of `overlay2`, and will -be removed in a future release. Users of the `aufs` storage driver are -recommended to migrate to a different storage driver, such as `overlay2`, which -is now the default storage driver. +The `aufs` storage driver is deprecated in favor of `overlay2`, and has been +removed in a Docker Engine v24.0. Users of the `aufs` storage driver must +migrate to a different storage driver, such as `overlay2`, before upgrading +to Docker Engine v24.0. -The `aufs` storage driver facilitates running Docker on distros that have no +The `aufs` storage driver facilitated running Docker on distros that have no support for OverlayFS, such as Ubuntu 14.04 LTS, which originally shipped with a 3.14 kernel. @@ -539,24 +539,6 @@ is available to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `aufs` storage driver. -#### Disabled by default in v23.0 - -Docker already prevented deprecated storage drivers from being automatically -selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0 release, the Docker Engine will -fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): - -```console -failed to start daemon: error initializing graphdriver: prior storage driver -aufs is deprecated and will be removed in a future release; update the the daemon -configuration and explicitly choose this storage driver to continue using it; -visit https://docs.docker.com/go/storage-driver/ for more information. -``` - -To continue using the storage driver, update the daemon configuration to use -explicitly use the given storage driver. Users are encouraged to migrate to -different storage driver. - ### Legacy overlay storage driver **Deprecated in Release: v18.09** @@ -564,33 +546,15 @@ different storage driver. The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive -inode consumption). The legacy `overlay` storage driver will be removed in a future -release. Users of the `overlay` storage driver should migrate to the `overlay2` -storage driver. +inode consumption). The legacy `overlay` storage driver has been removed in +Docker Engine v24.0. Users of the `overlay` storage driver should migrate to the +`overlay2` storage driver before upgrading to Docker Engine v24.0. The legacy `overlay` storage driver allowed using overlayFS-backed filesystems on pre 4.x kernels. Now that all supported distributions are able to run `overlay2` (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. -#### Disabled by default in v23.0 - -Docker already prevented deprecated storage drivers from being automatically -selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0 release, the Docker Engine will -fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): - -```console -failed to start daemon: error initializing graphdriver: prior storage driver -overlay is deprecated and will be removed in a future release; update the the daemon -configuration and explicitly choose this storage driver to continue using it; -visit https://docs.docker.com/go/storage-driver/ for more information. -``` - -To continue using the storage driver, update the daemon configuration to use -explicitly use the given storage driver. Users are encouraged to migrate to -different storage driver. - ### Device mapper storage driver **Deprecated in Release: v18.09** @@ -602,7 +566,7 @@ recommended to migrate to a different storage driver, such as `overlay2`, which is now the default storage driver. The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels -that have no support for other storage drivers (such as overlay2, or AUFS). +that have no support for other storage drivers (such as overlay2, or btrfs). Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there From c9aec4dba76b7fa95f27377f91cffb2c90361d0b Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 19 May 2023 17:21:34 +0200 Subject: [PATCH 427/534] docs: fix example for proxies in daemon.json commit 392b34fcec63d36ae32cc58c15d4fd8e5a28cfc5 added proxies to the example `daemon.json`, based on the implementation that was added in https://github.com/moby/moby/commit/427c7cc5f86364466c7173e8ca59b97c3876471d. However, a follow-up pull request changed the proxy-configuration in`daemon.json` to nest the configuration in a "proxies" struct, and the documentation was not updated accordingly; see: https://github.com/moby/moby/commit/101dafd049949a4da65dcefd495828a6644e1ce1 This patch fixes the example. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2a69c2e466..b39bdbaa9b 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -1252,9 +1252,11 @@ This is a full example of the allowed configuration options on Linux: "fixed-cidr-v6": "", "group": "", "hosts": [], - "http-proxy": "http://proxy.example.com:80", - "https-proxy": "https://proxy.example.com:443", - "no-proxy": "*.test.example.com,.example.org", + "proxies": { + "http-proxy": "http://proxy.example.com:80", + "https-proxy": "https://proxy.example.com:443", + "no-proxy": "*.test.example.com,.example.org", + }, "icc": false, "init": false, "init-path": "/usr/libexec/docker-init", From 5595f577b0f1da17c2ece9f5de2bb21c193876df Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 19 May 2023 01:42:52 +0200 Subject: [PATCH 428/534] docs: remove devicemapper Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 27 +- docs/reference/commandline/dockerd.md | 438 +------------------------- 2 files changed, 8 insertions(+), 457 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9a96be8d2e..0e257584be 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -559,11 +559,12 @@ backported), there is no reason to keep maintaining the `overlay` storage driver **Deprecated in Release: v18.09** **Disabled by default in Release: v23.0** +**Removed in Release: v25.0** -The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will -be removed in a future release. Users of the `devicemapper` storage driver are -recommended to migrate to a different storage driver, such as `overlay2`, which -is now the default storage driver. +The `devicemapper` storage driver is deprecated in favor of `overlay2`, and has +been removed in Docker Engine v25.0. Users of the `devicemapper` storage driver +must migrate to a different storage driver, such as `overlay2`, before upgrading +to Docker Engine v25.0. The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels that have no support for other storage drivers (such as overlay2, or btrfs). @@ -572,24 +573,6 @@ Now that support for `overlay2` is added to all supported distros (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to continue maintenance of the `devicemapper` storage driver. -#### Disabled by default in v23.0 - -Docker already prevented deprecated storage drivers from being automatically -selected on new installations, but continued to use these drivers when upgrading -existing installations. Starting with the v23.0 release, the Docker Engine will -fail to start if a deprecated storage driver is used (see [moby#43378](https://github.com/moby/moby/pull/43378): - -```console -failed to start daemon: error initializing graphdriver: prior storage driver -devicemapper is deprecated and will be removed in a future release; update the the daemon -configuration and explicitly choose this storage driver to continue using it; -visit https://docs.docker.com/go/storage-driver/ for more information. -``` - -To continue using the storage driver, update the daemon configuration to use -explicitly use the given storage driver. Users are encouraged to migrate to -different storage driver. - ### Use of reserved namespaces in engine labels **Deprecated in Release: v18.06** diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index b39bdbaa9b..6c49514ec4 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -323,7 +323,7 @@ $ docker -H tcp://127.0.0.1:2375 pull ubuntu ### Daemon storage-driver On Linux, the Docker daemon has support for several different image layer storage -drivers: `overlay2`, `fuse-overlayfs`, `btrfs`, `zfs`, and `devicemapper`. +drivers: `overlay2`, `fuse-overlayfs`, `btrfs`, and `zfs`. `overlay2` is the preferred storage driver for all currently supported Linux distributions, and is selected by default. Unless users have a strong reason to prefer another storage driver, @@ -336,440 +336,8 @@ On Windows, the Docker daemon only supports the `windowsfilter` storage driver. ### Options per storage driver Particular storage-driver can be configured with options specified with -`--storage-opt` flags. Options for `devicemapper` are prefixed with `dm`, -options for `zfs` start with `zfs`, and options for `btrfs` start with `btrfs`. - -#### Devicemapper options - -This is an example of the configuration file for devicemapper on Linux: - -```json -{ - "storage-driver": "devicemapper", - "storage-opts": [ - "dm.thinpooldev=/dev/mapper/thin-pool", - "dm.use_deferred_deletion=true", - "dm.use_deferred_removal=true" - ] -} -``` - -##### `dm.thinpooldev` - -Specifies a custom block storage device to use for the thin pool. - -If using a block device for device mapper storage, it is best to use `lvm` -to create and manage the thin-pool volume. This volume is then handed to Docker -to exclusively create snapshot volumes needed for images and containers. - -Managing the thin-pool outside of Engine makes for the most feature-rich -method of having Docker utilize device mapper thin provisioning as the -backing storage for Docker containers. The highlights of the lvm-based -thin-pool management feature include: automatic or interactive thin-pool -resize support, dynamically changing thin-pool features, automatic thinp -metadata checking when lvm activates the thin-pool, etc. - -As a fallback if no thin pool is provided, loopback files are -created. Loopback is very slow, but can be used without any -pre-configuration of storage. It is strongly recommended that you do -not use loopback in production. Ensure your Engine daemon has a -`--storage-opt dm.thinpooldev` argument provided. - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.thinpooldev=/dev/mapper/thin-pool -``` - -##### `dm.directlvm_device` - -As an alternative to providing a thin pool as above, Docker can setup a block -device for you. - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.directlvm_device=/dev/xvdf -``` - -##### `dm.thinp_percent` - -Sets the percentage of passed in block device to use for storage. - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.thinp_percent=95 -``` - -##### `dm.thinp_metapercent` - -Sets the percentage of the passed in block device to use for metadata storage. - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.thinp_metapercent=1 -``` - -##### `dm.thinp_autoextend_threshold` - -Sets the value of the percentage of space used before `lvm` attempts to -autoextend the available space [100 = disabled] - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.thinp_autoextend_threshold=80 -``` - -##### `dm.thinp_autoextend_percent` - -Sets the value percentage value to increase the thin pool by when `lvm` -attempts to autoextend the available space [100 = disabled] - -###### Example: - -```console -$ sudo dockerd --storage-opt dm.thinp_autoextend_percent=20 -``` - - -##### `dm.basesize` - -Specifies the size to use when creating the base device, which limits the -size of images and containers. The default value is 10G. Note, thin devices -are inherently "sparse", so a 10G device which is mostly empty doesn't use -10 GB of space on the pool. However, the filesystem will use more space for -the empty case the larger the device is. - -The base device size can be increased at daemon restart which will allow -all future images and containers (based on those new images) to be of the -new base device size. - -###### Examples - -```console -$ sudo dockerd --storage-opt dm.basesize=50G -``` - -This will increase the base device size to 50G. The Docker daemon will throw an -error if existing base device size is larger than 50G. A user can use -this option to expand the base device size however shrinking is not permitted. - -This value affects the system-wide "base" empty filesystem -that may already be initialized and inherited by pulled images. Typically, -a change to this value requires additional steps to take effect: - -```console -$ sudo service docker stop - -$ sudo rm -rf /var/lib/docker - -$ sudo service docker start -``` - - -##### `dm.loopdatasize` - -> **Note** -> -> This option configures devicemapper loopback, which should not -> be used in production. - -Specifies the size to use when creating the loopback file for the -"data" device which is used for the thin pool. The default size is -100G. The file is sparse, so it will not initially take up this -much space. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.loopdatasize=200G -``` - -##### `dm.loopmetadatasize` - -> **Note** -> -> This option configures devicemapper loopback, which should not -> be used in production. - -Specifies the size to use when creating the loopback file for the -"metadata" device which is used for the thin pool. The default size -is 2G. The file is sparse, so it will not initially take up -this much space. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.loopmetadatasize=4G -``` - -##### `dm.fs` - -Specifies the filesystem type to use for the base device. The supported -options are "ext4" and "xfs". The default is "xfs" - -###### Example - -```console -$ sudo dockerd --storage-opt dm.fs=ext4 -``` - -##### `dm.mkfsarg` - -Specifies extra mkfs arguments to be used when creating the base device. - -###### Example - -```console -$ sudo dockerd --storage-opt "dm.mkfsarg=-O ^has_journal" -``` - -##### `dm.mountopt` - -Specifies extra mount options used when mounting the thin devices. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.mountopt=nodiscard -``` - -##### `dm.datadev` - -(Deprecated, use `dm.thinpooldev`) - -Specifies a custom blockdevice to use for data for the thin pool. - -If using a block device for device mapper storage, ideally both `datadev` and -`metadatadev` should be specified to completely avoid using the loopback -device. - -###### Example - -```console -$ sudo dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 -``` - -##### `dm.metadatadev` - -(Deprecated, use `dm.thinpooldev`) - -Specifies a custom blockdevice to use for metadata for the thin pool. - -For best performance the metadata should be on a different spindle than the -data, or even better on an SSD. - -If setting up a new metadata pool it is required to be valid. This can be -achieved by zeroing the first 4k to indicate empty metadata, like this: - -```console -$ dd if=/dev/zero of=$metadata_dev bs=4096 count=1 -``` - -###### Example - -```console -$ sudo dockerd \ - --storage-opt dm.datadev=/dev/sdb1 \ - --storage-opt dm.metadatadev=/dev/sdc1 -``` - -##### `dm.blocksize` - -Specifies a custom blocksize to use for the thin pool. The default -blocksize is 64K. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.blocksize=512K -``` - -##### `dm.blkdiscard` - -Enables or disables the use of `blkdiscard` when removing devicemapper -devices. This is enabled by default (only) if using loopback devices and is -required to resparsify the loopback file on image/container removal. - -Disabling this on loopback can lead to *much* faster container removal -times, but will make the space used in `/var/lib/docker` directory not be -returned to the system for other use when containers are removed. - -###### Examples - -```console -$ sudo dockerd --storage-opt dm.blkdiscard=false -``` - -##### `dm.override_udev_sync_check` - -Overrides the `udev` synchronization checks between `devicemapper` and `udev`. -`udev` is the device manager for the Linux kernel. - -To view the `udev` sync support of a Docker daemon that is using the -`devicemapper` driver, run: - -```console -$ docker info -<...> -Udev Sync Supported: true -<...> -``` - -When `udev` sync support is `true`, then `devicemapper` and udev can -coordinate the activation and deactivation of devices for containers. - -When `udev` sync support is `false`, a race condition occurs between -the`devicemapper` and `udev` during create and cleanup. The race condition -results in errors and failures. (For information on these failures, see -[docker#4036](https://github.com/docker/docker/issues/4036)) - -To allow the `docker` daemon to start, regardless of `udev` sync not being -supported, set `dm.override_udev_sync_check` to true: - -```console -$ sudo dockerd --storage-opt dm.override_udev_sync_check=true -``` - -When this value is `true`, the `devicemapper` continues and simply warns -you the errors are happening. - -> **Note** -> -> The ideal is to pursue a `docker` daemon and environment that does -> support synchronizing with `udev`. For further discussion on this -> topic, see [docker#4036](https://github.com/docker/docker/issues/4036). -> Otherwise, set this flag for migrating existing Docker daemons to -> a daemon with a supported environment. - -##### `dm.use_deferred_removal` - -Enables use of deferred device removal if `libdm` and the kernel driver -support the mechanism. - -Deferred device removal means that if device is busy when devices are -being removed/deactivated, then a deferred removal is scheduled on -device. And devices automatically go away when last user of the device -exits. - -For example, when a container exits, its associated thin device is removed. -If that device has leaked into some other mount namespace and can't be -removed, the container exit still succeeds and this option causes the -system to schedule the device for deferred removal. It does not wait in a -loop trying to remove a busy device. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.use_deferred_removal=true -``` - -##### `dm.use_deferred_deletion` - -Enables use of deferred device deletion for thin pool devices. By default, -thin pool device deletion is synchronous. Before a container is deleted, -the Docker daemon removes any associated devices. If the storage driver -can not remove a device, the container deletion fails and daemon returns. - -```console -Error deleting container: Error response from daemon: Cannot destroy container -``` - -To avoid this failure, enable both deferred device deletion and deferred -device removal on the daemon. - -```console -$ sudo dockerd \ - --storage-opt dm.use_deferred_deletion=true \ - --storage-opt dm.use_deferred_removal=true -``` - -With these two options enabled, if a device is busy when the driver is -deleting a container, the driver marks the device as deleted. Later, when -the device isn't in use, the driver deletes it. - -In general it should be safe to enable this option by default. It will help -when unintentional leaking of mount point happens across multiple mount -namespaces. - -##### `dm.min_free_space` - -Specifies the min free space percent in a thin pool require for new device -creation to succeed. This check applies to both free data space as well -as free metadata space. Valid values are from 0% - 99%. Value 0% disables -free space checking logic. If user does not specify a value for this option, -the Engine uses a default value of 10%. - -Whenever a new a thin pool device is created (during `docker pull` or during -container creation), the Engine checks if the minimum free space is -available. If sufficient space is unavailable, then device creation fails -and any relevant `docker` operation fails. - -To recover from this error, you must create more free space in the thin pool -to recover from the error. You can create free space by deleting some images -and containers from the thin pool. You can also add more storage to the thin -pool. - -To add more space to a LVM (logical volume management) thin pool, just add -more storage to the volume group container thin pool; this should automatically -resolve any errors. If your configuration uses loop devices, then stop the -Engine daemon, grow the size of loop files and restart the daemon to resolve -the issue. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.min_free_space=10% -``` - -##### `dm.xfs_nospace_max_retries` - -Specifies the maximum number of retries XFS should attempt to complete -IO when ENOSPC (no space) error is returned by underlying storage device. - -By default XFS retries infinitely for IO to finish and this can result -in unkillable process. To change this behavior one can set -xfs_nospace_max_retries to say 0 and XFS will not retry IO after getting -ENOSPC and will shutdown filesystem. - -###### Example - -```console -$ sudo dockerd --storage-opt dm.xfs_nospace_max_retries=0 -``` - -##### `dm.libdm_log_level` - -Specifies the maxmimum `libdm` log level that will be forwarded to the -`dockerd` log (as specified by `--log-level`). This option is primarily -intended for debugging problems involving `libdm`. Using values other than the -defaults may cause false-positive warnings to be logged. - -Values specified must fall within the range of valid `libdm` log levels. At the -time of writing, the following is the list of `libdm` log levels as well as -their corresponding levels when output by `dockerd`. - -| `libdm` Level | Value | `--log-level` | -|---------------|------:|---------------| -| `_LOG_FATAL` | 2 | error | -| `_LOG_ERR` | 3 | error | -| `_LOG_WARN` | 4 | warn | -| `_LOG_NOTICE` | 5 | info | -| `_LOG_INFO` | 6 | info | -| `_LOG_DEBUG` | 7 | debug | - -###### Example - -```console -$ sudo dockerd \ - --log-level debug \ - --storage-opt dm.libdm_log_level=7 -``` +`--storage-opt` flags. Options for `zfs` start with `zfs`, and options for +`btrfs` start with `btrfs`. #### ZFS options From 0b65082f2b5688b29b465988108e11d71eded485 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Mon, 12 Jun 2023 22:20:53 +0200 Subject: [PATCH 429/534] docs: update the runtime configuration section Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 258 +++++++++++++++++++++++--- 1 file changed, 228 insertions(+), 30 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 6c49514ec4..ff40ddf2ba 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -396,36 +396,197 @@ Defaults to 20G. C:\> dockerd --storage-opt size=40G ``` -### Docker runtime execution options +### Runtime options The Docker daemon relies on a [OCI](https://github.com/opencontainers/runtime-spec) compliant runtime (invoked via the `containerd` daemon) as its interface to the Linux kernel `namespaces`, `cgroups`, and `SELinux`. -By default, the Docker daemon automatically starts `containerd`. If you want to -control `containerd` startup, manually start `containerd` and pass the path to -the `containerd` socket using the `--containerd` flag. For example: +#### Configure container runtimes + +By default, the Docker daemon uses runc as a container runtime. +You can configure the daemon to add additional runtimes. + +containerd shims installed on `PATH` can be used directly, without the need +to edit the daemon's configuration. For example, if you install the Kata +Containers shim (`containerd-shim-kata-v2`) on `PATH`, then you can select that +runtime with `docker run` without having to edit the daemon's configuration: ```console -$ sudo dockerd --containerd /var/run/dev/docker-containerd.sock +$ docker run --runtime io.containerd.kata.v2 ``` -Runtimes can be registered with the daemon either via the -configuration file or using the `--add-runtime` command line argument. +Container runtimes that don't implement containerd shims, or containerd shims +installed outside of `PATH`, must be registered with the daemon, either via the +configuration file or using the `--add-runtime` command line flag. -The following is an example adding 2 runtimes via the configuration: +For examples on how to use other container runtimes, see +[Alternative container runtimes](https://docs.docker.com/engine/alternative-container-runtimes) + +##### Configure runtimes using `daemon.json` + +To register and configure container runtimes using the daemon's configuration +file, add the runtimes as entries under `runtimes`: ```json { - "default-runtime": "runc", "runtimes": { - "custom": { - "path": "/usr/local/bin/my-runc-replacement", - "runtimeArgs": [ - "--debug" - ] + "": {} + } +} +``` + +The key of the entry (`` in the previous example) represents the name +of the runtime. This is the name that you reference when you run a container, +using `docker run --runtime `. + +The runtime entry contains an object specifying the configuration for your +runtime. The properties of the object depends on what kind of runtime you're +looking to register: + +- If the runtime implements its own containerd shim, the object shall contain + a `runtimeType` field and an optional `options` field. + + ```json + { + "runtimes": { + "": { + "runtimeType": "", + "options": {} + } + } + } + ``` + + See [Configure shims](#configure-containerd-shims). + +- If the runtime is designed to be a drop-in replacement for runc, + the object contains a `path` field, and an optional `runtimeArgs` field. + + ```json + { + "runtimes": { + "": { + "path": "/path/to/bin", + "runtimeArgs": ["...args"] + } + } + } + ``` + + See [Configure runc drop-in replacements](#configure-runc-drop-in-replacements). + +After changing the runtimes configuration in the configuration file, +you must reload or restart the daemon for changes to take effect: + +```console +$ sudo systemctl reload dockerd +``` + +##### Configure containerd shims + +If the runtime that you want to register implements a containerd shim, +or if you want to register a runtime which uses the runc shim, +use the following format for the runtime entry: + +```json +{ + "runtimes": { + "": { + "runtimeType": "", + "options": {} + } + } +} +``` + +`runtimeType` refers to either: + +- A fully qualified name of a containerd shim. + + The fully qualified name of a shim is the same as the `runtime_type` used to + register the runtime in containerd's CRI configuration. + For example, `io.containerd.runsc.v1`. + +- The path of a containerd shim binary. + + This option is useful if you installed the containerd shim binary outside of + `PATH`. + +`options` is optional. It lets you specify the runtime configuration that you +want to use for the shim. The configuration parameters that you can specify in +`options` depends on the runtime you're registering. For most shims, +the supported configuration options are `TypeUrl` and `ConfigPath`. +For example: + +```json +{ + "runtimes": { + "gvisor": { + "runtimeType": "io.containerd.runsc.v1", + "options": { + "TypeUrl": "io.containerd.runsc.v1.options", + "ConfigPath": "/etc/containerd/runsc.toml", + } + } + } +} +``` + +You can configure multiple runtimes using the same runtimeType. For example: + +```json +{ + "runtimes": { + "gvisor-foo": { + "runtimeType": "io.containerd.runsc.v1", + "options": { + "TypeUrl": "io.containerd.runsc.v1.options", + "ConfigPath": "/etc/containerd/runsc-foo.toml" + } }, + "gvisor-bar": { + "runtimeType": "io.containerd.runsc.v1", + "options": { + "TypeUrl": "io.containerd.runsc.v1.options", + "ConfigPath": "/etc/containerd/runsc-bar.toml" + } + } + } +} +``` + +The `options` field takes a special set of configuration parameters when used +with `"runtimeType": "io.containerd.runc.v2"`. For more information about runc +parameters, refer to the runc configuration section in +[CRI Plugin Config Guide](https://github.com/containerd/containerd/blob/v1.7.2/docs/cri/config.md#full-configuration). + +##### Configure runc drop-in replacements + +If the runtime that you want to register can act as a drop-in replacement for +runc, you can register the runtime either using the daemon configuration file, +or using the `--add-runtime` flag for the `dockerd` cli. + +When you use the configuration file, the entry uses the following format: + +```json +{ + "runtimes": { + "": { + "path": "/path/to/binary", + "runtimeArgs": ["...args"] + } + } +} +``` + +Where `path` is either the absolute path to the runtime executable, or the name +of an executable installed on `PATH`: + +```json +{ + "runtimes": { "runc": { "path": "runc" } @@ -433,24 +594,58 @@ The following is an example adding 2 runtimes via the configuration: } ``` -This is the same example via the command line: +And `runtimeArgs` lets you optionally pass additional arguments to the runtime. +Entries with this format use the containerd runc shim to invoke a custom +runtime binary. + +When you use the `--add-runtime` CLI flag, use the following format: ```console -$ sudo dockerd --add-runtime runc=runc --add-runtime custom=/usr/local/bin/my-runc-replacement +$ sudo dockerd --add-runtime = ``` -> **Note** -> -> Defining runtime arguments via the command line is not supported. +Defining runtime arguments via the command line is not supported. -#### Options for the runtime +For an example configuration for a runc drop-in replacment, see +[Alternative container runtimes > youki](https://docs.docker.com/engine/alternative-runtimes/#youki) -You can configure the runtime using options specified -with the `--exec-opt` flag. All the flag's options have the `native` prefix. A -single `native.cgroupdriver` option is available. +##### Configure the default container runtime -The `native.cgroupdriver` option specifies the management of the container's -cgroups. You can only specify `cgroupfs` or `systemd`. If you specify +You can specify either the name of a fully qualified containerd runtime shim, +or the name of a registered runtime. You can specify the default runtime either +using the daemon configuration file, or using the `--default-runtime` flag for +the `dockerd` cli. + +When you use the configuration file, the entry uses the following format: + +```json +{ + "default-runtime": "io.containerd.runsc.v1" +} +``` + +When you use the `--default-runtime` CLI flag, use the following format: + +```console +$ dockerd --default-runtime io.containerd.runsc.v1 +``` + +#### Run containerd standalone + +By default, the Docker daemon automatically starts `containerd`. If you want to +control `containerd` startup, manually start `containerd` and pass the path to +the `containerd` socket using the `--containerd` flag. For example: + +```console +$ sudo dockerd --containerd /run/containerd/containerd.sock +``` + +#### Configure cgroup driver + +You can configure how the runtime should manage container cgroups, using the +`--exec-opt native.cgroupdriver` CLI flag. + +You can only specify `cgroupfs` or `systemd`. If you specify `systemd` and it is not available, the system errors out. If you omit the `native.cgroupdriver` option,` cgroupfs` is used on cgroup v1 hosts, `systemd` is used on cgroup v2 hosts with systemd available. @@ -463,16 +658,19 @@ $ sudo dockerd --exec-opt native.cgroupdriver=systemd Setting this option applies to all containers the daemon launches. -Also Windows Container makes use of `--exec-opt` for special purpose. Docker user -can specify default container isolation technology with this, for example: +#### Configure container isolation technology (Windows) + +For Windows containers, you can specify the default container isolation +technology to use, using the `--exec-opt isolation` flag. + +The following example makes `hyperv` the default isolation technology: ```console > dockerd --exec-opt isolation=hyperv ``` -Will make `hyperv` the default isolation technology on Windows. If no isolation -value is specified on daemon start, on Windows client, the default is -`hyperv`, and on Windows server, the default is `process`. +If no isolation value is specified on daemon start, on Windows client, +the default is `hyperv`, and on Windows server, the default is `process`. ### Daemon DNS options From 2cbdb0d2ec41475f38d9b1808fb02eac670be1fa Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Mon, 26 Jun 2023 14:55:29 +0200 Subject: [PATCH 430/534] docs: fix broken link Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ff40ddf2ba..e0d3d30260 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -422,7 +422,7 @@ installed outside of `PATH`, must be registered with the daemon, either via the configuration file or using the `--add-runtime` command line flag. For examples on how to use other container runtimes, see -[Alternative container runtimes](https://docs.docker.com/engine/alternative-container-runtimes) +[Alternative container runtimes](https://docs.docker.com/engine/alternative-runtimes/) ##### Configure runtimes using `daemon.json` From 23bd42899476917c3cc1b1e0830b5a57b03697e8 Mon Sep 17 00:00:00 2001 From: Zhiwei Liang Date: Fri, 30 Jun 2023 21:39:58 -0400 Subject: [PATCH 431/534] Add Linode docker volume plugin Signed-off-by: Zhiwei Liang --- docs/extend/legacy_plugins.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index d3981bd076..ffba4ced24 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -74,6 +74,7 @@ The sections below provide an inexhaustive overview of available plugins. | [Infinit volume plugin](https://infinit.sh/documentation/docker/volume-plugin) | A volume plugin that makes it easy to mount and manage Infinit volumes using Docker. | | [IPFS Volume Plugin](https://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. | | [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. | +| [Linode Volume Plugin](https://github.com/linode/docker-volume-linode) | A plugin that adds the ability to manage Linode Block Storage as Docker Volumes from within a Linode. | | [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. | | [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. | | [Netshare plugin](https://github.com/ContainX/docker-volume-netshare) | A volume plugin that provides volume management for NFS 3/4, AWS EFS and CIFS file systems. | From 4fac95d18a23333ba82664561d7e0638a336ddb3 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Mon, 3 Jul 2023 16:34:55 +0200 Subject: [PATCH 432/534] docs: document special host-gateway value for add-host Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index e0d3d30260..2549d5d95d 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -833,6 +833,25 @@ the host. For details about how to use this feature, as well as limitations, see [Isolate containers with a user namespace](https://docs.docker.com/engine/security/userns-remap/). +### Configure host gateway IP + +The Docker daemon supports a special `host-gateway` value for the `--add-host` +flag for the `docker run` and `docker build` commands. This value resolves to +the host's gateway IP and lets containers connect to services running on the +host. + +By default, `host-gateway` resolves to the IP address of the default bridge. +You can configure this to resolve to a different IP using the `--host-gateway-ip` +flag for the dockerd command line interface, or the `host-gateway-ip` key in +the daemon configuration file. + +```console +$ dockerd --host-gateway-ip 192.0.2.0 +$ docker run -it --add-host host.docker.internal:host-gateway \ + busybox ping host.docker.internal +PING host.docker.internal (192.0.2.0): 56 data bytes +``` + ### Miscellaneous options IP masquerading uses address translation to allow containers without a public From e3a279723e68c0a55c0f74dca84b70072132837e Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 10 Jul 2023 20:18:28 +0200 Subject: [PATCH 433/534] search: deprecate "is-automated" filter and "IsAutomated" field The IsAutomated field is being deprecated by Docker Hub's search API and will always be "false" in future. This patch: - Deprecates the field and the related "is-automated" filter - Removes the "AUTOMATED" column from the default output of "docker search" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 0e257584be..b16dd15437 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field--and-is--automated-filter-on-docker-search) | v25.0 | - | | Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | @@ -106,6 +107,21 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | + +### IsAutomated field, and "is-automated" filter on docker search + +**Deprecated in Release: v25.0** +**Target For Removal In Release: v26.0** + +The "is_automated" field has been deprecated by Docker Hub's search API. +Consequently, the `IsAutomated` field in image search will always be set +to `false` in future, and searching for "is-automated=true" will yield no +results. + +The `AUTOMATED` column has been removed from the default `docker search` +and `docker image search` output in v25.0, and the corresponding `IsAutomated` +templating option will be removed in v26.0. + ### OOM-score adjust for the daemon **Deprecated in Release: v24.0** From 3b8c575b8dd27a8915809c9bf9667fe9d05f9951 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Fri, 3 Nov 2023 14:20:54 +0100 Subject: [PATCH 434/534] docs: add default-network-opt daemon option Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 50 +++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2549d5d95d..ce4aadcb71 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -46,6 +46,7 @@ Options: --default-gateway ip Container default gateway IPv4 address --default-gateway-v6 ip Container default gateway IPv6 address --default-ipc-mode string Default mode for containers ipc ("shareable" | "private") (default "private") + --default-network-opt mapmap Default network options (default map[]) --default-runtime string Default OCI runtime for containers (default "runc") --default-shm-size bytes Default shm size for containers (default 64MiB) --default-ulimit ulimit Default ulimits for containers (default []) @@ -1017,6 +1018,7 @@ This is a full example of the allowed configuration options on Linux: "default-cgroupns-mode": "private", "default-gateway": "", "default-gateway-v6": "", + "default-network-opts": {}, "default-runtime": "runc", "default-shm-size": "64M", "default-ulimits": { @@ -1134,6 +1136,7 @@ This is a full example of the allowed configuration options on Windows: "containerd-plugin-namespace": "docker-plugins", "data-root": "", "debug": true, + "default-network-opts": {}, "default-runtime": "", "default-ulimits": {}, "dns": [], @@ -1279,3 +1282,50 @@ $ sudo dockerd \ --data-root=/var/lib/docker-bootstrap \ --exec-root=/var/run/docker-bootstrap ``` + +### Default network options + +The `default-network-opts` key in the `daemon.json` configuration file, and the +equivalent `--default-network-opt` CLI flag, let you specify default values for +driver network driver options for new networks. + +The following example shows how to configure options for the `bridge` driver +using the `daemon.json` file. + +```json +{ + "default-network-opts": { + "bridge": { + "com.docker.network.bridge.host_binding_ipv4": "127.0.0.1", + "com.docker.network.bridge.mtu": "1234" + } + } +} +``` + +This example uses the `bridge` network driver. Refer to the +[bridge network driver page](https://docs.docker.com/network/drivers/bridge/#options) +for an overview of available driver options. + +After changing the configuration and restarting the daemon, new networks that +you create use these option configurations as defaults. + +```console +$ docker network create mynet +$ docker network inspect mynet --format "{{json .Options}}" +{"com.docker.network.bridge.host_binding_ipv4":"127.0.0.1","com.docker.network.bridge.mtu":"1234"} +``` + +Note that changing this daemon configuration doesn't affect pre-existing +networks. + +Using the `--default-network-opt` CLI flag is useful for testing and debugging +purposes, but you should prefer using the `daemon.json` file for persistent +daemon configuration. The CLI flag expects a value with the following format: +`driver=opt=value`, for example: + +```console +$ sudo dockerd \ + --default-network-opt bridge=com.docker.network.bridge.host_binding_ipv4=127.0.0.1 \ + --default-network-opt bridge=com.docker.network.bridge.mtu=1234 +``` From 1ad75aa3e132dadfededeb81f07c4ebec12675ad Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 6 Nov 2023 18:25:49 +0100 Subject: [PATCH 435/534] docs: update redirect metadata for hugo docs.docker.com switched from Jekyll to Hugo, which uses "aliases" instead of "redirect_from". Signed-off-by: Sebastiaan van Stijn --- docs/extend/legacy_plugins.md | 2 +- docs/extend/plugins_authorization.md | 2 +- docs/reference/commandline/dockerd.md | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index d3981bd076..6c842ac473 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -1,5 +1,5 @@ --- -redirect_from: +aliases: - "/engine/extend/plugins/" description: "How to add additional functionality to Docker with plugins extensions" keywords: "Examples, Usage, plugins, docker, documentation, user guide" diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 2fc444e101..203c1bf2e2 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -1,7 +1,7 @@ --- description: "How to create authorization plugins to manage access control to your Docker daemon." keywords: "security, authorization, authentication, docker, documentation, plugin, extend" -redirect_from: +aliases: - "/engine/extend/authorization/" --- diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index ce4aadcb71..1c1b99115c 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -2,7 +2,7 @@ title: "dockerd" description: "The daemon command description and usage" keywords: "container, daemon, runtime" -redirect_from: +aliases: - /engine/reference/commandline/daemon/ --- From f4965f29d127566ff98a6ac8841e1037caf7e678 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 27 Nov 2023 21:58:45 +0100 Subject: [PATCH 436/534] docs: remove "{% raw %}" / "{% endraw %}" Jekyl (liquid) leftovers Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 -- docs/extend/plugins_graphdriver.md | 4 ---- docs/extend/plugins_logging.md | 2 +- docs/extend/plugins_services.md | 2 -- 4 files changed, 1 insertion(+), 9 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index b16dd15437..3dfb37effe 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -817,9 +817,7 @@ Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` a `fluentd-tag` have been deprecated in favor of the generic `tag` option. ```console -{% raw %} $ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" -{% endraw %} ``` diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index f0c88f734f..f6558d7f1e 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -311,9 +311,7 @@ and `Parent`. `Parent` may be an empty string, in which case there is no parent. **Response**: ``` -{% raw %} {{ TAR STREAM }} -{% endraw %} ``` ### /GraphDriver.Changes @@ -357,9 +355,7 @@ Respond with a non-empty string error if an error occurred. **Request**: ``` -{% raw %} {{ TAR STREAM }} -{% endraw %} ``` Extract the changeset from the given diff into the layer with the specified `ID` diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md index 6405dd26cf..d6ce96cfb3 100644 --- a/docs/extend/plugins_logging.md +++ b/docs/extend/plugins_logging.md @@ -213,7 +213,7 @@ to determine what set of logs to read. **Response**: ``` -{% raw %}{{ log stream }}{% endraw %} +{{ log stream }} ``` The response should be the encoded log message using the same format as the diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md index 20889dd976..f0c2dc503d 100644 --- a/docs/extend/plugins_services.md +++ b/docs/extend/plugins_services.md @@ -69,10 +69,8 @@ node1 is the manager and node2 is the worker. 4. Observe the task getting scheduled in node 2: ```console - {% raw %} $ docker ps --format '{{.ID}}\t {{.Status}} {{.Names}} {{.Command}}' 83fc1e842599 Up 2 days my-service.1.9jn59qzn7nbc3m0zt1hij12xs "top" - {% endraw %} ``` ### Network plugins From 25246caa65165a51ee895d988b32f81d73892043 Mon Sep 17 00:00:00 2001 From: TheRealGramdalf <79593869+TheRealGramdalf@users.noreply.github.com> Date: Fri, 1 Dec 2023 05:17:22 +0000 Subject: [PATCH 437/534] Fix typo in dockerd reference documentation Signed-off-by: Graeme Wiebe --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 1c1b99115c..2aa4eabaad 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -205,7 +205,7 @@ find examples of using Systemd socket activation with Docker and Systemd in the You can configure the Docker daemon to listen to multiple sockets at the same time using multiple `-H` options: -The example below runs the daemon listenin on the default unix socket, and +The example below runs the daemon listening on the default unix socket, and on 2 specific IP addresses on this host: ```console From 570df3b38d44032517c2aade9a957b377cc6a396 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Dec 2023 22:30:28 +0100 Subject: [PATCH 438/534] docs/deprecate.md: add version for devicemapper removal to table commit 5595f577b0f1da17c2ece9f5de2bb21c193876df updated the deprecation status for devicemapper to "removed", but forgot to update the status in the table. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 3dfb37effe..7a8d441e75 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -76,7 +76,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [Support for the `overlay2.override_kernel_check` storage option](#support-for-the-overlay2override_kernel_check-storage-option) | v19.03 | v24.0 | | Removed | [AuFS storage driver](#aufs-storage-driver) | v19.03 | v24.0 | | Removed | [Legacy "overlay" storage driver](#legacy-overlay-storage-driver) | v18.09 | v24.0 | -| Disabled | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | - | +| Removed | [Device mapper storage driver](#device-mapper-storage-driver) | v18.09 | v25.0 | | Removed | [Use of reserved namespaces in engine labels](#use-of-reserved-namespaces-in-engine-labels) | v18.06 | v20.10 | | Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | | Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | From f757d7cf67592999e62b6862f0773542cd647610 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 7 Dec 2023 22:34:38 +0100 Subject: [PATCH 439/534] docs/deprecated.md: add version for "-g" / "--graph" removal commit b482413fb6e45866504f2bf7ae56a36d240df870 updated the deprecation status for these options, but forgot to update the status in the table. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7a8d441e75..38c416503e 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -81,7 +81,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--disable-legacy-registry` override daemon option](#--disable-legacy-registry-override-daemon-option) | v17.12 | v19.03 | | Removed | [Interacting with V1 registries](#interacting-with-v1-registries) | v17.06 | v17.12 | | Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | -| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | - | +| Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | v23.0 | | Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | | Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | | Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | From 447b97a703afdb6662935adc26aaa82c457f092c Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 13 Dec 2023 09:12:02 +0100 Subject: [PATCH 440/534] docs/deprecated: mark logentries log-driver for removal The service has been discontinued on November 15, 2022: > Dear Logentries user, > > We have identified you as the owner of, or collaborator of, a Logentries > account. > > The Logentries service will be discontinued on November 15th, 2022. This > means that your Logentries account access will be removed and all your > log data will be permanently deleted on this date. > > Next Steps > If you are interested in an alternative Rapid7 log management solution, > InsightOps will be available for purchase through December 16th, 2022. > Please note, there is no support to migrate your existing Logentries > account to InsightOps. > > Thank you for being a valued user of Logentries. > > Thank you, > Rapid7 Customer Success Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 38c416503e..834c68a6ee 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -51,6 +51,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field--and-is--automated-filter-on-docker-search) | v25.0 | - | +| Deprecated | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | @@ -122,6 +123,15 @@ The `AUTOMATED` column has been removed from the default `docker search` and `docker image search` output in v25.0, and the corresponding `IsAutomated` templating option will be removed in v26.0. +### Logentries logging driver + +**Deprecated in Release: v24.0** +**Target For Removal In Release: v25.0** + +The logentries service SaaS was shut down on November 15, 2022, rendering +this logging driver non-functional. Users should no longer use this logging +driver, and the driver will be removed in Docker 25.0. + ### OOM-score adjust for the daemon **Deprecated in Release: v24.0** From 60c503dfadbbdc7d1b957a72d82173b6d50e93fc Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 13 Dec 2023 13:19:33 +0100 Subject: [PATCH 441/534] docs/deprecated: mark logentries logging-driver as "removed" This driver has been deprecated and removed because the service is no longer operational. Update the deprecated features docs to mention it's been fully removed in Docker v25.0. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 834c68a6ee..95eee9df3c 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -51,7 +51,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field--and-is--automated-filter-on-docker-search) | v25.0 | - | -| Deprecated | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | +| Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | @@ -126,11 +126,13 @@ templating option will be removed in v26.0. ### Logentries logging driver **Deprecated in Release: v24.0** -**Target For Removal In Release: v25.0** +**Removed in Release: v25.0** The logentries service SaaS was shut down on November 15, 2022, rendering this logging driver non-functional. Users should no longer use this logging -driver, and the driver will be removed in Docker 25.0. +driver, and the driver has been removed in Docker 25.0. Existing containers +using this logging-driver are migrated to use the "local" logging driver +after upgrading. ### OOM-score adjust for the daemon From 22c88d536b0c73de0f1b0e5777a68336cdd1f65a Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 13 Dec 2023 15:16:56 +0100 Subject: [PATCH 442/534] docs: minor editorial improvements, typo fixes Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 367 +++++++++++++------------- 1 file changed, 180 insertions(+), 187 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 2aa4eabaad..313332bde3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -124,7 +124,7 @@ To run the daemon with debug output, use `dockerd --debug` or add `"debug": true to [the `daemon.json` file](#daemon-configuration-file). > **Enabling experimental features** -> +> > Enable experimental features by starting `dockerd` with the `--experimental` > flag or adding `"experimental": true` to the `daemon.json` file. @@ -137,7 +137,7 @@ by the `dockerd` command line: |:--------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | | `DOCKER_DRIVER` | The storage driver to use. | -| `DOCKER_RAMDISK` | If set this disables 'pivot_root'. | +| `DOCKER_RAMDISK` | If set this disables `pivot_root`. | | `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | | `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | | `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | @@ -150,7 +150,7 @@ by the `dockerd` command line: ### Proxy configuration > **Note** -> +> > Refer to the [Docker Desktop manual](https://docs.docker.com/desktop/networking/#httphttps-proxy-support) > if you are running [Docker Desktop](https://docs.docker.com/desktop/). @@ -160,10 +160,10 @@ operations such as pulling and pushing images. The daemon can be configured in three ways: 1. Using environment variables (`HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY`). -2. Using the "http-proxy", "https-proxy", and "no-proxy" fields in the - [daemon configuration file](#daemon-configuration-file) (Docker Engine 23.0 or newer). +2. Using the `http-proxy`, `https-proxy`, and `no-proxy` fields in the + [daemon configuration file](#daemon-configuration-file) (Docker Engine version 23.0 or later). 3. Using the `--http-proxy`, `--https-proxy`, and `--no-proxy` command-line - options. (Docker Engine 23.0 or newer). + options. (Docker Engine version 23.0 or later). The command-line and configuration file options take precedence over environment variables. Refer to [control and configure Docker with systemd](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) @@ -178,11 +178,11 @@ By default, a `unix` domain socket (or IPC socket) is created at `/var/run/docker.sock`, requiring either `root` permission, or `docker` group membership. -If you need to access the Docker daemon remotely, you need to enable the `tcp` -Socket. Beware that the default setup provides un-encrypted and -un-authenticated direct access to the Docker daemon - and should be secured -either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/https/), or by -putting a secure web proxy in front of it. You can listen on port `2375` on all +If you need to access the Docker daemon remotely, you need to enable the tcp +Socket. When using a TCP socket, the Docker daemon provides un-encrypted and +un-authenticated direct access to the Docker daemon by default. You should secure +the daemon either using the [built in HTTPS encrypted socket](https://docs.docker.com/engine/security/protect-access/), +or by putting a secure web proxy in front of it. You can listen on port `2375` on all network interfaces with `-H tcp://0.0.0.0:2375`, or on a particular network interface using its IP address: `-H tcp://192.168.59.103:2375`. It is conventional to use port `2375` for un-encrypted, and port `2376` for encrypted @@ -191,28 +191,28 @@ communication with the daemon. > **Note** > > If you're using an HTTPS encrypted socket, keep in mind that only -> TLS1.0 and greater are supported. Protocols SSLv3 and under are not -> supported anymore for security reasons. +> TLS version 1.0 and higher is supported. Protocols SSLv3 and below are not +> supported for security reasons. -On Systemd based systems, you can communicate with the daemon via -[Systemd socket activation](https://0pointer.de/blog/projects/socket-activation.html), -use `dockerd -H fd://`. Using `fd://` will work perfectly for most setups but +On systemd based systems, you can communicate with the daemon via +[systemd socket activation](https://0pointer.de/blog/projects/socket-activation.html), +with `dockerd -H fd://`. Using `fd://` works for most setups, but you can also specify individual sockets: `dockerd -H fd://3`. If the -specified socket activated files aren't found, then Docker will exit. You can -find examples of using Systemd socket activation with Docker and Systemd in the +specified socket activated files aren't found, the daemon exits. You can +find examples of using systemd socket activation with Docker and systemd in the [Docker source tree](https://github.com/docker/docker/tree/master/contrib/init/systemd/). You can configure the Docker daemon to listen to multiple sockets at the same time using multiple `-H` options: -The example below runs the daemon listening on the default unix socket, and +The example below runs the daemon listening on the default Unix socket, and on 2 specific IP addresses on this host: ```console $ sudo dockerd -H unix:///var/run/docker.sock -H tcp://192.168.59.106 -H tcp://10.10.10.2 ``` -The Docker client will honor the `DOCKER_HOST` environment variable to set the +The Docker client honors the `DOCKER_HOST` environment variable to set the `-H` flag for the client. Use **one** of the following commands: ```console @@ -236,7 +236,7 @@ $ export DOCKER_TLS_VERIFY=1 $ docker ps ``` -The Docker client will honor the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` +The Docker client honors the `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables (or the lowercase versions thereof). `HTTPS_PROXY` takes precedence over `HTTP_PROXY`. @@ -258,29 +258,28 @@ supported. If your key is protected with passphrase, you need to set up > **Warning** > -> Changing the default `docker` daemon binding to a -> TCP port or Unix *docker* user group will increase your security risks -> by allowing non-root users to gain *root* access on the host. Make sure -> you control access to `docker`. If you are binding -> to a TCP port, anyone with access to that port has full Docker access; -> so it is not advisable on an open network. -{: .warning :} +> Changing the default `docker` daemon binding to a TCP port or Unix `docker` +> user group introduces security risks, as it may allow non-root users to gain +> root access on the host. Make sure you control access to `docker`. If you are +> binding to a TCP port, anyone with access to that port has full Docker +> access; so it's not advisable on an open network. +{ .warning } -With `-H` it is possible to make the Docker daemon to listen on a -specific IP and port. By default, it will listen on -`unix:///var/run/docker.sock` to allow only local connections by the -*root* user. You *could* set it to `0.0.0.0:2375` or a specific host IP -to give access to everybody, but that is **not recommended** because -then it is trivial for someone to gain root access to the host where the -daemon is running. +With `-H` it's possible to make the Docker daemon to listen on a specific IP +and port. By default, it listens on `unix:///var/run/docker.sock` to allow +only local connections by the root user. You could set it to `0.0.0.0:2375` or +a specific host IP to give access to everybody, but that isn't recommended +because someone could gain root access to the host where the daemon is running. Similarly, the Docker client can use `-H` to connect to a custom port. -The Docker client will default to connecting to `unix:///var/run/docker.sock` +The Docker client defaults to connecting to `unix:///var/run/docker.sock` on Linux, and `tcp://127.0.0.1:2376` on Windows. `-H` accepts host and port assignment in the following format: - tcp://[host]:[port][path] or unix://path +```text +tcp://[host]:[port][path] or unix://path +``` For example: @@ -293,7 +292,7 @@ For example: - `unix://path/to/socket` -> Unix socket located at `path/to/socket` -`-H`, when empty, will default to the same value as +`-H`, when empty, defaults to the same value as when no `-H` was passed in. `-H` also accepts short form for TCP bindings: `host:` or `host:port` or `:port` @@ -344,9 +343,8 @@ Particular storage-driver can be configured with options specified with ##### `zfs.fsname` -Set zfs filesystem under which docker will create its own datasets. -By default docker will pick up the zfs filesystem where docker graph -(`/var/lib/docker`) is located. +Specifies the ZFS filesystem that the daemon should use to create its datasets. +By default, the ZFS filesystem in `/var/lib/docker` is used. ###### Example @@ -360,8 +358,8 @@ $ sudo dockerd -s zfs --storage-opt zfs.fsname=zroot/docker Specifies the minimum size to use when creating the subvolume which is used for containers. If user uses disk quota for btrfs when creating or running -a container with **--storage-opt size** option, docker should ensure the -**size** cannot be smaller than **btrfs.min_space**. +a container with **--storage-opt size** option, Docker should ensure the +**size** can't be smaller than **btrfs.min_space**. ###### Example @@ -374,8 +372,8 @@ $ sudo dockerd -s btrfs --storage-opt btrfs.min_space=10G ##### `overlay2.size` Sets the default max size of the container. It is supported only when the -backing fs is `xfs` and mounted with `pquota` mount option. Under these -conditions the user can pass any size less than the backing fs size. +backing filesystem is `xfs` and mounted with `pquota` mount option. Under these +conditions the user can pass any size less than the backing filesystem size. ###### Example @@ -687,7 +685,7 @@ To set the DNS search domain for all Docker containers, use: $ sudo dockerd --dns-search example.com ``` -### Allow push of nondistributable artifacts +### Allow push of non-distributable artifacts Some images (e.g., Windows base images) contain artifacts whose distribution is restricted by license. When these images are pushed to a registry, restricted @@ -697,38 +695,41 @@ To override this behavior for specific registries, use the `--allow-nondistributable-artifacts` option in one of the following forms: * `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon - to push nondistributable artifacts to myregistry:5000. + to push non-distributable artifacts to myregistry:5000. * `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to - push nondistributable artifacts to all registries whose resolved IP address + push non-distributable artifacts to all registries whose resolved IP address is within the subnet described by the CIDR syntax. This option can be used multiple times. -This option is useful when pushing images containing nondistributable artifacts +This option is useful when pushing images containing non-distributable artifacts to a registry on an air-gapped network so hosts on that network can pull the images without connecting to another server. -> **Warning**: Nondistributable artifacts typically have restrictions on how +> **Warning** +> +> Non-distributable artifacts typically have restrictions on how > and where they can be distributed and shared. Only use this feature to push > artifacts to private registries and ensure that you are in compliance with -> any terms that cover redistributing nondistributable artifacts. +> any terms that cover redistributing non-distributable artifacts. +{ .warning } ### Insecure registries -Docker considers a private registry either secure or insecure. In the rest of -this section, *registry* is used for *private registry*, and `myregistry:5000` -is a placeholder example for a private registry. +In this section, "registry" refers to a private registry, and `myregistry:5000` +is a placeholder example of a private registry. +Docker considers a private registry either secure or insecure. A secure registry uses TLS and a copy of its CA certificate is placed on the Docker host at `/etc/docker/certs.d/myregistry:5000/ca.crt`. An insecure registry is either not using TLS (i.e., listening on plain text HTTP), or is using TLS with a CA certificate not known by the Docker daemon. The latter can -happen when the certificate was not found under +happen when the certificate wasn't found under `/etc/docker/certs.d/myregistry:5000/`, or if the certificate verification failed (i.e., wrong CA). -By default, Docker assumes all, but local (see local registries below), -registries are secure. Communicating with an insecure registry is not possible +By default, Docker assumes all registries to be secure, except for local registries. +Communicating with an insecure registry isn't possible if Docker assumes that registry is secure. In order to communicate with an insecure registry, the Docker daemon requires `--insecure-registry` in one of the following two forms: @@ -742,34 +743,33 @@ the following two forms: The flag can be used multiple times to allow multiple registries to be marked as insecure. -If an insecure registry is not marked as insecure, `docker pull`, -`docker push`, and `docker search` will result in an error message prompting +If an insecure registry isn't marked as insecure, `docker pull`, +`docker push`, and `docker search` result in error messages, prompting the user to either secure or pass the `--insecure-registry` flag to the Docker daemon as described above. Local registries, whose IP address falls in the 127.0.0.0/8 range, are -automatically marked as insecure as of Docker 1.3.2. It is not recommended to +automatically marked as insecure as of Docker 1.3.2. It isn't recommended to rely on this, as it may change in the future. Enabling `--insecure-registry`, i.e., allowing un-encrypted and/or untrusted -communication, can be useful when running a local registry. However, -because its use creates security vulnerabilities it should ONLY be enabled for -testing purposes. For increased security, users should add their CA to their +communication, can be useful when running a local registry. However, +because its use creates security vulnerabilities it should only be enabled for +testing purposes. For increased security, users should add their CA to their system's list of trusted CAs instead of enabling `--insecure-registry`. #### Legacy Registries Operations against registries supporting only the legacy v1 protocol are no longer -supported. Specifically, the daemon will not attempt `push`, `pull` and `login` +supported. Specifically, the daemon doesn't attempt to push, pull or sign in to v1 registries. The exception to this is `search` which can still be performed on v1 registries. - ### Running a Docker daemon behind an HTTPS_PROXY -When running inside a LAN that uses an `HTTPS` proxy, the Docker Hub -certificates will be replaced by the proxy's certificates. These certificates -need to be added to your Docker host's configuration: +When running inside a LAN that uses an `HTTPS` proxy, the proxy's certificates +replace Docker Hub's certificates. These certificates must be added to your +Docker host's configuration: 1. Install the `ca-certificates` package for your distribution 2. Ask your network admin for the proxy's CA certificate and append them to @@ -778,21 +778,20 @@ need to be added to your Docker host's configuration: The `username:` and `password@` are optional - and are only needed if your proxy is set up to require authentication. -This will only add the proxy and authentication to the Docker daemon's requests - -your `docker build`s and running containers will need extra configuration to -use the proxy +This only adds the proxy and authentication to the Docker daemon's requests. +To use the proxy when building images and running containers, see +[Configure Docker to use a proxy server](https://docs.docker.com/network/proxy/) ### Default `ulimit` settings -`--default-ulimit` allows you to set the default `ulimit` options to use for +The `--default-ulimit` flag lets you set the default `ulimit` options to use for all containers. It takes the same options as `--ulimit` for `docker run`. If -these defaults are not set, `ulimit` settings will be inherited, if not set on -`docker run`, from the Docker daemon. Any `--ulimit` options passed to -`docker run` will overwrite these defaults. +these defaults aren't set, `ulimit` settings are inherited from the Docker daemon. +Any `--ulimit` options passed to `docker run` override the daemon defaults. -Be careful setting `nproc` with the `ulimit` flag as `nproc` is designed by Linux to -set the maximum number of processes available to a user, not to a container. For details -please check the [run](run.md) reference. +Be careful setting `nproc` with the `ulimit` flag, as `nproc` is designed by Linux to +set the maximum number of processes available to a user, not to a container. +For details, see [`docker run` reference](run.md#ulimit). ### Access authorization @@ -818,17 +817,16 @@ allow the request for it to complete. For information about how to create an authorization plugin, refer to the [authorization plugin](../../extend/plugins_authorization.md) section. - ### Daemon user namespace options The Linux kernel [user namespace support](https://man7.org/linux/man-pages/man7/user_namespaces.7.html) provides additional security by enabling a process, and therefore a container, to have a unique range of user and group IDs which are outside the traditional -user and group range utilized by the host system. Potentially the most important -security improvement is that, by default, container processes running as the -`root` user will have expected administrative privilege (with some restrictions) -inside the container but will effectively be mapped to an unprivileged `uid` on +user and group range utilized by the host system. One of the most important +security improvements is that, by default, container processes running as the +`root` user have expected administrative privileges it expects (with some restrictions) +inside the container, but are effectively mapped to an unprivileged `uid` on the host. For details about how to use this feature, as well as limitations, see @@ -856,29 +854,23 @@ PING host.docker.internal (192.0.2.0): 56 data bytes ### Miscellaneous options IP masquerading uses address translation to allow containers without a public -IP to talk to other machines on the Internet. This may interfere with some -network topologies and can be disabled with `--ip-masq=false`. +IP to talk to other machines on the internet. This may interfere with some +network topologies, and can be disabled with `--ip-masq=false`. -Docker supports softlinks for the Docker data directory (`/var/lib/docker`) and +Docker supports soft links for the Docker data directory (`/var/lib/docker`) and for `/var/lib/docker/tmp`. The `DOCKER_TMPDIR` and the data directory can be set like this: -```console -$ DOCKER_TMPDIR=/mnt/disk2/tmp /usr/local/bin/dockerd --data-root /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 -``` - -or - ```console $ export DOCKER_TMPDIR=/mnt/disk2/tmp -$ /usr/local/bin/dockerd --data-root /var/lib/docker -H unix:// > /var/lib/docker-machine/docker.log 2>&1 +$ sudo -E dockerd --data-root /var/lib/docker -H unix:// ```` #### Default cgroup parent -The `--cgroup-parent` option allows you to set the default cgroup parent -to use for containers. If this option is not set, it defaults to `/docker` for -fs cgroup driver and `system.slice` for systemd cgroup driver. +The `--cgroup-parent` option lets you set the default cgroup parent +for containers. If this option isn't set, it defaults to `/docker` for +the cgroupfs driver, and `system.slice` for the systemd cgroup driver. If the cgroup has a leading forward slash (`/`), the cgroup is created under the root cgroup, otherwise the cgroup is created under the daemon @@ -889,7 +881,7 @@ Assuming the daemon is running in cgroup `daemoncgroup`, `/sys/fs/cgroup/memory/foobar`, whereas using `--cgroup-parent=foobar` creates the cgroup in `/sys/fs/cgroup/memory/daemoncgroup/foobar` -The systemd cgroup driver has different rules for `--cgroup-parent`. Systemd +The systemd cgroup driver has different rules for `--cgroup-parent`. systemd represents hierarchy by slice and the name of the slice encodes the location in the tree. So `--cgroup-parent` for systemd cgroups should be a slice name. A name can consist of a dash-separated series of names, which describes the path @@ -903,7 +895,7 @@ the `--cgroup-parent` option on the daemon. #### Daemon metrics -The `--metrics-addr` option takes a tcp address to serve the metrics API. +The `--metrics-addr` option takes a TCP address to serve the metrics API. This feature is still experimental, therefore, the daemon must be running in experimental mode for this feature to work. @@ -913,34 +905,24 @@ allowing you to make requests on the API at `127.0.0.1:9323/metrics` to receive Port `9323` is the [default port associated with Docker metrics](https://github.com/prometheus/prometheus/wiki/Default-port-allocations) -to avoid collisions with other prometheus exporters and services. +to avoid collisions with other Prometheus exporters and services. -If you are running a prometheus server you can add this address to your scrape configs -to have prometheus collect metrics on Docker. For more information -on prometheus refer to the [prometheus website](https://prometheus.io/). +If you are running a Prometheus server you can add this address to your scrape configs +to have Prometheus collect metrics on Docker. For more information, see +[Collect Docker metrics with Prometheus](https://docs.docker.com/config/daemon/prometheus/). -```yaml -scrape_configs: - - job_name: 'docker' - static_configs: - - targets: ['127.0.0.1:9323'] -``` - -Please note that this feature is still marked as experimental as metrics and metric -names could change while this feature is still in experimental. Please provide -feedback on what you would like to see collected in the API. - -#### Node Generic Resources +#### Node generic resources The `--node-generic-resources` option takes a list of key-value pair (`key=value`) that allows you to advertise user defined resources -in a swarm cluster. +in a Swarm cluster. The current expected use case is to advertise NVIDIA GPUs so that services requesting `NVIDIA-GPU=[0-16]` can land on a node that has enough GPUs for the task to run. Example of usage: + ```json { "node-generic-resources": [ @@ -958,8 +940,8 @@ except for flags that allow several entries, where it uses the plural of the flag name, e.g., `labels` for the `label` flag. The options set in the configuration file must not conflict with options set -via flags. The docker daemon fails to start if an option is duplicated between -the file and the flags, regardless of their value. We do this to avoid +using flags. The Docker daemon fails to start if an option is duplicated between +the file and the flags, regardless of their value. This is intentional, and avoids silently ignore changes introduced in configuration reloads. For example, the daemon fails to start if you set daemon labels in the configuration file and also set daemon labels via the `--label` flag. @@ -983,14 +965,13 @@ $ echo $? 1 ``` - ##### On Linux The default location of the configuration file on Linux is -`/etc/docker/daemon.json`. The `--config-file` flag can be used to specify a - non-default location. +`/etc/docker/daemon.json`. Use the `--config-file` flag to specify a +non-default location. -This is a full example of the allowed configuration options on Linux: +The following is a full example of the allowed configuration options on Linux: ```json { @@ -1109,22 +1090,22 @@ This is a full example of the allowed configuration options on Linux: } ``` -> **Note:** +> **Note** > -> You cannot set options in `daemon.json` that have already been set on +> You can't set options in `daemon.json` that have already been set on > daemon startup as a flag. -> On systems that use `systemd` to start the Docker daemon, `-H` is already set, so -> you cannot use the `hosts` key in `daemon.json` to add listening addresses. -> See ["custom Docker daemon options"](https://docs.docker.com/config/daemon/systemd/#custom-docker-daemon-options) for how -> to accomplish this task with a systemd drop-in file. +> On systems that use systemd to start the Docker daemon, `-H` is already set, so +> you can't use the `hosts` key in `daemon.json` to add listening addresses. +> See [custom Docker daemon options](https://docs.docker.com/config/daemon/systemd/#custom-docker-daemon-options) +> for an example on how to configure the daemon using systemd drop-in files. ##### On Windows The default location of the configuration file on Windows is - `%programdata%\docker\config\daemon.json`. The `--config-file` flag can be - used to specify a non-default location. +`%programdata%\docker\config\daemon.json`. Use the `--config-file` flag +to specify a non-default location. -This is a full example of the allowed configuration options on Windows: +The following is a full example of the allowed configuration options on Windows: ```json { @@ -1170,7 +1151,8 @@ This is a full example of the allowed configuration options on Windows: } ``` -The `default-runtime` option is by default unset, in which case dockerd will auto-detect the runtime. This detection is currently based on if the `containerd` flag is set. +The `default-runtime` option is by default unset, in which case dockerd automatically detects the runtime. +This detection is based on if the `containerd` flag is set. Accepted values: @@ -1178,60 +1160,69 @@ Accepted values: - `io.containerd.runhcs.v1` - This is uses the containerd `runhcs` shim to run the container and uses the v2 HCS API's in Windows. #### Feature options -The optional field `features` in `daemon.json` allows users to enable or disable specific -daemon features. For example, `{"features":{"buildkit": true}}` enables `buildkit` as the -default docker image builder. -The list of currently supported feature options: -- `buildkit`: It enables `buildkit` as default builder when set to `true` or disables it by -`false`. Note that if this option is not explicitly set in the daemon config file, then it -is up to the cli to determine which builder to invoke. +The optional field `features` in `daemon.json` lets you enable or disable specific +daemon features. + +```json +{ + "features": { + "some-feature": true, + "some-disabled-feature-enabled-by-default": false + } +} +``` + +The list of feature options include: + +- `containerd-snapshotter`: when set to `true`, the daemon uses containerd + snapshotters instead of the classic storage drivers for storing image and + container data. For more information, see + [containerd storage](https://docs.docker.com/storage/containerd/). #### Configuration reload behavior Some options can be reconfigured when the daemon is running without requiring -to restart the process. We use the `SIGHUP` signal in Linux to reload, and a global event -in Windows with the key `Global\docker-daemon-config-$PID`. The options can -be modified in the configuration file but still will check for conflicts with -the provided flags. The daemon fails to reconfigure itself -if there are conflicts, but it won't stop execution. +to restart the process. The daemon uses the `SIGHUP` signal in Linux to reload, +and a global event in Windows with the key `Global\docker-daemon-config-$PID`. +You can modify the options in the configuration file, but the daemon still +checks for conflicting settings with the specified CLI flags. The daemon fails +to reconfigure itself if there are conflicts, but it won't stop execution. The list of currently supported options that can be reconfigured is this: -- `debug`: it changes the daemon to debug mode when set to true. -- `labels`: it replaces the daemon labels with a new set of labels. -- `live-restore`: Enables [keeping containers alive during daemon downtime](https://docs.docker.com/config/containers/live-restore/). -- `max-concurrent-downloads`: it updates the max concurrent downloads for each pull. -- `max-concurrent-uploads`: it updates the max concurrent uploads for each push. -- `max-download-attempts`: it updates the max download attempts for each pull. -- `default-runtime`: it updates the runtime to be used if not is - specified at container creation. It defaults to "default" which is - the runtime shipped with the official docker packages. -- `runtimes`: it updates the list of available OCI runtimes that can - be used to run containers. -- `authorization-plugin`: it specifies the authorization plugins to use. -- `allow-nondistributable-artifacts`: Replaces the set of registries to which the daemon will push nondistributable artifacts with a new set of registries. -- `insecure-registries`: it replaces the daemon insecure registries with a new set of insecure registries. If some existing insecure registries in daemon's configuration are not in newly reloaded insecure registries, these existing ones will be removed from daemon's config. -- `registry-mirrors`: it replaces the daemon registry mirrors with a new set of registry mirrors. If some existing registry mirrors in daemon's configuration are not in newly reloaded registry mirrors, these existing ones will be removed from daemon's config. -- `shutdown-timeout`: it replaces the daemon's existing configuration timeout with a new timeout for shutting down all containers. -- `features`: it explicitly enables or disables specific features. +| Option | Description | +| ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| `debug` | Toggles debug mode of the daemon. | +| `labels` | Replaces the daemon labels with a new set of labels. | +| `live-restore` | Toggles [live restore](https://docs.docker.com/config/containers/live-restore/). | +| `max-concurrent-downloads` | Configures the max concurrent downloads for each pull. | +| `max-concurrent-uploads` | Configures the max concurrent uploads for each push. | +| `max-download-attempts` | Configures the max download attempts for each pull. | +| `default-runtime` | Configures the runtime to be used if not is specified at container creation. | +| `runtimes` | Configures the list of available OCI runtimes that can be used to run containers. | +| `authorization-plugin` | Specifies the authorization plugins to use. | +| `allow-nondistributable-artifacts` | Specifies a list of registries to which the daemon will push non-distributable artifacts. | +| `insecure-registries` | Specifies a list of registries that the daemon should consider insecure. | +| `registry-mirrors` | Specifies a list of registry mirrors. | +| `shutdown-timeout` | Configures the daemon's existing configuration timeout with a new timeout for shutting down all containers. | +| `features` | Enables or disables specific features. | ### Run multiple daemons -> **Note:** +> **Note** > -> Running multiple daemons on a single host is considered as "experimental". The user should be aware of -> unsolved problems. This solution may not work properly in some cases. Solutions are currently under development -> and will be delivered in the near future. +> Running multiple daemons on a single host is considered experimental. +> You may encounter unsolved problems, and things may not work as expected in some cases. This section describes how to run multiple Docker daemons on a single host. To -run multiple daemons, you must configure each daemon so that it does not +run multiple daemons, you must configure each daemon so that it doesn't conflict with other daemons on the same host. You can set these options either by providing them as flags, or by using a [daemon configuration file](#daemon-configuration-file). The following daemon options must be configured for each daemon: -```console +```text -b, --bridge= Attach containers to a network bridge --exec-root=/var/run/docker Root of the Docker execdriver --data-root=/var/lib/docker Root of persisted Docker data @@ -1245,30 +1236,32 @@ The following daemon options must be configured for each daemon: ``` When your daemons use different values for these flags, you can run them on the same host without any problems. -It is very important to properly understand the meaning of those options and to use them correctly. +It is important that you understand the meaning of these options and to use them correctly. -- The `-b, --bridge=` flag is set to `docker0` as default bridge network. It is created automatically when you install Docker. -If you are not using the default, you must create and configure the bridge manually or just set it to 'none': `--bridge=none` -- `--exec-root` is the path where the container state is stored. The default value is `/var/run/docker`. Specify the path for -your running daemon here. +- The `-b, --bridge=` flag is set to `docker0` as default bridge network. + It is created automatically when you install Docker. + If you aren't using the default, you must create and configure the bridge manually, or set it to 'none': `--bridge=none` +- `--exec-root` is the path where the container state is stored. + The default value is `/var/run/docker`. + Specify the path for your running daemon here. - `--data-root` is the path where persisted data such as images, volumes, and -cluster state are stored. The default value is `/var/lib/docker`. To avoid any -conflict with other daemons, set this parameter separately for each daemon. -- `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. Specify the path for your -pid file here. -- `--host=[]` specifies where the Docker daemon will listen for client connections. If unspecified, it defaults to `/var/run/docker.sock`. -- `--iptables=false` prevents the Docker daemon from adding iptables rules. If -multiple daemons manage iptables rules, they may overwrite rules set by another -daemon. Be aware that disabling this option requires you to manually add -iptables rules to expose container ports. If you prevent Docker from adding -iptables rules, Docker will also not add IP masquerading rules, even if you set -`--ip-masq` to `true`. Without IP masquerading rules, Docker containers will not be -able to connect to external hosts or the internet when using network other than -default bridge. -- `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. You can use it instead of -daemon flags. Specify the path for each daemon. + cluster state are stored. The default value is `/var/lib/docker`. To avoid any + conflict with other daemons, set this parameter separately for each daemon. +- `-p, --pidfile=/var/run/docker.pid` is the path where the process ID of the daemon is stored. + Specify the path for your PID file here. +- `--host=[]` specifies where the Docker daemon listens for client connections. + If unspecified, it defaults to `/var/run/docker.sock`. +- `--iptables=false` prevents the Docker daemon from adding iptables rules. If + multiple daemons manage iptables rules, they may overwrite rules set by another + daemon. Be aware that disabling this option requires you to manually add + iptables rules to expose container ports. If you prevent Docker from adding + iptables rules, Docker also doesn't add IP masquerading rules, even if you set + `--ip-masq` to `true`. Without IP masquerading rules, Docker containers can't + connect to external hosts or the internet when using network other than default bridge. +- `--config-file=/etc/docker/daemon.json` is the path where configuration file is stored. + You can use it instead of daemon flags. Specify the path for each daemon. - `--tls*` Docker daemon supports `--tlsverify` mode that enforces encrypted and authenticated remote connections. -The `--tls*` options enable use of specific certificates for individual daemons. + The `--tls*` options enable use of specific certificates for individual daemons. Example script for a separate “bootstrap” instance of the Docker daemon without network: From 38aeaa71b90aa84ca3603849531ffefd34e176f1 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 20 Dec 2023 21:08:32 +0100 Subject: [PATCH 443/534] docs(plugins): update Docker Store to Docker Hub Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/plugins_network.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 9d92a96400..46075e5734 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -60,7 +60,7 @@ $ docker run --network=mynet busybox top Network plugins are written by third parties, and are published by those third parties, either on -[Docker Store](https://store.docker.com/search?category=network&q=&type=plugin) +[Docker Hub](https://hub.docker.com/search?q=&type=plugin) or on the third party's site. ## Write a network plugin From 97d1319c67b24bc84639f57a9acbe86ca1d55dbb Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 20 Dec 2023 21:14:00 +0100 Subject: [PATCH 444/534] =?UTF-8?q?docs(plugins):=20add=20Kathar=C3=A1=20e?= =?UTF-8?q?ngine=20network=20plugin?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/legacy_plugins.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 6c842ac473..fb6ca0e4cf 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -49,10 +49,11 @@ The sections below provide an inexhaustive overview of available plugins. ### Network plugins | Plugin | Description | -|:-----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| :--------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. | | [Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. | | [Weave Network Plugin](https://www.weave.works/docs/net/latest/introducing-weave/) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. | +| [Kathará Network Plugin](https://github.com/KatharaFramework/NetworkPlugin) | Docker Network Plugin used by Kathará, an open source container-based network emulation system for showing interactive demos/lessons, testing production networks in a sandbox environment, or developing new network protocols. | ### Volume plugins From 754c6ace3cfa0a35e381ebfe11a64bf9a8e23d21 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 20 Dec 2023 21:14:56 +0100 Subject: [PATCH 445/534] docs(plugins): remove inline CSS markup Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/legacy_plugins.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index fb6ca0e4cf..39836dc7db 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -42,10 +42,6 @@ Follow the instructions in the plugin's documentation. The sections below provide an inexhaustive overview of available plugins. - - ### Network plugins | Plugin | Description | From 2692b0efbc3994aee9f6efaf85ba3787f87fafde Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 20 Dec 2023 21:15:49 +0100 Subject: [PATCH 446/534] docs(plugins): remove Weave engine network plugin Linked URL returned 404, looks like there's a v2 plugin now instead Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/legacy_plugins.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 39836dc7db..31a8a655fb 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -48,7 +48,6 @@ The sections below provide an inexhaustive overview of available plugins. | :--------------------------------------------------------------------------------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [Contiv Networking](https://github.com/contiv/netplugin) | An open source network plugin to provide infrastructure and security policies for a multi-tenant micro services deployment, while providing an integration to physical network for non-container workload. Contiv Networking implements the remote driver and IPAM APIs available in Docker 1.9 onwards. | | [Kuryr Network Plugin](https://github.com/openstack/kuryr) | A network plugin is developed as part of the OpenStack Kuryr project and implements the Docker networking (libnetwork) remote driver API by utilizing Neutron, the OpenStack networking service. It includes an IPAM driver as well. | -| [Weave Network Plugin](https://www.weave.works/docs/net/latest/introducing-weave/) | A network plugin that creates a virtual network that connects your Docker containers - across multiple hosts or clouds and enables automatic discovery of applications. Weave networks are resilient, partition tolerant, secure and work in partially connected networks, and other adverse environments - all configured with delightful simplicity. | | [Kathará Network Plugin](https://github.com/KatharaFramework/NetworkPlugin) | Docker Network Plugin used by Kathará, an open source container-based network emulation system for showing interactive demos/lessons, testing production networks in a sandbox environment, or developing new network protocols. | ### Volume plugins From 709ce56467f1a639155667ea28ce081e96a1328f Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Wed, 20 Dec 2023 22:45:15 +0100 Subject: [PATCH 447/534] docs: deprecate short cid in Aliases field Related to https://github.com/moby/moby/pull/46853 Co-Authored-by: Sebastiaan van Stijn Signed-off-by: Albin Kerouanton --- docs/deprecated.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 95eee9df3c..758f4014be 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field--and-is--automated-filter-on-docker-search) | v25.0 | - | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | @@ -109,6 +110,21 @@ The table below provides an overview of the current status of deprecated feature | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Container short ID in network Aliases field + +**Deprecated in Release: v25.0** +**Target For Remove In Release: v26.0** + +The `Aliases` field returned by `docker inspect` contains the container short +ID once the container is started. This behavior is deprecated in v25.0 but +kept until the next release, v26.0. Starting with that version, the `Aliases` +field will only contain the aliases set through the `docker container create` +and `docker run` flag `--network-alias`. + +A new field `DNSNames` containing the container name (if one was specified), +the hostname, the network aliases, as well as the container short ID, has been +introduced in v25.0 and should be used instead of the `Aliases` field. + ### IsAutomated field, and "is-automated" filter on docker search **Deprecated in Release: v25.0** From 88e7e9015da65a99c6e249449097aff8c3f89f37 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 20 Dec 2023 23:14:51 +0100 Subject: [PATCH 448/534] deprecated: fix anchor and add missing version Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 758f4014be..afe1941ee7 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -51,7 +51,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | -| Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field--and-is--automated-filter-on-docker-search) | v25.0 | - | +| Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | From 6175399738419cc065ed0e59644a13617bf2c77d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 20 Dec 2023 23:25:56 +0100 Subject: [PATCH 449/534] docs/deprecated: update "oom-score-adjust" status for daemon Using this option on the daemon will now produce an error (flag will be removed entirely in v26.0). Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index afe1941ee7..28ffbdcb8b 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,7 +53,7 @@ The table below provides an overview of the current status of deprecated feature | Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | -| Deprecated | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | +| Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0 | - | @@ -153,7 +153,7 @@ after upgrading. ### OOM-score adjust for the daemon **Deprecated in Release: v24.0** -**Target For Removal In Release: v25.0** +**Removed in Release: v25.0** The `oom-score-adjust` option was added to prevent the daemon from being OOM-killed before other processes. This option was mostly added as a From 2a631ed52e5ed8360efffb921f30c1e5dae35d38 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 11 Jan 2024 13:54:39 +0100 Subject: [PATCH 450/534] docs: dockerd: fix stray column-separateor in env-var table Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 313332bde3..f465530902 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -142,7 +142,7 @@ by the `dockerd` command line: | `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | | `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | | `HTTPS_PROXY` | Proxy URL for HTTPS requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | -| `MOBY_DISABLE_PIGZ` | Disables the use of [`unpigz`](https://linux.die.net/man/1/pigz) to decompress layers in parallel when pulling images, even if it is installed. | | +| `MOBY_DISABLE_PIGZ` | Disables the use of [`unpigz`](https://linux.die.net/man/1/pigz) to decompress layers in parallel when pulling images, even if it is installed. | | `NO_PROXY` | Comma-separated values specifying hosts that should be excluded from proxying. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | ## Examples From affb82f0c6d0aa26e5cf0437408984ff5cd43933 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 11 Jan 2024 13:55:10 +0100 Subject: [PATCH 451/534] docs: dockerd: rephrase environment-variables intro Slightly rephrase the intro (remove "easy reference"), and cross-reference the corresponding section on the docker CLI page. Signed-off-by: Sebastiaan van Stijn --- docs/reference/commandline/dockerd.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index f465530902..996e0253a0 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -130,8 +130,11 @@ to [the `daemon.json` file](#daemon-configuration-file). ### Environment variables -For easy reference, the following list of environment variables are supported -by the `dockerd` command line: +The following list of environment variables are supported by the `dockerd` daemon. +Some of these environment variables are supported both by the Docker Daemon and +the `docker` CLI. Refer to [Environment variables](cli.md#environment-variables) +in the CLI section to learn about environment variables supported by the +`docker` CLI. | Variable | Description | |:--------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| From 93f3312996351e1e354f1b48fed1dbc7d15a89ab Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Sat, 9 Dec 2023 02:24:30 +0100 Subject: [PATCH 452/534] deprecate old API versions (1.23 and lower) The Docker daemon provides a versioned API for backward compatibility with old clients. Docker clients can perform API-version negotiation to select the most recent API version supported by the daemon (downgrading to and older version of the API when necessary). API version negotiation was introduced in Docker v1.12.0 (API 1.24), and clients before that used a fixed API version. Docker Engine versions through v25.0 provide support for all [API versions](https://docs.docker.com/engine/api/#api-version-matrix) included in stable releases for a given platform. For Docker daemons on Linux, the earliest supported API version is 1.12 (corresponding with Docker Engine v1.0.0), whereas for Docker daemons on Windows, the earliest supported API version is 1.24 (corresponding with Docker Engine v1.12.0). Support for legacy API versions (providing old API versions on current versions of the Docker Engine) is primarily intended to provide compatibility with recent, but still supported versions of the client, which is a common scenario (the Docker daemon may be updated to the latest release, but not all clients may be up-to-date or vice versa). Support for API versions before that (API versions provided by EOL versions of the Docker Daemon) is provided on a "best effort" basis. Use of old API versions is very rare, and support for legacy API versions involves significant complexity (Docker 1.0.0 having been released 10 years ago). Because of this, we'll start deprecating support for legacy API versions. Docker Engine v25.0 by default disables API version older than 1.24 (aligning the minimum supported API version between Linux and Windows daemons). When connecting with a client that uses an API version version older than 1.24, the daemon returns an error. The following example configures the docker CLI to use API version 1.23, which produces an error: DOCKER_API_VERSION=1.23 docker version Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version An environment variable (`DOCKER_MIN_API_VERSION`) is introduced that allows re-enabling older API versions in the daemon. This environment variable must be set in the daemon's environment (for example, through a [systemd override file](https://docs.docker.com/config/daemon/systemd/)), and the specified API version must be supported by the daemon (`1.12` or higher on Linux, or `1.24` or higher on Windows). Support for API versions lower than `1.24` will be permanently removed in Docker Engine v26, and the minimum supported API version will be incrementally raised in releases following that. We do not recommend depending on the `DOCKER_MIN_API_VERSION` environment variable other than for exceptional cases where it's not possible to update old clients, and those clients must be supported. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 54 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 28ffbdcb8b..7a68c0f60b 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | @@ -109,6 +110,59 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Deprecate legacy API versions + +**Deprecated in Release: v25.0** +**Target For Removal In Release: v26.0** + +The Docker daemon provides a versioned API for backward compatibility with old +clients. Docker clients can perform API-version negotiation to select the most +recent API version supported by the daemon (downgrading to and older version of +the API when necessary). API version negotiation was introduced in Docker v1.12.0 +(API 1.24), and clients before that used a fixed API version. + +Docker Engine versions through v25.0 provide support for all [API versions](https://docs.docker.com/engine/api/#api-version-matrix) +included in stable releases for a given platform. For Docker daemons on Linux, +the earliest supported API version is 1.12 (corresponding with Docker Engine +v1.0.0), whereas for Docker daemons on Windows, the earliest supported API +version is 1.24 (corresponding with Docker Engine v1.12.0). + +Support for legacy API versions (providing old API versions on current versions +of the Docker Engine) is primarily intended to provide compatibility with recent, +but still supported versions of the client, which is a common scenario (the Docker +daemon may be updated to the latest release, but not all clients may be up-to-date +or vice versa). Support for API versions before that (API versions provided by +EOL versions of the Docker Daemon) is provided on a "best effort" basis. + +Use of old API versions is very rare, and support for legacy API versions +involves significant complexity (Docker 1.0.0 having been released 10 years ago). +Because of this, we'll start deprecating support for legacy API versions. + +Docker Engine v25.0 by default disables API version older than 1.24 (aligning +the minimum supported API version between Linux and Windows daemons). When +connecting with a client that uses an API version version older than 1.24, +the daemon returns an error. The following example configures the docker +CLI to use API version 1.23, which produces an error: + +```console +DOCKER_API_VERSION=1.23 docker version +Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version +``` + +An environment variable (`DOCKER_MIN_API_VERSION`) is introduced that allows +re-enabling older API versions in the daemon. This environment variable must +be set in the daemon's environment (for example, through a [systemd override +file](https://docs.docker.com/config/daemon/systemd/)), and the specified +API version must be supported by the daemon (`1.12` or higher on Linux, or +`1.24` or higher on Windows). + +Support for API versions lower than `1.24` will be permanently removed in Docker +Engine v26, and the minimum supported API version will be incrementally raised +in releases following that. + +We do not recommend depending on the `DOCKER_MIN_API_VERSION` environment +variable other than for exceptional cases where it's not possible to update +old clients, and those clients must be supported. ### Container short ID in network Aliases field From 31662c32130179edd725d40552772a6740ee5455 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Tue, 16 Jan 2024 20:19:06 +0100 Subject: [PATCH 453/534] docs: add "builder" key to full daemon.json example Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 996e0253a0..d4e59ac1f0 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -983,6 +983,17 @@ The following is a full example of the allowed configuration options on Linux: "authorization-plugins": [], "bip": "", "bridge": "", + "builder": { + "gc": { + "enabled": true, + "defaultKeepStorage": "10GB", + "policy": [ + { "keepStorage": "10GB", "filter": ["unused-for=2200h"] }, + { "keepStorage": "50GB", "filter": ["unused-for=3300h"] }, + { "keepStorage": "100GB", "all": true } + ] + } + }, "cgroup-parent": "", "containerd": "/run/containerd/containerd.sock", "containerd-namespace": "docker", From abd54e6b69330001c26dc161968cc23dd1ec38fa Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 10 Jan 2024 10:38:42 +0100 Subject: [PATCH 454/534] docs(plugins): fix typos and formatting Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/config.md | 191 +++++++++++++-------------- docs/extend/index.md | 127 +++++++++--------- docs/extend/legacy_plugins.md | 16 +-- docs/extend/plugin_api.md | 41 +++--- docs/extend/plugins_authorization.md | 31 ++--- docs/extend/plugins_logging.md | 67 +++++----- docs/extend/plugins_metrics.md | 34 ++--- docs/extend/plugins_network.md | 21 +-- docs/extend/plugins_volume.md | 78 ++++++----- 9 files changed, 266 insertions(+), 340 deletions(-) diff --git a/docs/extend/config.md b/docs/extend/config.md index 6af92c55b4..1f96d3e009 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -1,192 +1,185 @@ --- description: "How to develop and use a plugin with the managed plugin system" keywords: "API, Usage, plugins, documentation, developer" +title: Plugin Config Version 1 of Plugin V2 --- - - - -# Plugin Config Version 1 of Plugin V2 - This document outlines the format of the V0 plugin configuration. -Plugin configs describe the various constituents of a docker plugin. Plugin -configs can be serialized to JSON format with the following media types: +Plugin configs describe the various constituents of a Docker engine plugin. +Plugin configs can be serialized to JSON format with the following media types: | Config Type | Media Type | |-------------|-----------------------------------------| -| config | "application/vnd.docker.plugin.v1+json" | +| config | `application/vnd.docker.plugin.v1+json` | -## *Config* Field Descriptions +## Config Field Descriptions -Config provides the base accessible fields for working with V0 plugin format - in the registry. +Config provides the base accessible fields for working with V0 plugin format in +the registry. -- **`description`** *string* +- `description` string - description of the plugin + Description of the plugin -- **`documentation`** *string* +- `documentation` string - link to the documentation about the plugin + Link to the documentation about the plugin -- **`interface`** *PluginInterface* +- `interface` PluginInterface - interface implemented by the plugins, struct consisting of the following fields + Interface implemented by the plugins, struct consisting of the following fields: - - **`types`** *string array* + - `types` string array - types indicate what interface(s) the plugin currently implements. + Types indicate what interface(s) the plugin currently implements. - currently supported: + Supported types: - - **docker.volumedriver/1.0** + - `docker.volumedriver/1.0` - - **docker.networkdriver/1.0** + - `docker.networkdriver/1.0` - - **docker.ipamdriver/1.0** + - `docker.ipamdriver/1.0` - - **docker.authz/1.0** + - `docker.authz/1.0` - - **docker.logdriver/1.0** + - `docker.logdriver/1.0` - - **docker.metricscollector/1.0** + - `docker.metricscollector/1.0` - - **`socket`** *string* + - `socket` string - socket is the name of the socket the engine should use to communicate with the plugins. - the socket will be created in `/run/docker/plugins`. + Socket is the name of the socket the engine should use to communicate with the plugins. + the socket will be created in `/run/docker/plugins`. +- `entrypoint` string array -- **`entrypoint`** *string array* + Entrypoint of the plugin, see [`ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) - entrypoint of the plugin, see [`ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) +- `workdir` string -- **`workdir`** *string* + Working directory of the plugin, see [`WORKDIR`](https://docs.docker.com/engine/reference/builder/#workdir) - workdir of the plugin, see [`WORKDIR`](https://docs.docker.com/engine/reference/builder/#workdir) +- `network` PluginNetwork -- **`network`** *PluginNetwork* + Network of the plugin, struct consisting of the following fields: - network of the plugin, struct consisting of the following fields + - `type` string - - **`type`** *string* + Network type. - network type. + Supported types: - currently supported: + - `bridge` + - `host` + - `none` - - **bridge** - - **host** - - **none** +- `mounts` PluginMount array -- **`mounts`** *PluginMount array* + Mount of the plugin, struct consisting of the following fields. + See [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts). - mount of the plugin, struct consisting of the following fields, see [`MOUNTS`](https://github.com/opencontainers/runtime-spec/blob/master/config.md#mounts) + - `name` string - - **`name`** *string* + Name of the mount. - name of the mount. + - `description` string - - **`description`** *string* + Description of the mount. - description of the mount. + - `source` string - - **`source`** *string* + Source of the mount. - source of the mount. + - `destination` string - - **`destination`** *string* + Destination of the mount. - destination of the mount. + - `type` string - - **`type`** *string* + Mount type. - mount type. + - `options` string array - - **`options`** *string array* + Options of the mount. - options of the mount. +- `ipchost` Boolean -- **`ipchost`** *boolean* Access to host ipc namespace. -- **`pidhost`** *boolean* - Access to host pid namespace. -- **`propagatedMount`** *string* +- `pidhost` Boolean - path to be mounted as rshared, so that mounts under that path are visible to docker. This is useful for volume plugins. - This path will be bind-mounted outside of the plugin rootfs so it's contents - are preserved on upgrade. + Access to host PID namespace. -- **`env`** *PluginEnv array* +- `propagatedMount` string - env of the plugin, struct consisting of the following fields + Path to be mounted as rshared, so that mounts under that path are visible to + Docker. This is useful for volume plugins. This path will be bind-mounted + outside of the plugin rootfs so it's contents are preserved on upgrade. - - **`name`** *string* +- `env` PluginEnv array - name of the env. + Environment variables of the plugin, struct consisting of the following fields: - - **`description`** *string* + - `name` string - description of the env. + Name of the environment variable. - - **`value`** *string* + - `description` string - value of the env. + Description of the environment variable. -- **`args`** *PluginArgs* + - `value` string - args of the plugin, struct consisting of the following fields + Value of the environment variable. - - **`name`** *string* +- `args` PluginArgs - name of the args. + Arguments of the plugin, struct consisting of the following fields: - - **`description`** *string* + - `name` string - description of the args. + Name of the arguments. - - **`value`** *string array* + - `description` string - values of the args. + Description of the arguments. -- **`linux`** *PluginLinux* + - `value` string array - - **`capabilities`** *string array* + Values of the arguments. - capabilities of the plugin (*Linux only*), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) +- `linux` PluginLinux - - **`allowAllDevices`** *boolean* + - `capabilities` string array - If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. + Capabilities of the plugin (Linux only), see list [`here`](https://github.com/opencontainers/runc/blob/master/libcontainer/SPEC.md#security) - - **`devices`** *PluginDevice array* + - `allowAllDevices` Boolean - device of the plugin, (*Linux only*), struct consisting of the following fields, see [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices) + If `/dev` is bind mounted from the host, and allowAllDevices is set to true, the plugin will have `rwm` access to all devices on the host. - - **`name`** *string* + - `devices` PluginDevice array - name of the device. + Device of the plugin, (Linux only), struct consisting of the following fields. + See [`DEVICES`](https://github.com/opencontainers/runtime-spec/blob/master/config-linux.md#devices). - - **`description`** *string* + - `name` string - description of the device. + Name of the device. - - **`path`** *string* + - `description` string - path of the device. + Description of the device. + + - `path` string + + Path of the device. ## Example Config -*Example showing the 'tiborvass/sample-volume-plugin' plugin config.* +The following example shows the 'tiborvass/sample-volume-plugin' plugin config. ```json { diff --git a/docs/extend/index.md b/docs/extend/index.md index b0c2f29589..22ab0f3f30 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -1,24 +1,14 @@ --- +title: Docker Engine managed plugin system description: Develop and use a plugin with the managed plugin system keywords: "API, Usage, plugins, documentation, developer" --- - - -# Docker Engine managed plugin system - - [Installing and using a plugin](index.md#installing-and-using-a-plugin) - [Developing a plugin](index.md#developing-a-plugin) - [Debugging plugins](index.md#debugging-plugins) -Docker Engine's plugin system allows you to install, start, stop, and remove +Docker Engine's plugin system lets you install, start, stop, and remove plugins using Docker Engine. For information about legacy (non-managed) plugins, refer to @@ -49,78 +39,78 @@ enabled, and use it to create a volume. > **Note** > > This example is intended for instructional purposes only. Once the volume is -> created, your SSH password to the remote host will be exposed as plaintext -> when inspecting the volume. You should delete the volume as soon as you are -> done with the example. +> created, your SSH password to the remote host is exposed as plaintext when +> inspecting the volume. Delete the volume as soon as you are done with the +> example. -1. Install the `sshfs` plugin. +1. Install the `sshfs` plugin. - ```console - $ docker plugin install vieux/sshfs + ```console + $ docker plugin install vieux/sshfs - Plugin "vieux/sshfs" is requesting the following privileges: - - network: [host] - - capabilities: [CAP_SYS_ADMIN] - Do you grant the above permissions? [y/N] y + Plugin "vieux/sshfs" is requesting the following privileges: + - network: [host] + - capabilities: [CAP_SYS_ADMIN] + Do you grant the above permissions? [y/N] y - vieux/sshfs - ``` + vieux/sshfs + ``` - The plugin requests 2 privileges: + The plugin requests 2 privileges: - - It needs access to the `host` network. - - It needs the `CAP_SYS_ADMIN` capability, which allows the plugin to run - the `mount` command. + - It needs access to the `host` network. + - It needs the `CAP_SYS_ADMIN` capability, which allows the plugin to run + the `mount` command. -2. Check that the plugin is enabled in the output of `docker plugin ls`. +2. Check that the plugin is enabled in the output of `docker plugin ls`. - ```console - $ docker plugin ls + ```console + $ docker plugin ls - ID NAME TAG DESCRIPTION ENABLED - 69553ca1d789 vieux/sshfs latest the `sshfs` plugin true - ``` + ID NAME TAG DESCRIPTION ENABLED + 69553ca1d789 vieux/sshfs latest the `sshfs` plugin true + ``` -3. Create a volume using the plugin. - This example mounts the `/remote` directory on host `1.2.3.4` into a - volume named `sshvolume`. +3. Create a volume using the plugin. + This example mounts the `/remote` directory on host `1.2.3.4` into a + volume named `sshvolume`. - This volume can now be mounted into containers. + This volume can now be mounted into containers. - ```console - $ docker volume create \ - -d vieux/sshfs \ - --name sshvolume \ - -o sshcmd=user@1.2.3.4:/remote \ - -o password=$(cat file_containing_password_for_remote_host) + ```console + $ docker volume create \ + -d vieux/sshfs \ + --name sshvolume \ + -o sshcmd=user@1.2.3.4:/remote \ + -o password=$(cat file_containing_password_for_remote_host) - sshvolume - ``` + sshvolume + ``` -4. Verify that the volume was created successfully. +4. Verify that the volume was created successfully. - ```console - $ docker volume ls + ```console + $ docker volume ls - DRIVER NAME - vieux/sshfs sshvolume - ``` + DRIVER NAME + vieux/sshfs sshvolume + ``` -5. Start a container that uses the volume `sshvolume`. +5. Start a container that uses the volume `sshvolume`. - ```console - $ docker run --rm -v sshvolume:/data busybox ls /data + ```console + $ docker run --rm -v sshvolume:/data busybox ls /data - - ``` + + ``` -6. Remove the volume `sshvolume` +6. Remove the volume `sshvolume` - ```console - $ docker volume rm sshvolume + ```console + $ docker volume rm sshvolume - sshvolume - ``` + sshvolume + ``` To disable a plugin, use the `docker plugin disable` command. To completely remove it, use the `docker plugin remove` command. For other available @@ -134,8 +124,10 @@ commands and options, see the The `rootfs` directory represents the root filesystem of the plugin. In this example, it was created from a Dockerfile: -> **Note:** The `/run/docker/plugins` directory is mandatory inside of the -> plugin's filesystem for docker to communicate with the plugin. +> **Note** +> +> The `/run/docker/plugins` directory is mandatory inside of the +> plugin's filesystem for Docker to communicate with the plugin. ```console $ git clone https://github.com/vieux/docker-volume-sshfs @@ -219,11 +211,10 @@ INFO[0421] Path Called... Returned path /data/samplevol plugin=f52a3df433b9a INFO[0421] Unmount Called... Unmounted samplevol plugin=f52a3df433b9aceee436eaada0752f5797aab1de47e5485f1690a073b860ff62 ``` -#### Using docker-runc to obtain logfiles and shell into the plugin. +#### Using runc to obtain logfiles and shell into the plugin. -`docker-runc`, the default docker container runtime can be used for debugging -plugins. This is specifically useful to collect plugin logs if they are -redirected to a file. +Use `runc`, the default docker container runtime, for debugging plugins by +collecting plugin logs redirected to a file. ```console $ sudo runc --root /run/docker/runtime-runc/plugins.moby list diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 31a8a655fb..0086761c19 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -1,21 +1,11 @@ --- +title: Use Docker Engine plugins aliases: - "/engine/extend/plugins/" description: "How to add additional functionality to Docker with plugins extensions" keywords: "Examples, Usage, plugins, docker, documentation, user guide" --- - - -# Use Docker Engine plugins - This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker, refer to [Docker Engine plugin system](index.md). @@ -40,7 +30,7 @@ Follow the instructions in the plugin's documentation. ## Finding a plugin -The sections below provide an inexhaustive overview of available plugins. +The sections below provide an overview of available third-party plugins. ### Network plugins @@ -97,4 +87,4 @@ of the plugin for help. The Docker team may not be able to assist you. ## Writing a plugin If you are interested in writing a plugin for Docker, or seeing how they work -under the hood, see the [docker plugins reference](plugin_api.md). +under the hood, see the [Docker plugins reference](plugin_api.md). diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 812b465085..4600992c4e 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -1,19 +1,9 @@ --- +title: Docker Plugin API description: "How to write Docker plugins extensions " keywords: "API, Usage, plugins, documentation, developer" --- - - -# Docker Plugin API - Docker plugins are out-of-process extensions which add capabilities to the Docker Engine. @@ -26,8 +16,8 @@ If you just want to learn about or use Docker plugins, look ## What plugins are -A plugin is a process running on the same or a different host as the docker daemon, -which registers itself by placing a file on the same docker host in one of the plugin +A plugin is a process running on the same or a different host as the Docker daemon, +which registers itself by placing a file on the daemon host in one of the plugin directories described in [Plugin discovery](#plugin-discovery). Plugins have human-readable names, which are short, lowercase strings. For @@ -43,26 +33,26 @@ user or container tries to use one by name. There are three types of files which can be put in the plugin directory. -* `.sock` files are UNIX domain sockets. +* `.sock` files are Unix domain sockets. * `.spec` files are text files containing a URL, such as `unix:///other.sock` or `tcp://localhost:8080`. * `.json` files are text files containing a full json specification for the plugin. -Plugins with UNIX domain socket files must run on the same docker host, whereas -plugins with spec or json files can run on a different host if a remote URL is specified. +Plugins with Unix domain socket files must run on the same host as the Docker daemon. +Plugins with `.spec` or `.json` files can run on a different host if you specify a remote URL. -UNIX domain socket files must be located under `/run/docker/plugins`, whereas +Unix domain socket files must be located under `/run/docker/plugins`, whereas spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker/plugins`. The name of the file (excluding the extension) determines the plugin name. -For example, the `flocker` plugin might create a UNIX socket at +For example, the `flocker` plugin might create a Unix socket at `/run/docker/plugins/flocker.sock`. You can define each plugin into a separated subdirectory if you want to isolate definitions from each other. For example, you can create the `flocker` socket under `/run/docker/plugins/flocker/flocker.sock` and only mount `/run/docker/plugins/flocker` inside the `flocker` container. -Docker always searches for unix sockets in `/run/docker/plugins` first. It checks for spec or json files under +Docker always searches for Unix sockets in `/run/docker/plugins` first. It checks for spec or json files under `/etc/docker/plugins` and `/usr/lib/docker/plugins` if the socket doesn't exist. The directory scan stops as soon as it finds the first plugin definition with the given name. @@ -87,7 +77,7 @@ The `TLSConfig` field is optional and TLS will only be verified if this configur ## Plugin lifecycle -Plugins should be started before Docker, and stopped after Docker. For +Plugins should be started before Docker, and stopped after Docker. For example, when packaging a plugin for a platform which supports `systemd`, you might use [`systemd` dependencies]( https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Before=) to @@ -103,7 +93,7 @@ When a plugin is first referred to -- either by a user referring to it by name use a plugin being started -- Docker looks for the named plugin in the plugin directory and activates it with a handshake. See Handshake API below. -Plugins are *not* activated automatically at Docker daemon startup. Rather, +Plugins are not activated automatically at Docker daemon startup. Rather, they are activated only lazily, or on-demand, when they are needed. ## Systemd socket activation @@ -149,8 +139,8 @@ or if one of the plugin goes down accidentally). The Plugin API is RPC-style JSON over HTTP, much like webhooks. -Requests flow *from* the Docker daemon *to* the plugin. So the plugin needs to -implement an HTTP server and bind this to the UNIX socket mentioned in the +Requests flow from the Docker daemon to the plugin. The plugin needs to +implement an HTTP server and bind this to the Unix socket mentioned in the "plugin discovery" section. All requests are HTTP `POST` requests. @@ -164,9 +154,9 @@ Plugins are activated via the following "handshake" API call. ### /Plugin.Activate -**Request:** empty body +Request: empty body -**Response:** +Response: ```json { @@ -183,7 +173,6 @@ Possible values are: * [`NetworkDriver`](plugins_network.md) * [`VolumeDriver`](plugins_volume.md) - ## Plugin retries Attempts to call a method on a plugin are retried with an exponential backoff diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 203c1bf2e2..8a646e50fa 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -1,22 +1,12 @@ --- +title: Access authorization plugin description: "How to create authorization plugins to manage access control to your Docker daemon." keywords: "security, authorization, authentication, docker, documentation, plugin, extend" aliases: - "/engine/extend/authorization/" --- - - -# Access authorization plugin - -This document describes the Docker Engine plugins generally available in Docker +This document describes the Docker Engine plugins available in Docker Engine. To view information on plugins managed by Docker Engine, refer to [Docker Engine plugin system](index.md). @@ -41,7 +31,7 @@ third-party components using a generic API. The access authorization subsystem was built using this mechanism. Using this subsystem, you don't need to rebuild the Docker daemon to add an -authorization plugin. You can add a plugin to an installed Docker daemon. You do +authorization plugin. You can add a plugin to an installed Docker daemon. You do need to restart the Docker daemon to add a new plugin. An authorization plugin approves or denies requests to the Docker daemon based @@ -158,7 +148,7 @@ should implement the following two methods: #### /AuthZPlugin.AuthZReq -**Request**: +Request ```json { @@ -171,7 +161,7 @@ should implement the following two methods: } ``` -**Response**: +Response ```json { @@ -183,7 +173,7 @@ should implement the following two methods: #### /AuthZPlugin.AuthZRes -**Request**: +Request: ```json { @@ -199,7 +189,7 @@ should implement the following two methods: } ``` -**Response**: +Response: ```json { @@ -224,7 +214,6 @@ Request URI | string | The HTTP request URI including API Request headers | map[string]string | Request headers as key value pairs (without the authorization header) Request body | []byte | Raw request body - #### Plugin -> Daemon Name | Type | Description @@ -239,7 +228,6 @@ The plugin must support two authorization messages formats, one from the daemon #### Daemon -> Plugin - Name | Type | Description ----------------------- |------------------ |---------------------------------------------------- User | string | The user identification @@ -248,10 +236,9 @@ Request method | string | The HTTP method (GET/DELETE/POST) Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) Request headers | map[string]string | Request headers as key value pairs (without the authorization header) Request body | []byte | Raw request body -Response status code | int | Status code from the docker daemon +Response status code | int | Status code from the Docker daemon Response headers | map[string]string | Response headers as key value pairs -Response body | []byte | Raw docker daemon response body - +Response body | []byte | Raw Docker daemon response body #### Plugin -> Daemon diff --git a/docs/extend/plugins_logging.md b/docs/extend/plugins_logging.md index d6ce96cfb3..2d30571fe3 100644 --- a/docs/extend/plugins_logging.md +++ b/docs/extend/plugins_logging.md @@ -1,19 +1,9 @@ --- +title: Docker log driver plugins description: "Log driver plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide, logging" --- - - -# Docker log driver plugins - This document describes logging driver plugins for Docker. Logging drivers enables users to forward container logs to another service for @@ -46,20 +36,21 @@ receiving logs for. Logs will be streamed over the defined file in the request. On Linux this file is a FIFO. Logging plugins are not currently supported on Windows. -**Request**: +Request: + ```json { - "File": "/path/to/file/stream", - "Info": { - "ContainerID": "123456" - } + "File": "/path/to/file/stream", + "Info": { + "ContainerID": "123456" + } } ``` `File` is the path to the log stream that needs to be consumed. Each call to `StartLogging` should provide a different file path, even if it's a container that the plugin has already received logs for prior. The file is created by -docker with a randomly generated name. +Docker with a randomly generated name. `Info` is details about the container that's being logged. This is fairly free-form, but is defined by the following struct definition: @@ -81,14 +72,14 @@ type Info struct { } ``` - `ContainerID` will always be supplied with this struct, but other fields may be empty or missing. -**Response** +Response: + ```json { - "Err": "" + "Err": "" } ``` @@ -102,12 +93,12 @@ write to its stdio streams. Log stream messages are encoded as protocol buffers. The protobuf definitions are in the -[docker repository](https://github.com/docker/docker/blob/master/api/types/plugins/logdriver/entry.proto). +[moby repository](https://github.com/moby/moby/blob/master/api/types/plugins/logdriver/entry.proto). Since protocol buffers are not self-delimited you must decode them from the stream using the following stream format: -``` +```text [size][message] ``` @@ -127,17 +118,19 @@ losing log data. Requests on this endpoint does not mean that the container has been removed only that it has stopped. -**Request**: +Request: + ```json { - "File": "/path/to/file/stream" + "File": "/path/to/file/stream" } ``` -**Response**: +Response: + ```json { - "Err": "" + "Err": "" } ``` @@ -154,15 +147,17 @@ Logging plugins can implement two extra logging endpoints: Defines the capabilities of the log driver. You must implement this endpoint for Docker to be able to take advantage of any of the defined capabilities. -**Request**: +Request: + ```json {} ``` -**Response**: +Response: + ```json { - "ReadLogs": true + "ReadLogs": true } ``` @@ -180,14 +175,14 @@ called. In order for Docker to use this endpoint, the plugin must specify as much when `/LogDriver.Capabilities` is called. +Request: -**Request**: ```json { - "ReadConfig": {}, - "Info": { - "ContainerID": "123456" - } + "ReadConfig": {}, + "Info": { + "ContainerID": "123456" + } } ``` @@ -210,9 +205,9 @@ as they come in once the existing logs have been read. `Info` is the same type defined in `/LogDriver.StartLogging`. It should be used to determine what set of logs to read. -**Response**: +Response: -``` +```text {{ log stream }} ``` diff --git a/docs/extend/plugins_metrics.md b/docs/extend/plugins_metrics.md index c698a5a137..dc47e46926 100644 --- a/docs/extend/plugins_metrics.md +++ b/docs/extend/plugins_metrics.md @@ -1,20 +1,10 @@ --- +title: Docker metrics collector plugins description: "Metrics plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide, metrics" --- - - -# Docker metrics collector plugins - -Docker exposes internal metrics based on the prometheus format. Metrics plugins +Docker exposes internal metrics based on the Prometheus format. Metrics plugins enable accessing these metrics in a consistent way by providing a Unix socket at a predefined path where the plugin can scrape the metrics. @@ -44,17 +34,19 @@ plugin's rootfs. Signals to the plugin that the metrics socket is now available for scraping -**Request** +Request: + ```json {} ``` -The request has no playload. +The request has no payload. + +Response: -**Response** ```json { - "Err": "" + "Err": "" } ``` @@ -67,17 +59,19 @@ or an empty value for the `Err` field. Errors will only be logged. Signals to the plugin that the metrics socket is no longer available. This may happen when the daemon is shutting down. -**Request** +Request: + ```json {} ``` -The request has no playload. +The request has no payload. + +Response: -**Response** ```json { - "Err": "" + "Err": "" } ``` diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 46075e5734..e24fd26257 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -1,19 +1,9 @@ --- +title: Docker network driver plugins description: "Network driver plugins." keywords: "Examples, Usage, plugins, docker, documentation, user guide" --- - - -# Docker network driver plugins - This document describes Docker Engine network driver plugins generally available in Docker Engine. To view information on plugins managed by Docker Engine, refer to [Docker Engine plugin system](index.md). @@ -26,11 +16,11 @@ LibNetwork, which shares plugin infrastructure with Engine. Effectively, network driver plugins are activated in the same way as other plugins, and use the same kind of protocol. -## Network plugins and swarm mode +## Network plugins and Swarm mode -[Legacy plugins](legacy_plugins.md) do not work in swarm mode. However, -plugins written using the [v2 plugin system](index.md) do work in swarm mode, as -long as they are installed on each swarm worker node. +[Legacy plugins](legacy_plugins.md) do not work in Swarm mode. However, +plugins written using the [v2 plugin system](index.md) do work in Swarm mode, as +long as they are installed on each Swarm worker node. ## Use network driver plugins @@ -55,7 +45,6 @@ referring to that network will be sent to the plugin, $ docker run --network=mynet busybox top ``` - ## Find network plugins Network plugins are written by third parties, and are published by those diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 23e7cddf9e..3e42bc75aa 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -1,19 +1,9 @@ --- +title: Docker volume plugins description: "How to manage data with external volume plugins" keywords: "Examples, Usage, volume, docker, data, volumes, plugin, api" --- - - -# Docker volume plugins - Docker Engine volume plugins enable Engine deployments to be integrated with external storage systems such as Amazon EBS, and enable data volumes to persist beyond the lifetime of a single Docker host. See the @@ -50,7 +40,7 @@ beyond the lifetime of a single Docker host. See the ## Command-line changes To give a container access to a volume, use the `--volume` and `--volume-driver` -flags on the `docker container run` command. The `--volume` (or `-v`) flag +flags on the `docker container run` command. The `--volume` (or `-v`) flag accepts a volume name and path on the host, and the `--volume-driver` flag accepts a driver type. @@ -98,7 +88,8 @@ the volumes available by bind-mounting the provided paths into the containers. ### `/VolumeDriver.Create` -**Request**: +Request: + ```json { "Name": "volume_name", @@ -111,18 +102,20 @@ specified volume name. The plugin does not need to actually manifest the volume on the filesystem yet (until `Mount` is called). `Opts` is a map of driver specific options passed through from the user request. -**Response**: +Response: + ```json { "Err": "" } ``` -Respond with a string error if an error occurred. + Respond with a string error if an error occurred. ### `/VolumeDriver.Remove` -**Request**: +Request: + ```json { "Name": "volume_name" @@ -132,7 +125,8 @@ Respond with a string error if an error occurred. Delete the specified volume from disk. This request is issued when a user invokes `docker rm -v` to remove volumes associated with a container. -**Response**: +Response: + ```json { "Err": "" @@ -143,7 +137,8 @@ Respond with a string error if an error occurred. ### `/VolumeDriver.Mount` -**Request**: +Request: + ```json { "Name": "volume_name", @@ -158,9 +153,9 @@ at the first mount request and deprovision at the last corresponding unmount req `ID` is a unique ID for the caller that is requesting the mount. -**Response**: +Response: -- **v1**: +- v1 ```json { @@ -169,7 +164,7 @@ at the first mount request and deprovision at the last corresponding unmount req } ``` -- **v2**: +- v2 ```json { @@ -185,7 +180,7 @@ has been made available. ### `/VolumeDriver.Path` -**Request**: +Request: ```json { @@ -195,9 +190,9 @@ has been made available. Request the path to the volume with the given `volume_name`. -**Response**: +Response: -- **v1**: +- v1 ```json { @@ -206,7 +201,7 @@ Request the path to the volume with the given `volume_name`. } ``` -- **v2**: +- v2 ```json { @@ -223,7 +218,8 @@ is not provided. ### `/VolumeDriver.Unmount` -**Request**: +Request: + ```json { "Name": "volume_name", @@ -237,7 +233,8 @@ this point. `ID` is a unique ID for the caller that is requesting the mount. -**Response**: +Response: + ```json { "Err": "" @@ -246,10 +243,10 @@ this point. Respond with a string error if an error occurred. - ### `/VolumeDriver.Get` -**Request**: +Request: + ```json { "Name": "volume_name" @@ -258,10 +255,9 @@ Respond with a string error if an error occurred. Get info about `volume_name`. +Response: -**Response**: - -- **v1**: +- v1 ```json { @@ -274,7 +270,7 @@ Get info about `volume_name`. } ``` -- **v2**: +- v2 ```json { @@ -293,16 +289,17 @@ optional. ### /VolumeDriver.List -**Request**: +Request: + ```json {} ``` Get the list of volumes registered with the plugin. -**Response**: +Response: -- **v1**: +- v1 ```json { @@ -316,7 +313,7 @@ Get the list of volumes registered with the plugin. } ``` -- **v2**: +- v2 ```json { @@ -330,12 +327,12 @@ Get the list of volumes registered with the plugin. } ``` - Respond with a string error if an error occurred. `Mountpoint` is optional. ### /VolumeDriver.Capabilities -**Request**: +Request: + ```json {} ``` @@ -345,7 +342,8 @@ Get the list of capabilities the driver supports. The driver is not required to implement `Capabilities`. If it is not implemented, the default values are used. -**Response**: +Response: + ```json { "Capabilities": { From 69809ddbf2919436e6195f3d956df097ce080f53 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Thu, 18 Jan 2024 13:24:23 +0100 Subject: [PATCH 455/534] docs: add documentation for CDI - Add section about cdi-spec-dirs daemon configuration - Add subsection about cdi in --device section for docker run - Update `docker info` output example Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 39 +++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index d4e59ac1f0..9e9fd332f3 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -29,6 +29,7 @@ Options: --authorization-plugin list Authorization plugins to load --bip string Specify network bridge IP -b, --bridge string Attach containers to a network bridge + --cdi-spec-dir list CDI specification directories to use --cgroup-parent string Set parent cgroup for all containers --config-file string Daemon configuration file (default "/etc/docker/daemon.json") --containerd string containerd grpc address @@ -854,6 +855,44 @@ $ docker run -it --add-host host.docker.internal:host-gateway \ PING host.docker.internal (192.0.2.0): 56 data bytes ``` +### Enable CDI devices + +> **Note** +> +> This is experimental feature and as such doesn't represent a stable API. +> +> This feature isn't enabled by default. To this feature, set `features.cdi` to +> `true` in the `daemon.json` configuration file. + +Container Device Interface (CDI) is a +[standardized](https://github.com/cncf-tags/container-device-interface/blob/main/SPEC.md) +mechanism for container runtimes to create containers which are able to +interact with third party devices. + +The Docker daemon supports running containers with CDI devices if the requested +device specifications are available on the filesystem of the daemon. + +The default specification directors are: + +- `/etc/cdi/` for static CDI Specs +- `/var/run/cdi` for generated CDI Specs + +Alternatively, you can set custom locations for CDI specifications using the +`cdi-spec-dirs` option in the `daemon.json` configuration file, or the +`--cdi-spec-dir` flag for the `dockerd` CLI. + +```json +{ + "features": { + "cdi": true + }, + "cdi-spec-dirs": ["/etc/cdi/", "/var/run/cdi"] +} +``` + +When CDI is enabled for a daemon, you can view the configured CDI specification +directories using the `docker info` command. + ### Miscellaneous options IP masquerading uses address translation to allow containers without a public From f99ed8076b9cae0ab11036eaf63741a455cbf237 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Tue, 23 Jan 2024 14:12:33 +0100 Subject: [PATCH 456/534] docs: update host-gateway-ip to use daemon.json instead of cli flag Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/commandline/dockerd.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/commandline/dockerd.md index 9e9fd332f3..00397b7be8 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/commandline/dockerd.md @@ -849,7 +849,9 @@ flag for the dockerd command line interface, or the `host-gateway-ip` key in the daemon configuration file. ```console -$ dockerd --host-gateway-ip 192.0.2.0 +$ cat > /etc/docker/daemon.json +{ "host-gateway-ip": "192.0.2.0" } +$ sudo systemctl restart docker $ docker run -it --add-host host.docker.internal:host-gateway \ busybox ping host.docker.internal PING host.docker.internal (192.0.2.0): 56 data bytes @@ -1072,6 +1074,7 @@ The following is a full example of the allowed configuration options on Linux: "fixed-cidr": "", "fixed-cidr-v6": "", "group": "", + "host-gateway-ip": "", "hosts": [], "proxies": { "http-proxy": "http://proxy.example.com:80", @@ -1181,6 +1184,7 @@ The following is a full example of the allowed configuration options on Windows: "features": {}, "fixed-cidr": "", "group": "", + "host-gateway-ip": "", "hosts": [], "insecure-registries": [], "labels": [], From 679efc8c4dc4f52cdef5b05ee758226e7afb4eac Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Tue, 13 Feb 2024 13:40:53 +0100 Subject: [PATCH 457/534] docs: update url scheme for reference docs Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/config.md | 4 +-- docs/extend/plugins_authorization.md | 2 +- docs/reference/{commandline => }/dockerd.md | 28 ++++++++++----------- 3 files changed, 17 insertions(+), 17 deletions(-) rename docs/reference/{commandline => }/dockerd.md (98%) diff --git a/docs/extend/config.md b/docs/extend/config.md index 1f96d3e009..d2472a39f4 100644 --- a/docs/extend/config.md +++ b/docs/extend/config.md @@ -55,11 +55,11 @@ the registry. - `entrypoint` string array - Entrypoint of the plugin, see [`ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) + Entrypoint of the plugin, see [`ENTRYPOINT`](https://docs.docker.com/reference/dockerfile/#entrypoint) - `workdir` string - Working directory of the plugin, see [`WORKDIR`](https://docs.docker.com/engine/reference/builder/#workdir) + Working directory of the plugin, see [`WORKDIR`](https://docs.docker.com/reference/dockerfile/#workdir) - `network` PluginNetwork diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 8a646e50fa..79201ec236 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -104,7 +104,7 @@ Enable the authorization plugin with a dedicated command line flag in the `--authorization-plugin=PLUGIN_ID` format. The flag supplies a `PLUGIN_ID` value. This value can be the plugin’s socket or a path to a specification file. Authorization plugins can be loaded without restarting the daemon. Refer -to the [`dockerd` documentation](../reference/commandline/dockerd.md#configuration-reload-behavior) for more information. +to the [`dockerd` documentation](https://docs.docker.com/reference/cli/dockerd/#configuration-reload-behavior) for more information. ```console $ dockerd --authorization-plugin=plugin1 --authorization-plugin=plugin2,... diff --git a/docs/reference/commandline/dockerd.md b/docs/reference/dockerd.md similarity index 98% rename from docs/reference/commandline/dockerd.md rename to docs/reference/dockerd.md index 00397b7be8..5ada8973e7 100644 --- a/docs/reference/commandline/dockerd.md +++ b/docs/reference/dockerd.md @@ -133,21 +133,21 @@ to [the `daemon.json` file](#daemon-configuration-file). The following list of environment variables are supported by the `dockerd` daemon. Some of these environment variables are supported both by the Docker Daemon and -the `docker` CLI. Refer to [Environment variables](cli.md#environment-variables) +the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) in the CLI section to learn about environment variables supported by the `docker` CLI. -| Variable | Description | -|:--------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | -| `DOCKER_DRIVER` | The storage driver to use. | -| `DOCKER_RAMDISK` | If set this disables `pivot_root`. | -| `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](cli.md) and the `dockerd` daemon. | -| `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | -| `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | -| `HTTPS_PROXY` | Proxy URL for HTTPS requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | -| `MOBY_DISABLE_PIGZ` | Disables the use of [`unpigz`](https://linux.die.net/man/1/pigz) to decompress layers in parallel when pulling images, even if it is installed. | -| `NO_PROXY` | Comma-separated values specifying hosts that should be excluded from proxying. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | +| Variable | Description | +| :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](https://docs.docker.com/engine/reference/commandline/cli/) and the `dockerd` daemon. | +| `DOCKER_DRIVER` | The storage driver to use. | +| `DOCKER_RAMDISK` | If set this disables `pivot_root`. | +| `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](https://docs.docker.com/engine/reference/commandline/cli/) and the `dockerd` daemon. | +| `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | +| `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | +| `HTTPS_PROXY` | Proxy URL for HTTPS requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | +| `MOBY_DISABLE_PIGZ` | Disables the use of [`unpigz`](https://linux.die.net/man/1/pigz) to decompress layers in parallel when pulling images, even if it is installed. | +| `NO_PROXY` | Comma-separated values specifying hosts that should be excluded from proxying. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | ## Examples @@ -795,7 +795,7 @@ Any `--ulimit` options passed to `docker run` override the daemon defaults. Be careful setting `nproc` with the `ulimit` flag, as `nproc` is designed by Linux to set the maximum number of processes available to a user, not to a container. -For details, see [`docker run` reference](run.md#ulimit). +For details, see [`docker run` reference](https://docs.docker.com/reference/cli/docker/container/run/#ulimit). ### Access authorization @@ -819,7 +819,7 @@ If you have multiple plugins installed, each plugin, in order, must allow the request for it to complete. For information about how to create an authorization plugin, refer to the -[authorization plugin](../../extend/plugins_authorization.md) section. +[authorization plugin](https://docs.docker.com/engine/extend/plugins_authorization/) section. ### Daemon user namespace options From 00a6100c405825550fb7b569f40bc09a50b8a4a7 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 21 Feb 2024 12:29:31 +0100 Subject: [PATCH 458/534] docs: fix invalid daemon.json examples Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/dockerd.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 5ada8973e7..63b9ce8211 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1038,7 +1038,7 @@ The following is a full example of the allowed configuration options on Linux: "cgroup-parent": "", "containerd": "/run/containerd/containerd.sock", "containerd-namespace": "docker", - "containerd-plugin-namespace": "docker-plugins", + "containerd-plugins-namespace": "docker-plugins", "data-root": "", "debug": true, "default-address-pools": [ @@ -1079,7 +1079,7 @@ The following is a full example of the allowed configuration options on Linux: "proxies": { "http-proxy": "http://proxy.example.com:80", "https-proxy": "https://proxy.example.com:443", - "no-proxy": "*.test.example.com,.example.org", + "no-proxy": "*.test.example.com,.example.org" }, "icc": false, "init": false, @@ -1170,7 +1170,7 @@ The following is a full example of the allowed configuration options on Windows: "bridge": "", "containerd": "\\\\.\\pipe\\containerd-containerd", "containerd-namespace": "docker", - "containerd-plugin-namespace": "docker-plugins", + "containerd-plugins-namespace": "docker-plugins", "data-root": "", "debug": true, "default-network-opts": {}, From 5e795df81903b33311df45daba6043707da1ed6f Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Fri, 8 Mar 2024 14:50:15 +0100 Subject: [PATCH 459/534] docs: deprecate TCP connection without TLS Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/deprecated.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7a68c0f60b..1c986d8739 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | @@ -110,6 +111,33 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Unauthenticated TCP connections + +**Deprecated in Release: v26.0** +**Target For Removal In Release: v27.0** + +Configuring the Docker daemon to listen on a TCP address will require mandatory +TLS verification. This change aims to ensure secure communication by preventing +unauthorized access to the Docker daemon over potentially insecure networks. +This mandatory TLS requirement applies to all TCP addresses except `tcp://localhost`. + +In version 27.0 and later, specifying `--tls=false` or `--tlsverify=false` CLI flags +causes the daemon to fail to start if it's also configured to accept remote connections over TCP. +This also applies to the equivalent configuration options in `daemon.json`. + +To facilitate remote access to the Docker daemon over TCP, you'll need to +implement TLS verification. This secures the connection by encrypting data in +transit and providing a mechanism for mutual authentication. + +For environments remote daemon access isn't required, +we recommend binding the Docker daemon to a Unix socket. +For daemon's where remote access is required and where TLS encryption is not feasible, +you may want to consider using SSH as an alternative solution. + +For further information, assistance, and step-by-step instructions on +configuring TLS (or SSH) for the Docker daemon, refer to +[Protect the Docker daemon socket](https://docs.docker.com/engine/security/protect-access/). + ### Deprecate legacy API versions **Deprecated in Release: v25.0** From 4fb5267ad6d2e87a4b91b111ad3565df78879359 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 4 Mar 2024 09:14:25 +0100 Subject: [PATCH 460/534] deprecate legacy image formats Mark pulling legacy image formats as deprecated, and describe the DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE environment variable. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 34 +++++++++++++++++++++++++++------- 1 file changed, 27 insertions(+), 7 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7a68c0f60b..4b7b0f0742 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -72,7 +72,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | | Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | | Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0 | -| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v20.10 | +| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v27.0 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | @@ -558,15 +558,35 @@ for the old option will be removed in a future release. **Deprecated in Release: v19.03** -**Target For Removal In Release: v20.10** +**Disabled by default in Release: v26.0** -The image manifest -[v2 schema 1](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-1.md) -format is deprecated in favor of the -[v2 schema 2](https://github.com/docker/distribution/blob/fda42e5ef908bdba722d435ff1f330d40dfcd56c/docs/spec/manifest-v2-2.md) format. +**Target For Removal In Release: v27.0** -If the registry you are using still supports v2 schema 1, urge their administrators to move to v2 schema 2. +The image manifest [v2 schema 1](https://distribution.github.io/distribution/spec/deprecated-schema-v1/) +and "Docker Image v1" formats were deprecated in favor of the +[v2 schema 2](https://distribution.github.io/distribution/spec/manifest-v2-2/) +and [OCI image spec](https://github.com/opencontainers/image-spec/tree/v1.1.0) +formats. +These legacy formats should no longer be used, and users are recommended to +update images to use current formats, or to upgrade to more current images. +Starting with Docker v26.0, pulling these images is disabled by default, and +produces an error when attempting to pull the image: + +```console +$ docker pull ubuntu:10.04 +Error response from daemon: +[DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. +Suggest the author of docker.io/library/ubuntu:10.04 to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. +More information at https://docs.docker.com/go/deprecated-image-specs/ +``` + +An environment variable (`DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE`) is +added in Docker v26.0 that allows re-enabling support for these image formats +in the daemon. This environment variable must be set to a non-empty value in +the daemon's environment (for example, through a [systemd override file](https://docs.docker.com/config/daemon/systemd/)). +Support for the `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE` environment-variable +will be removed in Docker v27.0 after which this functionality is removed permanently. ### `docker engine` subcommands From c0a77a5c8f801cb07689060a855b4cf99532dbcc Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Tue, 5 Mar 2024 14:27:26 +0100 Subject: [PATCH 461/534] docs: typo fixes and other corrections Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/dockerd.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 63b9ce8211..580b1f0dec 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1347,7 +1347,7 @@ using the `daemon.json` file. "default-network-opts": { "bridge": { "com.docker.network.bridge.host_binding_ipv4": "127.0.0.1", - "com.docker.network.bridge.mtu": "1234" + "com.docker.network.driver.mtu": "1234" } } } @@ -1363,7 +1363,7 @@ you create use these option configurations as defaults. ```console $ docker network create mynet $ docker network inspect mynet --format "{{json .Options}}" -{"com.docker.network.bridge.host_binding_ipv4":"127.0.0.1","com.docker.network.bridge.mtu":"1234"} +{"com.docker.network.bridge.host_binding_ipv4":"127.0.0.1","com.docker.network.driver.mtu":"1234"} ``` Note that changing this daemon configuration doesn't affect pre-existing @@ -1377,5 +1377,5 @@ daemon configuration. The CLI flag expects a value with the following format: ```console $ sudo dockerd \ --default-network-opt bridge=com.docker.network.bridge.host_binding_ipv4=127.0.0.1 \ - --default-network-opt bridge=com.docker.network.bridge.mtu=1234 + --default-network-opt bridge=com.docker.network.driver.mtu=1234 ``` From 374285901b346e09f54080aeeb844e3c84e00955 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Thu, 22 Feb 2024 17:00:32 +0100 Subject: [PATCH 462/534] docs/deprecated: Deprecate Container fields in image inspect MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit See moby#46939 Signed-off-by: Paweł Gronowski --- docs/deprecated.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 0fa02e8e0a..a7267afbbd 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -51,6 +51,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | +| Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | @@ -138,6 +139,19 @@ For further information, assistance, and step-by-step instructions on configuring TLS (or SSH) for the Docker daemon, refer to [Protect the Docker daemon socket](https://docs.docker.com/engine/security/protect-access/). +### `Container` and `ContainerConfig` fields in Image inspect + +**Deprecated in Release: v25.0** +**Target For Removal In Release: v26.0** + +The `Container` and `ContainerConfig` fields returned by `docker inspect` are +mostly an implementation detail of the classic (non-BuildKit) image builder. +These fields are not portable and are empty when using the +BuildKit-based builder (enabled by default since v23.0). +These fields are deprecated in v25.0 and will be omitted starting from v26.0. +If image configuration of an image is needed, you can obtain it from the +`Config` field. + ### Deprecate legacy API versions **Deprecated in Release: v25.0** From b3415d3e7f1038d7f893f8efcc32fff65f0e862e Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Mon, 18 Mar 2024 10:39:14 +0100 Subject: [PATCH 463/534] docs/deprecated: update status of short cid alias deprecation Starting with API v1.45, the container short ID is removed from the container Aliases. Signed-off-by: Albin Kerouanton --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 0fa02e8e0a..c8c89e0ff3 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,7 +52,7 @@ The table below provides an overview of the current status of deprecated feature |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | -| Deprecated | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | +| Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | @@ -195,7 +195,7 @@ old clients, and those clients must be supported. ### Container short ID in network Aliases field **Deprecated in Release: v25.0** -**Target For Remove In Release: v26.0** +**Removed In Release: v26.0** The `Aliases` field returned by `docker inspect` contains the container short ID once the container is started. This behavior is deprecated in v25.0 but From 996062626f629b0c6c67012a56ecb27c5b2e29b6 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Tue, 26 Mar 2024 16:26:45 +0000 Subject: [PATCH 464/534] Feature option 'windows-dns-proxy' Document feature option 'windows-dns-proxy', which can be used to enable or disable forwarding of DNS requests from the daemon's internal resolver to external servers. Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 580b1f0dec..0cf727b137 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1236,6 +1236,12 @@ The list of feature options include: snapshotters instead of the classic storage drivers for storing image and container data. For more information, see [containerd storage](https://docs.docker.com/storage/containerd/). +- `windows-dns-proxy`: when set to `true`, the daemon's internal DNS resolver + will forward requests to external servers. Without this, most applications + running in the container will still be able to use secondary DNS servers + configured in the container itself, but `nslookup` won't be able to resolve + external names. The current default is `false`, it will change to `true` in + a future release. This option is only allowed on Windows. #### Configuration reload behavior From 46f2ec7daea420ad4279bf7f6b810b801577ada3 Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Fri, 19 Apr 2024 13:32:28 +0200 Subject: [PATCH 465/534] Warn about deprecation of windows-dns-proxy flag Signed-off-by: Albin Kerouanton --- docs/reference/dockerd.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 0cf727b137..17083e0e4e 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1243,6 +1243,9 @@ The list of feature options include: external names. The current default is `false`, it will change to `true` in a future release. This option is only allowed on Windows. + > **Warning** + > The `windows-dns-proxy` feature flag will be removed in a future release. + #### Configuration reload behavior Some options can be reconfigured when the daemon is running without requiring From 80db4364ae993c1ae5e9d813ed1bd1a212bdfd2b Mon Sep 17 00:00:00 2001 From: cncal Date: Fri, 17 May 2024 17:35:22 +0800 Subject: [PATCH 466/534] docs: fix json syntax error Signed-off-by: cncal --- docs/reference/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 17083e0e4e..f8bf348d54 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -530,7 +530,7 @@ For example: "runtimeType": "io.containerd.runsc.v1", "options": { "TypeUrl": "io.containerd.runsc.v1.options", - "ConfigPath": "/etc/containerd/runsc.toml", + "ConfigPath": "/etc/containerd/runsc.toml" } } } From 2953ec627bcddc7f93c6f95ecac4fa5229fa56b1 Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Wed, 22 May 2024 10:49:30 +0200 Subject: [PATCH 467/534] docs: deprecate daemon's api-cors-header flag Signed-off-by: Albin Kerouanton --- docs/deprecated.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index d89206d2eb..49c7f9904d 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | @@ -112,6 +113,26 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### API CORS headers + +**Deprecated in Release: v27.0** +**Target For Removal In Release: v28.0** + +The `api-cors-header` configuration option for the Docker daemon is insecure, +and is therefore deprecated and scheduled for removal. +Incorrectly setting this option could leave a window of opportunity +for unauthenticated cross-origin requests to be accepted by the daemon. + +Starting in Docker Engine v27.0, this flag can still be set, +but it has no effect unless the environment variable +`DOCKERD_DEPRECATED_CORS_HEADER` is also set to a non-empty value. + +This flag will be removed altogether in v28.0. + +This is a breaking change for authorization plugins and other programs +that depend on this option for accessing the Docker API from a browser. +If you need to access the API through a browser, use a reverse proxy. + ### Unauthenticated TCP connections **Deprecated in Release: v26.0** From 07c7fb57c6ed52c79ebbfdab78c331126c8fefa2 Mon Sep 17 00:00:00 2001 From: Grace Choi Date: Fri, 26 Apr 2024 13:16:51 -0500 Subject: [PATCH 468/534] Removed all mentions of "please" from docs and messages Signed-off-by: Grace Choi Signed-off-by: Pranjal Rai Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 5 +++-- docs/extend/EBS_volume.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 49c7f9904d..1210e0bda1 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -209,7 +209,8 @@ CLI to use API version 1.23, which produces an error: ```console DOCKER_API_VERSION=1.23 docker version -Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version +Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, +upgrade your client to a newer version ``` An environment variable (`DOCKER_MIN_API_VERSION`) is introduced that allows @@ -611,7 +612,7 @@ The `--log-opt fluentd-async-connect` option for the fluentd logging driver is A deprecation message is logged in the daemon logs if the old option is used: ```console -fluent#New: AsyncConnect is now deprecated, please use Async instead +fluent#New: AsyncConnect is now deprecated, use Async instead ``` Users are encouraged to use the `fluentd-async` option going forward, as support diff --git a/docs/extend/EBS_volume.md b/docs/extend/EBS_volume.md index e09df6c543..88de970ba2 100644 --- a/docs/extend/EBS_volume.md +++ b/docs/extend/EBS_volume.md @@ -122,7 +122,7 @@ We have to put the following JSON to `$TMPDIR/config.json`: } ``` -Please note a couple of points: +Note a couple of points: - `PropagatedMount` is needed so that the docker daemon can see mounts done by the rexray plugin from within the container, otherwise the docker daemon is not able to mount a docker volume. From bd10f33f23eb415dcc261d03904e843bc0b9c6c2 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 11 Jun 2024 17:05:25 +0200 Subject: [PATCH 469/534] Deprecate non-standard (config) fields in image inspect output The `Config` field returned by this endpoint (used for "image inspect") returns additional fields that are not part of the image's configuration and not part of the [Docker Image Spec] and the [OCI Image Spec]. These additional fields are included in the response, due to an implementation detail, where the [api/types.ImageInspec] type used for the response is using the [container.Config] type. The [container.Config] type is a superset of the image config, and while the image's Config is used as a _template_ for containers created from the image, the additional fields are set at runtime (from options passed when creating the container) and not taken from the image Config. These fields are never set (and always return the default value for the type), but are not omitted in the response when left empty. As these fields were not intended to be part of the image configuration response, they are deprecated, and will be removed from the API. The following fields are currently included in the API response, but are not part of the underlying image's Config, and deprecated: - `Hostname` - `Domainname` - `AttachStdin` - `AttachStdout` - `AttachStderr` - `Tty` - `OpenStdin` - `StdinOnce` - `Image` - `NetworkDisabled` (already omitted unless set) - `MacAddress` (already omitted unless set) - `StopTimeout` (already omitted unless set) [Docker image spec]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 [OCI Image Spec]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 [api/types.ImageInspec]: https://github.com/moby/moby/blob/v26.1.4/api/types/types.go#L87-L104 [container.Config]: https://github.com/moby/moby/blob/v26.1.4/api/types/container/config.go#L47-L82 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 49c7f9904d..0c31883888 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -50,6 +50,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | | Deprecated | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | @@ -113,6 +114,40 @@ The table below provides an overview of the current status of deprecated feature | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Non-standard fields in image inspect + +**Deprecated in Release: v27.0** +**Target For Removal In Release: v28.0** + +The `Config` field returned shown in `docker image inspect` (and as returned by +the `GET /images/{name}/json` API endpoint) returns additional fields that are +not part of the image's configuration and not part of the [Docker Image Spec] +and [OCI Image Specification]. + +These fields are never set (and always return the default value for the type), +but are not omitted in the response when left empty. As these fields were not +intended to be part of the image configuration response, they are deprecated, +and will be removed from the API in thee next release. + +The following fields are currently included in the API response, but are not +part of the underlying image's Config, and deprecated: + +- `Hostname` +- `Domainname` +- `AttachStdin` +- `AttachStdout` +- `AttachStderr` +- `Tty` +- `OpenStdin` +- `StdinOnce` +- `Image` +- `NetworkDisabled` (already omitted unless set) +- `MacAddress` (already omitted unless set) +- `StopTimeout` (already omitted unless set) + +[Docker image spec]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 +[OCI Image Spec]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 + ### API CORS headers **Deprecated in Release: v27.0** From a089d9b942ec6fac40a1a4e53aed82052fad842f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 12 Jun 2024 13:40:43 +0200 Subject: [PATCH 470/534] docs: extend deprecation target for non-TLS API No work has started yet on the deprecation, so extending the time- window. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1210e0bda1..baeb797a08 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -51,7 +51,7 @@ The table below provides an overview of the current status of deprecated feature | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | -| Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v27.0 | +| Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | @@ -136,7 +136,7 @@ If you need to access the API through a browser, use a reverse proxy. ### Unauthenticated TCP connections **Deprecated in Release: v26.0** -**Target For Removal In Release: v27.0** +**Target For Removal In Release: v28.0** Configuring the Docker daemon to listen on a TCP address will require mandatory TLS verification. This change aims to ensure secure communication by preventing From 33cebe67bb1d794a70f8e532559f9978c4241975 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 18 Jun 2024 16:18:07 +0200 Subject: [PATCH 471/534] docs: add deprecation warning to experimental grapdriver plugins Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugins_graphdriver.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md index f6558d7f1e..fbb54bd664 100644 --- a/docs/extend/plugins_graphdriver.md +++ b/docs/extend/plugins_graphdriver.md @@ -15,6 +15,15 @@ advisory: experimental # Graphdriver plugins +> **Deprecated** +> +> Support for graphdriver plugins is an experimental feature, which has been +> deprecated. Upcoming releases of the Docker Engine will no longer support +> graphdriver plugins, and users of this feature are recommended to instead +> use the containerd image store, and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters) +{ .important } + + ## Changelog ### 1.13.0 From d82b26847e86114be057718928db65a0e6715012 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 18 Jun 2024 16:41:55 +0200 Subject: [PATCH 472/534] deprecate experimental graphdriver plugins [Graphdriver plugins] are an experimental feature that allow extending the Docker Engine with custom storage drivers for storing images and containers. This feature was not maintained since its inception, and will no longer be supported in upcoming releases. Users of this feature are recommended to instead configure the Docker Engine to use the [containerd image store], and a custom [snapshotter]. [Graphdriver plugins]: https://github.com/docker/cli/blob/v26.1.4/docs/extend/plugins_graphdriver.md [containerd image store]: https://github.com/docker/docs/blob/31a16f1da2f4a4c94b5a82713aa012f052fb7e6d/content/storage/containerd.md [snapshotter]: https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 ++ docs/extend/index.md | 2 + docs/extend/plugins_graphdriver.md | 410 ----------------------------- 3 files changed, 18 insertions(+), 410 deletions(-) delete mode 100644 docs/extend/plugins_graphdriver.md diff --git a/docs/deprecated.md b/docs/deprecated.md index e0b6bf608b..64567be4be 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -52,6 +52,7 @@ The table below provides an overview of the current status of deprecated feature |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | | Deprecated | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | +| Deprecated | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | @@ -148,6 +149,21 @@ part of the underlying image's Config, and deprecated: [Docker image spec]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 [OCI Image Spec]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 +### Graphdriver plugins (experimental) + +**Deprecated in Release: v27.0** +**Target For Removal In Release: v28.0** + +[Graphdriver plugins](https://github.com/docker/cli/blob/v26.1.4/docs/extend/plugins_graphdriver.md) +are an experimental feature that allow extending the Docker Engine with custom +storage drivers for storing images and containers. This feature was not +maintained since its inception, and will no longer be supported in upcoming +releases. + +Users of this feature are recommended to instead configure the Docker Engine +to use the [containerd image store](https://docs.docker.com/storage/containerd/), +and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters) + ### API CORS headers **Deprecated in Release: v27.0** diff --git a/docs/extend/index.md b/docs/extend/index.md index 22ab0f3f30..c90a42cb4f 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -2,6 +2,8 @@ title: Docker Engine managed plugin system description: Develop and use a plugin with the managed plugin system keywords: "API, Usage, plugins, documentation, developer" +aliases: + - "/engine/extend/plugins_graphdriver/" --- - [Installing and using a plugin](index.md#installing-and-using-a-plugin) diff --git a/docs/extend/plugins_graphdriver.md b/docs/extend/plugins_graphdriver.md deleted file mode 100644 index fbb54bd664..0000000000 --- a/docs/extend/plugins_graphdriver.md +++ /dev/null @@ -1,410 +0,0 @@ ---- -description: "How to manage image and container filesystems with external plugins" -keywords: "Examples, Usage, storage, image, docker, data, graph, plugin, api" -advisory: experimental ---- - - - -# Graphdriver plugins - -> **Deprecated** -> -> Support for graphdriver plugins is an experimental feature, which has been -> deprecated. Upcoming releases of the Docker Engine will no longer support -> graphdriver plugins, and users of this feature are recommended to instead -> use the containerd image store, and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters) -{ .important } - - -## Changelog - -### 1.13.0 - -- Support v2 plugins - -# Docker graph driver plugins - -Docker graph driver plugins enable admins to use an external/out-of-process -graph driver for use with Docker engine. This is an alternative to using the -built-in storage drivers, such as overlay2. - -You need to install and enable the plugin and then restart the Docker daemon -before using the plugin. See the following example for the correct ordering -of steps. - -```console -$ docker plugin install cpuguy83/docker-overlay2-graphdriver-plugin # this command also enables the driver - -$ pkill dockerd -$ dockerd --experimental -s cpuguy83/docker-overlay2-graphdriver-plugin -``` - -# Write a graph driver plugin - -See the [plugin documentation](https://docs.docker.com/engine/extend/) for detailed information -on the underlying plugin protocol. - - -## Graph Driver plugin protocol - -If a plugin registers itself as a `GraphDriver` when activated, then it is -expected to provide the rootfs for containers as well as image layer storage. - -### /GraphDriver.Init - -**Request**: -```json -{ - "Home": "/graph/home/path", - "Opts": [], - "UIDMaps": [], - "GIDMaps": [] -} -``` - -Initialize the graph driver plugin with a home directory and array of options. -These are passed through from the user, but the plugin is not required to parse -or honor them. - -The request also includes a list of UID and GID mappings, structed as follows: -```json -{ - "ContainerID": 0, - "HostID": 0, - "Size": 0 -} -``` - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a non-empty string error if an error occurred. - - -### /GraphDriver.Capabilities - -**Request**: -```json -{} -``` - -Get behavioral characteristics of the graph driver. If a plugin does not handle -this request, the engine will use default values for all capabilities. - -**Response**: -```json -{ - "ReproducesExactDiffs": false, -} -``` - -Respond with values of capabilities: - -* **ReproducesExactDiffs** Defaults to false. Flags that this driver is capable -of reproducing exactly equivalent diffs for read-only filesystem layers. - - -### /GraphDriver.Create - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142", - "MountLabel": "", - "StorageOpt": {} -} -``` - -Create a new, empty, read-only filesystem layer with the specified -`ID`, `Parent` and `MountLabel`. If `Parent` is an empty string, there is no -parent layer. `StorageOpt` is map of strings which indicate storage options. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.CreateReadWrite - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142", - "MountLabel": "", - "StorageOpt": {} -} -``` - -Similar to `/GraphDriver.Create` but creates a read-write filesystem layer. - -### /GraphDriver.Remove - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" -} -``` - -Remove the filesystem layer with this given `ID`. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.Get - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "MountLabel": "" -} -``` - -Get the mountpoint for the layered filesystem referred to by the given `ID`. - -**Response**: -```json -{ - "Dir": "/var/mygraph/46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Err": "" -} -``` - -Respond with the absolute path to the mounted layered filesystem. -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.Put - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" -} -``` - -Release the system resources for the specified `ID`, such as unmounting the -filesystem layer. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.Exists - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" -} -``` - -Determine if a filesystem layer with the specified `ID` exists. - -**Response**: -```json -{ - "Exists": true -} -``` - -Respond with a boolean for whether or not the filesystem layer with the specified -`ID` exists. - -### /GraphDriver.Status - -**Request**: -```json -{} -``` - -Get low-level diagnostic information about the graph driver. - -**Response**: -```json -{ - "Status": [[]] -} -``` - -Respond with a 2-D array with key/value pairs for the underlying status -information. - - -### /GraphDriver.GetMetadata - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187" -} -``` - -Get low-level diagnostic information about the layered filesystem with the -with the specified `ID` - -**Response**: -```json -{ - "Metadata": {}, - "Err": "" -} -``` - -Respond with a set of key/value pairs containing the low-level diagnostic -information about the layered filesystem. -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.Cleanup - -**Request**: -```json -{} -``` - -Perform necessary tasks to release resources help by the plugin, such as -unmounting all the layered file systems. - -**Response**: -```json -{ - "Err": "" -} -``` - -Respond with a non-empty string error if an error occurred. - - -### /GraphDriver.Diff - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" -} -``` - -Get an archive of the changes between the filesystem layers specified by the `ID` -and `Parent`. `Parent` may be an empty string, in which case there is no parent. - -**Response**: - -``` -{{ TAR STREAM }} -``` - -### /GraphDriver.Changes - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" -} -``` - -Get a list of changes between the filesystem layers specified by the `ID` and -`Parent`. If `Parent` is an empty string, there is no parent. - -**Response**: -```json -{ - "Changes": [{}], - "Err": "" -} -``` - -Respond with a list of changes. The structure of a change is: -```json - "Path": "/some/path", - "Kind": 0, -``` - -Where the `Path` is the filesystem path within the layered filesystem that is -changed and `Kind` is an integer specifying the type of change that occurred: - -- 0 - Modified -- 1 - Added -- 2 - Deleted - -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.ApplyDiff - -**Request**: - -``` -{{ TAR STREAM }} -``` - -Extract the changeset from the given diff into the layer with the specified `ID` -and `Parent` - -**Query Parameters**: - -- id (required)- the `ID` of the new filesystem layer to extract the diff to -- parent (required)- the `Parent` of the given `ID` - -**Response**: -```json -{ - "Size": 512366, - "Err": "" -} -``` - -Respond with the size of the new layer in bytes. -Respond with a non-empty string error if an error occurred. - -### /GraphDriver.DiffSize - -**Request**: -```json -{ - "ID": "46fe8644f2572fd1e505364f7581e0c9dbc7f14640bd1fb6ce97714fb6fc5187", - "Parent": "2cd9c322cb78a55e8212aa3ea8425a4180236d7106938ec921d0935a4b8ca142" -} -``` - -Calculate the changes between the specified `ID` - -**Response**: -```json -{ - "Size": 512366, - "Err": "" -} -``` - -Respond with the size changes between the specified `ID` and `Parent` -Respond with a non-empty string error if an error occurred. From ecf640af7e9625bf25438021d65a87f2ef38fa9f Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 21 Jun 2024 12:51:06 +0200 Subject: [PATCH 473/534] docs/deprecated: update status for graphdriver-plugins Add a mention of the feature being disabled by default, and the DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS env-var. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 64567be4be..e17879025d 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -152,6 +152,7 @@ part of the underlying image's Config, and deprecated: ### Graphdriver plugins (experimental) **Deprecated in Release: v27.0** +**Disabled by default in Release: v27.0** **Target For Removal In Release: v28.0** [Graphdriver plugins](https://github.com/docker/cli/blob/v26.1.4/docs/extend/plugins_graphdriver.md) @@ -160,8 +161,14 @@ storage drivers for storing images and containers. This feature was not maintained since its inception, and will no longer be supported in upcoming releases. -Users of this feature are recommended to instead configure the Docker Engine -to use the [containerd image store](https://docs.docker.com/storage/containerd/), +Support for graphdriver plugins is disabled by default in v27.0, and will be +removed v28.0. An `DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS` environment variable +is provided in v27.0 to re-enable the feature. This environment variable must +be set to a non-empty value in the daemon's environment. + +The `DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS` environment variable, along with +support for graphdriver plugins, will be removed in v28.0. Users of this feature +are recommended to instead configure the Docker Engine to use the [containerd image store](https://docs.docker.com/storage/containerd/) and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters) ### API CORS headers @@ -700,7 +707,7 @@ An environment variable (`DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE`) is added in Docker v26.0 that allows re-enabling support for these image formats in the daemon. This environment variable must be set to a non-empty value in the daemon's environment (for example, through a [systemd override file](https://docs.docker.com/config/daemon/systemd/)). -Support for the `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE` environment-variable +Support for the `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE` environment variable will be removed in Docker v27.0 after which this functionality is removed permanently. ### `docker engine` subcommands From ec6cd968082a438cb54c6bb50ef12c99ff935d5e Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Fri, 12 Apr 2024 11:10:58 +0200 Subject: [PATCH 474/534] docs: update links to docker cli reference Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/index.md | 2 +- docs/reference/dockerd.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/extend/index.md b/docs/extend/index.md index c90a42cb4f..7264d59bae 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -117,7 +117,7 @@ enabled, and use it to create a volume. To disable a plugin, use the `docker plugin disable` command. To completely remove it, use the `docker plugin remove` command. For other available commands and options, see the -[command line reference](https://docs.docker.com/engine/reference/commandline/cli/). +[command line reference](https://docs.docker.com/reference/cli/docker/). ## Developing a plugin diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index f8bf348d54..89de47e0ae 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -133,16 +133,16 @@ to [the `daemon.json` file](#daemon-configuration-file). The following list of environment variables are supported by the `dockerd` daemon. Some of these environment variables are supported both by the Docker Daemon and -the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) +the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/reference/cli/docker/#environment-variables) in the CLI section to learn about environment variables supported by the `docker` CLI. | Variable | Description | | :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](https://docs.docker.com/engine/reference/commandline/cli/) and the `dockerd` daemon. | +| `DOCKER_CERT_PATH` | Location of your authentication keys. This variable is used both by the [`docker` CLI](https://docs.docker.com/reference/cli/docker/) and the `dockerd` daemon. | | `DOCKER_DRIVER` | The storage driver to use. | | `DOCKER_RAMDISK` | If set this disables `pivot_root`. | -| `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](https://docs.docker.com/engine/reference/commandline/cli/) and the `dockerd` daemon. | +| `DOCKER_TLS_VERIFY` | When set Docker uses TLS and verifies the remote. This variable is used both by the [`docker` CLI](https://docs.docker.com/reference/cli/docker/) and the `dockerd` daemon. | | `DOCKER_TMPDIR` | Location for temporary files created by the daemon. | | `HTTP_PROXY` | Proxy URL for HTTP requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | | `HTTPS_PROXY` | Proxy URL for HTTPS requests unless overridden by NoProxy. See the [Go specification](https://pkg.go.dev/golang.org/x/net/http/httpproxy#Config) for details. | From c98c647d28d8571c4303751034dc1fbf6e22198d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 31 May 2024 13:04:52 +0200 Subject: [PATCH 475/534] add support for DOCKER_CUSTOM_HEADERS env-var (experimental) This environment variable allows for setting additional headers to be sent by the client. Headers set through this environment variable are added to headers set through the config-file (through the HttpHeaders field). This environment variable can be used in situations where headers must be set for a specific invocation of the CLI, but should not be set by default, and therefore cannot be set in the config-file. WARNING: If both config and environment-variable are set, the environment variable currently overrides all headers set in the configuration file. This behavior may change in a future update, as we are considering the environment variable to be appending to existing headers (and to only override headers with the same name). Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 89de47e0ae..d0be72e45b 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -133,9 +133,8 @@ to [the `daemon.json` file](#daemon-configuration-file). The following list of environment variables are supported by the `dockerd` daemon. Some of these environment variables are supported both by the Docker Daemon and -the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/reference/cli/docker/#environment-variables) -in the CLI section to learn about environment variables supported by the -`docker` CLI. +the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) +to learn about environment variables supported by the `docker` CLI. | Variable | Description | | :------------------ | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | From ef0b32a673077c7d23e5d5e0c9e791d53e960b75 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Sun, 11 Aug 2024 16:43:21 +0200 Subject: [PATCH 476/534] docs: update internal links after refactor Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/dockerd.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index d0be72e45b..8b472c67f1 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -133,7 +133,7 @@ to [the `daemon.json` file](#daemon-configuration-file). The following list of environment variables are supported by the `dockerd` daemon. Some of these environment variables are supported both by the Docker Daemon and -the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/engine/reference/commandline/cli/#environment-variables) +the `docker` CLI. Refer to [Environment variables](https://docs.docker.com/reference/cli/docker/#environment-variables) to learn about environment variables supported by the `docker` CLI. | Variable | Description | @@ -169,7 +169,7 @@ in three ways: options. (Docker Engine version 23.0 or later). The command-line and configuration file options take precedence over environment -variables. Refer to [control and configure Docker with systemd](https://docs.docker.com/config/daemon/systemd/#httphttps-proxy) +variables. Refer to [control and configure Docker with systemd](https://docs.docker.com/engine/daemon/proxy/) to set these environment variables on a host using `systemd`. ### Daemon socket option @@ -332,7 +332,7 @@ drivers: `overlay2`, `fuse-overlayfs`, `btrfs`, and `zfs`. and is selected by default. Unless users have a strong reason to prefer another storage driver, `overlay2` should be used. -You can find out more about storage drivers and how to select one in [Select a storage driver](https://docs.docker.com/storage/storagedriver/select-storage-driver/). +You can find out more about storage drivers and how to select one in [Select a storage driver](https://docs.docker.com/engine/storage/drivers/select-storage-driver/). On Windows, the Docker daemon only supports the `windowsfilter` storage driver. @@ -424,7 +424,7 @@ installed outside of `PATH`, must be registered with the daemon, either via the configuration file or using the `--add-runtime` command line flag. For examples on how to use other container runtimes, see -[Alternative container runtimes](https://docs.docker.com/engine/alternative-runtimes/) +[Alternative container runtimes](https://docs.docker.com/engine/daemon/alternative-runtimes/) ##### Configure runtimes using `daemon.json` @@ -609,7 +609,7 @@ $ sudo dockerd --add-runtime = Defining runtime arguments via the command line is not supported. For an example configuration for a runc drop-in replacment, see -[Alternative container runtimes > youki](https://docs.docker.com/engine/alternative-runtimes/#youki) +[Alternative container runtimes > youki](https://docs.docker.com/engine/daemon/alternative-runtimes/#youki) ##### Configure the default container runtime @@ -783,7 +783,7 @@ Docker host's configuration: This only adds the proxy and authentication to the Docker daemon's requests. To use the proxy when building images and running containers, see -[Configure Docker to use a proxy server](https://docs.docker.com/network/proxy/) +[Configure Docker to use a proxy server](https://docs.docker.com/engine/cli/proxy/) ### Default `ulimit` settings @@ -952,7 +952,7 @@ to avoid collisions with other Prometheus exporters and services. If you are running a Prometheus server you can add this address to your scrape configs to have Prometheus collect metrics on Docker. For more information, see -[Collect Docker metrics with Prometheus](https://docs.docker.com/config/daemon/prometheus/). +[Collect Docker metrics with Prometheus](https://docs.docker.com/engine/daemon/prometheus/). #### Node generic resources @@ -1151,7 +1151,7 @@ The following is a full example of the allowed configuration options on Linux: > daemon startup as a flag. > On systems that use systemd to start the Docker daemon, `-H` is already set, so > you can't use the `hosts` key in `daemon.json` to add listening addresses. -> See [custom Docker daemon options](https://docs.docker.com/config/daemon/systemd/#custom-docker-daemon-options) +> See [custom Docker daemon options](https://docs.docker.com/engine/daemon/proxy/#systemd-unit-file) > for an example on how to configure the daemon using systemd drop-in files. ##### On Windows @@ -1234,7 +1234,7 @@ The list of feature options include: - `containerd-snapshotter`: when set to `true`, the daemon uses containerd snapshotters instead of the classic storage drivers for storing image and container data. For more information, see - [containerd storage](https://docs.docker.com/storage/containerd/). + [containerd storage](https://docs.docker.com/engine/storage/containerd/). - `windows-dns-proxy`: when set to `true`, the daemon's internal DNS resolver will forward requests to external servers. Without this, most applications running in the container will still be able to use secondary DNS servers @@ -1260,7 +1260,7 @@ The list of currently supported options that can be reconfigured is this: | ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `debug` | Toggles debug mode of the daemon. | | `labels` | Replaces the daemon labels with a new set of labels. | -| `live-restore` | Toggles [live restore](https://docs.docker.com/config/containers/live-restore/). | +| `live-restore` | Toggles [live restore](https://docs.docker.com/engine/containers/live-restore/). | | `max-concurrent-downloads` | Configures the max concurrent downloads for each pull. | | `max-concurrent-uploads` | Configures the max concurrent uploads for each push. | | `max-download-attempts` | Configures the max download attempts for each pull. | @@ -1362,7 +1362,7 @@ using the `daemon.json` file. ``` This example uses the `bridge` network driver. Refer to the -[bridge network driver page](https://docs.docker.com/network/drivers/bridge/#options) +[bridge network driver page](https://docs.docker.com/engine/network/drivers/bridge/#options) for an overview of available driver options. After changing the configuration and restarting the daemon, new networks that From 122e5b6c439da0cdd72d6dbefb0e3cf8b25b1da2 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Fri, 16 Aug 2024 11:02:10 +0200 Subject: [PATCH 477/534] docs: use gh alert syntax for callouts Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/deprecated.md | 3 +-- docs/extend/index.md | 9 +++------ docs/extend/plugins_authorization.md | 3 +-- docs/extend/plugins_metrics.md | 3 +-- docs/extend/plugins_volume.md | 3 +-- docs/reference/dockerd.md | 24 +++++++++--------------- 6 files changed, 16 insertions(+), 29 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index e17879025d..1f28bbcff2 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -562,8 +562,7 @@ Docker API v1.42 and up now ignores this option when set. Older versions of the API continue to accept the option, but depending on the OCI runtime used, may take no effect. -> **Note** -> +> [!NOTE] > While not deprecated (yet) in Docker, the OCI runtime specification also > deprecated the `memory.kmem.tcp.limit_in_bytes` option. When using `runc` as > runtime, this option takes no effect. The linux kernel did not explicitly diff --git a/docs/extend/index.md b/docs/extend/index.md index 7264d59bae..8a4f9885d7 100644 --- a/docs/extend/index.md +++ b/docs/extend/index.md @@ -16,8 +16,7 @@ plugins using Docker Engine. For information about legacy (non-managed) plugins, refer to [Understand legacy Docker Engine plugins](legacy_plugins.md). -> **Note** -> +> [!NOTE] > Docker Engine managed plugins are currently not supported on Windows daemons. ## Installing and using a plugin @@ -38,8 +37,7 @@ operation, such as creating a volume. In the following example, you install the `sshfs` plugin, verify that it is enabled, and use it to create a volume. -> **Note** -> +> [!NOTE] > This example is intended for instructional purposes only. Once the volume is > created, your SSH password to the remote host is exposed as plaintext when > inspecting the volume. Delete the volume as soon as you are done with the @@ -126,8 +124,7 @@ commands and options, see the The `rootfs` directory represents the root filesystem of the plugin. In this example, it was created from a Dockerfile: -> **Note** -> +> [!NOTE] > The `/run/docker/plugins` directory is mandatory inside of the > plugin's filesystem for Docker to communicate with the plugin. diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 79201ec236..1fc6f78441 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -43,8 +43,7 @@ Authorization plugins must follow the rules described in [Docker Plugin API](plu Each plugin must reside within directories described under the [Plugin discovery](plugin_api.md#plugin-discovery) section. -> **Note** -> +> [!NOTE] > The abbreviations `AuthZ` and `AuthN` mean authorization and authentication > respectively. diff --git a/docs/extend/plugins_metrics.md b/docs/extend/plugins_metrics.md index dc47e46926..a744406f7d 100644 --- a/docs/extend/plugins_metrics.md +++ b/docs/extend/plugins_metrics.md @@ -8,8 +8,7 @@ Docker exposes internal metrics based on the Prometheus format. Metrics plugins enable accessing these metrics in a consistent way by providing a Unix socket at a predefined path where the plugin can scrape the metrics. -> **Note** -> +> [!NOTE] > While the plugin interface for metrics is non-experimental, the naming of the > metrics and metric labels is still considered experimental and may change in a > future version. diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 3e42bc75aa..8da7deb742 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -80,8 +80,7 @@ provide the Docker Daemon with writeable paths on the host filesystem. The Docke daemon provides these paths to containers to consume. The Docker daemon makes the volumes available by bind-mounting the provided paths into the containers. -> **Note** -> +> [!NOTE] > Volume plugins should *not* write data to the `/var/lib/docker/` directory, > including `/var/lib/docker/volumes`. The `/var/lib/docker/` directory is > reserved for Docker. diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 8b472c67f1..c13c20a864 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -124,6 +124,7 @@ type `dockerd`. To run the daemon with debug output, use `dockerd --debug` or add `"debug": true` to [the `daemon.json` file](#daemon-configuration-file). +> [!NOTE] > **Enabling experimental features** > > Enable experimental features by starting `dockerd` with the `--experimental` @@ -152,8 +153,7 @@ to learn about environment variables supported by the `docker` CLI. ### Proxy configuration -> **Note** -> +> [!NOTE] > Refer to the [Docker Desktop manual](https://docs.docker.com/desktop/networking/#httphttps-proxy-support) > if you are running [Docker Desktop](https://docs.docker.com/desktop/). @@ -191,8 +191,7 @@ interface using its IP address: `-H tcp://192.168.59.103:2375`. It is conventional to use port `2375` for un-encrypted, and port `2376` for encrypted communication with the daemon. -> **Note** -> +> [!NOTE] > If you're using an HTTPS encrypted socket, keep in mind that only > TLS version 1.0 and higher is supported. Protocols SSLv3 and below are not > supported for security reasons. @@ -259,8 +258,7 @@ supported. If your key is protected with passphrase, you need to set up #### Bind Docker to another host/port or a Unix socket -> **Warning** -> +> [!WARNING] > Changing the default `docker` daemon binding to a TCP port or Unix `docker` > user group introduces security risks, as it may allow non-root users to gain > root access on the host. Make sure you control access to `docker`. If you are @@ -709,8 +707,7 @@ This option is useful when pushing images containing non-distributable artifacts to a registry on an air-gapped network so hosts on that network can pull the images without connecting to another server. -> **Warning** -> +> [!WARNING] > Non-distributable artifacts typically have restrictions on how > and where they can be distributed and shared. Only use this feature to push > artifacts to private registries and ensure that you are in compliance with @@ -858,8 +855,7 @@ PING host.docker.internal (192.0.2.0): 56 data bytes ### Enable CDI devices -> **Note** -> +> [!NOTE] > This is experimental feature and as such doesn't represent a stable API. > > This feature isn't enabled by default. To this feature, set `features.cdi` to @@ -1145,8 +1141,7 @@ The following is a full example of the allowed configuration options on Linux: } ``` -> **Note** -> +> [!NOTE] > You can't set options in `daemon.json` that have already been set on > daemon startup as a flag. > On systems that use systemd to start the Docker daemon, `-H` is already set, so @@ -1242,7 +1237,7 @@ The list of feature options include: external names. The current default is `false`, it will change to `true` in a future release. This option is only allowed on Windows. - > **Warning** + > [!WARNING] > The `windows-dns-proxy` feature flag will be removed in a future release. #### Configuration reload behavior @@ -1275,8 +1270,7 @@ The list of currently supported options that can be reconfigured is this: ### Run multiple daemons -> **Note** -> +> [!NOTE] > Running multiple daemons on a single host is considered experimental. > You may encounter unsolved problems, and things may not work as expected in some cases. From b921b6ff2cd2d2453244a87ec74d146dbabc4440 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 4 Sep 2024 11:19:55 +0200 Subject: [PATCH 478/534] docs: rename plugins index file and add linkTitle We publish this page on docs.docker.com, and hugo expects index pages for sections to be named _index.md. We currently rename the page when we mount it to the docs repo but might as well change the filename in the source. Also adds a linkTitle to the page, which is a shorter title that will be used in the sidebar navigation and breadcrumbs. Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/{index.md => _index.md} | 1 + docs/extend/legacy_plugins.md | 4 ++-- docs/extend/plugin_api.md | 2 +- docs/extend/plugins_authorization.md | 2 +- docs/extend/plugins_network.md | 6 +++--- 5 files changed, 8 insertions(+), 7 deletions(-) rename docs/extend/{index.md => _index.md} (99%) diff --git a/docs/extend/index.md b/docs/extend/_index.md similarity index 99% rename from docs/extend/index.md rename to docs/extend/_index.md index 8a4f9885d7..57528444d1 100644 --- a/docs/extend/index.md +++ b/docs/extend/_index.md @@ -1,5 +1,6 @@ --- title: Docker Engine managed plugin system +linkTitle: Docker Engine plugins description: Develop and use a plugin with the managed plugin system keywords: "API, Usage, plugins, documentation, developer" aliases: diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 830a7dce9c..2c6ba4bc1a 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -8,7 +8,7 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide" This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker, -refer to [Docker Engine plugin system](index.md). +refer to [Docker Engine plugin system](_index.md). You can extend the capabilities of the Docker Engine by loading third-party plugins. This page explains the types of plugins and provides links to several @@ -16,7 +16,7 @@ volume and network plugins for Docker. ## Types of plugins -Plugins extend Docker's functionality. They come in specific types. For +Plugins extend Docker's functionality. They come in specific types. For example, a [volume plugin](plugins_volume.md) might enable Docker volumes to persist across multiple Docker hosts and a [network plugin](plugins_network.md) might provide network plumbing. diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index 4600992c4e..a0b0a73675 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -8,7 +8,7 @@ Docker plugins are out-of-process extensions which add capabilities to the Docker Engine. This document describes the Docker Engine plugin API. To view information on -plugins managed by Docker Engine, refer to [Docker Engine plugin system](index.md). +plugins managed by Docker Engine, refer to [Docker Engine plugin system](_index.md). This page is intended for people who want to develop their own Docker plugin. If you just want to learn about or use Docker plugins, look diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 1fc6f78441..b8678debd5 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -8,7 +8,7 @@ aliases: This document describes the Docker Engine plugins available in Docker Engine. To view information on plugins managed by Docker Engine, -refer to [Docker Engine plugin system](index.md). +refer to [Docker Engine plugin system](_index.md). Docker's out-of-the-box authorization model is all or nothing. Any user with permission to access the Docker daemon can run any Docker client command. The diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index e24fd26257..8f94546b01 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -6,12 +6,12 @@ keywords: "Examples, Usage, plugins, docker, documentation, user guide" This document describes Docker Engine network driver plugins generally available in Docker Engine. To view information on plugins -managed by Docker Engine, refer to [Docker Engine plugin system](index.md). +managed by Docker Engine, refer to [Docker Engine plugin system](_index.md). Docker Engine network plugins enable Engine deployments to be extended to support a wide range of networking technologies, such as VXLAN, IPVLAN, MACVLAN or something completely different. Network driver plugins are supported via the -LibNetwork project. Each plugin is implemented as a "remote driver" for +LibNetwork project. Each plugin is implemented as a "remote driver" for LibNetwork, which shares plugin infrastructure with Engine. Effectively, network driver plugins are activated in the same way as other plugins, and use the same kind of protocol. @@ -19,7 +19,7 @@ kind of protocol. ## Network plugins and Swarm mode [Legacy plugins](legacy_plugins.md) do not work in Swarm mode. However, -plugins written using the [v2 plugin system](index.md) do work in Swarm mode, as +plugins written using the [v2 plugin system](_index.md) do work in Swarm mode, as long as they are installed on each Swarm worker node. ## Use network driver plugins From d1f852fbd4578040be11751584a3b46d7d390d55 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 4 Sep 2024 13:35:08 +0200 Subject: [PATCH 479/534] docs: add front matter title to deprecated.md Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1f28bbcff2..7f598ba8bc 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -1,4 +1,6 @@ --- +title: Deprecated Docker Engine features +linkTitle: Deprecated features aliases: ["/engine/misc/deprecated/"] description: "Deprecated Features." keywords: "docker, documentation, about, technology, deprecate" @@ -13,8 +15,6 @@ keywords: "docker, documentation, about, technology, deprecate" will be rejected. --> -# Deprecated Engine Features - This page provides an overview of features that are deprecated in Engine. Changes in packaging, and supported (Linux) distributions are not included. To learn about end of support for Linux distributions, refer to the From 9c2b0b46a72437865f1eea40d3687bd53e297c17 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Wed, 4 Sep 2024 13:45:57 +0200 Subject: [PATCH 480/534] chore: fix style/lint issues in deprecated.md Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/deprecated.md | 211 +++++++++++++++++++++++---------------------- 1 file changed, 106 insertions(+), 105 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 7f598ba8bc..2748dd9254 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -20,7 +20,7 @@ in packaging, and supported (Linux) distributions are not included. To learn about end of support for Linux distributions, refer to the [release notes](https://docs.docker.com/engine/release-notes/). -## Feature Deprecation Policy +## Feature deprecation policy As changes are made to Docker there may be times when existing features need to be removed or replaced with newer features. Before an existing feature is removed @@ -32,21 +32,24 @@ Users are expected to take note of the list of deprecated features each release and plan their migration away from those features, and (if applicable) towards the replacement features as soon as possible. -## Deprecated Engine Features +## Deprecated engine features -The table below provides an overview of the current status of deprecated features: +The following table provides an overview of the current status of deprecated features: - **Deprecated**: the feature is marked "deprecated" and should no longer be used. + The feature may be removed, disabled, or change behavior in a future release. - The _"Deprecated"_ column contains the release in which the feature was marked + The _"Deprecated"_ column contains the release in which the feature was marked deprecated, whereas the _"Remove"_ column contains a tentative release in which the feature is to be removed. If no release is included in the _"Remove"_ column, the release is yet to be decided on. -- **Removed**: the feature was removed, disabled, or hidden. Refer to the linked - section for details. Some features are "soft" deprecated, which means that they - remain functional for backward compatibility, and to allow users to migrate to - alternatives. In such cases, a warning may be printed, and users should not rely - on this feature. + +- **Removed**: the feature was removed, disabled, or hidden. + + Refer to the linked section for details. Some features are "soft" deprecated, + which means that they remain functional for backward compatibility, and to + allow users to migrate to alternatives. In such cases, a warning may be + printed, and users should not rely on this feature. | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| @@ -57,10 +60,10 @@ The table below provides an overview of the current status of deprecated feature | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | -| Deprecated | [IsAutomated field, and "is-automated" filter on docker search](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | +| Deprecated | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | -| Removed | [Buildkit build information](#buildkit-build-information) | v23.0 | v24.0 | +| Removed | [BuildKit build information](#buildkit-build-information) | v23.0 | v24.0 | | Deprecated | [Legacy builder for Linux images](#legacy-builder-for-linux-images) | v23.0 | - | | Deprecated | [Legacy builder fallback](#legacy-builder-fallback) | v23.0 | - | | Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0 | @@ -91,7 +94,7 @@ The table below provides an overview of the current status of deprecated feature | Removed | [Asynchronous `service create` and `service update` as default](#asynchronous-service-create-and-service-update-as-default) | v17.05 | v17.10 | | Removed | [`-g` and `--graph` flags on `dockerd`](#-g-and---graph-flags-on-dockerd) | v17.05 | v23.0 | | Deprecated | [Top-level network properties in NetworkSettings](#top-level-network-properties-in-networksettings) | v1.13 | v17.12 | -| Removed | [`filter` param for `/images/json` endpoint](#filter-param-for-imagesjson-endpoint) | v1.13 | v20.10 | +| Removed | [`filter` option for `/images/json` endpoint](#filter-option-for-imagesjson-endpoint) | v1.13 | v20.10 | | Removed | [`repository:shortid` image references](#repositoryshortid-image-references) | v1.13 | v17.12 | | Removed | [`docker daemon` subcommand](#docker-daemon-subcommand) | v1.13 | v17.12 | | Removed | [Duplicate keys with conflicting values in engine labels](#duplicate-keys-with-conflicting-values-in-engine-labels) | v1.13 | v17.12 | @@ -122,8 +125,8 @@ The table below provides an overview of the current status of deprecated feature The `Config` field returned shown in `docker image inspect` (and as returned by the `GET /images/{name}/json` API endpoint) returns additional fields that are -not part of the image's configuration and not part of the [Docker Image Spec] -and [OCI Image Specification]. +not part of the image's configuration and not part of the [Docker image specification] +and [OCI image specification]. These fields are never set (and always return the default value for the type), but are not omitted in the response when left empty. As these fields were not @@ -131,7 +134,7 @@ intended to be part of the image configuration response, they are deprecated, and will be removed from the API in thee next release. The following fields are currently included in the API response, but are not -part of the underlying image's Config, and deprecated: +part of the underlying image's `Config` field, and deprecated: - `Hostname` - `Domainname` @@ -146,8 +149,8 @@ part of the underlying image's Config, and deprecated: - `MacAddress` (already omitted unless set) - `StopTimeout` (already omitted unless set) -[Docker image spec]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 -[OCI Image Spec]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 +[Docker image specification]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 +[OCI image specification]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 ### Graphdriver plugins (experimental) @@ -211,7 +214,7 @@ transit and providing a mechanism for mutual authentication. For environments remote daemon access isn't required, we recommend binding the Docker daemon to a Unix socket. -For daemon's where remote access is required and where TLS encryption is not feasible, +For daemons where remote access is required and where TLS encryption is not feasible, you may want to consider using SSH as an alternative solution. For further information, assistance, and step-by-step instructions on @@ -255,19 +258,19 @@ daemon may be updated to the latest release, but not all clients may be up-to-da or vice versa). Support for API versions before that (API versions provided by EOL versions of the Docker Daemon) is provided on a "best effort" basis. -Use of old API versions is very rare, and support for legacy API versions +Use of old API versions is rare, and support for legacy API versions involves significant complexity (Docker 1.0.0 having been released 10 years ago). Because of this, we'll start deprecating support for legacy API versions. Docker Engine v25.0 by default disables API version older than 1.24 (aligning the minimum supported API version between Linux and Windows daemons). When -connecting with a client that uses an API version version older than 1.24, -the daemon returns an error. The following example configures the docker +connecting with a client that uses an API version older than 1.24, +the daemon returns an error. The following example configures the Docker CLI to use API version 1.23, which produces an error: ```console DOCKER_API_VERSION=1.23 docker version -Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, +Error response from daemon: client version 1.23 is too old. Minimum supported API version is 1.24, upgrade your client to a newer version ``` @@ -301,12 +304,12 @@ A new field `DNSNames` containing the container name (if one was specified), the hostname, the network aliases, as well as the container short ID, has been introduced in v25.0 and should be used instead of the `Aliases` field. -### IsAutomated field, and "is-automated" filter on docker search +### IsAutomated field, and `is-automated` filter on `docker search` **Deprecated in Release: v25.0** **Target For Removal In Release: v26.0** -The "is_automated" field has been deprecated by Docker Hub's search API. +The `is_automated` field has been deprecated by Docker Hub's search API. Consequently, the `IsAutomated` field in image search will always be set to `false` in future, and searching for "is-automated=true" will yield no results. @@ -346,7 +349,7 @@ Users currently depending on this feature are recommended to adjust the daemon's OOM score using systemd or through other means, when starting the daemon. -### Buildkit build information +### BuildKit build information **Deprecated in Release: v23.0** **Removed in Release: v24.0** @@ -354,7 +357,7 @@ the daemon. [Build information](https://github.com/moby/buildkit/blob/v0.11/docs/buildinfo.md) structures have been introduced in [BuildKit v0.10.0](https://github.com/moby/buildkit/releases/tag/v0.10.0) and are generated with build metadata that allows you to see all the sources -(images, git repositories) that were used by the build with their exact +(images, Git repositories) that were used by the build with their exact versions and also the configuration that was passed to the build. This information is also embedded into the image configuration if one is generated. @@ -389,7 +392,7 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co > `docker build` continues to use the classic builder to build native Windows > images on Windows daemons. -### Legacy builder fallback +### Legacy builder fallback **Deprecated in Release: v23.0** @@ -402,11 +405,11 @@ To provide a smooth transition to BuildKit as the default builder, Docker v23.0 has an automatic fallback for some situations, or produces an error to assist users to resolve the problem. -In situations where the user did not explicitly opt-in to use BuildKit (i.e., +In situations where the user did not explicitly opt-in to use BuildKit (i.e., `DOCKER_BUILDKIT=1` is not set), the CLI automatically falls back to the classic builder, but prints a deprecation warning: -``` +```text DEPRECATED: The legacy builder is deprecated and will be removed in a future release. Install the buildx component to build images with BuildKit: https://docs.docker.com/go/buildx/ @@ -420,7 +423,7 @@ and use BuildKit for your builds, or opt-out of using BuildKit with `DOCKER_BUIL If you opted-in to use BuildKit (`DOCKER_BUILDKIT=1`), but the Buildx component is missing, an error is printed instead, and the `docker build` command fails: -``` +```text ERROR: BuildKit is enabled but the buildx component is missing or broken. Install the buildx component to build images with BuildKit: https://docs.docker.com/go/buildx/ @@ -470,27 +473,27 @@ to decrypt the private key, and store it un-encrypted to continue using it. Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from -the cli, and options related to this functionality are now either ignored, or may +the CLI, and options related to this functionality are now either ignored, or may produce an error. The following command-line flags are removed from the `docker context` subcommands: - `--default-stack-orchestrator` - swarm is now the only (and default) orchestrator for stacks. -- `--kubernetes` - the kubernetes endpoint can no longer be stored in `docker context`. +- `--kubernetes` - the Kubernetes endpoint can no longer be stored in `docker context`. - `--kubeconfig` - exporting a context as a kubeconfig file is no longer supported. The output produced by the `docker context inspect` subcommand no longer contains information about `StackOrchestrator` and `Kubernetes` endpoints for new contexts. The following command-line flags are removed from the `docker stack` subcommands: - + - `--kubeconfig` - using a kubeconfig file as context is no longer supported. -- `--namespace` - configuring the kubernetes namespace for stacks is no longer supported. +- `--namespace` - configuring the Kubernetes namespace for stacks is no longer supported. - `--orchestrator` - swarm is now the only (and default) orchestrator for stacks. The `DOCKER_STACK_ORCHESTRATOR`, `DOCKER_ORCHESTRATOR`, and `KUBECONFIG` environment variables, as well as the `stackOrchestrator` option in the `~/.docker/config.json` -cli configuration file are no longer used, and ignored. +CLI configuration file are no longer used, and ignored. ### Pulling images from non-compliant image registries @@ -532,7 +535,7 @@ major release. The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after its introduction, the feature never reached completeness, and development has -now stopped in favor of running docker natively on Linux in WSL2. +now stopped in favor of running Docker natively on Linux in WSL2. Developers who want to run Linux workloads on a Windows host are encouraged to use [Docker Desktop with WSL2](https://docs.docker.com/docker-for-windows/wsl/) instead. @@ -565,11 +568,11 @@ take no effect. > [!NOTE] > While not deprecated (yet) in Docker, the OCI runtime specification also > deprecated the `memory.kmem.tcp.limit_in_bytes` option. When using `runc` as -> runtime, this option takes no effect. The linux kernel did not explicitly +> runtime, this option takes no effect. The Linux kernel did not explicitly > deprecate this feature, and there is a tracking ticket in the `runc` issue > tracker to determine if this option should be reinstated or if this was an > oversight of the Linux kernel maintainers (see [opencontainers/runc#3174](https://github.com/opencontainers/runc/issues/3174)). -> +> > The `memory.kmem.tcp.limit_in_bytes` option is only supported with cgroups v1, > and not available on installations running with cgroups v2. This option is > only supported by the API, and not exposed on the `docker` command-line. @@ -588,7 +591,7 @@ networks using an external key/value store. The corresponding`--cluster-advertis **Deprecated in Release: v20.10** **Removed in Release: v23.0** -The docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials +The Docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this file with a new CLI configuration file, located in `~/.docker/config.json`. When implementing the new configuration file, the old file (and file-format) was kept @@ -722,8 +725,7 @@ to Docker Enterprise, using an image-based distribution of the Docker Engine. This feature was only available on Linux, and only when executed on a local node. Given the limitations of this feature, and the feature not getting widely adopted, the `docker engine` subcommands will be removed, in favor of installation through -standard package managers. - +standard package managers. ### Top-level `docker deploy` subcommand (experimental) @@ -736,7 +738,6 @@ The top-level `docker deploy` command (using the "Docker Application Bundle" 17.03, but superseded by support for Docker Compose files using the `docker stack deploy` subcommand. - ### `docker stack deploy` using "dab" files (experimental) **Deprecated in Release: v19.03** @@ -744,7 +745,7 @@ subcommand. **Removed in Release: v20.10** With no development being done on this feature, and no active use of the file -format, support for the DAB file format and the top-level docker deploy command +format, support for the DAB file format and the top-level `docker deploy` command (hidden by default in 19.03), will be removed, in favour of `docker stack deploy` using compose files. @@ -785,12 +786,12 @@ maintenance of the `aufs` storage driver. The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive -inode consumption). The legacy `overlay` storage driver has been removed in +inode consumption). The legacy `overlay` storage driver has been removed in Docker Engine v24.0. Users of the `overlay` storage driver should migrate to the `overlay2` storage driver before upgrading to Docker Engine v24.0. The legacy `overlay` storage driver allowed using overlayFS-backed filesystems -on pre 4.x kernels. Now that all supported distributions are able to run `overlay2` +on kernels older than v4.x. Now that all supported distributions are able to run `overlay2` (as they are either on kernel 4.x, or have support for multiple lowerdirs backported), there is no reason to keep maintaining the `overlay` storage driver. @@ -824,7 +825,6 @@ were always documented to be reserved, but there was never any enforcement. Usage of these namespaces will now cause a warning in the engine logs to discourage their use, and will error instead in v20.10 and above. - ### `--disable-legacy-registry` override daemon option **Disabled In Release: v17.12** @@ -835,7 +835,6 @@ The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print an error when used. For this error to be printed, the flag itself is still present, but hidden. The flag has been removed in Docker 19.03. - ### Interacting with V1 registries **Disabled By Default In Release: v17.06** @@ -843,7 +842,7 @@ but hidden. The flag has been removed in Docker 19.03. **Removed In Release: v17.12** Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the -docker daemon from `pull`, `push`, and `login` operations against v1 +Docker daemon from `pull`, `push`, and `login` operations against v1 registries. Though enabled by default, this signals the intent to deprecate the v1 protocol. @@ -855,7 +854,6 @@ Starting with Docker 17.12, support for V1 registries has been removed, and the `--disable-legacy-registry` flag can no longer be used, and `dockerd` will fail to start when set. - ### Asynchronous `service create` and `service update` as default **Deprecated In Release: v17.05** @@ -895,20 +893,22 @@ about the default ("bridge") network; These properties are deprecated in favor of per-network properties in `NetworkSettings.Networks`. These properties were already "deprecated" in -docker 1.9, but kept around for backward compatibility. +Docker 1.9, but kept around for backward compatibility. Refer to [#17538](https://github.com/docker/docker/pull/17538) for further information. -### `filter` param for `/images/json` endpoint +### `filter` option for `/images/json` endpoint + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Removed In Release: v20.10** -The `filter` param to filter the list of image by reference (name or name:tag) +The `filter` option to filter the list of image by reference (name or name:tag) is now implemented as a regular filter, named `reference`. ### `repository:shortid` image references + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Removed In Release: v17.12** @@ -920,6 +920,7 @@ Support for the `repository:shortid` notation to reference images was removed in Docker 17.12. ### `docker daemon` subcommand + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Removed In Release: v17.12** @@ -927,6 +928,7 @@ in Docker 17.12. The daemon is moved to a separate binary (`dockerd`), and should be used instead. ### Duplicate keys with conflicting values in engine labels + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Removed In Release: v17.12** @@ -935,11 +937,13 @@ When setting duplicate keys with conflicting values, an error will be produced, will fail to start. ### `MAINTAINER` in Dockerfile + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** `MAINTAINER` was an early very limited form of `LABEL` which should be used instead. ### API calls without a version + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Target For Removal In Release: v17.12** @@ -949,6 +953,7 @@ future Engine versions. Instead of just requesting, for example, the URL `/containers/json`, you must now request `/v1.25/containers/json`. ### Backing filesystem without `d_type` support for overlay/overlay2 + **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** **Removed In Release: v17.12** @@ -963,7 +968,6 @@ backing filesystem without `d_type` support. Refer to [#27358](https://github.com/docker/docker/issues/27358) for details. - ### `--automated` and `--stars` flags on `docker search` **Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -973,7 +977,6 @@ Refer to [#27358](https://github.com/docker/docker/issues/27358) for details. The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=` and `docker search --filter=stars=...` instead. - ### `-h` shorthand for `--help` **Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -986,13 +989,15 @@ on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on "usage" output of subcommands, nor documented, and is now marked "deprecated". ### `-e` and `--email` flags on `docker login` + **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Removed In Release: [v17.06](https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce)** -The docker login command is removing the ability to automatically register for an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. +The `docker login` no longer automatically registers an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` + **Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** **Target For Removal In Release: v17.06** @@ -1000,12 +1005,14 @@ The docker login command is removing the ability to automatically register for a The flag `--security-opt` doesn't use the colon separator (`:`) anymore to divide keys and values, it uses the equal symbol (`=`) for consistency with other similar flags, like `--storage-opt`. ### Ambiguous event fields in API + **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. See the events API documentation for the new format. ### `-f` flag on `docker tag` + **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -1013,6 +1020,7 @@ See the events API documentation for the new format. To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is no longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start + **Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -1029,8 +1037,8 @@ defining it at container creation (`POST /containers/create`). The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. - ### Driver-specific log tags + **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -1043,8 +1051,8 @@ Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` a $ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ``` - ### Docker Content Trust ENV passphrase variables name change + **Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** @@ -1054,7 +1062,6 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the - DOCKER_CONTENT_TRUST_OFFLINE_PASSPHRASE is now named DOCKER_CONTENT_TRUST_ROOT_PASSPHRASE - DOCKER_CONTENT_TRUST_TAGGING_PASSPHRASE is now named DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE - ### `/containers/(id or name)/copy` endpoint **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** @@ -1063,63 +1070,61 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. - ### LXC built-in exec driver + **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** **Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. - ### Old Command Line Options + **Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** **Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -The flags `-d` and `--daemon` are deprecated in favor of the `daemon` subcommand: - - docker daemon -H ... +The flags `-d` and `--daemon` are deprecated. Use the separate `dockerd` binary instead. The following single-dash (`-opt`) variant of certain command line options are deprecated and replaced with double-dash options (`--opt`): - docker attach -nostdin - docker attach -sig-proxy - docker build -no-cache - docker build -rm - docker commit -author - docker commit -run - docker events -since - docker history -notrunc - docker images -notrunc - docker inspect -format - docker ps -beforeId - docker ps -notrunc - docker ps -sinceId - docker rm -link - docker run -cidfile - docker run -dns - docker run -entrypoint - docker run -expose - docker run -link - docker run -lxc-conf - docker run -n - docker run -privileged - docker run -volumes-from - docker search -notrunc - docker search -stars - docker search -t - docker search -trusted - docker tag -force +- `docker attach -nostdin` +- `docker attach -sig-proxy` +- `docker build -no-cache` +- `docker build -rm` +- `docker commit -author` +- `docker commit -run` +- `docker events -since` +- `docker history -notrunc` +- `docker images -notrunc` +- `docker inspect -format` +- `docker ps -beforeId` +- `docker ps -notrunc` +- `docker ps -sinceId` +- `docker rm -link` +- `docker run -cidfile` +- `docker run -dns` +- `docker run -entrypoint` +- `docker run -expose` +- `docker run -link` +- `docker run -lxc-conf` +- `docker run -n` +- `docker run -privileged` +- `docker run -volumes-from` +- `docker search -notrunc` +- `docker search -stars` +- `docker search -t` +- `docker search -trusted` +- `docker tag -force` The following double-dash options are deprecated and have no replacement: - docker run --cpuset - docker run --networking - docker ps --since-id - docker ps --before-id - docker search --trusted +- `docker run --cpuset` +- `docker run --networking` +- `docker ps --since-id` +- `docker ps --before-id` +- `docker search --trusted` **Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** @@ -1127,11 +1132,7 @@ The following double-dash options are deprecated and have no replacement: The single-dash (`-help`) was removed, in favor of the double-dash `--help` - docker -help - docker [COMMAND] -help - - -### `--api-enable-cors` flag on dockerd +### `--api-enable-cors` flag on `dockerd` **Deprecated In Release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** @@ -1140,19 +1141,19 @@ The single-dash (`-help`) was removed, in favor of the double-dash `--help` The flag `--api-enable-cors` is deprecated since v1.6.0. Use the flag `--api-cors-header` instead. -### `--run` flag on docker commit +### `--run` flag on `docker commit` **Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** **Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -The flag `--run` of the docker commit (and its short version `-run`) were deprecated in favor +The flag `--run` of the `docker commit` command (and its short version `-run`) were deprecated in favor of the `--changes` flag that allows to pass `Dockerfile` commands. - ### Three arguments form in `docker import` + **Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** **Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated since November 2013. It's no more supported. +The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated since November 2013. It's no longer supported. From 3ae4ecc97e44a5bbabf24f8fd4a392f1ef15ac72 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 16 Sep 2024 12:19:27 +0200 Subject: [PATCH 481/534] docs: dockerd: remove --api-cors-header (deprecated) Updates 2953ec627bcddc7f93c6f95ecac4fa5229fa56b1 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 9 +++++---- docs/reference/dockerd.md | 2 -- 2 files changed, 5 insertions(+), 6 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 2748dd9254..735ff4f862 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -54,7 +54,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | -| Deprecated | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | +| Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Deprecated | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | @@ -177,18 +177,19 @@ and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18 ### API CORS headers **Deprecated in Release: v27.0** -**Target For Removal In Release: v28.0** +**Disabled by default in Release: v27.0** +**Removed in release: v28.0** The `api-cors-header` configuration option for the Docker daemon is insecure, and is therefore deprecated and scheduled for removal. Incorrectly setting this option could leave a window of opportunity for unauthenticated cross-origin requests to be accepted by the daemon. -Starting in Docker Engine v27.0, this flag can still be set, +In Docker Engine v27.0, this flag can still be set, but it has no effect unless the environment variable `DOCKERD_DEPRECATED_CORS_HEADER` is also set to a non-empty value. -This flag will be removed altogether in v28.0. +This flag has been removed altogether in v28.0. This is a breaking change for authorization plugins and other programs that depend on this option for accessing the Docker API from a browser. diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index c13c20a864..6292e57e79 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -25,7 +25,6 @@ A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) --allow-nondistributable-artifacts list Allow push of nondistributable artifacts to registry - --api-cors-header string Set CORS headers in the Engine API --authorization-plugin list Authorization plugins to load --bip string Specify network bridge IP -b, --bridge string Attach containers to a network bridge @@ -1015,7 +1014,6 @@ The following is a full example of the allowed configuration options on Linux: ```json { "allow-nondistributable-artifacts": [], - "api-cors-header": "", "authorization-plugins": [], "bip": "", "bridge": "", From c4d6bdce5e7448f228cecc195590f4201c03d911 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 16 Sep 2024 11:28:41 +0200 Subject: [PATCH 482/534] docs/reference: dockerd: add docs for --feature option Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index c13c20a864..a6b6464aeb 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -57,6 +57,7 @@ Options: --exec-opt list Runtime execution options --exec-root string Root directory for execution state files (default "/var/run/docker") --experimental Enable experimental features + --feature map Enable feature in the daemon --fixed-cidr string IPv4 subnet for fixed IPs --fixed-cidr-v6 string IPv6 subnet for fixed IPs -G, --group string Group for the unix socket (default "docker") @@ -971,6 +972,36 @@ Example of usage: } ``` +### Enable feature in the daemon (--feature) + +The `--feature` option lets you enable or disable a feature in the daemon. +This option corresponds with the "features" field in the [daemon.json configuration file](#daemon-configuration-file). +Features should only be configured either through the `--feature` command line +option or through the "features" field in the configuration file; using both +the command-line option and the "features" field in the configuration +file produces an error. The feature option can be specified multiple times +to configure multiple features. The `--feature` option accepts a name and +optional boolean value. When omitting the value, the default is `true`. + +The following example runs the daemon with the `cdi` and `containerd-snapshotter` +features enabled. The `cdi` option is provided with a value; + +```console +$ dockerd --feature cdi=true --feature containerd-snapshotter +``` + +The following example is the equivalent using the `daemon.json` configuration +file; + +```json +{ + "features": { + "cdi": true, + "containerd-snapshotter": true + } +} +``` + ### Daemon configuration file The `--config-file` option allows you to set any configuration option @@ -1065,7 +1096,10 @@ The following is a full example of the allowed configuration options on Linux: "exec-opts": [], "exec-root": "", "experimental": false, - "features": {}, + "features": { + "cdi": true, + "containerd-snapshotter": true + }, "fixed-cidr": "", "fixed-cidr-v6": "", "group": "", From e516dd9a4794917412d99761a45ff2e5364a32cb Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 16 Sep 2024 13:16:13 +0200 Subject: [PATCH 483/534] docs: dockerd: add documentation for --log-format option Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index c13c20a864..942b029a9f 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -79,6 +79,7 @@ Options: --label list Set key=value labels to the daemon --live-restore Enable live restore of docker when containers are still running --log-driver string Default driver for container logs (default "json-file") + --log-format string Set the logging format ("text"|"json") (default "text") -l, --log-level string Set the logging level ("debug"|"info"|"warn"|"error"|"fatal") (default "info") --log-opt map Default log driver options for containers (default map[]) --max-concurrent-downloads int Set the max concurrent downloads (default 3) @@ -890,6 +891,33 @@ Alternatively, you can set custom locations for CDI specifications using the When CDI is enabled for a daemon, you can view the configured CDI specification directories using the `docker info` command. +#### Daemon logging format + +The `--log-format` option or "log-format" option in the [daemon configuration file](#daemon-configuration-file) +lets you set the format for logs produced by the daemon. The logging format should +only be configured either through the `--log-format` command line option or +through the "log-format" field in the configuration file; using both +the command-line option and the "log-format" field in the configuration +file produces an error. If this option is not set, the default is "text". + +The following example configures the daemon through the `--log-format` command +line option to use `json` formatted logs; + +```console +$ dockerd --log-format=json +# ... +{"level":"info","msg":"API listen on /var/run/docker.sock","time":"2024-09-16T11:06:08.558145428Z"} +``` + +The following example shows a `daemon.json` configuration file with the +"log-format" set; + +```json +{ + "log-format": "json" +} +``` + ### Miscellaneous options IP masquerading uses address translation to allow containers without a public @@ -1089,6 +1117,7 @@ The following is a full example of the allowed configuration options on Linux: "labels": [], "live-restore": true, "log-driver": "json-file", + "log-format": "text", "log-level": "", "log-opts": { "cache-disabled": "false", @@ -1183,6 +1212,7 @@ The following is a full example of the allowed configuration options on Windows: "insecure-registries": [], "labels": [], "log-driver": "", + "log-format": "text", "log-level": "", "max-concurrent-downloads": 3, "max-concurrent-uploads": 5, From bcd6efa9f8234b04ccfce3e6aeb61b4cae43a5b8 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 27 Sep 2024 16:26:33 +0200 Subject: [PATCH 484/534] cli/command/container: stop, restart: rename "--time" to "--timeout" This renames the `--time` flag as used on `docker stop` and `docker restart` to `--timeout`, bringing it in line with other uses for this property, such as `--stop-timeout` on `docker run`. The `--time` option is deprecated and hidden, but will be kept for backward compatibility, as these options existed for a long time. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 735ff4f862..9c99fad610 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | | Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | | Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Deprecated | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | @@ -118,6 +119,16 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### `--time` option on `docker stop` and `docker restart` + +**Deprecated in Release: v28.0** + +The `--time` option for the `docker stop`, `docker container stop`, `docker restart`, +and `docker container restart` commands has been renamed to `--timeout` for +consistency with other uses of timeout options. The `--time` option is now +deprecated and hidden, but remains functional for backward compatibility. +Users are encouraged to migrate to using the `--timeout` option instead. + ### Non-standard fields in image inspect **Deprecated in Release: v27.0** From e7570294a8f7943e26945e86618922177d9043b6 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Wed, 6 Nov 2024 14:39:08 +0000 Subject: [PATCH 485/534] Add --bip6 to dockerd cmdline ref Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 5f57756e88..a0cc96ddfd 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -26,7 +26,8 @@ Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) --allow-nondistributable-artifacts list Allow push of nondistributable artifacts to registry --authorization-plugin list Authorization plugins to load - --bip string Specify network bridge IP + --bip string Specify default-bridge IPv4 network + --bip6 string Specify default-bridge IPv6 network -b, --bridge string Attach containers to a network bridge --cdi-spec-dir list CDI specification directories to use --cgroup-parent string Set parent cgroup for all containers @@ -1075,6 +1076,7 @@ The following is a full example of the allowed configuration options on Linux: "allow-nondistributable-artifacts": [], "authorization-plugins": [], "bip": "", + "bip6": "", "bridge": "", "builder": { "gc": { From 3ce26f6f11ca420b26356830286e71fcbb304e07 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Thu, 28 Nov 2024 15:01:09 +0100 Subject: [PATCH 486/534] docs: fix janky rendering of toc on docs.docker.com Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/dockerd.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index a0cc96ddfd..de8011c2b6 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -892,7 +892,7 @@ Alternatively, you can set custom locations for CDI specifications using the When CDI is enabled for a daemon, you can view the configured CDI specification directories using the `docker info` command. -#### Daemon logging format +#### Daemon logging format {#log-format} The `--log-format` option or "log-format" option in the [daemon configuration file](#daemon-configuration-file) lets you set the format for logs produced by the daemon. The logging format should @@ -1000,7 +1000,7 @@ Example of usage: } ``` -### Enable feature in the daemon (--feature) +### Enable feature in the daemon (--feature) {#feature} The `--feature` option lets you enable or disable a feature in the daemon. This option corresponds with the "features" field in the [daemon.json configuration file](#daemon-configuration-file). From ea810dda5bde39f9047b401d71b11d968f191afc Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Wed, 6 Nov 2024 14:47:53 +0000 Subject: [PATCH 487/534] Add --ip-filter-forward-drop Added to the dockerd cmdline ref and its manpage. Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index de8011c2b6..97b9d23e3f 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -72,7 +72,8 @@ Options: --init-path string Path to the docker-init binary --insecure-registry list Enable insecure registry communication --ip ip Default IP when binding container ports (default 0.0.0.0) - --ip-forward Enable net.ipv4.ip_forward (default true) + --ip-forward Enable IP forwarding in system configuration (default true) + --ip-forward-no-drop Do not set the filter-FORWARD policy to DROP when enabling IP forwarding --ip-masq Enable IP masquerading (default true) --ip6tables Enable addition of ip6tables rules (experimental) --iptables Enable addition of iptables rules (default true) From 54c7d98766aa457b2844c29a2625a3d3f57a363c Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Mon, 11 Nov 2024 11:57:39 +0000 Subject: [PATCH 488/534] Docs: host-gateway-ip daemon option IPv4+IPv6 The host-gateway-ip daemon option now accepts two addresses, one IPv4 and one IPv6. Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 97b9d23e3f..bc378a1993 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -63,8 +63,8 @@ Options: -G, --group string Group for the unix socket (default "docker") --help Print usage -H, --host list Daemon socket(s) to connect to - --host-gateway-ip ip IP address that the special 'host-gateway' string in --add-host resolves to. - Defaults to the IP address of the default bridge + --host-gateway-ip list IP addresses that the special 'host-gateway' string in --add-host resolves to. + Defaults to the IP addresses of the default bridge --http-proxy string HTTP proxy URL to use for outgoing traffic --https-proxy string HTTPS proxy URL to use for outgoing traffic --icc Enable inter-container communication (default true) @@ -839,21 +839,34 @@ For details about how to use this feature, as well as limitations, see The Docker daemon supports a special `host-gateway` value for the `--add-host` flag for the `docker run` and `docker build` commands. This value resolves to -the host's gateway IP and lets containers connect to services running on the +addresses on the host, so that containers can connect to services running on the host. -By default, `host-gateway` resolves to the IP address of the default bridge. +By default, `host-gateway` resolves to the IPv4 address of the default bridge, +and its IPv6 address if it has one. + You can configure this to resolve to a different IP using the `--host-gateway-ip` flag for the dockerd command line interface, or the `host-gateway-ip` key in the daemon configuration file. +To supply both IPv4 and IPv6 addresses on the command line, use two +`--host-gateway-ip` options. + +To supply addresses in the daemon configuration file, use `"host-gateway-ips"` +with a JSON array, as shown below. For compatibility with older versions of the +daemon, a single IP address can also be specified as a JSON string in option +`"host-gateway-ip"`. + ```console $ cat > /etc/docker/daemon.json -{ "host-gateway-ip": "192.0.2.0" } +{ "host-gateway-ips": ["192.0.2.1", "2001:db8::1111"]} $ sudo systemctl restart docker $ docker run -it --add-host host.docker.internal:host-gateway \ busybox ping host.docker.internal -PING host.docker.internal (192.0.2.0): 56 data bytes +PING host.docker.internal (192.0.2.1): 56 data bytes +$ docker run -it --add-host host.docker.internal:host-gateway \ + busybox ping -6 host.docker.internal +PING host.docker.internal (2001:db8::1111): 56 data bytes ``` ### Enable CDI devices From e0f9b3b9dfff84b63b95da38eb4e8fdf0d33ec88 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 3 Jan 2025 20:20:47 +0100 Subject: [PATCH 489/534] docs: dockerd: --oom-score-adj flag This flag was deprecated in docker v24.0, and no longer functional since v25.0; fully removed in v26.0, so we can remove the docs for this. Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index bc378a1993..75ae2b31ba 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -93,7 +93,6 @@ Options: --no-new-privileges Set no-new-privileges by default for new containers --no-proxy string Comma-separated list of hosts or IP addresses for which the proxy is skipped --node-generic-resource list Advertise user-defined resource - --oom-score-adjust int Set the oom_score_adj for the daemon -p, --pidfile string Path to use for daemon PID file (default "/var/run/docker.pid") --raw-logs Full timestamps without ANSI coloring --registry-mirror list Preferred registry mirror @@ -1186,7 +1185,6 @@ The following is a full example of the allowed configuration options on Linux: "NVIDIA-GPU=UUID1", "NVIDIA-GPU=UUID2" ], - "oom-score-adjust": 0, "pidfile": "", "raw-logs": false, "registry-mirrors": [], From d0604885f61f9fe76310e6db38f8096ed16c8ca8 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 3 Jan 2025 20:50:48 +0100 Subject: [PATCH 490/534] Deprecate configuration for pushing non-distributable artifacts Non-distributable artifacts (also called foreign layers) were introduced in docker v1.12 to accommodate Windows images for which the EULA did not allow layers to be distributed through registries other than those hosted by Microsoft. The concept of foreign / non-distributable layers was adopted by the OCI distribution spec in [oci#233]. These restrictions were relaxed later to allow distributing these images through non-public registries, for which a configuration was added in Docker v17.0.6.0. In 2022, Microsoft updated the EULA and [removed these restrictions][msft-3645201], followed by the OCI distribution specification deprecating foreign layers in [oci#965]. In 2023, Microsoft [removed the use of foreign data layers][msft-3846833] for their images, making this functionality obsolete. Docker v28.0 deprecates the `--allow-nondistributable-artifacts` daemon flag and corresponding `allow-nondistributable-artifacts` field in `daemon.json`. Setting either option no longer takes an effect, but a deprecation warning log is added to raise awareness about the deprecation. This warning is planned to become an error in the Docker v29.0. Users currently using these options are therefore recommended to remove this option from their configuration to prevent the daemon from starting when upgrading to Docker v29.0. The `AllowNondistributableArtifactsCIDRs` and `AllowNondistributableArtifactsHostnames` fields in the `RegistryConfig` of the `GET /info` API response are also deprecated. For API version v1.48 and lower, the fields are still included in the response but always `null`. In API version v1.49 and higher, the field will be omitted entirely. [oci#233]: https://github.com/opencontainers/image-spec/pull/233 [oci#965]: https://github.com/opencontainers/image-spec/pull/965 [msft-3645201]: https://techcommunity.microsoft.com/blog/containers/announcing-windows-container-base-image-redistribution-rights-change/3645201 [msft-3846833]: https://techcommunity.microsoft.com/blog/containers/announcing-removal-of-foreign-layers-from-windows-container-images/3846833 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 40 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 40 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9c99fad610..f27ea7c6a3 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Configuration for pushing non-distributable artifacts](#configuration-for-pushing-non-distributable-artifacts) | v28.0 | v29.0 | | Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | | Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | | Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | @@ -119,6 +120,45 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +## Configuration for pushing non-distributable artifacts + +**Deprecated in Release: v28.0** +**Target For Removal In Release: v29.0** + +Non-distributable artifacts (also called foreign layers) were introduced in +docker v1.12 to accommodate Windows images for which the EULA did not allow +layers to be distributed through registries other than those hosted by Microsoft. +The concept of foreign / non-distributable layers was adopted by the OCI distribution +spec in [oci#233]. These restrictions were relaxed later to allow distributing +these images through non-public registries, for which a configuration was added +in Docker v17.0.6.0. + +In 2022, Microsoft updated the EULA and [removed these restrictions][msft-3645201], +followed by the OCI distribution specification deprecating foreign layers in [oci#965]. +In 2023, Microsoft [removed the use of foreign data layers][msft-3846833] for their images, +making this functionality obsolete. + +Docker v28.0 deprecates the `--allow-nondistributable-artifacts` daemon flag and +corresponding `allow-nondistributable-artifacts` field in `daemon.json`. Setting +either option no longer takes an effect, but a deprecation warning log is added +to raise awareness about the deprecation. This warning is planned to become an +error in the Docker v29.0. + +Users currently using these options are therefore recommended to remove this +option from their configuration to prevent the daemon from starting when +upgrading to Docker v29.0. + +The `AllowNondistributableArtifactsCIDRs` and `AllowNondistributableArtifactsHostnames` +fields in the `RegistryConfig` of the `GET /info` API response are also deprecated. +For API version v1.48 and lower, the fields are still included in the response +but always `null`. In API version v1.49 and higher, the field will be omitted +entirely. + +[oci#233]: https://github.com/opencontainers/image-spec/pull/233 +[oci#965]: https://github.com/opencontainers/image-spec/pull/965 +[msft-3645201]: https://techcommunity.microsoft.com/blog/containers/announcing-windows-container-base-image-redistribution-rights-change/3645201 +[msft-3846833]: https://techcommunity.microsoft.com/blog/containers/announcing-removal-of-foreign-layers-from-windows-container-images/3846833 + ### `--time` option on `docker stop` and `docker restart` **Deprecated in Release: v28.0** From 2901bbdf6df352a0124f41e49e0ebb9c88b728e2 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 3 Jan 2025 21:08:11 +0100 Subject: [PATCH 491/534] docs, man: remove --allow-nondistributable-artifacts flag This flag is deprecated and no longer functional. Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 32 -------------------------------- 1 file changed, 32 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 75ae2b31ba..3c36462399 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -24,7 +24,6 @@ A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) - --allow-nondistributable-artifacts list Allow push of nondistributable artifacts to registry --authorization-plugin list Authorization plugins to load --bip string Specify default-bridge IPv4 network --bip6 string Specify default-bridge IPv6 network @@ -688,34 +687,6 @@ To set the DNS search domain for all Docker containers, use: $ sudo dockerd --dns-search example.com ``` -### Allow push of non-distributable artifacts - -Some images (e.g., Windows base images) contain artifacts whose distribution is -restricted by license. When these images are pushed to a registry, restricted -artifacts are not included. - -To override this behavior for specific registries, use the -`--allow-nondistributable-artifacts` option in one of the following forms: - -* `--allow-nondistributable-artifacts myregistry:5000` tells the Docker daemon - to push non-distributable artifacts to myregistry:5000. -* `--allow-nondistributable-artifacts 10.1.0.0/16` tells the Docker daemon to - push non-distributable artifacts to all registries whose resolved IP address - is within the subnet described by the CIDR syntax. - -This option can be used multiple times. - -This option is useful when pushing images containing non-distributable artifacts -to a registry on an air-gapped network so hosts on that network can pull the -images without connecting to another server. - -> [!WARNING] -> Non-distributable artifacts typically have restrictions on how -> and where they can be distributed and shared. Only use this feature to push -> artifacts to private registries and ensure that you are in compliance with -> any terms that cover redistributing non-distributable artifacts. -{ .warning } - ### Insecure registries In this section, "registry" refers to a private registry, and `myregistry:5000` @@ -1086,7 +1057,6 @@ The following is a full example of the allowed configuration options on Linux: ```json { - "allow-nondistributable-artifacts": [], "authorization-plugins": [], "bip": "", "bip6": "", @@ -1234,7 +1204,6 @@ The following is a full example of the allowed configuration options on Windows: ```json { - "allow-nondistributable-artifacts": [], "authorization-plugins": [], "bridge": "", "containerd": "\\\\.\\pipe\\containerd-containerd", @@ -1338,7 +1307,6 @@ The list of currently supported options that can be reconfigured is this: | `default-runtime` | Configures the runtime to be used if not is specified at container creation. | | `runtimes` | Configures the list of available OCI runtimes that can be used to run containers. | | `authorization-plugin` | Specifies the authorization plugins to use. | -| `allow-nondistributable-artifacts` | Specifies a list of registries to which the daemon will push non-distributable artifacts. | | `insecure-registries` | Specifies a list of registries that the daemon should consider insecure. | | `registry-mirrors` | Specifies a list of registry mirrors. | | `shutdown-timeout` | Configures the daemon's existing configuration timeout with a new timeout for shutting down all containers. | From 78770876fc82530620b47bd8f59aeb23d4cbbe06 Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Mon, 13 Jan 2025 18:38:09 +0100 Subject: [PATCH 492/534] docs: mark logger opt 'fluentd-async-connect' as removed Signed-off-by: Albin Kerouanton --- docs/deprecated.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9c99fad610..5381f218b6 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -79,7 +79,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | -| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | - | +| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | v28.0 | | Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0 | | Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v27.0 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | @@ -678,6 +678,7 @@ files. ### `fluentd-async-connect` log opt **Deprecated in Release: v20.10** +**Removed in Release: v28.0** The `--log-opt fluentd-async-connect` option for the fluentd logging driver is [deprecated in favor of `--log-opt fluentd-async`](https://github.com/moby/moby/pull/39086). From f14942f599415d0e85eb2fa0ad5af8560c3a2445 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Mon, 24 Feb 2025 15:18:56 +0000 Subject: [PATCH 493/534] Update dockerd command line ref, changes in 28.0 Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 3c36462399..f3c607a573 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -96,7 +96,7 @@ Options: --raw-logs Full timestamps without ANSI coloring --registry-mirror list Preferred registry mirror --rootless Enable rootless mode; typically used with RootlessKit - --seccomp-profile string Path to seccomp profile. Use "unconfined" to disable the default seccomp profile (default "builtin") + --seccomp-profile string Path to seccomp profile. Set to "unconfined" to disable the default seccomp profile (default "builtin") --selinux-enabled Enable selinux support --shutdown-timeout int Set the default shutdown timeout (default 15) -s, --storage-driver string Storage driver to use From 94cbaa8103d3d9b7140ed9cbeb1b54736faf1526 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Mon, 24 Feb 2025 15:19:34 +0000 Subject: [PATCH 494/534] Update dockerd command line ref, default bridge opts Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index f3c607a573..b55b66c30b 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -18,15 +18,15 @@ aliases: # daemon ```markdown -Usage: dockerd [OPTIONS] +Usage: dockerd [OPTIONS] A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) --authorization-plugin list Authorization plugins to load - --bip string Specify default-bridge IPv4 network - --bip6 string Specify default-bridge IPv6 network + --bip string IPv4 address for the default bridge + --bip6 string IPv6 address for the default bridge -b, --bridge string Attach containers to a network bridge --cdi-spec-dir list CDI specification directories to use --cgroup-parent string Set parent cgroup for all containers @@ -43,8 +43,8 @@ Options: -D, --debug Enable debug mode --default-address-pool pool-options Default address pools for node specific local networks --default-cgroupns-mode string Default mode for containers cgroup namespace ("host" | "private") (default "private") - --default-gateway ip Container default gateway IPv4 address - --default-gateway-v6 ip Container default gateway IPv6 address + --default-gateway ip Default gateway IPv4 address for the default bridge network + --default-gateway-v6 ip Default gateway IPv6 address for the default bridge network --default-ipc-mode string Default mode for containers ipc ("shareable" | "private") (default "private") --default-network-opt mapmap Default network options (default map[]) --default-runtime string Default OCI runtime for containers (default "runc") @@ -57,8 +57,8 @@ Options: --exec-root string Root directory for execution state files (default "/var/run/docker") --experimental Enable experimental features --feature map Enable feature in the daemon - --fixed-cidr string IPv4 subnet for fixed IPs - --fixed-cidr-v6 string IPv6 subnet for fixed IPs + --fixed-cidr string IPv4 subnet for the default bridge network + --fixed-cidr-v6 string IPv6 subnet for the default bridge network -G, --group string Group for the unix socket (default "docker") --help Print usage -H, --host list Daemon socket(s) to connect to @@ -66,17 +66,17 @@ Options: Defaults to the IP addresses of the default bridge --http-proxy string HTTP proxy URL to use for outgoing traffic --https-proxy string HTTPS proxy URL to use for outgoing traffic - --icc Enable inter-container communication (default true) + --icc Enable inter-container communication for the default bridge network (default true) --init Run an init in the container to forward signals and reap processes --init-path string Path to the docker-init binary --insecure-registry list Enable insecure registry communication - --ip ip Default IP when binding container ports (default 0.0.0.0) + --ip ip Host IP for port publishing from the default bridge network (default 0.0.0.0) --ip-forward Enable IP forwarding in system configuration (default true) --ip-forward-no-drop Do not set the filter-FORWARD policy to DROP when enabling IP forwarding - --ip-masq Enable IP masquerading (default true) - --ip6tables Enable addition of ip6tables rules (experimental) + --ip-masq Enable IP masquerading for the default bridge network (default true) + --ip6tables Enable addition of ip6tables rules (default true) --iptables Enable addition of iptables rules (default true) - --ipv6 Enable IPv6 networking + --ipv6 Enable IPv6 networking for the default bridge network --label list Set key=value labels to the daemon --live-restore Enable live restore of docker when containers are still running --log-driver string Default driver for container logs (default "json-file") @@ -87,7 +87,7 @@ Options: --max-concurrent-uploads int Set the max concurrent uploads (default 5) --max-download-attempts int Set the max download attempts for each pull (default 5) --metrics-addr string Set default address and port to serve the metrics api on - --mtu int Set the containers network MTU (default 1500) + --mtu int Set the MTU for the default "bridge" network (default 1500) --network-control-plane-mtu int Network Control plane MTU (default 1500) --no-new-privileges Set no-new-privileges by default for new containers --no-proxy string Comma-separated list of hosts or IP addresses for which the proxy is skipped From 19e484218aa5fc50c91cdd21526e163ed643de8b Mon Sep 17 00:00:00 2001 From: aevesdocker Date: Tue, 8 Apr 2025 12:30:55 +0100 Subject: [PATCH 495/534] docs: replace sshfs with rclone Signed-off-by: aevesdocker Signed-off-by: Sebastiaan van Stijn --- docs/extend/_index.md | 64 ++++++++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 28 deletions(-) diff --git a/docs/extend/_index.md b/docs/extend/_index.md index 57528444d1..8c85b10485 100644 --- a/docs/extend/_index.md +++ b/docs/extend/_index.md @@ -35,31 +35,38 @@ Plugins that start successfully are listed as enabled in the output. After a plugin is installed, you can use it as an option for another Docker operation, such as creating a volume. -In the following example, you install the `sshfs` plugin, verify that it is +In the following example, you install the [`rclone` plugin](https://rclone.org/docker/), verify that it is enabled, and use it to create a volume. > [!NOTE] -> This example is intended for instructional purposes only. Once the volume is -> created, your SSH password to the remote host is exposed as plaintext when -> inspecting the volume. Delete the volume as soon as you are done with the -> example. +> This example is intended for instructional purposes only. -1. Install the `sshfs` plugin. +1. Set up the pre-requisite directories. By default they must exist on the host at the following locations: + + - `/var/lib/docker-plugins/rclone/config`. Reserved for the `rclone.conf` config file and must exist even if it's empty and the config file is not present. + - `/var/lib/docker-plugins/rclone/cache`. Holds the plugin state file as well as optional VFS caches. + +2. Install the `rclone` plugin. ```console - $ docker plugin install vieux/sshfs + $ docker plugin install rclone/docker-volume-rclone --alias rclone - Plugin "vieux/sshfs" is requesting the following privileges: - - network: [host] - - capabilities: [CAP_SYS_ADMIN] - Do you grant the above permissions? [y/N] y - - vieux/sshfs + Plugin "rclone/docker-volume-rclone" is requesting the following privileges: + - network: [host] + - mount: [/var/lib/docker-plugins/rclone/config] + - mount: [/var/lib/docker-plugins/rclone/cache] + - device: [/dev/fuse] + - capabilities: [CAP_SYS_ADMIN] + Do you grant the above permissions? [y/N] ``` - The plugin requests 2 privileges: + The plugin requests 5 privileges: - It needs access to the `host` network. + - Access to pre-requisite directories to mount to store: + - Your Rclone config files + - Temporary cache data + - Gives access to the FUSE (Filesystem in Userspace) device. This is required because Rclone uses FUSE to mount remote storage as if it were a local filesystem. - It needs the `CAP_SYS_ADMIN` capability, which allows the plugin to run the `mount` command. @@ -68,24 +75,25 @@ enabled, and use it to create a volume. ```console $ docker plugin ls - ID NAME TAG DESCRIPTION ENABLED - 69553ca1d789 vieux/sshfs latest the `sshfs` plugin true + ID NAME DESCRIPTION ENABLED + aede66158353 rclone:latest Rclone volume plugin for Docker true ``` 3. Create a volume using the plugin. This example mounts the `/remote` directory on host `1.2.3.4` into a - volume named `sshvolume`. + volume named `rclonevolume`. This volume can now be mounted into containers. ```console $ docker volume create \ - -d vieux/sshfs \ - --name sshvolume \ - -o sshcmd=user@1.2.3.4:/remote \ - -o password=$(cat file_containing_password_for_remote_host) - - sshvolume + -d rclone \ + --name rclonevolume \ + -o type=sftp \ + -o path=remote \ + -o sftp-host=1.2.3.4 \ + -o sftp-user=user \ + -o "sftp-password=$(cat file_containing_password_for_remote_host)" ``` 4. Verify that the volume was created successfully. @@ -94,21 +102,21 @@ enabled, and use it to create a volume. $ docker volume ls DRIVER NAME - vieux/sshfs sshvolume + rclone rclonevolume ``` -5. Start a container that uses the volume `sshvolume`. +5. Start a container that uses the volume `rclonevolume`. ```console - $ docker run --rm -v sshvolume:/data busybox ls /data + $ docker run --rm -v rclonevolume:/data busybox ls /data ``` -6. Remove the volume `sshvolume` +6. Remove the volume `rclonevolume` ```console - $ docker volume rm sshvolume + $ docker volume rm rclonevolume sshvolume ``` From 2b10a5c392f44f76219e857055ed6ab12b6811d1 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Thu, 1 May 2025 10:46:21 +0100 Subject: [PATCH 496/534] Add Linux daemon option --allow-direct-routing Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index b55b66c30b..f905f0a4cf 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -24,6 +24,7 @@ A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) + --allow-direct-routing Allow remote access to published ports on container IP addresses --authorization-plugin list Authorization plugins to load --bip string IPv4 address for the default bridge --bip6 string IPv6 address for the default bridge @@ -1057,6 +1058,7 @@ The following is a full example of the allowed configuration options on Linux: ```json { + "allow-direct-routing": false, "authorization-plugins": [], "bip": "", "bip6": "", From ace0c1429a8e3fd9713623730ec497a46af9e7b7 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 31 Mar 2025 17:45:54 +0200 Subject: [PATCH 497/534] search: remove deprecated "IsAutomated" placeholder IsAutomated was deprecated in e3a279723e68c0a55c0f74dca84b70072132837e (docker v25.0), and marked for removal in docker 26.0 (which we missed). This removes it. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 30fe60f0ea..ce4de636be 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -62,7 +62,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | -| Deprecated | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v26.0 | +| Deprecated | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [BuildKit build information](#buildkit-build-information) | v23.0 | v24.0 | @@ -359,7 +359,7 @@ introduced in v25.0 and should be used instead of the `Aliases` field. ### IsAutomated field, and `is-automated` filter on `docker search` **Deprecated in Release: v25.0** -**Target For Removal In Release: v26.0** +**Removed In Release: v28.2** The `is_automated` field has been deprecated by Docker Hub's search API. Consequently, the `IsAutomated` field in image search will always be set @@ -368,7 +368,7 @@ results. The `AUTOMATED` column has been removed from the default `docker search` and `docker image search` output in v25.0, and the corresponding `IsAutomated` -templating option will be removed in v26.0. +templating has been removed in v28.2. ### Logentries logging driver From 456f6c1f173eaeb8c94c728f17f5e5fc45bbca8a Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 00:34:13 +0200 Subject: [PATCH 498/534] docs: fix deprecation status for IsAutomated Follow-up to ace0c1429a8e3fd9713623730ec497a46af9e7b7, where I forgot to update the status column. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index ce4de636be..76116209e6 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -62,7 +62,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | -| Deprecated | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | +| Removed | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | | Removed | [BuildKit build information](#buildkit-build-information) | v23.0 | v24.0 | From 5b3778638788be4a6a4477b6aa3d0c98741676af Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 01:10:13 +0200 Subject: [PATCH 499/534] docs: move deprecation status of legacy schema1 images to "removed" Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index ce4de636be..5c18dd3eab 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -82,7 +82,7 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | | Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | v28.0 | | Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0 | -| Deprecated | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v27.0 | +| Removed | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v28.2 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | | Removed | [Top-level `docker deploy` subcommand (experimental)](#top-level-docker-deploy-subcommand-experimental) | v19.03 | v20.10 | | Removed | [`docker stack deploy` using "dab" files (experimental)](#docker-stack-deploy-using-dab-files-experimental) | v19.03 | v20.10 | @@ -737,7 +737,8 @@ for the old option will be removed in a future release. **Disabled by default in Release: v26.0** -**Target For Removal In Release: v27.0** +**Removed in Release: v28.2** + The image manifest [v2 schema 1](https://distribution.github.io/distribution/spec/deprecated-schema-v1/) and "Docker Image v1" formats were deprecated in favor of the @@ -748,23 +749,17 @@ formats. These legacy formats should no longer be used, and users are recommended to update images to use current formats, or to upgrade to more current images. Starting with Docker v26.0, pulling these images is disabled by default, and -produces an error when attempting to pull the image: +support has been removed in v28.2. Attempting to pull a legacy image now +produces an error: ```console $ docker pull ubuntu:10.04 Error response from daemon: -[DEPRECATION NOTICE] Docker Image Format v1 and Docker Image manifest version 2, schema 1 support is disabled by default and will be removed in an upcoming release. +Docker Image Format v1 and Docker Image manifest version 2, schema 1 support has been removed. Suggest the author of docker.io/library/ubuntu:10.04 to upgrade the image to the OCI Format or Docker Image manifest v2, schema 2. More information at https://docs.docker.com/go/deprecated-image-specs/ ``` -An environment variable (`DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE`) is -added in Docker v26.0 that allows re-enabling support for these image formats -in the daemon. This environment variable must be set to a non-empty value in -the daemon's environment (for example, through a [systemd override file](https://docs.docker.com/config/daemon/systemd/)). -Support for the `DOCKER_ENABLE_DEPRECATED_PULL_SCHEMA_1_IMAGE` environment variable -will be removed in Docker v27.0 after which this functionality is removed permanently. - ### `docker engine` subcommands **Deprecated in Release: v19.03** From b6298a2c214a6617ad7c0da67920e19c56a18798 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 11:26:57 +0200 Subject: [PATCH 500/534] docs: update deprecation status of graphdriver-plugins to "removed" This functionality, was removed and the DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS no longer can be used in v28.0; https://github.com/moby/moby/commit/42ca9154e9b8f8120011add4c09c6edcecd1e6f5 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 18 ++++++------------ 1 file changed, 6 insertions(+), 12 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 124939f0fe..7d6edd424f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -57,7 +57,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | | Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | | Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | -| Deprecated | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | +| Removed | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | | Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | @@ -210,19 +210,13 @@ part of the underlying image's `Config` field, and deprecated: **Target For Removal In Release: v28.0** [Graphdriver plugins](https://github.com/docker/cli/blob/v26.1.4/docs/extend/plugins_graphdriver.md) -are an experimental feature that allow extending the Docker Engine with custom +were an experimental feature that allowed extending the Docker Engine with custom storage drivers for storing images and containers. This feature was not -maintained since its inception, and will no longer be supported in upcoming -releases. +maintained since its inception. -Support for graphdriver plugins is disabled by default in v27.0, and will be -removed v28.0. An `DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS` environment variable -is provided in v27.0 to re-enable the feature. This environment variable must -be set to a non-empty value in the daemon's environment. - -The `DOCKERD_DEPRECATED_GRAPHDRIVER_PLUGINS` environment variable, along with -support for graphdriver plugins, will be removed in v28.0. Users of this feature -are recommended to instead configure the Docker Engine to use the [containerd image store](https://docs.docker.com/storage/containerd/) +Support for graphdriver plugins was disabled by default in v27.0, and removed +in v28.0. Users of this feature are recommended to instead configure the Docker +Engine to use the [containerd image store](https://docs.docker.com/storage/containerd/) and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18/docs/snapshotters) ### API CORS headers From e986150ae68587b7d90595c84319886ce708702d Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 11:45:57 +0200 Subject: [PATCH 501/534] docs: deprecated: set Container and ContainerConfig fields to "removed" These fields have been removed in v26.0 (API v1.45 and up), and are always omitted when using the containerd image store; https://github.com/moby/moby/commit/03cddc62f4bcd48fbc3a31dd94f2bf84e44840dd Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 124939f0fe..6009d68500 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -59,7 +59,7 @@ The following table provides an overview of the current status of deprecated fea | Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Deprecated | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | -| Deprecated | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | +| Removed | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Removed | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | @@ -276,15 +276,15 @@ configuring TLS (or SSH) for the Docker daemon, refer to ### `Container` and `ContainerConfig` fields in Image inspect **Deprecated in Release: v25.0** -**Target For Removal In Release: v26.0** +**Removed In Release: v26.0** The `Container` and `ContainerConfig` fields returned by `docker inspect` are mostly an implementation detail of the classic (non-BuildKit) image builder. These fields are not portable and are empty when using the BuildKit-based builder (enabled by default since v23.0). -These fields are deprecated in v25.0 and will be omitted starting from v26.0. -If image configuration of an image is needed, you can obtain it from the -`Config` field. +These fields are deprecated in v25.0 and are omitted starting from v26.0 ( +API version v1.45 and up). If image configuration of an image is needed, +you can obtain it from the `Config` field. ### Deprecate legacy API versions From 03f46014d03a16e739548eb67912ac962ee82fcc Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 13:28:18 +0200 Subject: [PATCH 502/534] docs: deprecated: mark `fluentd-async-connect` as "removed" The daemon still has migration code in place, but no longer accepts the option for new containers, so marking it as "removed"; https://github.com/moby/moby//commit/49ec488036d9702df8432ec0e6f33c170a4a2bbe Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 67e0aef42c..24a5ae4391 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -80,7 +80,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [CLI plugins support](#cli-plugins-support) | v20.10 | - | | Deprecated | [Dockerfile legacy `ENV name value` syntax](#dockerfile-legacy-env-name-value-syntax) | v20.10 | - | | Removed | [`docker build --stream` flag (experimental)](#docker-build---stream-flag-experimental) | v20.10 | v20.10 | -| Deprecated | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | v28.0 | +| Removed | [`fluentd-async-connect` log opt](#fluentd-async-connect-log-opt) | v20.10 | v28.0 | | Removed | [Configuration options for experimental CLI features](#configuration-options-for-experimental-cli-features) | v19.03 | v23.0 | | Removed | [Pushing and pulling with image manifest v2 schema 1](#pushing-and-pulling-with-image-manifest-v2-schema-1) | v19.03 | v28.2 | | Removed | [`docker engine` subcommands](#docker-engine-subcommands) | v19.03 | v20.10 | @@ -723,7 +723,7 @@ fluent#New: AsyncConnect is now deprecated, use Async instead ``` Users are encouraged to use the `fluentd-async` option going forward, as support -for the old option will be removed in a future release. +for the old option has been removed. ### Pushing and pulling with image manifest v2 schema 1 From e0fe2666a12985c2762f769dc1c5753d7d483dce Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Thu, 22 May 2025 13:37:44 +0200 Subject: [PATCH 503/534] docs: deprecated: mark API < v1.24 as "removed" Support for API versions lower than v1.24 was removed in v26.0. The DOCKER_MIN_API_VERSION environment-variable is still present in the docker daemon, but can currently only be used to raise the minimum version. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 67e0aef42c..abed4e2c5b 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -60,7 +60,7 @@ The following table provides an overview of the current status of deprecated fea | Removed | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | | Removed | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | -| Deprecated | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | +| Removed | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | | Removed | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | @@ -320,20 +320,22 @@ Error response from daemon: client version 1.23 is too old. Minimum supported AP upgrade your client to a newer version ``` +Support for API versions lower than `1.24` has been permanently removed in Docker +Engine v26, and the minimum supported API version will be incrementally raised +in releases following that. + + + ### Container short ID in network Aliases field From e92e21a5001ead368d909225fb1b37bdae822582 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 27 May 2025 16:14:00 +0200 Subject: [PATCH 504/534] docs: deprecated: update status for non-standard fields in image inspect Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 15 ++++++++------- 1 file changed, 8 insertions(+), 7 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 93ec094067..9a6324d32f 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -55,7 +55,7 @@ The following table provides an overview of the current status of deprecated fea |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| | Deprecated | [Configuration for pushing non-distributable artifacts](#configuration-for-pushing-non-distributable-artifacts) | v28.0 | v29.0 | | Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | -| Deprecated | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.0 | +| Removed | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.2 | | Removed | [API CORS headers](#api-cors-headers) | v27.0 | v28.0 | | Removed | [Graphdriver plugins (experimental)](#graphdriver-plugins-experimental) | v27.0 | v28.0 | | Deprecated | [Unauthenticated TCP connections](#unauthenticated-tcp-connections) | v26.0 | v28.0 | @@ -172,7 +172,7 @@ Users are encouraged to migrate to using the `--timeout` option instead. ### Non-standard fields in image inspect **Deprecated in Release: v27.0** -**Target For Removal In Release: v28.0** +**Removed In Release: v28.2** The `Config` field returned shown in `docker image inspect` (and as returned by the `GET /images/{name}/json` API endpoint) returns additional fields that are @@ -184,8 +184,9 @@ but are not omitted in the response when left empty. As these fields were not intended to be part of the image configuration response, they are deprecated, and will be removed from the API in thee next release. -The following fields are currently included in the API response, but are not -part of the underlying image's `Config` field, and deprecated: +The following fields are not part of the underlying image's `Config` field, and +removed in the API response for API v1.50 and newer, corresponding with v28.2. +They continue to be included when using clients that use an older API version: - `Hostname` - `Domainname` @@ -196,9 +197,9 @@ part of the underlying image's `Config` field, and deprecated: - `OpenStdin` - `StdinOnce` - `Image` -- `NetworkDisabled` (already omitted unless set) -- `MacAddress` (already omitted unless set) -- `StopTimeout` (already omitted unless set) +- `NetworkDisabled` (omitted unless set on older API versions) +- `MacAddress` (omitted unless set on older API versions) +- `StopTimeout` (omitted unless set on older API versions) [Docker image specification]: https://github.com/moby/docker-image-spec/blob/v1.3.1/specs-go/v1/image.go#L19-L32 [OCI image specification]: https://github.com/opencontainers/image-spec/blob/v1.1.0/specs-go/v1/config.go#L24-L62 From 8cb287fdefccaacf8e441ffde960c01e327955ea Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Wed, 28 May 2025 10:05:40 +0200 Subject: [PATCH 505/534] docs: deprecated: fallback for non-OCI-compliant registries is removed GitHub deprecated the legacy registry, and it was [sunset on Feb 24th, 2025][1] in favor of GitHub Container Registry (GHCR) (ghcr.io), so the fallback was removed. [1]: https://github.blog/changelog/2025-01-23-legacy-docker-registry-closing-down/ Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9a6324d32f..120e950d09 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -71,7 +71,7 @@ The following table provides an overview of the current status of deprecated fea | Removed | [Btrfs storage driver on CentOS 7 and RHEL 7](#btrfs-storage-driver-on-centos-7-and-rhel-7) | v20.10 | v23.0 | | Removed | [Support for encrypted TLS private keys](#support-for-encrypted-tls-private-keys) | v20.10 | v23.0 | | Removed | [Kubernetes stack and context support](#kubernetes-stack-and-context-support) | v20.10 | v23.0 | -| Deprecated | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | - | +| Removed | [Pulling images from non-compliant image registries](#pulling-images-from-non-compliant-image-registries) | v20.10 | v28.2 | | Removed | [Linux containers on Windows (LCOW)](#linux-containers-on-windows-lcow-experimental) | v20.10 | v23.0 | | Deprecated | [BLKIO weight options with cgroups v1](#blkio-weight-options-with-cgroups-v1) | v20.10 | - | | Removed | [Kernel memory limit](#kernel-memory-limit) | v20.10 | v23.0 | @@ -547,6 +547,7 @@ CLI configuration file are no longer used, and ignored. ### Pulling images from non-compliant image registries **Deprecated in Release: v20.10** +**Removed in Release: v28.2** Docker Engine v20.10 and up includes optimizations to verify if images in the local image cache need updating before pulling, preventing the Docker Engine @@ -556,7 +557,7 @@ image registry to conform to the [Open Container Initiative Distribution Specifi While most registries conform to the specification, we encountered some registries to be non-compliant, resulting in `docker pull` to fail. -As a temporary solution, Docker Engine v20.10 includes a fallback mechanism to +As a temporary solution, Docker Engine v20.10 added a fallback mechanism to allow `docker pull` to be functional when using a non-compliant registry. A warning message is printed in this situation: @@ -565,16 +566,13 @@ warning message is printed in this situation: pull by tag. This fallback is DEPRECATED, and will be removed in a future release. -The fallback is added to allow users to either migrate their images to a compliant -registry, or for these registries to become compliant. +The fallback was added to allow users to either migrate their images to a +compliant registry, or for these registries to become compliant. -Note that this fallback only addresses failures on `docker pull`. Other commands, -such as `docker stack deploy`, or pulling images with `containerd` will continue -to fail. - -Given that other functionality is still broken with these registries, we consider -this fallback a _temporary_ solution, and will remove the fallback in an upcoming -major release. +GitHub deprecated the legacy `docker.pkg.github.com` registry, and it was +[sunset on Feb 24th, 2025](https://github.blog/changelog/2025-01-23-legacy-docker-registry-closing-down/) +in favor of GitHub Container Registry (GHCR, ghcr.io), making this fallback +no longer needed. ### Linux containers on Windows (LCOW) (experimental) From 06060588b257fbed4f7023ead26a7dff2affe6c9 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Fri, 30 May 2025 18:20:53 +0200 Subject: [PATCH 506/534] docs: fix link to live-restore Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index f905f0a4cf..ce97427a19 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1302,7 +1302,7 @@ The list of currently supported options that can be reconfigured is this: | ---------------------------------- | ----------------------------------------------------------------------------------------------------------- | | `debug` | Toggles debug mode of the daemon. | | `labels` | Replaces the daemon labels with a new set of labels. | -| `live-restore` | Toggles [live restore](https://docs.docker.com/engine/containers/live-restore/). | +| `live-restore` | Toggles [live restore](https://docs.docker.com/engine/daemon/live-restore/). | | `max-concurrent-downloads` | Configures the max concurrent downloads for each pull. | | `max-concurrent-uploads` | Configures the max concurrent uploads for each push. | | `max-download-attempts` | Configures the max download attempts for each pull. | From e68b5c4ab0274bcba8f149410aa4a16de231a1ce Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Fri, 13 Jun 2025 13:27:41 +0200 Subject: [PATCH 507/534] docs: deprecate empty Config fields in image inspect API MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Image config fields like Cmd, Entrypoint, Env, etc. will be omitted from /images/{name}/json response when empty, starting in v29.0. Signed-off-by: Paweł Gronowski --- docs/deprecated.md | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 120e950d09..1a3c353da6 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Empty/nil fields in image Config from inspect API](#emptynil-fields-in-image-config-from-inspect-api) | v28.3 | v29.0 | | Deprecated | [Configuration for pushing non-distributable artifacts](#configuration-for-pushing-non-distributable-artifacts) | v28.0 | v29.0 | | Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | | Removed | [Non-standard fields in image inspect](#non-standard-fields-in-image-inspect) | v27.0 | v28.2 | @@ -120,7 +121,34 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | -## Configuration for pushing non-distributable artifacts +### Empty/nil fields in image Config from inspect API + +**Deprecated in Release: v28.3** +**Target For Removal In Release: v29.0** + +The `Config` field returned by `docker image inspect` (and the `GET /images/{name}/json` +API endpoint) currently includes certain fields even when they are empty or nil. +Starting in Docker v29.0, the following fields will be omitted from the API response +when they contain empty or default values: + +- `Cmd` +- `Entrypoint` +- `Env` +- `Labels` +- `OnBuild` +- `User` +- `Volumes` +- `WorkingDir` + +Applications consuming the image inspect API should be updated to handle the +absence of these fields gracefully, treating missing fields as having their +default/empty values. + +For API version corresponding to Docker v29.0, these fields will be omitted when +empty. They will continue to be included when using clients that request an older +API version for backward compatibility. + +### Configuration for pushing non-distributable artifacts **Deprecated in Release: v28.0** **Target For Removal In Release: v29.0** From c58786f00731011caff88260f5f94050246182c3 Mon Sep 17 00:00:00 2001 From: ArthurFlag Date: Thu, 3 Jul 2025 09:25:39 +0200 Subject: [PATCH 508/534] docs: cdi is not experimental anymore Signed-off-by: ArthurFlag --- docs/reference/dockerd.md | 37 ++++++++++++++++++++++--------------- 1 file changed, 22 insertions(+), 15 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index ce97427a19..1e2b01633c 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -840,42 +840,49 @@ $ docker run -it --add-host host.docker.internal:host-gateway \ PING host.docker.internal (2001:db8::1111): 56 data bytes ``` -### Enable CDI devices - -> [!NOTE] -> This is experimental feature and as such doesn't represent a stable API. -> -> This feature isn't enabled by default. To this feature, set `features.cdi` to -> `true` in the `daemon.json` configuration file. +### Configure CDI devices Container Device Interface (CDI) is a [standardized](https://github.com/cncf-tags/container-device-interface/blob/main/SPEC.md) mechanism for container runtimes to create containers which are able to interact with third party devices. +CDI is currently only supported for Linux containers and is enabled by default +since Docker Engine 28.3.0. + The Docker daemon supports running containers with CDI devices if the requested device specifications are available on the filesystem of the daemon. -The default specification directors are: +The default specification directories are: - `/etc/cdi/` for static CDI Specs - `/var/run/cdi` for generated CDI Specs -Alternatively, you can set custom locations for CDI specifications using the +#### Set custom locations + +To set custom locations for CDI specifications, use the `cdi-spec-dirs` option in the `daemon.json` configuration file, or the -`--cdi-spec-dir` flag for the `dockerd` CLI. +`--cdi-spec-dir` flag for the `dockerd` CLI: ```json { - "features": { - "cdi": true - }, "cdi-spec-dirs": ["/etc/cdi/", "/var/run/cdi"] } ``` -When CDI is enabled for a daemon, you can view the configured CDI specification -directories using the `docker info` command. +You can view the configured CDI specification directories using the `docker info` command. + +#### Disable CDI devices + +The feature in enabled by default. To disable it, use the `cdi` options in the `deamon.json` file: + +```json +"features": { + "cdi": false +}, +``` + +To check the status of the CDI devices, run `docker info`. #### Daemon logging format {#log-format} From 142973f592093e11c640f972a27f70bbb86f5610 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Wed, 23 Jul 2025 11:13:20 +0100 Subject: [PATCH 509/534] dockerd.md: add --bridge-accept-fwmark Related to https://github.com/moby/moby/commit/cf1695bef13b4eeef71d106a979825aaaed8a79f Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 1e2b01633c..10bcc2da43 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -29,6 +29,7 @@ Options: --bip string IPv4 address for the default bridge --bip6 string IPv6 address for the default bridge -b, --bridge string Attach containers to a network bridge + --bridge-accept-fwmark string In bridge networks, accept packets with this firewall mark/mask --cdi-spec-dir list CDI specification directories to use --cgroup-parent string Set parent cgroup for all containers --config-file string Daemon configuration file (default "/etc/docker/daemon.json") @@ -1070,6 +1071,7 @@ The following is a full example of the allowed configuration options on Linux: "bip": "", "bip6": "", "bridge": "", + "bridge-accept-fwmark": "", "builder": { "gc": { "enabled": true, From 18fdd4a8329ec2f8787c32b00c35112147877224 Mon Sep 17 00:00:00 2001 From: Rob Murray Date: Wed, 23 Jul 2025 11:26:39 +0100 Subject: [PATCH 510/534] dockerd.md: Add --firewall-backend Related to https://github.com/moby/moby/commit/39ab39327417e1feebbc48a98748579ff8872e45 Signed-off-by: Rob Murray --- docs/reference/dockerd.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 10bcc2da43..3af8457251 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -59,6 +59,7 @@ Options: --exec-root string Root directory for execution state files (default "/var/run/docker") --experimental Enable experimental features --feature map Enable feature in the daemon + --firewall-backend string Firewall backend to use, iptables or nftables --fixed-cidr string IPv4 subnet for the default bridge network --fixed-cidr-v6 string IPv6 subnet for the default bridge network -G, --group string Group for the unix socket (default "docker") @@ -1122,6 +1123,7 @@ The following is a full example of the allowed configuration options on Linux: "cdi": true, "containerd-snapshotter": true }, + "firewall-backend": "", "fixed-cidr": "", "fixed-cidr-v6": "", "group": "", From 06ea4fa2d804d9439b654cc688d4827197d333e1 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 5 Aug 2025 23:23:38 +0200 Subject: [PATCH 511/534] remove deprecated `bind-nonrecursive` option for `--mount` The `bind-nonrecursive` option was replaced with the [`bind-recursive`] option (see [cli-4316], [cli-4671]). The option was still accepted, but printed a deprecation warning: bind-nonrecursive is deprecated, use bind-recursive=disabled instead In the v29.0 release, this warning is removed, and returned as an error. Users should use the equivalent `bind-recursive=disabled` option instead. [`bind-recursive`]: https://docs.docker.com/engine/storage/bind-mounts/#recursive-mounts [cli-4316]: https://github.com/docker/cli/pull/4316 [cli-4671]: https://github.com/docker/cli/pull/4671 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 1a3c353da6..803f9a63b6 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -63,6 +63,7 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`Container` and `ContainerConfig` fields in Image inspect](#container-and-containerconfig-fields-in-image-inspect) | v25.0 | v26.0 | | Removed | [Deprecate legacy API versions](#deprecate-legacy-api-versions) | v25.0 | v26.0 | | Removed | [Container short ID in network Aliases field](#container-short-id-in-network-aliases-field) | v25.0 | v26.0 | +| Removed | [Mount `bind-nonrecursive` option](#mount-bind-nonrecursive-option) | v25.0 | v29.0 | | Removed | [IsAutomated field, and `is-automated` filter on `docker search`](#isautomated-field-and-is-automated-filter-on-docker-search) | v25.0 | v28.2 | | Removed | [logentries logging driver](#logentries-logging-driver) | v24.0 | v25.0 | | Removed | [OOM-score adjust for the daemon](#oom-score-adjust-for-the-daemon) | v24.0 | v25.0 | @@ -381,6 +382,26 @@ A new field `DNSNames` containing the container name (if one was specified), the hostname, the network aliases, as well as the container short ID, has been introduced in v25.0 and should be used instead of the `Aliases` field. +### Mount `bind-nonrecursive` option + +**Deprecated in Release: v25.0** +**Removed In Release: v29.0** + +The `bind-nonrecursive` option was replaced with the [`bind-recursive`] +option (see [cli-4316], [cli-4671]). The option was still accepted, but +printed a deprecation warning: + +```console +bind-nonrecursive is deprecated, use bind-recursive=disabled instead +``` + +In the v29.0 release, this warning is removed, and returned as an error. +Users should use the equivalent `bind-recursive=disabled` option instead. + +[`bind-recursive`]: https://docs.docker.com/engine/storage/bind-mounts/#recursive-mounts +[cli-4316]: https://github.com/docker/cli/pull/4316 +[cli-4671]: https://github.com/docker/cli/pull/4671 + ### IsAutomated field, and `is-automated` filter on `docker search` **Deprecated in Release: v25.0** From 9ef6c8c1d0b15f474ba934ddc61793b9caa4b6ff Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 18 Aug 2025 17:59:16 +0200 Subject: [PATCH 512/534] Deprecate special handling for quoted values for TLS flags The `--tlscacert`, `--tlscert`, and `--tlskey` command-line flags had non-standard behavior for handling values contained in quotes (`"` or `'`). Normally, quotes are handled by the shell, for example, in the following example, the shell takes care of handling quotes before passing the values to the `docker` CLI: docker --some-option "some-value-in-quotes" ... However, when passing values using an equal sign (`=`), this may not happen and values may be handled including quotes; docker --some-option="some-value-in-quotes" ... This caused issues with "Docker Machine", which used this format as part of its `docker-machine config` output, and the CLI carried special, non-standard handling for these flags. Docker Machine reached EOL, and this special handling made the processing of flag values inconsistent with other flags used, so this behavior is deprecated. Users depending on this behavior are recommended to specify the quoted values using a space between the flag and its value, as illustrated above. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 803f9a63b6..b5a82804d4 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Special handling for quoted values for TLS flags](#special-handling-for-quoted-values-for-tls-flags) | v28.4 | v29.0 | | Deprecated | [Empty/nil fields in image Config from inspect API](#emptynil-fields-in-image-config-from-inspect-api) | v28.3 | v29.0 | | Deprecated | [Configuration for pushing non-distributable artifacts](#configuration-for-pushing-non-distributable-artifacts) | v28.0 | v29.0 | | Deprecated | [`--time` option on `docker stop` and `docker restart`](#--time-option-on-docker-stop-and-docker-restart) | v28.0 | - | @@ -122,6 +123,39 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | + +### Special handling for quoted values for TLS flags + +**Deprecated in Release: v28.4** +**Target For Removal In Release: v29.0** + +The `--tlscacert`, `--tlscert`, and `--tlskey` command-line flags had +non-standard behavior for handling values contained in quotes (`"` or `'`). +Normally, quotes are handled by the shell, for example, in the following +example, the shell takes care of handling quotes before passing the values +to the `docker` CLI: + +```console +docker --some-option "some-value-in-quotes" ... +``` + +However, when passing values using an equal sign (`=`), this may not happen +and values may be handled including quotes; + +```console +docker --some-option="some-value-in-quotes" ... +``` + +This caused issues with "Docker Machine", which used this format as part +of its `docker-machine config` output, and the CLI carried special, non-standard +handling for these flags. + +Docker Machine reached EOL, and this special handling made the processing +of flag values inconsistent with other flags used, so this behavior is +deprecated. Users depending on this behavior are recommended to specify +the quoted values using a space between the flag and its value, as illustrated +above. + ### Empty/nil fields in image Config from inspect API **Deprecated in Release: v28.3** From 39ed8e24a5a93e02fde0787f1c3c0690ebd634b8 Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Thu, 14 Aug 2025 15:33:53 +0200 Subject: [PATCH 513/534] docs/deprecated: legacy links env vars Signed-off-by: Albin Kerouanton Co-authored-by: Sebastiaan van Stijn Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index b5a82804d4..9e0ee14047 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Legacy links environment variables](#legacy-links-environment-variables) | v28.4 | v30.0 | | Deprecated | [Special handling for quoted values for TLS flags](#special-handling-for-quoted-values-for-tls-flags) | v28.4 | v29.0 | | Deprecated | [Empty/nil fields in image Config from inspect API](#emptynil-fields-in-image-config-from-inspect-api) | v28.3 | v29.0 | | Deprecated | [Configuration for pushing non-distributable artifacts](#configuration-for-pushing-non-distributable-artifacts) | v28.0 | v29.0 | @@ -123,6 +124,25 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Legacy links environment variables + +**Deprecated in Release: v28.4** +**Disabled by default in Release: v29.0** +**Target For Removal In Release: v30.0** + +Containers attached to the default bridge network can specify "legacy links" (e.g. +using `--links` on the CLI) to get access to other containers attached to that +network. The linking container (i.e., the container created with `--links`) automatically +gets environment variables that specify the IP address and port mappings of the linked +container. However, these environment variables are prefixed with the linked +container's names, making them impractical. + +Starting with Docker v29.0, these environment variables are no longer set by +default. Users who still depend on them can start Docker Engine with the +environment variable `DOCKER_KEEP_DEPRECATED_LEGACY_LINKS_ENV_VARS=1` set. + +Support for legacy links environment variables, as well as the `DOCKER_KEEP_DEPRECATED_LEGACY_LINKS_ENV_VARS` +will be removed in Docker Engine v30.0. ### Special handling for quoted values for TLS flags From 91d99d18d07e2c19c97ab385f6af754fac8e9252 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 19 Aug 2025 14:26:05 +0200 Subject: [PATCH 514/534] docs: deprecated: fix formatting of deprecated/removed in - Use sentence-case to follow our docs guidelines. - Add newlines to prevent these being rendered on a single line. Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 302 +++++++++++++++++++++++++-------------------- 1 file changed, 167 insertions(+), 135 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index 9e0ee14047..d2a76a4c14 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -126,9 +126,11 @@ The following table provides an overview of the current status of deprecated fea ### Legacy links environment variables -**Deprecated in Release: v28.4** -**Disabled by default in Release: v29.0** -**Target For Removal In Release: v30.0** +**Deprecated in release: v28.4** + +**Disabled by default in release: v29.0** + +**Target for removal in release: v30.0** Containers attached to the default bridge network can specify "legacy links" (e.g. using `--links` on the CLI) to get access to other containers attached to that @@ -146,8 +148,9 @@ will be removed in Docker Engine v30.0. ### Special handling for quoted values for TLS flags -**Deprecated in Release: v28.4** -**Target For Removal In Release: v29.0** +**Deprecated in release: v28.4** + +**Target for removal in release: v29.0** The `--tlscacert`, `--tlscert`, and `--tlskey` command-line flags had non-standard behavior for handling values contained in quotes (`"` or `'`). @@ -178,8 +181,9 @@ above. ### Empty/nil fields in image Config from inspect API -**Deprecated in Release: v28.3** -**Target For Removal In Release: v29.0** +**Deprecated in release: v28.3** + +**Target for removal in release: v29.0** The `Config` field returned by `docker image inspect` (and the `GET /images/{name}/json` API endpoint) currently includes certain fields even when they are empty or nil. @@ -205,8 +209,9 @@ API version for backward compatibility. ### Configuration for pushing non-distributable artifacts -**Deprecated in Release: v28.0** -**Target For Removal In Release: v29.0** +**Deprecated in release: v28.0** + +**Target for removal in release: v29.0** Non-distributable artifacts (also called foreign layers) were introduced in docker v1.12 to accommodate Windows images for which the EULA did not allow @@ -244,7 +249,7 @@ entirely. ### `--time` option on `docker stop` and `docker restart` -**Deprecated in Release: v28.0** +**Deprecated in release: v28.0** The `--time` option for the `docker stop`, `docker container stop`, `docker restart`, and `docker container restart` commands has been renamed to `--timeout` for @@ -254,8 +259,9 @@ Users are encouraged to migrate to using the `--timeout` option instead. ### Non-standard fields in image inspect -**Deprecated in Release: v27.0** -**Removed In Release: v28.2** +**Deprecated in release: v27.0** + +**Removed in release: v28.2** The `Config` field returned shown in `docker image inspect` (and as returned by the `GET /images/{name}/json` API endpoint) returns additional fields that are @@ -289,9 +295,11 @@ They continue to be included when using clients that use an older API version: ### Graphdriver plugins (experimental) -**Deprecated in Release: v27.0** -**Disabled by default in Release: v27.0** -**Target For Removal In Release: v28.0** +**Deprecated in**: v27.0**. + +**Disabled by default in release: v27.0** + +**Target for removal in release: v28.0** [Graphdriver plugins](https://github.com/docker/cli/blob/v26.1.4/docs/extend/plugins_graphdriver.md) were an experimental feature that allowed extending the Docker Engine with custom @@ -305,8 +313,10 @@ and a custom [snapshotter](https://github.com/containerd/containerd/tree/v1.7.18 ### API CORS headers -**Deprecated in Release: v27.0** -**Disabled by default in Release: v27.0** +**Deprecated in release: v27.0** + +**Disabled by default in release: v27.0** + **Removed in release: v28.0** The `api-cors-header` configuration option for the Docker daemon is insecure, @@ -326,8 +336,9 @@ If you need to access the API through a browser, use a reverse proxy. ### Unauthenticated TCP connections -**Deprecated in Release: v26.0** -**Target For Removal In Release: v28.0** +**Deprecated in release: v26.0** + +**Target for removal in release: v28.0** Configuring the Docker daemon to listen on a TCP address will require mandatory TLS verification. This change aims to ensure secure communication by preventing @@ -353,8 +364,9 @@ configuring TLS (or SSH) for the Docker daemon, refer to ### `Container` and `ContainerConfig` fields in Image inspect -**Deprecated in Release: v25.0** -**Removed In Release: v26.0** +**Deprecated in release: v25.0** + +**Removed in release: v26.0** The `Container` and `ContainerConfig` fields returned by `docker inspect` are mostly an implementation detail of the classic (non-BuildKit) image builder. @@ -366,8 +378,9 @@ you can obtain it from the `Config` field. ### Deprecate legacy API versions -**Deprecated in Release: v25.0** -**Target For Removal In Release: v26.0** +**Deprecated in release: v25.0** + +**Target for removal in release: v26.0** The Docker daemon provides a versioned API for backward compatibility with old clients. Docker clients can perform API-version negotiation to select the most @@ -423,8 +436,9 @@ old clients, and those clients must be supported. ### Container short ID in network Aliases field -**Deprecated in Release: v25.0** -**Removed In Release: v26.0** +**Deprecated in release: v25.0** + +**Removed in release: v26.0** The `Aliases` field returned by `docker inspect` contains the container short ID once the container is started. This behavior is deprecated in v25.0 but @@ -438,8 +452,9 @@ introduced in v25.0 and should be used instead of the `Aliases` field. ### Mount `bind-nonrecursive` option -**Deprecated in Release: v25.0** -**Removed In Release: v29.0** +**Deprecated in release: v25.0** + +**Removed in release: v29.0** The `bind-nonrecursive` option was replaced with the [`bind-recursive`] option (see [cli-4316], [cli-4671]). The option was still accepted, but @@ -458,8 +473,9 @@ Users should use the equivalent `bind-recursive=disabled` option instead. ### IsAutomated field, and `is-automated` filter on `docker search` -**Deprecated in Release: v25.0** -**Removed In Release: v28.2** +**Deprecated in release: v25.0** + +**Removed in release: v28.2** The `is_automated` field has been deprecated by Docker Hub's search API. Consequently, the `IsAutomated` field in image search will always be set @@ -472,8 +488,9 @@ templating has been removed in v28.2. ### Logentries logging driver -**Deprecated in Release: v24.0** -**Removed in Release: v25.0** +**Deprecated in release: v24.0** + +**Removed in release: v25.0** The logentries service SaaS was shut down on November 15, 2022, rendering this logging driver non-functional. Users should no longer use this logging @@ -483,8 +500,9 @@ after upgrading. ### OOM-score adjust for the daemon -**Deprecated in Release: v24.0** -**Removed in Release: v25.0** +**Deprecated in release: v24.0** + +**Removed in release: v25.0** The `oom-score-adjust` option was added to prevent the daemon from being OOM-killed before other processes. This option was mostly added as a @@ -503,8 +521,9 @@ the daemon. ### BuildKit build information -**Deprecated in Release: v23.0** -**Removed in Release: v24.0** +**Deprecated in release: v23.0** + +**Removed in release: v24.0** [Build information](https://github.com/moby/buildkit/blob/v0.11/docs/buildinfo.md) structures have been introduced in [BuildKit v0.10.0](https://github.com/moby/buildkit/releases/tag/v0.10.0) @@ -515,7 +534,7 @@ information is also embedded into the image configuration if one is generated. ### Legacy builder for Linux images -**Deprecated in Release: v23.0** +**Deprecated in release: v23.0** Docker v23.0 now uses BuildKit by default to build Linux images, and uses the [Buildx](https://docs.docker.com/buildx/working-with-buildx/) CLI component for @@ -546,7 +565,7 @@ you to report issues in the [BuildKit issue tracker on GitHub](https://github.co ### Legacy builder fallback -**Deprecated in Release: v23.0** +**Deprecated in release: v23.0** [Docker v23.0 now uses BuildKit by default to build Linux images](#legacy-builder-for-linux-images), which requires the Buildx component to build images with BuildKit. There may be @@ -592,7 +611,7 @@ be possible in a future release. ### Btrfs storage driver on CentOS 7 and RHEL 7 -**Removed in Release: v23.0** +**Removed in release: v23.0** The `btrfs` storage driver on CentOS and RHEL was provided as a technology preview by CentOS and RHEL, but has been deprecated since the [Red Hat Enterprise Linux 7.4 release](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/ch-btrfs), @@ -604,9 +623,9 @@ of Docker will no longer provide this driver. ### Support for encrypted TLS private keys -**Deprecated in Release: v20.10** +**Deprecated in release: v20.10** -**Removed in Release: v23.0** +**Removed in release: v23.0** Use of encrypted TLS private keys has been deprecated, and has been removed. Golang has deprecated support for legacy PEM encryption (as specified in @@ -620,8 +639,9 @@ to decrypt the private key, and store it un-encrypted to continue using it. ### Kubernetes stack and context support -**Deprecated in Release: v20.10** -**Removed in Release: v23.0** +**Deprecated in release: v20.10** + +**Removed in release: v23.0** Following the deprecation of [Compose on Kubernetes](https://github.com/docker/compose-on-kubernetes), support for Kubernetes in the `stack` and `context` commands has been removed from @@ -649,8 +669,9 @@ CLI configuration file are no longer used, and ignored. ### Pulling images from non-compliant image registries -**Deprecated in Release: v20.10** -**Removed in Release: v28.2** +**Deprecated in release: v20.10** + +**Removed in release: v28.2** Docker Engine v20.10 and up includes optimizations to verify if images in the local image cache need updating before pulling, preventing the Docker Engine @@ -679,8 +700,9 @@ no longer needed. ### Linux containers on Windows (LCOW) (experimental) -**Deprecated in Release: v20.10** -**Removed in Release: v23.0** +**Deprecated in release: v20.10** + +**Removed in release: v23.0** The experimental feature to run Linux containers on Windows (LCOW) was introduced as a technical preview in Docker 17.09. While many enhancements were made after @@ -692,7 +714,7 @@ Developers who want to run Linux workloads on a Windows host are encouraged to u ### BLKIO weight options with cgroups v1 -**Deprecated in Release: v20.10** +**Deprecated in release: v20.10** Specifying blkio weight (`docker run --blkio-weight` and `docker run --blkio-weight-device`) is now marked as deprecated when using cgroups v1 because the corresponding features @@ -702,8 +724,9 @@ When using cgroups v2, the `--blkio-weight` options are implemented using ### Kernel memory limit -**Deprecated in Release: v20.10** -**Removed in Release: v23.0** +**Deprecated in release: v20.10** + +**Removed in release: v23.0** Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). @@ -729,8 +752,9 @@ take no effect. ### Classic Swarm and overlay networks using cluster store -**Deprecated in Release: v20.10** -**Removed in Release: v23.0** +**Deprecated in release: v20.10** + +**Removed in release: v23.0** Standalone ("classic") Swarm has been deprecated, and with that the use of overlay networks using an external key/value store. The corresponding`--cluster-advertise`, @@ -738,8 +762,9 @@ networks using an external key/value store. The corresponding`--cluster-advertis ### Support for legacy `~/.dockercfg` configuration files -**Deprecated in Release: v20.10** -**Removed in Release: v23.0** +**Deprecated in release: v20.10** + +**Removed in release: v23.0** The Docker CLI up until v1.7.0 used the `~/.dockercfg` file to store credentials after authenticating to a registry (`docker login`). Docker v1.7.0 replaced this @@ -754,9 +779,9 @@ been removed. ### Configuration options for experimental CLI features -**Deprecated in Release: v19.03** +**Deprecated in release: v19.03** -**Removed in Release: v23.0** +**Removed in release: v23.0** The `DOCKER_CLI_EXPERIMENTAL` environment variable and the corresponding `experimental` field in the CLI configuration file are deprecated. Experimental features are @@ -768,13 +793,13 @@ format. ### CLI plugins support -**Deprecated in Release: v20.10** +**Deprecated in release: v20.10** CLI Plugin API is now marked as deprecated. ### Dockerfile legacy `ENV name value` syntax -**Deprecated in Release: v20.10** +**Deprecated in release: v20.10** The Dockerfile `ENV` instruction allows values to be set using either `ENV name=value` or `ENV name value`. The latter (`ENV name value`) form can be ambiguous, for example, @@ -798,8 +823,9 @@ ENV ONE="" TWO="" THREE="world" ### `docker build --stream` flag (experimental) -**Deprecated in Release: v20.10** -**Removed in Release: v20.10** +**Deprecated in release: v20.10** + +**Removed in release: v20.10** Docker v17.07 introduced an experimental `--stream` flag on `docker build` which allowed the build-context to be incrementally sent to the daemon, instead of @@ -815,8 +841,9 @@ files. ### `fluentd-async-connect` log opt -**Deprecated in Release: v20.10** -**Removed in Release: v28.0** +**Deprecated in release: v20.10** + +**Removed in release: v28.0** The `--log-opt fluentd-async-connect` option for the fluentd logging driver is [deprecated in favor of `--log-opt fluentd-async`](https://github.com/moby/moby/pull/39086). @@ -831,11 +858,11 @@ for the old option has been removed. ### Pushing and pulling with image manifest v2 schema 1 -**Deprecated in Release: v19.03** +**Deprecated in release: v19.03** -**Disabled by default in Release: v26.0** +**Disabled by default in release: v26.0** -**Removed in Release: v28.2** +**Removed in release: v28.2** The image manifest [v2 schema 1](https://distribution.github.io/distribution/spec/deprecated-schema-v1/) @@ -860,9 +887,9 @@ More information at https://docs.docker.com/go/deprecated-image-specs/ ### `docker engine` subcommands -**Deprecated in Release: v19.03** +**Deprecated in release: v19.03** -**Removed in Release: v20.10** +**Removed in release: v20.10** The `docker engine activate`, `docker engine check`, and `docker engine update` provided an alternative installation method to upgrade Docker Community engines @@ -875,9 +902,9 @@ standard package managers. ### Top-level `docker deploy` subcommand (experimental) -**Deprecated in Release: v19.03** +**Deprecated in release: v19.03** -**Removed in Release: v20.10** +**Removed in release: v20.10** The top-level `docker deploy` command (using the "Docker Application Bundle" (.dab) file format was introduced as an experimental feature in Docker 1.13 / @@ -886,9 +913,9 @@ subcommand. ### `docker stack deploy` using "dab" files (experimental) -**Deprecated in Release: v19.03** +**Deprecated in release: v19.03** -**Removed in Release: v20.10** +**Removed in release: v20.10** With no development being done on this feature, and no active use of the file format, support for the DAB file format and the top-level `docker deploy` command @@ -897,8 +924,9 @@ using compose files. ### Support for the `overlay2.override_kernel_check` storage option -**Deprecated in Release: v19.03** -**Removed in Release: v24.0** +**Deprecated in release: v19.03** + +**Removed in release: v24.0** This daemon configuration option disabled the Linux kernel version check used to detect if the kernel supported OverlayFS with multiple lower dirs, which is @@ -908,8 +936,9 @@ option was no longer used. ### AuFS storage driver -**Deprecated in Release: v19.03** -**Removed in Release: v24.0** +**Deprecated in release: v19.03** + +**Removed in release: v24.0** The `aufs` storage driver is deprecated in favor of `overlay2`, and has been removed in a Docker Engine v24.0. Users of the `aufs` storage driver must @@ -927,8 +956,9 @@ maintenance of the `aufs` storage driver. ### Legacy overlay storage driver -**Deprecated in Release: v18.09** -**Removed in Release: v24.0** +**Deprecated in release: v18.09** + +**Removed in release: v24.0** The `overlay` storage driver is deprecated in favor of the `overlay2` storage driver, which has all the benefits of `overlay`, without its limitations (excessive @@ -943,9 +973,11 @@ backported), there is no reason to keep maintaining the `overlay` storage driver ### Device mapper storage driver -**Deprecated in Release: v18.09** -**Disabled by default in Release: v23.0** -**Removed in Release: v25.0** +**Deprecated in release: v18.09** + +**Disabled by default in release: v23.0** + +**Removed in release: v25.0** The `devicemapper` storage driver is deprecated in favor of `overlay2`, and has been removed in Docker Engine v25.0. Users of the `devicemapper` storage driver @@ -961,9 +993,9 @@ is no reason to continue maintenance of the `devicemapper` storage driver. ### Use of reserved namespaces in engine labels -**Deprecated in Release: v18.06** +**Deprecated in release: v18.06** -**Removed In Release: v20.10** +**Removed in release: v20.10** The namespaces `com.docker.*`, `io.docker.*`, and `org.dockerproject.*` in engine labels were always documented to be reserved, but there was never any enforcement. @@ -975,7 +1007,7 @@ use, and will error instead in v20.10 and above. **Disabled In Release: v17.12** -**Removed In Release: v19.03** +**Removed in release: v19.03** The `--disable-legacy-registry` flag was disabled in Docker 17.12 and will print an error when used. For this error to be printed, the flag itself is still present, @@ -983,9 +1015,9 @@ but hidden. The flag has been removed in Docker 19.03. ### Interacting with V1 registries -**Disabled By Default In Release: v17.06** +**Disabled by default in release: v17.06** -**Removed In Release: v17.12** +**Removed in release: v17.12** Version 1.8.3 added a flag (`--disable-legacy-registry=false`) which prevents the Docker daemon from `pull`, `push`, and `login` operations against v1 @@ -1002,7 +1034,7 @@ start when set. ### Asynchronous `service create` and `service update` as default -**Deprecated In Release: v17.05** +**Deprecated in release: v17.05** **Disabled by default in release: [v17.10](https://github.com/docker/docker-ce/releases/tag/v17.10.0-ce)** @@ -1016,9 +1048,9 @@ and `docker service scale` in Docker 17.10. ### `-g` and `--graph` flags on `dockerd` -**Deprecated In Release: v17.05** +**Deprecated in release: v17.05** -**Removed In Release: v23.0** +**Removed in release: v23.0** The `-g` or `--graph` flag for the `dockerd` or `docker daemon` command was used to indicate the directory in which to store persistent data and resource @@ -1027,9 +1059,9 @@ flag. These flags were deprecated and hidden in v17.05, and removed in v23.0. ### Top-level network properties in NetworkSettings -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Target for removal in release: v17.12** When inspecting a container, `NetworkSettings` contains top-level information about the default ("bridge") network; @@ -1046,18 +1078,18 @@ information. ### `filter` option for `/images/json` endpoint -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v20.10** +**Removed in release: v20.10** The `filter` option to filter the list of image by reference (name or name:tag) is now implemented as a regular filter, named `reference`. ### `repository:shortid` image references -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v17.12** +**Removed in release: v17.12** The `repository:shortid` syntax for referencing images is very little used, collides with tag references, and can be confused with digest references. @@ -1067,32 +1099,32 @@ in Docker 17.12. ### `docker daemon` subcommand -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v17.12** +**Removed in release: v17.12** The daemon is moved to a separate binary (`dockerd`), and should be used instead. ### Duplicate keys with conflicting values in engine labels -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v17.12** +**Removed in release: v17.12** When setting duplicate keys with conflicting values, an error will be produced, and the daemon will fail to start. ### `MAINTAINER` in Dockerfile -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** `MAINTAINER` was an early very limited form of `LABEL` which should be used instead. ### API calls without a version -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Target For Removal In Release: v17.12** +**Target for removal in release: v17.12** API versions should be supplied to all API calls to ensure compatibility with future Engine versions. Instead of just requesting, for example, the URL @@ -1100,9 +1132,9 @@ future Engine versions. Instead of just requesting, for example, the URL ### Backing filesystem without `d_type` support for overlay/overlay2 -**Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Deprecated in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** -**Removed In Release: v17.12** +**Removed in release: v17.12** The overlay and overlay2 storage driver does not work as expected if the backing filesystem does not support `d_type`. For example, XFS does not support `d_type` @@ -1116,18 +1148,18 @@ Refer to [#27358](https://github.com/docker/docker/issues/27358) for details. ### `--automated` and `--stars` flags on `docker search` -**Deprecated in Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Deprecated in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Removed In Release: v20.10** +**Removed in release: v20.10** The `docker search --automated` and `docker search --stars` options are deprecated. Use `docker search --filter=is-automated=` and `docker search --filter=stars=...` instead. ### `-h` shorthand for `--help` -**Deprecated In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Deprecated in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** -**Target For Removal In Release: v17.09** +**Target for removal in release: v17.09** The shorthand (`-h`) is less common than `--help` on Linux and cannot be used on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on @@ -1136,58 +1168,58 @@ on all subcommands (due to it conflicting with, e.g. `-h` / `--hostname` on ### `-e` and `--email` flags on `docker login` -**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** +**Deprecated in release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Removed In Release: [v17.06](https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce)** +**Removed in release: [v17.06](https://github.com/docker/docker-ce/releases/tag/v17.06.0-ce)** The `docker login` no longer automatically registers an account with the target registry if the given username doesn't exist. Due to this change, the email flag is no longer required, and will be deprecated. ### Separator (`:`) of `--security-opt` flag on `docker run` -**Deprecated In Release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** +**Deprecated in release: [v1.11.0](https://github.com/docker/docker/releases/tag/v1.11.0)** -**Target For Removal In Release: v17.06** +**Target for removal in release: v17.06** The flag `--security-opt` doesn't use the colon separator (`:`) anymore to divide keys and values, it uses the equal symbol (`=`) for consistency with other similar flags, like `--storage-opt`. ### Ambiguous event fields in API -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Deprecated in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The fields `ID`, `Status` and `From` in the events API have been deprecated in favor of a more rich structure. See the events API documentation for the new format. ### `-f` flag on `docker tag` -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Deprecated in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** To make tagging consistent across the various `docker` commands, the `-f` flag on the `docker tag` command is deprecated. It is no longer necessary to specify `-f` to move a tag from one image to another. Nor will `docker` generate an error if the `-f` flag is missing and the specified tag is already in use. ### HostConfig at API container start -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Deprecated in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Passing an `HostConfig` to `POST /containers/{name}/start` is deprecated in favor of defining it at container creation (`POST /containers/create`). ### `--before` and `--since` flags on `docker ps` -**Deprecated In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Deprecated in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker ps --before` and `docker ps --since` options are deprecated. Use `docker ps --filter=before=...` and `docker ps --filter=since=...` instead. ### Driver-specific log tags -**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** +**Deprecated in release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Log tags are now generated in a standard way across different logging drivers. Because of which, the driver specific log tag options `syslog-tag`, `gelf-tag` and @@ -1199,9 +1231,9 @@ $ docker --log-driver=syslog --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ### Docker Content Trust ENV passphrase variables name change -**Deprecated In Release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** +**Deprecated in release: [v1.9.0](https://github.com/docker/docker/releases/tag/v1.9.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the Tagging key has been renamed to Repository key. Due to this renaming, we're also changing the corresponding environment variables @@ -1210,25 +1242,25 @@ Since 1.9, Docker Content Trust Offline key has been renamed to Root key and the ### `/containers/(id or name)/copy` endpoint -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** +**Deprecated in release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The endpoint `/containers/(id or name)/copy` is deprecated in favor of `/containers/(id or name)/archive`. ### LXC built-in exec driver -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** +**Deprecated in release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Removed in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The built-in LXC execution driver, the lxc-conf flag, and API fields have been removed. ### Old Command Line Options -**Deprecated In Release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** +**Deprecated in release: [v1.8.0](https://github.com/docker/docker/releases/tag/v1.8.0)** -**Removed In Release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** +**Removed in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** The flags `-d` and `--daemon` are deprecated. Use the separate `dockerd` binary instead. @@ -1272,34 +1304,34 @@ The following double-dash options are deprecated and have no replacement: - `docker ps --before-id` - `docker search --trusted` -**Deprecated In Release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** +**Deprecated in release: [v1.5.0](https://github.com/docker/docker/releases/tag/v1.5.0)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The single-dash (`-help`) was removed, in favor of the double-dash `--help` ### `--api-enable-cors` flag on `dockerd` -**Deprecated In Release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** +**Deprecated in release: [v1.6.0](https://github.com/docker/docker/releases/tag/v1.6.0)** -**Removed In Release: [v17.09](https://github.com/docker/docker-ce/releases/tag/v17.09.0-ce)** +**Removed in release: [v17.09](https://github.com/docker/docker-ce/releases/tag/v17.09.0-ce)** The flag `--api-enable-cors` is deprecated since v1.6.0. Use the flag `--api-cors-header` instead. ### `--run` flag on `docker commit` -**Deprecated In Release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** +**Deprecated in release: [v0.10.0](https://github.com/docker/docker/releases/tag/v0.10.0)** -**Removed In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** +**Removed in release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)** The flag `--run` of the `docker commit` command (and its short version `-run`) were deprecated in favor of the `--changes` flag that allows to pass `Dockerfile` commands. ### Three arguments form in `docker import` -**Deprecated In Release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** +**Deprecated in release: [v0.6.7](https://github.com/docker/docker/releases/tag/v0.6.7)** -**Removed In Release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** +**Removed in release: [v1.12.0](https://github.com/docker/docker/releases/tag/v1.12.0)** The `docker import` command format `file|URL|- [REPOSITORY [TAG]]` is deprecated since November 2013. It's no longer supported. From c13c6d7f490ddbc05709da96363bc22ad70badd1 Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Tue, 16 Sep 2025 13:50:16 +0200 Subject: [PATCH 515/534] docs: deprecated: complete deprecation of kernel-memory limit Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 20 +++----------------- 1 file changed, 3 insertions(+), 17 deletions(-) diff --git a/docs/deprecated.md b/docs/deprecated.md index d2a76a4c14..d2c686732c 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -730,25 +730,11 @@ When using cgroups v2, the `--blkio-weight` options are implemented using Specifying kernel memory limit (`docker run --kernel-memory`) is no longer supported because the [Linux kernel deprecated `kmem.limit_in_bytes` in v5.4](https://github.com/torvalds/linux/commit/0158115f702b0ba208ab0b5adf44cae99b3ebcc7). -The OCI runtime specification now marks this option (as well as `--kernel-memory-tcp`) -as ["NOT RECOMMENDED"](https://github.com/opencontainers/runtime-spec/pull/1093), +The OCI runtime specification now marks this option as ["NOT RECOMMENDED"](https://github.com/opencontainers/runtime-spec/pull/1093), and OCI runtimes such as `runc` no longer support this option. -Docker API v1.42 and up now ignores this option when set. Older versions of the -API continue to accept the option, but depending on the OCI runtime used, may -take no effect. - -> [!NOTE] -> While not deprecated (yet) in Docker, the OCI runtime specification also -> deprecated the `memory.kmem.tcp.limit_in_bytes` option. When using `runc` as -> runtime, this option takes no effect. The Linux kernel did not explicitly -> deprecate this feature, and there is a tracking ticket in the `runc` issue -> tracker to determine if this option should be reinstated or if this was an -> oversight of the Linux kernel maintainers (see [opencontainers/runc#3174](https://github.com/opencontainers/runc/issues/3174)). -> -> The `memory.kmem.tcp.limit_in_bytes` option is only supported with cgroups v1, -> and not available on installations running with cgroups v2. This option is -> only supported by the API, and not exposed on the `docker` command-line. +The Docker API no longer handles the kernel-memory fields, and Docker CLI v29.0 +removes the `--kernel-memory` option. ### Classic Swarm and overlay networks using cluster store From 67c5b7e28e47099df2f06dfb91cbcc1f949b03ed Mon Sep 17 00:00:00 2001 From: Sebastiaan van Stijn Date: Mon, 22 Sep 2025 15:27:47 +0200 Subject: [PATCH 516/534] deprecate "--pause" flag on docker commit in favor of "--no-pause" Commit [moby@17d870b] (API v1.13, docker v1.1.0) changed the default to pause containers during commit, keeping the behavior opt-in for older API versions. This version-gate was removed in [moby@1b1147e] because API versions lower than v1.23 were no longer supported. This patch deprecates the `--pause` flag in favor of a `--no-pause` flag to be more explicit on the default. The old `--pause` flag is marked deprecated but still functional. Using the deprecated flag will print a warning, and an error is produced when trying to use both the old and new flag; docker commit --pause mycontainer Flag --pause has been deprecated, and enabled by default. Use --no-pause to disable pausing during commit. docker commit --pause=false mycontainer Flag --pause has been deprecated, and enabled by default. Use --no-pause to disable pausing during commit. docker commit --pause --no-pause mycontainer Flag --pause has been deprecated, use --no-pause instead conflicting options: --no-pause and --pause cannot be used together [moby@17d870b]: https://github.com/moby/moby/commit/17d870bed5ef997c30da1e8b9843f4e84202f8d4 [moby@1b1147e]: https://github.com/moby/moby/commit/1b1147e46b732caeaed4ae365cd56ccbfdf40233 Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index d2c686732c..21e05457f1 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [`--pause` option on `docker commit`](#--pause-option-on-docker-commit) | v29.0 | v30.0 | | Deprecated | [Legacy links environment variables](#legacy-links-environment-variables) | v28.4 | v30.0 | | Deprecated | [Special handling for quoted values for TLS flags](#special-handling-for-quoted-values-for-tls-flags) | v28.4 | v29.0 | | Deprecated | [Empty/nil fields in image Config from inspect API](#emptynil-fields-in-image-config-from-inspect-api) | v28.3 | v29.0 | @@ -124,6 +125,19 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### `--pause` option on `docker commit` + +**Deprecated in release: v29.0** + +**Target for removal in release: v30.0** + +The `--pause` option is enabled by default since Docker v1.1.0 to prevent +committing containers in an inconsistent state, but can be disabled by +setting the `--pause=false` option. In docker CLI v29.0 this flag is +replaced by a `--no-pause` flag instead. The `--pause` option is still +functional in the v29.0 release, printing a deprecation warning, but +will be removed in docker CLI v30. + ### Legacy links environment variables **Deprecated in release: v28.4** From b738d7055d2182b148bb01eb8405602286804492 Mon Sep 17 00:00:00 2001 From: Akihiro Suda Date: Sat, 1 Nov 2025 00:54:06 +0900 Subject: [PATCH 517/534] docs: deprecated: deprecate cgroup v1 See - moby/moby issue 51111 - moby/moby PR 51360 Signed-off-by: Akihiro Suda --- docs/deprecated.md | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 21e05457f1..4271bbf0ac 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -53,6 +53,7 @@ The following table provides an overview of the current status of deprecated fea | Status | Feature | Deprecated | Remove | |------------|------------------------------------------------------------------------------------------------------------------------------------|------------|--------| +| Deprecated | [Support for cgroup v1](#support-for-cgroup-v1) | v29.0 | - | | Deprecated | [`--pause` option on `docker commit`](#--pause-option-on-docker-commit) | v29.0 | v30.0 | | Deprecated | [Legacy links environment variables](#legacy-links-environment-variables) | v28.4 | v30.0 | | Deprecated | [Special handling for quoted values for TLS flags](#special-handling-for-quoted-values-for-tls-flags) | v28.4 | v29.0 | @@ -125,6 +126,26 @@ The following table provides an overview of the current status of deprecated fea | Removed | [`--run` flag on `docker commit`](#--run-flag-on-docker-commit) | v0.10 | v1.13 | | Removed | [Three arguments form in `docker import`](#three-arguments-form-in-docker-import) | v0.6.7 | v1.12 | +### Support for cgroup v1 + +**Deprecated in release: v29.0** + +Support for cgroup v1 is deprecated in the v29.0 release, however, it will continue +to be supported until May 2029. +The latest release in May 2029 may not necessarily support cgroup v1, +but there will be at least one maintained branch with the support for cgroup v1. + +The cgroup version currently in use can be checked by running the `docker info` command: + +```console +$ docker info +<...> +Server: + <...> + Cgroup Version: 2 + <...> +``` + ### `--pause` option on `docker commit` **Deprecated in release: v29.0** From 0bc0c6dec17e82f0c3c179bbb8d11a6b2c82fd88 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Mon, 3 Nov 2025 11:18:00 +0100 Subject: [PATCH 518/534] docs: update link to libnetwork protocol doc Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/extend/plugins_network.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/docs/extend/plugins_network.md b/docs/extend/plugins_network.md index 8f94546b01..0ff0a64b1b 100644 --- a/docs/extend/plugins_network.md +++ b/docs/extend/plugins_network.md @@ -61,11 +61,4 @@ plugin protocol The network driver protocol, in addition to the plugin activation call, is documented as part of libnetwork: -[https://github.com/moby/moby/blob/master/libnetwork/docs/remote.md](https://github.com/moby/moby/blob/master/libnetwork/docs/remote.md). - -## Related Information - -To interact with the Docker maintainers and other interested users, see the IRC channel `#docker-network`. - -- [Docker networks feature overview](https://docs.docker.com/engine/userguide/networking/) -- The [LibNetwork](https://github.com/docker/libnetwork) project +[https://github.com/moby/moby/blob/master/daemon/libnetwork/docs/remote.md](https://github.com/moby/moby/blob/master/daemon/libnetwork/docs/remote.md). From b76862dbe11bda5f6052d7317b08c858b5ee01f4 Mon Sep 17 00:00:00 2001 From: David Karlsson <35727626+dvdksn@users.noreply.github.com> Date: Mon, 27 Oct 2025 14:37:38 +0100 Subject: [PATCH 519/534] docs: update buildgc example config to use new buildkit v0.17 options Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> --- docs/reference/dockerd.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 3af8457251..f2ed191104 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1076,11 +1076,11 @@ The following is a full example of the allowed configuration options on Linux: "builder": { "gc": { "enabled": true, - "defaultKeepStorage": "10GB", + "defaultReservedSpace": "10GB", "policy": [ - { "keepStorage": "10GB", "filter": ["unused-for=2200h"] }, - { "keepStorage": "50GB", "filter": ["unused-for=3300h"] }, - { "keepStorage": "100GB", "all": true } + { "maxUsedSpace": "512MB", "keepDuration": "48h", "filter": [ "type=source.local" ] }, + { "reservedSpace": "10GB", "maxUsedSpace": "100GB", "keepDuration": "1440h" }, + { "reservedSpace": "50GB", "minFreeSpace": "20GB", "maxUsedSpace": "200GB", "all": true } ] } }, From 4b81397da3a7d99d76a7f9e84d0199eea319943b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Thu, 15 Jan 2026 12:50:04 +0100 Subject: [PATCH 520/534] docs: Fix daemon.json typo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Paweł Gronowski --- docs/reference/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index f2ed191104..68d5870d79 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -876,7 +876,7 @@ You can view the configured CDI specification directories using the `docker info #### Disable CDI devices -The feature in enabled by default. To disable it, use the `cdi` options in the `deamon.json` file: +The feature in enabled by default. To disable it, use the `cdi` options in the `daemon.json` file: ```json "features": { From 4c1e6d67a8fe5e68dcf6f4740d835d665cc3fcdf Mon Sep 17 00:00:00 2001 From: Vineet Kumar Date: Wed, 4 Feb 2026 23:10:40 +0530 Subject: [PATCH 521/534] docs: fix typo in dockerd.md for 'replacement' Signed-off-by: Vineet Kumar --- docs/reference/dockerd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 68d5870d79..32ed51f9d0 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -610,7 +610,7 @@ $ sudo dockerd --add-runtime = Defining runtime arguments via the command line is not supported. -For an example configuration for a runc drop-in replacment, see +For an example configuration for a runc drop-in replacement, see [Alternative container runtimes > youki](https://docs.docker.com/engine/daemon/alternative-runtimes/#youki) ##### Configure the default container runtime From 29ed59fe0d6bafb0bdc42a4bd0ff33cac60b071c Mon Sep 17 00:00:00 2001 From: Ludovic Temgoua Abanda Date: Mon, 16 Feb 2026 22:23:04 +0100 Subject: [PATCH 522/534] docs: use generic myplugin example for plugin documentation - Removed EOL Flocker plugin reference from plugin documentations. - docs: use generic myplugin example instead of VolumeDriver Co-authored-by: Ludovic Temgoua Abanda Signed-off-by: Ludovic Temgoua Abanda Signed-off-by: Sebastiaan van Stijn --- docs/extend/plugin_api.md | 10 +++++----- docs/extend/plugins_volume.md | 17 ++++++++++++----- 2 files changed, 17 insertions(+), 10 deletions(-) diff --git a/docs/extend/plugin_api.md b/docs/extend/plugin_api.md index a0b0a73675..0cf2d9e1d6 100644 --- a/docs/extend/plugin_api.md +++ b/docs/extend/plugin_api.md @@ -21,7 +21,7 @@ which registers itself by placing a file on the daemon host in one of the plugin directories described in [Plugin discovery](#plugin-discovery). Plugins have human-readable names, which are short, lowercase strings. For -example, `flocker` or `weave`. +example, `myplugin`. Plugins can run inside or outside containers. Currently running them outside containers is recommended. @@ -45,12 +45,12 @@ spec files can be located either under `/etc/docker/plugins` or `/usr/lib/docker The name of the file (excluding the extension) determines the plugin name. -For example, the `flocker` plugin might create a Unix socket at -`/run/docker/plugins/flocker.sock`. +For example, a plugin named `myplugin` might create a Unix socket at +`/run/docker/plugins/myplugin.sock`. You can define each plugin into a separated subdirectory if you want to isolate definitions from each other. -For example, you can create the `flocker` socket under `/run/docker/plugins/flocker/flocker.sock` and only -mount `/run/docker/plugins/flocker` inside the `flocker` container. +For example, you can create the `myplugin` socket under `/run/docker/plugins/myplugin/myplugin.sock` and only +mount `/run/docker/plugins/myplugin` inside the `myplugin` container. Docker always searches for Unix sockets in `/run/docker/plugins` first. It checks for spec or json files under `/etc/docker/plugins` and `/usr/lib/docker/plugins` if the socket doesn't exist. The directory scan stops as diff --git a/docs/extend/plugins_volume.md b/docs/extend/plugins_volume.md index 8da7deb742..688350f623 100644 --- a/docs/extend/plugins_volume.md +++ b/docs/extend/plugins_volume.md @@ -45,7 +45,7 @@ accepts a volume name and path on the host, and the `--volume-driver` flag accepts a driver type. ```console -$ docker volume create --driver=flocker volumename +$ docker volume create --driver=myplugin volumename $ docker container run -it --volume volumename:/data busybox sh ``` @@ -61,11 +61,18 @@ separated by a colon (`:`) character. - The `Mountpoint` is the path on the host (v1) or in the plugin (v2) where the volume has been made available. -### `volumedriver` +### `--volume-driver` -Specifying a `volumedriver` in conjunction with a `volumename` allows you to -use plugins such as [Flocker](https://github.com/ScatterHQ/flocker) to manage -volumes external to a single host, such as those on EBS. +Specifying the `--volume-driver` flag together with a volume name (using +`--volume`) allows you to use plugins to manage volumes for the container. + +The `--volume-driver` flag is used as a default for all volumes created for +the container, including anonymous volumes. Use the [`--mount`] flag with +the [`volume-driver`] option to specify the driver to use for each volume +individually. + +[`--mount`]: https://docs.docker.com/reference/cli/docker/container/run/#mount +[`volume-driver`]: https://docs.docker.com/engine/storage/volumes/#start-a-container-which-creates-a-volume-using-a-volume-driver ## Create a VolumeDriver From c7a8cbf2625e96bbc060ce2f5657e969739a236f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Alexandre=20Vallie=CC=80res-Lagace=CC=81?= Date: Wed, 18 Mar 2026 15:05:40 -0400 Subject: [PATCH 523/534] Updated example tokens in swarm docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prevent security scanners from detecting them as eaked secrets. Co-authored-by: David Karlsson <35727626+dvdksn@users.noreply.github.com> Signed-off-by: Alexandre Vallières-Lagacé Signed-off-by: Alexandre Vallières-Lagacé --- docs/extend/plugins_services.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/extend/plugins_services.md b/docs/extend/plugins_services.md index f0c2dc503d..fb914047b8 100644 --- a/docs/extend/plugins_services.md +++ b/docs/extend/plugins_services.md @@ -38,7 +38,7 @@ node1 is the manager and node2 is the worker. ```console $ docker swarm join \ - --token SWMTKN-1-49nj1cmql0jkz5s954yi3oex3nedyz0fb0xx14ie39trti4wxv-8vxv8rssmk743ojnwacrr2e7c \ + --token SWMTKN-1-aabbccdd00112233aabbccdd00112233aabbccdd00112233aa-aabbccdd00112233... \ 192.168.99.100:2377 ``` From 76928c202509b4987b42119c2097523fcc5bbed2 Mon Sep 17 00:00:00 2001 From: Varun5711 Date: Sun, 12 Apr 2026 20:03:15 +0530 Subject: [PATCH 524/534] docs: refresh legacy plugin references Signed-off-by: Varun Hotani --- docs/extend/legacy_plugins.md | 28 ++++++++++++++++++---------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/docs/extend/legacy_plugins.md b/docs/extend/legacy_plugins.md index 2c6ba4bc1a..11269df404 100644 --- a/docs/extend/legacy_plugins.md +++ b/docs/extend/legacy_plugins.md @@ -10,6 +10,14 @@ This document describes the Docker Engine plugins generally available in Docker Engine. To view information on plugins managed by Docker, refer to [Docker Engine plugin system](_index.md). +> [!NOTE] +> Legacy plugins are superseded by Docker Engine's managed plugin system. +> For plugins installed and managed by Docker, use +> [`docker plugin install`](https://docs.docker.com/reference/cli/docker/plugin/install/) +> and the [Docker Engine plugin system](_index.md). The third-party plugins +> listed on this page are provided for historical reference, and archived +> projects are marked accordingly. + You can extend the capabilities of the Docker Engine by loading third-party plugins. This page explains the types of plugins and provides links to several volume and network plugins for Docker. @@ -30,7 +38,8 @@ Follow the instructions in the plugin's documentation. ## Finding a plugin -The sections below provide an overview of available third-party plugins. +The sections below provide an overview of third-party plugins that use the +legacy plugin model. ### Network plugins @@ -44,22 +53,21 @@ The sections below provide an overview of available third-party plugins. | Plugin | Description | |:---------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). | +| [Azure File Storage plugin](https://github.com/Azure/azurefile-dockervolumedriver) (archived) | Lets you mount Microsoft [Azure File Storage](https://azure.microsoft.com/blog/azure-file-storage-now-generally-available/) shares to Docker containers as volumes using the SMB 3.0 protocol. [Learn more](https://azure.microsoft.com/blog/persistent-docker-volumes-with-azure-file-storage/). | | [BeeGFS Volume Plugin](https://github.com/RedCoolBeans/docker-volume-beegfs) | An open source volume plugin to create persistent volumes in a BeeGFS parallel file system. | | [Blockbridge plugin](https://github.com/blockbridge/blockbridge-docker-volume) | A volume plugin that provides access to an extensible set of container-based persistent storage options. It supports single and multi-host Docker environments with features that include tenant isolation, automated provisioning, encryption, secure deletion, snapshots and QoS. | | [Contiv Volume Plugin](https://github.com/contiv/volplugin) | An open source volume plugin that provides multi-tenant, persistent, distributed storage with intent based consumption. It has support for Ceph and NFS. | -| [Convoy plugin](https://github.com/rancher/convoy) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. | +| [Convoy plugin](https://github.com/rancher/convoy) (archived) | A volume plugin for a variety of storage back-ends including device mapper and NFS. It's a simple standalone executable written in Go and provides the framework to support vendor-specific extensions such as snapshots, backups and restore. | | [DigitalOcean Block Storage plugin](https://github.com/omallo/docker-volume-plugin-dostorage) | Integrates DigitalOcean's [block storage solution](https://www.digitalocean.com/products/storage/) into the Docker ecosystem by automatically attaching a given block storage volume to a DigitalOcean droplet and making the contents of the volume available to Docker containers running on that droplet. | | [DRBD plugin](https://www.drbd.org/en/supported-projects/docker) | A volume plugin that provides highly available storage replicated by [DRBD](https://www.drbd.org). Data written to the docker volume is replicated in a cluster of DRBD nodes. | -| [Flocker plugin](https://github.com/ScatterHQ/flocker) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. | -| [Fuxi Volume Plugin](https://github.com/openstack/fuxi) | A volume plugin that is developed as part of the OpenStack Kuryr project and implements the Docker volume plugin API by utilizing Cinder, the OpenStack block storage service. | -| [gce-docker plugin](https://github.com/mcuadros/gce-docker) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). | -| [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. | +| [Flocker plugin](https://github.com/ScatterHQ/flocker) (archived) | A volume plugin that provides multi-host portable volumes for Docker, enabling you to run databases and other stateful containers and move them around across a cluster of machines. | +| [Fuxi Volume Plugin](https://github.com/openstack-archive/fuxi) (archived) | A volume plugin that is developed as part of the OpenStack Kuryr project and implements the Docker volume plugin API by utilizing Cinder, the OpenStack block storage service. | +| [gce-docker plugin](https://github.com/mcuadros/gce-docker) (archived) | A volume plugin able to attach, format and mount Google Compute [persistent-disks](https://cloud.google.com/compute/docs/disks/persistent-disks). | +| [GlusterFS plugin](https://github.com/calavera/docker-volume-glusterfs) (archived) | A volume plugin that provides multi-host volumes management for Docker using GlusterFS. | | [Horcrux Volume Plugin](https://github.com/muthu-r/horcrux) | A volume plugin that allows on-demand, version controlled access to your data. Horcrux is an open-source plugin, written in Go, and supports SCP, [Minio](https://www.minio.io) and Amazon S3. | | [HPE 3Par Volume Plugin](https://github.com/hpe-storage/python-hpedockerplugin/) | A volume plugin that supports HPE 3Par and StoreVirtual iSCSI storage arrays. | -| [Infinit volume plugin](https://infinit.sh/documentation/docker/volume-plugin) | A volume plugin that makes it easy to mount and manage Infinit volumes using Docker. | -| [IPFS Volume Plugin](https://github.com/vdemeester/docker-volume-ipfs) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. | -| [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) | A plugin that provides credentials and secret management using Keywhiz as a central repository. | +| [IPFS Volume Plugin](https://github.com/vdemeester/docker-volume-ipfs) (archived) | An open source volume plugin that allows using an [ipfs](https://ipfs.io/) filesystem as a volume. | +| [Keywhiz plugin](https://github.com/calavera/docker-volume-keywhiz) (archived) | A plugin that provides credentials and secret management using Keywhiz as a central repository. | | [Linode Volume Plugin](https://github.com/linode/docker-volume-linode) | A plugin that adds the ability to manage Linode Block Storage as Docker Volumes from within a Linode. | | [Local Persist Plugin](https://github.com/CWSpear/local-persist) | A volume plugin that extends the default `local` driver's functionality by allowing you specify a mountpoint anywhere on the host, which enables the files to *always persist*, even if the volume is removed via `docker volume rm`. | | [NetApp Plugin](https://github.com/NetApp/netappdvp) (nDVP) | A volume plugin that provides direct integration with the Docker ecosystem for the NetApp storage portfolio. The nDVP package supports the provisioning and management of storage resources from the storage platform to Docker hosts, with a robust framework for adding additional platforms in the future. | From 0b902c7eedb77d65ef1f456bb95fc5e73ad0954a Mon Sep 17 00:00:00 2001 From: Matteo Panzeri Date: Sat, 9 May 2026 23:00:51 +0200 Subject: [PATCH 525/534] docs: clarify 64 KiB response-body buffer in authz plugin docs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds a "Response body size and partial buffering" subsection to docs/extend/plugins_authorization.md documenting the 64 KiB maxBufferSize constant in the daemon's internal responseModifier (pkg/authorization/response.go in moby/moby) and the practical implications for plugins that use ResponseBody inspection. The existing docs (lines 81-87) say streaming endpoints such as logs and events send only the HTTP request to plugins, but don't explain the underlying mechanism. Plugin authors building response-body redaction or content-filtering can be surprised when the same effect happens on non-listed endpoints whose response is produced through multiple writes exceeding the buffer or via an io.WriteFlusher. The 64 KiB buffer is observable from the public moby source, so this PR is documentation catching up to existing behavior — not a contract change. Signed-off-by: Matteo Panzeri --- docs/extend/plugins_authorization.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index b8678debd5..232d5f1e6d 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -86,6 +86,31 @@ passed to the authorization plugins. For commands that return chunked HTTP response, such as `logs` and `events`, only the HTTP request is sent to the authorization plugins. +### Response body size and partial buffering + +The internal buffer that holds the response body between the daemon's HTTP +handler and the plugin's response authorization callback (`responseModifier`, +defined in [`pkg/authorization/response.go`](https://github.com/moby/moby/blob/master/pkg/authorization/response.go)) +has a fixed capacity of 64 KiB (`maxBufferSize`). + +For most non-streaming endpoints the full response is buffered for plugin +inspection regardless of total size, because Go's `encoding/json` encoder +serializes the complete payload into a single underlying write. The +streaming-response exclusion noted above (for example, `logs` and `events`) +is the practical effect of this 64 KiB threshold combined with the +`io.WriteFlusher` write pattern used by streaming handlers, where each write +is immediately drained to the client and is therefore no longer available +for plugin inspection by the time the handler returns. + +> [!NOTE] +> Plugins that depend on `ResponseBody` inspection for redaction or +> content-filtering should restrict their policies to endpoints whose +> response is produced as a single write (typical of REST-style API +> responses). For commands whose responses are streamed or are likely to +> exceed the buffer through multiple writes, do not rely on `ResponseBody` +> for security-relevant decisions; perform the filtering in a separate +> layer in front of the daemon. + During request/response processing, some authorization flows might need to do additional queries to the Docker daemon. To complete such flows, plugins can call the daemon API similar to a regular user. To enable these From 3b1b0a591494a2e3be2d30ac19ce3c966372433b Mon Sep 17 00:00:00 2001 From: Alexandre LEVAVASSEUR Date: Thu, 29 Jun 2023 12:55:43 +0200 Subject: [PATCH 526/534] metrics-addr is not experimental since 20.10 See https://github.com/moby/moby/commit/f337a8d21d9902772a766f57475a5512405c86c5 Signed-off-by: Alexandre LEVAVASSEUR --- docs/reference/dockerd.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 32ed51f9d0..c0ef206cda 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -958,8 +958,6 @@ the `--cgroup-parent` option on the daemon. #### Daemon metrics The `--metrics-addr` option takes a TCP address to serve the metrics API. -This feature is still experimental, therefore, the daemon must be running in experimental -mode for this feature to work. To serve the metrics API on `localhost:9323` you would specify `--metrics-addr 127.0.0.1:9323`, allowing you to make requests on the API at `127.0.0.1:9323/metrics` to receive metrics in the From f2e3b57ce39b17691990a249c81b2e7dba0b4110 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ren=C3=A9=20Hermenau?= Date: Thu, 1 Jul 2021 19:14:28 +0200 Subject: [PATCH 527/534] Update dockerd.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit daemon.json does not exist on a clean install. The doc should reflect that. Signed-off-by: René Hermenau Signed-off-by: Sebastiaan van Stijn --- docs/reference/dockerd.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index c0ef206cda..1dcb191626 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -1057,9 +1057,9 @@ $ echo $? ##### On Linux -The default location of the configuration file on Linux is -`/etc/docker/daemon.json`. Use the `--config-file` flag to specify a -non-default location. +The default location of the configuration file on Linux is `/etc/docker/daemon.json`. +If the file does not exist, you need to create it first. Use the `--config-file` +flag to specify a non-default location. The following is a full example of the allowed configuration options on Linux: @@ -1208,8 +1208,9 @@ The following is a full example of the allowed configuration options on Linux: ##### On Windows The default location of the configuration file on Windows is -`%programdata%\docker\config\daemon.json`. Use the `--config-file` flag -to specify a non-default location. +`%programdata%\docker\config\daemon.json`. If the file does not exist, you need +to create it first. Use the `--config-file` flag to specify a non-default +location. The following is a full example of the allowed configuration options on Windows: From 36013a3d4b9a99c2913903b6187d65b3effdff3a Mon Sep 17 00:00:00 2001 From: mickael emirkanian Date: Fri, 15 May 2026 14:07:45 -0400 Subject: [PATCH 528/534] docs: clarify authz content type update based on the logic in https://github.com/moby/moby/blob/0686f57c3d942ce4440f9ed7f2e955de3687dd4e/pkg/authorization/authz.go#L177 Signed-off-by: mickael emirkanian --- docs/extend/plugins_authorization.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 232d5f1e6d..2380d370db 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -75,8 +75,10 @@ Each request sent to the plugin includes the authenticated user, the HTTP headers, and the request/response body. Only the user name and the authentication method used are passed to the plugin. Most importantly, no user credentials or tokens are passed. Finally, not all request/response bodies -are sent to the authorization plugin. Only those request/response bodies where -the `Content-Type` is either `text/*` or `application/json` are sent. +are sent to the authorization plugin. Only request/response bodies where +the `Content-Type` is `application/json` are sent to the authorization plugin; +bodies of any other `Content-Type` are not visible to the plugin and cannot +be used for enforcement, even though the daemon may still act on this data. For commands that can potentially hijack the HTTP connection (`HTTP Upgrade`), such as `exec`, the authorization plugin is only called for the From 7b7867bc3c2a9c4e7b86eb61bd59abf6ac2ef4ae Mon Sep 17 00:00:00 2001 From: mickael emirkanian Date: Tue, 19 May 2026 15:44:59 -0400 Subject: [PATCH 529/534] docs: further clarify authz plugins Signed-off-by: mickael emirkanian --- docs/extend/plugins_authorization.md | 13 ++++++++----- 1 file changed, 8 insertions(+), 5 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 2380d370db..d887a30a6d 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -74,11 +74,14 @@ The sequence diagrams below depict an allow and deny authorization flow: Each request sent to the plugin includes the authenticated user, the HTTP headers, and the request/response body. Only the user name and the authentication method used are passed to the plugin. Most importantly, no user -credentials or tokens are passed. Finally, not all request/response bodies -are sent to the authorization plugin. Only request/response bodies where -the `Content-Type` is `application/json` are sent to the authorization plugin; -bodies of any other `Content-Type` are not visible to the plugin and cannot -be used for enforcement, even though the daemon may still act on this data. +credentials or tokens are passed. + +> [!NOTE] +> Authorization plugins enforce requests to the Docker daemon's HTTP API only. gRPC method +> calls, whether dispatched natively or upgraded through `POST /grpc`, are not subject to authorization. +> Furthermore, HTTP request/response bodies where the `Content-Type` is `application/json` are forwarded; +> bodies of any other type are not visible to the plugin and cannot be used for enforcement, +> even though the daemon acts on this data. For commands that can potentially hijack the HTTP connection (`HTTP Upgrade`), such as `exec`, the authorization plugin is only called for the From 91b92582cbb4e54791136e3eb12edc59876df34c Mon Sep 17 00:00:00 2001 From: mickael emirkanian Date: Tue, 2 Jun 2026 15:10:23 -0400 Subject: [PATCH 530/534] docs: recommend default deny and clarify requesturi field Signed-off-by: mickael emirkanian --- docs/extend/plugins_authorization.md | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index d887a30a6d..8c84d6e384 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -91,6 +91,10 @@ passed to the authorization plugins. For commands that return chunked HTTP response, such as `logs` and `events`, only the HTTP request is sent to the authorization plugins. +The Engine's authorization middleware fails closed: when a plugin returns an error or returns `Allow: false`, +the request is denied and the error is surfaced to the client. Plugins should also fail closed: if the plugin +cannot confidently evaluate a request, it should return an error or `Allow: false`. + ### Response body size and partial buffering The internal buffer that holds the response body between the daemon's HTTP @@ -239,7 +243,7 @@ Name | Type | Description User | string | The user identification Authentication method | string | The authentication method used Request method | enum | The HTTP method (GET/DELETE/POST) -Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) +Request URI | string | The HTTP request URI including API version, as sent by the client (e.g., v.1.17/containers/json) Request headers | map[string]string | Request headers as key value pairs (without the authorization header) Request body | []byte | Raw request body @@ -262,7 +266,7 @@ Name | Type | Description User | string | The user identification Authentication method | string | The authentication method used Request method | string | The HTTP method (GET/DELETE/POST) -Request URI | string | The HTTP request URI including API version (e.g., v.1.17/containers/json) +Request URI | string | The HTTP request URI including API version, as sent by the client (e.g., v.1.17/containers/json) Request headers | map[string]string | Request headers as key value pairs (without the authorization header) Request body | []byte | Raw request body Response status code | int | Status code from the Docker daemon From 786cfe95eae79a1b5a73717fb2071c79426b4433 Mon Sep 17 00:00:00 2001 From: Albin Kerouanton Date: Thu, 22 Feb 2024 12:13:16 +0100 Subject: [PATCH 531/534] Add a deprecation warning about --link on default nw Signed-off-by: Albin Kerouanton Signed-off-by: Sebastiaan van Stijn --- docs/deprecated.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/deprecated.md b/docs/deprecated.md index 4271bbf0ac..37ffcbc470 100644 --- a/docs/deprecated.md +++ b/docs/deprecated.md @@ -113,6 +113,7 @@ The following table provides an overview of the current status of deprecated fea | Deprecated | [`-h` shorthand for `--help`](#-h-shorthand-for---help) | v1.12 | v17.09 | | Removed | [`-e` and `--email` flags on `docker login`](#-e-and---email-flags-on-docker-login) | v1.11 | v17.06 | | Deprecated | [Separator (`:`) of `--security-opt` flag on `docker run`](#separator--of---security-opt-flag-on-docker-run) | v1.11 | v17.06 | +| Deprecated | [Links on the default bridge network](#links-on-the-default-bridge-network) | v1.10 | - | | Deprecated | [Ambiguous event fields in API](#ambiguous-event-fields-in-api) | v1.10 | - | | Removed | [`-f` flag on `docker tag`](#-f-flag-on-docker-tag) | v1.10 | v1.12 | | Removed | [HostConfig at API container start](#hostconfig-at-api-container-start) | v1.10 | v1.12 | @@ -1203,6 +1204,19 @@ The `docker login` no longer automatically registers an account with the target The flag `--security-opt` doesn't use the colon separator (`:`) anymore to divide keys and values, it uses the equal symbol (`=`) for consistency with other similar flags, like `--storage-opt`. +### Links on the default bridge network + +**Deprecated in release: v1.10** +**Target for removal in release: v30.0** + +The `--link` option on `docker create` and `docker run`, when used with no +`--network` specified, was deprecated in v1.10 and will be removed in a future +release. Custom networks should be used instead. Docker 29.6 added a deprecation +warning when this option is used for the default bridge network. + +Note that the `--link` option is still supported when a non-default network +is used. + ### Ambiguous event fields in API **Deprecated in release: [v1.10.0](https://github.com/docker/docker/releases/tag/v1.10.0)** From 2a39834a810c7c00389ee98f044c2b7953d621c4 Mon Sep 17 00:00:00 2001 From: mickael emirkanian Date: Tue, 7 Jul 2026 14:49:50 -0400 Subject: [PATCH 532/534] docs(authz): clarify daemon parsing semantics Signed-off-by: mickael emirkanian --- docs/extend/plugins_authorization.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/docs/extend/plugins_authorization.md b/docs/extend/plugins_authorization.md index 8c84d6e384..605fac16fa 100644 --- a/docs/extend/plugins_authorization.md +++ b/docs/extend/plugins_authorization.md @@ -95,6 +95,18 @@ The Engine's authorization middleware fails closed: when a plugin returns an err the request is denied and the error is surfaced to the client. Plugins should also fail closed: if the plugin cannot confidently evaluate a request, it should return an error or `Allow: false`. +> [!WARNING] +> Because the plugin receives the [**raw** request body](#authzpluginauthzreq) from the daemon, it must +> apply the same decoding semantics as the daemon to be sure it evaluates the request the daemon will +> act on. The daemon decodes JSON with Go's [`encoding/json.Unmarshal`](https://pkg.go.dev/encoding/json#Unmarshal). +> +> The same requirement applies to the response body. Plugins that depend on `ResponseBody` +> inspection for redaction or content-filtering should restrict their policies to endpoints +> whose response is produced as a single write (typical of REST-style API responses). For +> commands whose responses are streamed or are likely to exceed the [buffer](#response-body-size-and-partial-buffering) through multiple +> writes, do not rely on `ResponseBody` for security-relevant decisions; perform the filtering +> in a separate layer in front of the daemon. + ### Response body size and partial buffering The internal buffer that holds the response body between the daemon's HTTP @@ -111,15 +123,6 @@ is the practical effect of this 64 KiB threshold combined with the is immediately drained to the client and is therefore no longer available for plugin inspection by the time the handler returns. -> [!NOTE] -> Plugins that depend on `ResponseBody` inspection for redaction or -> content-filtering should restrict their policies to endpoints whose -> response is produced as a single write (typical of REST-style API -> responses). For commands whose responses are streamed or are likely to -> exceed the buffer through multiple writes, do not rely on `ResponseBody` -> for security-relevant decisions; perform the filtering in a separate -> layer in front of the daemon. - During request/response processing, some authorization flows might need to do additional queries to the Docker daemon. To complete such flows, plugins can call the daemon API similar to a regular user. To enable these From 5a4c2b9e2d58ca58df5ae85ef2a1dc457f9f1284 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Tue, 1 Sep 2026 16:19:27 +0200 Subject: [PATCH 533/534] dockerd: Document `default-stop-timeout` MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Paweł Gronowski --- docs/reference/dockerd.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 1dcb191626..71cb56212b 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -51,6 +51,7 @@ Options: --default-network-opt mapmap Default network options (default map[]) --default-runtime string Default OCI runtime for containers (default "runc") --default-shm-size bytes Default shm size for containers (default 64MiB) + --default-stop-timeout int Set the default container stop timeout in seconds (0 to terminate immediately) (default 10) --default-ulimit ulimit Default ulimits for containers (default []) --dns list DNS server to use --dns-opt list DNS options to use @@ -758,6 +759,18 @@ This only adds the proxy and authentication to the Docker daemon's requests. To use the proxy when building images and running containers, see [Configure Docker to use a proxy server](https://docs.docker.com/engine/cli/proxy/) +### Default container stop timeout + +The `--default-stop-timeout` flag sets the number of seconds to wait for a +container to stop when the container doesn't have a `--stop-timeout` configured. +The default is `10` seconds on Linux and `30` seconds on Windows. A value of `0` +doesn't wait before forcefully terminating the container. Negative values are +invalid. + +You can also configure the timeout with the `default-stop-timeout` option in the +[daemon configuration file](#daemon-configuration-file). You can reload this +option without restarting the daemon. + ### Default `ulimit` settings The `--default-ulimit` flag lets you set the default `ulimit` options to use for @@ -1104,6 +1117,7 @@ The following is a full example of the allowed configuration options on Linux: "default-network-opts": {}, "default-runtime": "runc", "default-shm-size": "64M", + "default-stop-timeout": 10, "default-ulimits": { "nofile": { "Hard": 64000, @@ -1225,6 +1239,7 @@ The following is a full example of the allowed configuration options on Windows: "debug": true, "default-network-opts": {}, "default-runtime": "", + "default-stop-timeout": 30, "default-ulimits": {}, "dns": [], "dns-opts": [], @@ -1317,6 +1332,7 @@ The list of currently supported options that can be reconfigured is this: | `max-concurrent-uploads` | Configures the max concurrent uploads for each push. | | `max-download-attempts` | Configures the max download attempts for each pull. | | `default-runtime` | Configures the runtime to be used if not is specified at container creation. | +| `default-stop-timeout` | Configures the timeout for stopping containers that have no container-specific timeout. | | `runtimes` | Configures the list of available OCI runtimes that can be used to run containers. | | `authorization-plugin` | Specifies the authorization plugins to use. | | `insecure-registries` | Specifies a list of registries that the daemon should consider insecure. | From e7e68bc95fbb22000163fa0dd90cfb04026b5d0d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Pawe=C5=82=20Gronowski?= Date: Tue, 1 Sep 2026 16:33:04 +0200 Subject: [PATCH 534/534] dockerd: Document custom AppArmor profiles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Paweł Gronowski --- docs/reference/dockerd.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/docs/reference/dockerd.md b/docs/reference/dockerd.md index 71cb56212b..61a342c655 100644 --- a/docs/reference/dockerd.md +++ b/docs/reference/dockerd.md @@ -25,6 +25,7 @@ A self-sufficient runtime for containers. Options: --add-runtime runtime Register an additional OCI compatible runtime (default []) --allow-direct-routing Allow remote access to published ports on container IP addresses + --apparmor-profile string Path to AppArmor profile template --authorization-plugin list Authorization plugins to load --bip string IPv4 address for the default bridge --bip6 string IPv6 address for the default bridge @@ -806,6 +807,24 @@ allow the request for it to complete. For information about how to create an authorization plugin, refer to the [authorization plugin](https://docs.docker.com/engine/extend/plugins_authorization/) section. +### Configure the default AppArmor profile + +The `--apparmor-profile` option sets the path to an AppArmor profile definition +for Docker's default `docker-default` container profile. This option is only +supported on Linux hosts. The file can be a static profile or a Go template. + +The equivalent `daemon.json` configuration is: + +```json +{ + "apparmor-profile": "/etc/docker/apparmor/docker-default" +} +``` + +Restart the daemon to apply changes. For template values, safe file placement, +security considerations, and verification, see +[Customize the default AppArmor profile](https://docs.docker.com/engine/security/apparmor/#customize-the-default-profile). + ### Daemon user namespace options The Linux kernel @@ -1079,6 +1098,7 @@ The following is a full example of the allowed configuration options on Linux: ```json { "allow-direct-routing": false, + "apparmor-profile": "", "authorization-plugins": [], "bip": "", "bip6": "",