Commit Graph

43 Commits

Author SHA1 Message Date
Henry Wang
0198b87fcf Implement parallel unpack
Signed-off-by: Henry Wang <henwang@amazon.com>
2025-10-24 17:54:26 +00:00
Henry Wang
748cd9f037 Prevent goroutine hangs during ProgressTracker shutdown
Signed-off-by: Henry Wang <henwang@amazon.com>
2025-09-25 18:50:14 +00:00
Derek McGowan
479cf42ca2 Add extraction progress to transfer service
Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-09-10 20:07:40 -07:00
yashsingh74
19a7130613 Disable QF1003: could use tagged switch on base (staticcheck)
Signed-off-by: yashsingh74 <yashsingh1774@gmail.com>
2025-06-18 14:16:41 +05:30
Derek McGowan
697d776768 Decode grpc errors in the transfer client proxy
Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-06-11 18:00:01 -07:00
Mike Brown
4d7381e47c Merge pull request #11927 from dmcgowan/enhance-transfer-unpack-selection
Update transfer supported platforms logic
2025-06-09 02:11:51 +00:00
Fu Wei
f482992fdc Merge pull request #11911 from dmcgowan/transfer-internal-interface
Remove internal interface from unpacker interface
2025-06-03 01:37:22 +00:00
Derek McGowan
48cec3659f Update transfer supported platforms logic
Allow selection of a non-default snapshotter when no unpack
snapshotter is specified.

Add more logging to help figure out why an unpack configuration is not
used.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-06-02 15:40:54 -07:00
Derek McGowan
00edba6aa8 Remove internal interface from unpacker interface
Avoid using internal interfaces for exported types and always prefer
using interfaces over structs to limit unused imports.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-05-28 14:23:24 -07:00
Adrien Delorme
77562a8e4a fix import for local transfer service
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-05-28 11:55:30 +02:00
Derek McGowan
6e17198f07 Add descriptor to transfer pull image events
Include the descriptor for the image target on image events.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-05-21 17:04:44 -07:00
Phil Estes
c213879967 Merge pull request #11770 from azr/log-fetch-parameters
fix multipart layer fetch in client: pass config down
2025-05-05 12:51:17 +00:00
Danny Canter
33ee060a35 Use Go 1.19 atomic wrappers everywhere
We've long been able to use these and they have a couple
great benefits:

1. Forces you to always access them atomically. With the pointer
variants it's completely valid to access the regular ol' int64/uint32
etc. without using the atomic.* methods. These wrappers don't provide
access to the underlying value so it forces correct usage always.

2. Conveys intent much better. Seeing the type be atomic.Int32 immediately
lets the reader know that this var will be used in a concurrent context,
and we no longer need comments like "this MUST be accessed atomically"
or similar.

Signed-off-by: Danny Canter <danny@dcantah.dev>
2025-05-03 03:46:20 -07:00
Adrien Delorme
413702b7e4 fix comment
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-04-30 10:20:26 +02:00
Wei Fu
b89733812c core/transfer/local: should not mark complete if it's not found
For the progress tracker, after creating job, it doesn't mean that we
have created ingest object for that job. There is race condition. We
should not mark job as completed if blob is not found.

Fixes: #11758

Signed-off-by: Wei Fu <fuweid89@gmail.com>
2025-04-28 15:22:01 -04:00
Fu Wei
ef7bdf18a1 Merge pull request #10177 from azr/azr/parallel-layer-fetch
Multipart layer fetch
2025-04-24 20:35:00 +00:00
Adrien Delorme
72c8c7708c only keep one setting: concurrent_layer_fetch_buffer
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-04-24 11:41:33 +02:00
Adrien Delorme
024775dab1 set dl options on resolver
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-04-24 11:41:33 +02:00
Adrien Delorme
88116b1911 remove max_dl_operations setting
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-04-24 11:39:42 +02:00
Adrien Delorme
755a4ac6f8 update
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
2025-04-24 11:39:42 +02:00
Adrien Delorme
f9af08820b perf(pull): multipart layer fetch
Signed-off-by: Adrien Delorme <azr@users.noreply.github.com>
Co-Authored-By: Corentin REGAL <143578+co42@users.noreply.github.com>
2025-04-24 11:39:42 +02:00
Maksym Pavlenko
17b6e1ef85 Allow streaming to client
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2025-04-23 18:03:29 -07:00
Maksym Pavlenko
bd8e6c727d Enable http debug and tracing for non local puller
Signed-off-by: Maksym Pavlenko <pavlenko.maksym@gmail.com>
2025-04-23 17:39:57 -07:00
Tony Fang
b694be29a0 Update CRI image service to pull using transfer service
- adds a transfer service progress reporter to handle timeouts. Also other test fixes
- fallback to local image pull when configuration conflict

Signed-off-by: Tony Fang <nhfang@amazon.com>

Co-authored-by: Swagat Bora <sbora@amazon.com>
2025-04-23 18:18:27 +00:00
Kohei Tokunaga
00fee4adb4 Transfer Service: enable remote snapshotters
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2025-01-15 15:49:48 +09:00
Kazuyoshi Kato
f38df49348 Merge pull request #11227 from dmcgowan/unpack-snapshotter-exports
Add snapshotter exports to unpack platform
2025-01-09 00:23:46 +00:00
Derek McGowan
63f6047282 Add snapshotter exports to unpack platform
Allows the unpack operation to be optimized based on the specific
snapshotter configuration.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2025-01-07 16:04:44 -08:00
Austin Vazquez
bee64b2b93 Remove loop variable copies
Signed-off-by: Austin Vazquez <macedonv@amazon.com>
2024-12-23 23:14:49 -07:00
Sebastiaan van Stijn
ea8265fb1e core/transfer/local: remove logrus import
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2024-06-17 12:37:14 +02:00
Maksym Pavlenko
34d3c17ae2 Merge pull request #10291 from ktock/push-platform-conf
Transfer: Push: Enable to specify platforms
2024-06-05 21:28:09 +00:00
Kohei Tokunaga
5611fdd4af Transfer: Push: Enable to specify platforms
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-06-04 10:02:13 +09:00
Kohei Tokunaga
0e8cc91463 Transfer: Push: fix failure on pushing duplicated blobs
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-05-31 18:08:45 +09:00
Abel Feng
b8dfb4d8f5 cri: support io by streaming api
Signed-off-by: Abel Feng <fshb1988@gmail.com>
2024-05-07 20:26:40 +08:00
Kazuyoshi Kato
d65976b687 Merge pull request #10163 from dmcgowan/transfer-ttrpc-support
Add support for ttrpc in transfer and streaming service
2024-05-03 18:15:52 +00:00
Fu Wei
857dc6f89e Merge pull request #10162 from dmcgowan/cleanup-local-transfer
Cleanup local transfer interface
2024-05-03 10:40:46 +00:00
Derek McGowan
05a3171bb4 Update transfer proxy to support ttrpc
Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-05-02 23:16:51 -07:00
Derek McGowan
fe01cad201 Cleanup local transfer interface
Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-05-02 15:45:49 -07:00
Derek McGowan
2ac2b9c909 Make api a Go sub-module
Allow the api to stay at the same v1 go package name and keep using a
1.x version number. This indicates the API is still at 1.x and allows
sharing proto types with containerd 1.6 and 1.7 releases.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-05-02 11:03:00 -07:00
Kohei Tokunaga
4332794384 Transfer: Registry: Enable plain HTTP
Currenlty transfer service doesn't handle plain HTTP connection.
This commit fixes this issue by propagating
`(core/remotes/docker/config).HostOptions.DefaultScheme` from client to the
transfer service.
This commit also fixes ctr to use this feature for "--plain-http" flag.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-04-03 10:46:10 +09:00
Derek McGowan
3a8c27dff8 Merge pull request #9908 from ktock/transfer-host-dir
Transfer: Registry: Enable to use registry configuration diretory
2024-04-02 18:59:43 +00:00
Edgar Lee
34c545824f Automatically decompress archives for transfer service import
Signed-off-by: Edgar Lee <edgarhinshunlee@gmail.com>
2024-03-15 10:14:19 +08:00
Kohei Tokunaga
7a3b7fba59 Transfer: Registry: Enable to use registry configuration diretory
Currently transfer service isn't aware of configurations of hosts directory and
ctr's `--hosts-dir` doesn't work.
This commit fixes this issue by using `config.ConfigureHosts` instead of
`docker.ConfigureDefaultRegistries`.
This commit also fixes ctr to use this feature for "--hosts-dir" flag.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2024-03-04 15:43:30 +09:00
Derek McGowan
f46aea6187 Move transfer and unpack packages
Packages related to transfer and unpacking provide core interfaces which
use other core interfaces and part of common functionality.

Signed-off-by: Derek McGowan <derek@mcg.dev>
2024-02-07 22:40:15 -08:00