Import the log module directly, instead of using the aliases that are
provided in the log package. This helps find code-paths that still depend
on the old location, and that haven't migrated yet to the new module.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
In order to make sure that we don't publish task exit events for init
processes before we do for execs in that container, we added logic to
`processExits` in 892dc54bd2 to skip these
and let the pending exec's `handleStarted` closure process them.
However, the conditional logic in `processExits` added was faulty - we
should only defer processing of exit events related to init processes,
not other execs. Due to this missing condition,
892dc54bd2 introduced a bug where, if
there are many concurrent execs for the same container/init pid, exec
exits are skipped and then never published, resulting in hanging
clients.
This commit adds the missing logic to `processExits`.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
(cherry picked from commit 6d00c3ada8)
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
For a given container, as long as the init process is the init process
of that PID namespace, we always receive the exits for execs before we
receive them for the init process.
It's important that we uphold this invariant for the outside world by
always emitting a TastExit event for a container's exec before we emit
one for the init process because this is the expected behavior from
callers, and changing this creates issues - such as Docker, which will
delete the container after receiving a TaskExit for the init process,
and then not be able to handle the exec's exit after having deleted
the container (see: https://github.com/containerd/containerd/issues/9719).
Since 5cd6210ad0, if an exec is starting
at the same time that an init exits, if the exec is an "early exit"
i.e. we haven't emitted a TaskStart for it/put it in `s.running` by the
time we receive it's exit, we notify concurrent calls to `s.Start()` of
the exit and continue processing exits, which will cause us to process
the Init's exit before the exec, and emit it, which we don't want to do.
This commit introduces a map `s.pendingExecs` to keep track of the
number of pending execs keyed by container, which allows us to skip
processing exits for inits if there are pending execs, and instead
have the closure returned by `s.preStart` handle the init exit after
emitting the exec's exit.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
(cherry picked from commit 892dc54bd2)
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
Fixes ae7021300
In ae7021300 the WritePidFile and WriteAddress functions were
changed to use AtomicFile instead of os.CreateFile. However,
AtomicFile creates a temporary file and then changes its permissions
with os.Chmod which alters the previously observed behavior of
os.CreateFile which takes the system's umask into account.
This means that on Linux-based systems these files suddenly
became world writable (#9363). The address file has since been
removed, but pid-file was still created as world writable. This
commit explicitly requests 0644 permissions as even on systems
without default umask of 0022 there is no reason to have these
two files world writable.
Signed-off-by: Jaroslav Jindrak <dzejrou@gmail.com>
(cherry picked from commit 9d328410a5)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
```go
// Delete the initial process and container
func (s *Service) Delete(ctx context.Context, r *ptypes.Empty) (*shimapi.DeleteResponse, error) {
p, err := s.getInitProcess()
if err != nil {
return nil, err
}
if err := p.Delete(ctx); err != nil {
return nil, errdefs.ToGRPC(err)
}
// The client might canceled the request but the shim service still
// moved on. The `delete(s.processes, s.id)` was executed
// successfully. So the next Delete call will return `container
// must be created` error. The client side should ignore this
// issue.
s.mu.Lock()
delete(s.processes, s.id)
s.mu.Unlock()
s.platform.Close()
return &shimapi.DeleteResponse{
ExitStatus: uint32(p.ExitStatus()),
ExitedAt: protobuf.ToTimestamp(p.ExitedAt()),
Pid: uint32(p.Pid()),
}, nil
}
```
introduced by #9004fixes: #9309
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 449912857d)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
After pr #8617, create handler of containerd-shim-runc-v2 will
call handleStarted() to record the init process and handle its exit.
Init process wouldn't quit so early in normal circumstances. But if
this screnario occurs, handleStarted() will call
handleProcessExit(), which will cause deadlock because create() had
acquired s.mu, and handleProcessExit() will try to lock it again.
So, I added a parameter muLocked to handleStarted to indicate whether
or not s.mu is currently locked, and thus deciding whether or not to
lock it when calling handleProcessExit.
Fix: #9103
Signed-off-by: Chen Yiyang <cyyzero@qq.com>
(cherry picked from commit 68dd47ef70)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previous code has already called `getContainer()`, just pass it into
`s.getContainerPids` to reduce unnecessary lock and map lookup.
Signed-off-by: Chen Yiyang <cyyzero@qq.com>
(cherry picked from commit 6604ff6c55)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
eventSendMu is causing severe lock contention when multiple processes
start and exit concurrently. Replace it with a different scheme for
maintaining causality w.r.t. start and exit events for a process which
does not rely on big locks for synchronization.
Keep track of all processes for which a Task(Exec)Start event has been
published and have not yet exited in a map, keyed by their PID.
Processing exits then is as simple as looking up which process
corresponds to the PID. If there are no started processes known with
that PID, the PID must either belong to a process which was started by
s.Start() and before the s.Start() call has added the process to the map
of running processes, or a reparented process which we don't care about.
Handle the former case by having each s.Start() call subscribe to exit
events before starting the process. It checks if the PID has exited in
the time between it starting the process and publishing the TaskStart
event, handling the exit if it has. Exit events for reparented processes
received when no s.Start() calls are in flight are immediately
discarded, and events received during an s.Start() call are discarded
when the s.Start() call returns.
Co-authored-by: Laura Brehm <laurabrehm@hey.com>
Signed-off-by: Cory Snider <csnider@mirantis.com>
(cherry picked from commit 5cd6210ad0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
As we know, shim starts twice. The first time we execute the shim with the
`start` arguments, the `-debug` argument is passed to the shim
process correctly. But the second time we execute shim, the debug flag
is ignored.
Signed-off-by: wllenyj <wllenyj@linux.alibaba.com>
(cherry picked from commit f710505f4a)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Some changes to account for 91800c4e9a
and 894af07e37
runtime/v2/shim/shim.go:178:18: directive `//nolint:staticcheck // Ignore SA4023 as some platforms always return error` is unused for linter "staticcheck" (nolintlint)
if err != nil { //nolint:staticcheck // Ignore SA4023 as some platforms always return error
^
runtime/v2/shim/shim.go:264:18: directive `//nolint:staticcheck // Ignore SA4023 as some platforms always return error` is unused for linter "staticcheck" (nolintlint)
if err != nil { //nolint:staticcheck // Ignore SA4023 as some platforms always return error
^
runtime/v2/shim/shim.go:269:39: directive `//nolint:staticcheck // Ignore SA4023 as some platforms always return error` is unused for linter "staticcheck" (nolintlint)
if err := subreaper(); err != nil { //nolint:staticcheck // Ignore SA4023 as some platforms always return error
^
runtime/v2/shim/shim.go:448:67: directive `//nolint:staticcheck // Ignore SA4023 as some platforms always return error` is unused for linter "staticcheck" (nolintlint)
if err := serve(ctx, server, signals, sd.Shutdown); err != nil { //nolint:staticcheck // Ignore SA4023 as some platforms always return error
^
runtime/v2/shim/shim.go:480:18: directive `//nolint:staticcheck // Ignore SA4023 as some platforms always return error` is unused for linter "staticcheck" (nolintlint)
if err != nil { //nolint:staticcheck // Ignore SA4023 as some platforms always return error
^
integration/main_test.go:446:31: directive `//nolint:unused` is unused for linter "unused" (nolintlint)
func KillPid(pid int) error { //nolint:unused
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
If there is any unskipable error during setuping shim plugins, we should
fail return error to prevent from leaky shim instance. For example,
there is error during init task plugin, the shim ttrpc server will not
contain any shim API method. The any call to the shim will receive that
failed to create shim task: service containerd.task.v2.Task: not implemented
Then containerd can't use `Shutdown` to let the shim close. The shim
will be leaky. And also fail return if there is no ttrpc service.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit b297775eaf)
Signed-off-by: Qiutong Song <songqt01@gmail.com>
Currently, the runc shimv2 commandline manager doesn't support ttrpc
server's customized option, for example, the ttrpc server interceptor.
This commit is to allow the task plugin can return the
`UnaryServerInterceptor` option to the manager so that the task plugin
can do enhancement before handling the incoming request, like API-level
failpoint control.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 822cc51d84)
Signed-off-by: Qiutong Song <songqt01@gmail.com>
If containerd-shim-runc-v1 process dead abnormally, such as received
kill -s 9 signal, panic or other unkown reasons, the containerd-shim-runc-v1
server can not reap runc container and forward init process exit event.
This will lead the container leaked in dockerd. When shim dead, containerd
will clean dead shim, here read init process pid and forward exit event
with pid at the same time.
Related to: #6402
Signed-off-by: Jeff Zvier <zvier20@gmail.com>
Signed-off-by: Wei Fu <fuweid89@gmail.com>
(cherry picked from commit 7901f4b331)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
If containerd-shim-runc-v2 process dead abnormally, such as received
kill 9 signal, panic or other unkown reasons, the containerd-shim-runc-v2
server can not reap runc container and forward init process exit event.
This will lead the container leaked in dockerd. When shim dead, containerd
will clean dead shim, here read init process pid and forward exit event
with pid at the same time.
Signed-off-by: Jeff Zvier <zvier20@gmail.com>
(cherry picked from commit 356ca75757)
Signed-off-by: Wei Fu <fuweid89@gmail.com>
This causes sigint/sigterm to trigger a shutdown of the shim.
It is needed because otherwise the v2 shim hangs system shutdown.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
ShimV2 has shim.Delete command to cleanup task's temporary resource,
like bundle folder. Since the shim server exits and no persistent store
is for task's exit code, the result of shim.Delete is always 137 exit
code, like the task has been killed.
And the result of shim.Delete can be used as task event only when the
shim server is killed somehow after container is running. Therefore,
dockerd, which watches task exit event to update status of container,
can report correct status.
Back to the issue #6429, the container is not running because the
entrypoint is not found. Based on this design, we should not send
137 exitcode event to subscriber.
This commit is aimed to remove shim instance first and then the
`cleanupAfterDeadShim` should not send event.
Similar Issue: #4769Fix#6429
Signed-off-by: Wei Fu <fuweid89@gmail.com>
Sebastiaan van Stijn