mirror of
https://github.com/kubernetes/kubernetes.git
synced 2026-11-06 05:17:08 +00:00
kubelet: read OOM events in-tree and drop the cadvisor oomparser dependency
The kubelet used cadvisor's lib/utils/oomparser to watch the kernel log for OOM kills and to produce the container_oom_events_total metric. That made the kubelet depend on the cadvisor lib module for OOM handling. This moves OOM handling in-tree: - The OOM kill parser is forked from cadvisor into third_party/forked/cadvisor/oomparser. It reads /dev/kmsg and extracts OOM events, adapted to take a context.Context and log through klog. - The kmsg reader is forked from github.com/euank/go-kmsg-parser into third_party/forked/go-kmsg-parser/kmsgparser. - The kubelet OOM watcher reads OOM events through the forked parser and records the container_oom_events_total metric in-tree. - cadvisor/lib is bumped to v0.60.3, which no longer ships the OOM parser. This removes the transitive github.com/euank/go-kmsg-parser v2.0.0+incompatible dependency and the vendored cadvisor lib/utils/oomparser package. Behavior is unchanged. OOM events are still emitted and the metric still reports the same counts.
This commit is contained in:
18 files changed
+900
-516
No files matched your search
+190
@@ -0,0 +1,190 @@
|
||||
Copyright 2014 The cAdvisor Authors
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
Vendored
File renamed without changes.
@@ -29,7 +29,7 @@ require (
|
||||
github.com/go-logr/logr v1.4.3
|
||||
github.com/go-openapi/jsonreference v0.21.4
|
||||
github.com/godbus/dbus/v5 v5.2.2
|
||||
github.com/google/cadvisor/lib v0.60.2
|
||||
github.com/google/cadvisor/lib v0.60.3
|
||||
github.com/google/cel-go v0.27.0
|
||||
github.com/google/gnostic-models v0.7.0
|
||||
github.com/google/go-cmp v0.7.0
|
||||
@@ -142,7 +142,6 @@ require (
|
||||
github.com/coreos/go-semver v0.3.1 // indirect
|
||||
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/euank/go-kmsg-parser v2.0.0+incompatible // indirect
|
||||
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f // indirect
|
||||
github.com/fatih/camelcase v1.0.0 // indirect
|
||||
github.com/felixge/httpsnoop v1.0.4 // indirect
|
||||
|
||||
@@ -87,8 +87,6 @@ github.com/envoyproxy/go-control-plane v0.14.0/go.mod h1:NcS5X47pLl/hfqxU70yPwL9
|
||||
github.com/envoyproxy/go-control-plane/envoy v1.37.0/go.mod h1:DReE9MMrmecPy+YvQOAOHNYMALuowAnbjjEMkkWOi6A=
|
||||
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4=
|
||||
github.com/envoyproxy/protoc-gen-validate v1.3.3/go.mod h1:TsndJ/ngyIdQRhMcVVGDDHINPLWB7C82oDArY51KfB0=
|
||||
github.com/euank/go-kmsg-parser v2.0.0+incompatible h1:cHD53+PLQuuQyLZeriD1V/esuG4MuU0Pjs5y6iknohY=
|
||||
github.com/euank/go-kmsg-parser v2.0.0+incompatible/go.mod h1:MhmAMZ8V4CYH4ybgdRwPr2TU5ThnS43puaKEMpja1uw=
|
||||
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f h1:Wl78ApPPB2Wvf/TIe2xdyJxTlb6obmF18d8QdkxNDu4=
|
||||
github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f/go.mod h1:OSYXu++VVOHnXeitef/D8n/6y4QV8uLHSFXX4NeXMGc=
|
||||
github.com/fatih/camelcase v1.0.0 h1:hxNvNX/xYBp0ovncs8WyWZrOrpBNub/JfaMvbURyft8=
|
||||
@@ -165,8 +163,8 @@ github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
|
||||
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
|
||||
github.com/google/cadvisor/lib v0.60.2 h1:FwbbVJbrUtz7bfPVGoNwH9x0GptU5dTfqGGR1TnsMFM=
|
||||
github.com/google/cadvisor/lib v0.60.2/go.mod h1:UgBpDLDHEAq4SpDw2zmWMDtU1pNFMd9n68jt9SmbcuM=
|
||||
github.com/google/cadvisor/lib v0.60.3 h1:WTyANRHy5g0QZ9eMMf0jIqVBnLSX+HbNwifgO55hdCo=
|
||||
github.com/google/cadvisor/lib v0.60.3/go.mod h1:MWLbvpPnJSMU6g34UT28CDbvcC/bgZBLYhGcfeZREhw=
|
||||
github.com/google/cel-go v0.27.0 h1:e7ih85+4qVrBuqQWTW4FKSqZYokVuc3HnhH5keboFTo=
|
||||
github.com/google/cel-go v0.27.0/go.mod h1:tTJ11FWqnhw5KKpnWpvW9CJC3Y9GK4EIS0WXnBbebzw=
|
||||
github.com/google/gnostic-models v0.7.0 h1:qwTtogB15McXDaNqTZdzPJRHvaVJlAl+HVQnLmJEJxo=
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
/*
|
||||
Copyright 2025 The Kubernetes Authors.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package oom
|
||||
|
||||
import "sync"
|
||||
|
||||
// oomEventCounts holds the cumulative number of OOM kills observed per
|
||||
// container, keyed by the cgroup name. The OOM watcher
|
||||
// writes it and the kubelet metrics server reads it to generate the
|
||||
// container_oom_events_total metric.
|
||||
//
|
||||
// ponytail: process global because the OOM watcher is a singleton. If more than
|
||||
// one watcher ever exists, pass an explicit counter to NewWatcher and to the
|
||||
// metrics server instead.
|
||||
//
|
||||
// ponytail: entries are never deleted, so the map grows by one for each distinct
|
||||
// cgroup that ever OOMs. OOM kills are rare and cgroup names are unique per
|
||||
// container, so this is a slow leak, not stale metrics (the collector only emits
|
||||
// for live containers). Prune against the live container set if it matters.
|
||||
var oomEventCounts = struct {
|
||||
sync.Mutex
|
||||
byContainer map[string]uint64
|
||||
}{byContainer: make(map[string]uint64)}
|
||||
|
||||
// recordOOMKill increments the OOM kill counter for the given cgroup name.
|
||||
func recordOOMKill(containerName string) {
|
||||
oomEventCounts.Lock()
|
||||
defer oomEventCounts.Unlock()
|
||||
oomEventCounts.byContainer[containerName]++
|
||||
}
|
||||
|
||||
// OOMEventsForContainer returns the number of OOM kills observed for the given
|
||||
// cgroup name. It returns 0 for an unknown container and on
|
||||
// platforms where the OOM watcher does not run.
|
||||
func OOMEventsForContainer(containerName string) uint64 {
|
||||
oomEventCounts.Lock()
|
||||
defer oomEventCounts.Unlock()
|
||||
return oomEventCounts.byContainer[containerName]
|
||||
}
|
||||
@@ -26,12 +26,11 @@ import (
|
||||
"k8s.io/apimachinery/pkg/util/runtime"
|
||||
"k8s.io/client-go/tools/record"
|
||||
"k8s.io/klog/v2"
|
||||
|
||||
"github.com/google/cadvisor/lib/utils/oomparser"
|
||||
"k8s.io/kubernetes/third_party/forked/cadvisor/oomparser"
|
||||
)
|
||||
|
||||
type streamer interface {
|
||||
StreamOoms(chan<- *oomparser.OomInstance)
|
||||
StreamOoms(ctx context.Context, outStream chan<- *oomparser.OomInstance)
|
||||
}
|
||||
|
||||
var _ streamer = &oomparser.OomParser{}
|
||||
@@ -43,8 +42,8 @@ type realWatcher struct {
|
||||
|
||||
var _ Watcher = &realWatcher{}
|
||||
|
||||
// NewWatcher creates and initializes a OOMWatcher backed by Cadvisor as
|
||||
// the oom streamer.
|
||||
// NewWatcher creates and initializes an OOMWatcher backed by the kernel log
|
||||
// (/dev/kmsg) oom streamer.
|
||||
func NewWatcher(recorder record.EventRecorderLogger) (Watcher, error) {
|
||||
// for test purpose
|
||||
_, ok := recorder.(*record.FakeRecorder)
|
||||
@@ -73,13 +72,17 @@ const (
|
||||
// Start watches for system oom's and records an event for every system oom encountered.
|
||||
func (ow *realWatcher) Start(ctx context.Context, ref *v1.ObjectReference) error {
|
||||
outStream := make(chan *oomparser.OomInstance, 10)
|
||||
go ow.oomStreamer.StreamOoms(outStream)
|
||||
go ow.oomStreamer.StreamOoms(ctx, outStream)
|
||||
|
||||
go func() {
|
||||
logger := klog.FromContext(ctx)
|
||||
defer runtime.HandleCrashWithContext(ctx)
|
||||
|
||||
for event := range outStream {
|
||||
// Count every OOM kill per container to back the
|
||||
// container_oom_events_total metric.
|
||||
recordOOMKill(event.ContainerName)
|
||||
|
||||
if event.VictimContainerName == recordEventContainerName {
|
||||
logger.V(1).Info("Got sys oom event", "event", event)
|
||||
eventMsg := "System OOM encountered"
|
||||
|
||||
@@ -17,6 +17,7 @@ limitations under the License.
|
||||
package oom
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -24,8 +25,8 @@ import (
|
||||
v1 "k8s.io/api/core/v1"
|
||||
"k8s.io/client-go/tools/record"
|
||||
"k8s.io/kubernetes/test/utils/ktesting"
|
||||
"k8s.io/kubernetes/third_party/forked/cadvisor/oomparser"
|
||||
|
||||
"github.com/google/cadvisor/lib/utils/oomparser"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
@@ -34,9 +35,9 @@ type fakeStreamer struct {
|
||||
oomInstancesToStream []*oomparser.OomInstance
|
||||
}
|
||||
|
||||
func (fs *fakeStreamer) StreamOoms(outStream chan<- *oomparser.OomInstance) {
|
||||
for _, oomInstance := range fs.oomInstancesToStream {
|
||||
outStream <- oomInstance
|
||||
func (fs *fakeStreamer) StreamOoms(_ context.Context, outStream chan<- *oomparser.OomInstance) {
|
||||
for _, oi := range fs.oomInstancesToStream {
|
||||
outStream <- oi
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -95,6 +95,7 @@ import (
|
||||
kubeletcadvisor "k8s.io/kubernetes/pkg/kubelet/cadvisor"
|
||||
kubecontainer "k8s.io/kubernetes/pkg/kubelet/container"
|
||||
"k8s.io/kubernetes/pkg/kubelet/metrics/collectors"
|
||||
oomwatcher "k8s.io/kubernetes/pkg/kubelet/oom"
|
||||
"k8s.io/kubernetes/pkg/kubelet/prober"
|
||||
servermetrics "k8s.io/kubernetes/pkg/kubelet/server/metrics"
|
||||
"k8s.io/kubernetes/pkg/kubelet/server/stats"
|
||||
@@ -530,7 +531,7 @@ func (s *Server) InstallAuthNotRequiredHandlers(ctx context.Context) {
|
||||
Count: 1,
|
||||
Recursive: true,
|
||||
}
|
||||
r.RawMustRegister(metrics.NewPrometheusCollector(prometheusHostAdapter{s.host}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))
|
||||
r.RawMustRegister(metrics.NewPrometheusCollector(oomEventsHostAdapter{prometheusHostAdapter{s.host}}, containerPrometheusLabelsFunc(s.host), includedMetrics, clock.RealClock{}, cadvisorOpts))
|
||||
servermetrics.SetMetricsProvider(servermetrics.CAdvisorMetricsProvider)
|
||||
}
|
||||
s.restfulCont.Handle(cadvisorMetricsPath,
|
||||
@@ -1321,6 +1322,25 @@ func (a prometheusHostAdapter) GetMachineInfo() (*cadvisorapi.MachineInfo, error
|
||||
return a.host.GetCachedMachineInfo()
|
||||
}
|
||||
|
||||
// oomEventsHostAdapter wraps prometheusHostAdapter and replaces the per-container
|
||||
// OOM event count with the count maintained by the kubelet OOM watcher
|
||||
// (pkg/kubelet/oom), so that container_oom_events_total is generated by the
|
||||
// kubelet rather than by cadvisor.
|
||||
type oomEventsHostAdapter struct {
|
||||
prometheusHostAdapter
|
||||
}
|
||||
|
||||
func (a oomEventsHostAdapter) GetRequestedContainersInfo(containerName string, options cadvisorapi.RequestOptions) (map[string]*cadvisorapi.ContainerInfo, error) {
|
||||
infos, err := a.prometheusHostAdapter.GetRequestedContainersInfo(containerName, options)
|
||||
for _, info := range infos {
|
||||
count := oomwatcher.OOMEventsForContainer(info.Name)
|
||||
for _, stats := range info.Stats {
|
||||
stats.OOMEvents = count
|
||||
}
|
||||
}
|
||||
return infos, err
|
||||
}
|
||||
|
||||
func containerPrometheusLabelsFunc(s stats.Provider) metrics.ContainerLabelsFunc {
|
||||
// containerPrometheusLabels maps cAdvisor labels to prometheus labels.
|
||||
return func(c *cadvisorapi.ContainerInfo) map[string]string {
|
||||
|
||||
Vendored
+190
@@ -0,0 +1,190 @@
|
||||
Copyright 2014 The cAdvisor Authors
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name) to the interfaces of,
|
||||
the Work and Derivative Works thereof.
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
@@ -0,0 +1,164 @@
|
||||
//go:build linux
|
||||
|
||||
/*
|
||||
Copyright 2014 Google Inc. All Rights Reserved.
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
// Package oomparser is forked from github.com/google/cadvisor (utils/oomparser).
|
||||
// It reads the kernel ring buffer (/dev/kmsg) and extracts OOM kill events. It
|
||||
// is adapted to take a context.Context, log through klog, and read the kernel
|
||||
// log through the forked go-kmsg-parser in this tree.
|
||||
package oomparser
|
||||
|
||||
import (
|
||||
"context"
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"k8s.io/klog/v2"
|
||||
"k8s.io/kubernetes/third_party/forked/go-kmsg-parser/kmsgparser"
|
||||
)
|
||||
|
||||
var (
|
||||
legacyContainerRegexp = regexp.MustCompile(`Task in (.*) killed as a result of limit of (.*)`)
|
||||
// Starting in 5.0 linux kernels, the OOM message changed.
|
||||
containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)
|
||||
lastLineRegexp = regexp.MustCompile(`Killed process ([0-9]+) \((.+)\)`)
|
||||
firstLineRegexp = regexp.MustCompile(`invoked oom-killer:`)
|
||||
)
|
||||
|
||||
// OomInstance holds the information extracted from a single kernel OOM kill.
|
||||
type OomInstance struct {
|
||||
// process id of the killed process
|
||||
Pid int
|
||||
// the name of the killed process
|
||||
ProcessName string
|
||||
// the time the process was reported killed, accurate to the minute
|
||||
TimeOfDeath time.Time
|
||||
// the absolute name of the container that OOMed
|
||||
ContainerName string
|
||||
// the absolute name of the container that was killed due to the OOM
|
||||
VictimContainerName string
|
||||
// the constraint that triggered the OOM (CONSTRAINT_NONE, CONSTRAINT_CPUSET,
|
||||
// CONSTRAINT_MEMORY_POLICY, CONSTRAINT_MEMCG)
|
||||
Constraint string
|
||||
}
|
||||
|
||||
func getLegacyContainerName(line string, o *OomInstance) {
|
||||
parsed := legacyContainerRegexp.FindStringSubmatch(line)
|
||||
if parsed == nil {
|
||||
return
|
||||
}
|
||||
o.ContainerName = path.Join("/", parsed[1])
|
||||
o.VictimContainerName = path.Join("/", parsed[2])
|
||||
}
|
||||
|
||||
// getContainerName fills the container fields from a line. It returns true when
|
||||
// the line was the modern oom-kill line that also carries the pid and task name.
|
||||
func getContainerName(line string, o *OomInstance) (bool, error) {
|
||||
parsed := containerRegexp.FindStringSubmatch(line)
|
||||
if parsed == nil {
|
||||
// Fall back to the pre-5.0 kernel format.
|
||||
getLegacyContainerName(line, o)
|
||||
return false, nil
|
||||
}
|
||||
o.ContainerName = parsed[6]
|
||||
o.VictimContainerName = parsed[5]
|
||||
o.Constraint = parsed[1]
|
||||
pid, err := strconv.Atoi(parsed[8])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
o.Pid = pid
|
||||
o.ProcessName = parsed[7]
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// getProcessNamePid fills the pid and process name from the "Killed process" line.
|
||||
func getProcessNamePid(line string, o *OomInstance) (bool, error) {
|
||||
parsed := lastLineRegexp.FindStringSubmatch(line)
|
||||
if parsed == nil {
|
||||
return false, nil
|
||||
}
|
||||
pid, err := strconv.Atoi(parsed[1])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
o.Pid = pid
|
||||
o.ProcessName = parsed[2]
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func checkIfStartOfOomMessages(line string) bool {
|
||||
return firstLineRegexp.MatchString(line)
|
||||
}
|
||||
|
||||
// OomParser reads the kernel ring buffer (/dev/kmsg) and extracts OOM events.
|
||||
type OomParser struct {
|
||||
parser kmsgparser.Parser
|
||||
}
|
||||
|
||||
// New creates an OomParser backed by the kernel log (/dev/kmsg).
|
||||
func New() (*OomParser, error) {
|
||||
parser, err := kmsgparser.NewParser()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &OomParser{parser: parser}, nil
|
||||
}
|
||||
|
||||
// StreamOoms writes an OomInstance to outStream for every OOM event found in the
|
||||
// kernel log. It blocks and should be called from a goroutine.
|
||||
func (p *OomParser) StreamOoms(ctx context.Context, outStream chan<- *OomInstance) {
|
||||
logger := klog.FromContext(ctx)
|
||||
kmsgEntries := make(chan kmsgparser.Message, 32)
|
||||
go func() {
|
||||
defer func() { _ = p.parser.Close() }()
|
||||
if err := p.parser.Parse(kmsgEntries); err != nil {
|
||||
logger.Error(err, "Exiting kmsg parse; OOM events will not be reported")
|
||||
}
|
||||
}()
|
||||
|
||||
for msg := range kmsgEntries {
|
||||
if !checkIfStartOfOomMessages(msg.Message) {
|
||||
continue
|
||||
}
|
||||
oom := &OomInstance{
|
||||
ContainerName: "/",
|
||||
VictimContainerName: "/",
|
||||
TimeOfDeath: msg.Timestamp,
|
||||
}
|
||||
for msg := range kmsgEntries {
|
||||
finished, err := getContainerName(msg.Message, oom)
|
||||
if err != nil {
|
||||
logger.Error(err, "Failed to parse OOM container name")
|
||||
}
|
||||
if !finished {
|
||||
finished, err = getProcessNamePid(msg.Message, oom)
|
||||
if err != nil {
|
||||
logger.Error(err, "Failed to parse OOM process name and pid")
|
||||
}
|
||||
}
|
||||
if finished {
|
||||
oom.TimeOfDeath = msg.Timestamp
|
||||
break
|
||||
}
|
||||
}
|
||||
outStream <- oom
|
||||
}
|
||||
logger.Error(nil, "Stopped receiving OOM notifications; OOM events will not be reported")
|
||||
}
|
||||
Vendored
-4
@@ -1,5 +1,3 @@
|
||||
= vendor/github.com/euank/go-kmsg-parser licensed under: =
|
||||
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
@@ -201,5 +199,3 @@
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
|
||||
= vendor/github.com/euank/go-kmsg-parser/LICENSE e3fc50a88d0a364313df4b21ef20c29e
|
||||
@@ -0,0 +1,261 @@
|
||||
/*
|
||||
Copyright 2016 Euan Kemp
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
// Package kmsgparser implements a parser for the Linux `/dev/kmsg` format.
|
||||
// More information about this format may be found here:
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
// Some parts of it are slightly inspired by rsyslog's contrib module:
|
||||
// https://github.com/rsyslog/rsyslog/blob/v8.22.0/contrib/imkmsg/kmsg.c
|
||||
//
|
||||
// This package is forked from github.com/euank/go-kmsg-parser/v3 at v3.0.0:
|
||||
// https://github.com/euank/go-kmsg-parser/blob/v3.0.0/kmsgparser/kmsgparser.go
|
||||
// The upstream Logger abstraction (log.go) is dropped; the parser logs through
|
||||
// klog instead.
|
||||
package kmsgparser
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"k8s.io/klog/v2"
|
||||
)
|
||||
|
||||
// Parser is a parser for the kernel ring buffer found at /dev/kmsg
|
||||
type Parser interface {
|
||||
// SeekEnd moves the parser to the end of the kmsg queue.
|
||||
SeekEnd() error
|
||||
// Parse reads from kmsg and provides a channel of messages.
|
||||
// Parse will always close the provided channel before returning.
|
||||
// Parse may be canceled by calling 'Close' on the parser.
|
||||
//
|
||||
// The caller should drain the channel after calling [Close]. The caller must
|
||||
// not close the channel passed in.
|
||||
Parse(chan<- Message) error
|
||||
|
||||
Close() error
|
||||
}
|
||||
|
||||
// Message represents a given kmsg logline, including its timestamp (as
|
||||
// calculated based on offset from boot time), its possibly multi-line body,
|
||||
// and so on. More information about these mssages may be found here:
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
type Message struct {
|
||||
Priority int
|
||||
SequenceNumber int
|
||||
Timestamp time.Time
|
||||
Message string
|
||||
}
|
||||
|
||||
// NewParser constructs a new parser with the given Options.
|
||||
func NewParser(opts ...Option) (Parser, error) {
|
||||
f, err := os.Open("/dev/kmsg")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
bootTime, err := getBootTime()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
p := &parser{
|
||||
kmsgReader: f,
|
||||
bootTime: bootTime,
|
||||
follow: true,
|
||||
}
|
||||
for _, opt := range opts {
|
||||
opt(p)
|
||||
}
|
||||
return p, nil
|
||||
}
|
||||
|
||||
// Option is a configuration option for [NewParser]
|
||||
type Option func(p *parser)
|
||||
|
||||
// WithNoFollow configures [Parser] to stop reading and close the channel when
|
||||
// it reaches the end of the kmsg buffer.
|
||||
func WithNoFollow() func(p *parser) {
|
||||
return func(p *parser) {
|
||||
p.follow = false
|
||||
}
|
||||
}
|
||||
|
||||
type parser struct {
|
||||
kmsgReader *os.File
|
||||
bootTime time.Time
|
||||
// follow indicates whether we should stop when we hit the end of the kmsg
|
||||
// buffer, or keep reading. Similar to dmesg --follow.
|
||||
// Default true
|
||||
follow bool
|
||||
}
|
||||
|
||||
func getBootTime() (time.Time, error) {
|
||||
var sysinfo syscall.Sysinfo_t
|
||||
err := syscall.Sysinfo(&sysinfo)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("could not get boot time: %v", err)
|
||||
}
|
||||
// sysinfo only has seconds
|
||||
return time.Now().Add(-1 * (time.Duration(sysinfo.Uptime) * time.Second)), nil
|
||||
}
|
||||
|
||||
func (p *parser) SeekEnd() error {
|
||||
_, err := p.kmsgReader.Seek(0, io.SeekEnd)
|
||||
return err
|
||||
}
|
||||
|
||||
func (p *parser) Close() error {
|
||||
return p.kmsgReader.Close()
|
||||
}
|
||||
|
||||
// Parse reads from the provided reader and provide a channel of messages
|
||||
// parsed.
|
||||
// If the provided reader *is not* a proper Linux kmsg device, Parse might not
|
||||
// behave correctly since it relies on specific behavior of `/dev/kmsg`
|
||||
//
|
||||
// The caller should typically run 'Parse' in a goroutine.
|
||||
func (p *parser) Parse(msgs chan<- Message) error {
|
||||
// with follow (dmesg --follow), we can use go's usual way of reading thing (i.e. epoll + nonblocking IO).
|
||||
if p.follow {
|
||||
return p.readFollow(msgs)
|
||||
}
|
||||
return p.readNofollow(msgs)
|
||||
}
|
||||
|
||||
func (p *parser) readFollow(msgs chan<- Message) error {
|
||||
defer close(msgs)
|
||||
msg := make([]byte, 8192)
|
||||
for {
|
||||
// Each read call gives us one full message.
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
n, err := p.kmsgReader.Read(msg)
|
||||
switch {
|
||||
case err == nil:
|
||||
case errors.Is(err, syscall.EPIPE):
|
||||
klog.Warningf("short read from kmsg; skipping")
|
||||
continue
|
||||
case errors.Is(err, io.EOF), errors.Is(err, os.ErrClosed):
|
||||
// someone closed us
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("unexpected kmsg reading error: %w", err)
|
||||
}
|
||||
msgStr := string(msg[:n])
|
||||
message, err := p.parseMessage(msgStr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("malformed kmsg message: %w", err)
|
||||
}
|
||||
msgs <- message
|
||||
}
|
||||
}
|
||||
|
||||
func (p *parser) readNofollow(msgs chan<- Message) error {
|
||||
defer close(msgs)
|
||||
|
||||
rawReader, err := p.kmsgReader.SyscallConn()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// we're forced to put the fd into non-blocking mode to be able to detect the
|
||||
// end of the buffer, but to not use go's built-in epoll
|
||||
if ctrlErr := rawReader.Control(func(fd uintptr) {
|
||||
err = syscall.SetNonblock(int(fd), true)
|
||||
}); ctrlErr != nil {
|
||||
return fmt.Errorf("error calling control on kmsg reader: %w", ctrlErr)
|
||||
}
|
||||
if err != nil {
|
||||
return fmt.Errorf("unable to set nonblocking on fd: %w", err)
|
||||
}
|
||||
msg := make([]byte, 8192)
|
||||
for {
|
||||
// Each read call gives us one full message.
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
var err error
|
||||
var n int
|
||||
readErr := rawReader.Read(func(fd uintptr) bool {
|
||||
n, err = syscall.Read(int(fd), msg)
|
||||
return true
|
||||
})
|
||||
if readErr != nil && err == nil {
|
||||
err = readErr
|
||||
}
|
||||
switch {
|
||||
case err == nil:
|
||||
case errors.Is(err, syscall.EPIPE):
|
||||
klog.Warningf("short read from kmsg; skipping")
|
||||
continue
|
||||
case errors.Is(err, syscall.EAGAIN):
|
||||
// end of ring buffer in nofollow mode, we're done
|
||||
return nil
|
||||
default:
|
||||
return fmt.Errorf("unexpected kmsg reading error: %w", err)
|
||||
}
|
||||
msgStr := string(msg[:n])
|
||||
|
||||
message, err := p.parseMessage(msgStr)
|
||||
if err != nil {
|
||||
return fmt.Errorf("malformed kmsg message: %w", err)
|
||||
}
|
||||
msgs <- message
|
||||
}
|
||||
}
|
||||
|
||||
func (p *parser) parseMessage(input string) (Message, error) {
|
||||
// Format:
|
||||
// PRIORITY,SEQUENCE_NUM,TIMESTAMP,-;MESSAGE
|
||||
parts := strings.SplitN(input, ";", 2)
|
||||
if len(parts) != 2 {
|
||||
return Message{}, fmt.Errorf("invalid kmsg; must contain a ';'")
|
||||
}
|
||||
|
||||
metadata, message := parts[0], parts[1]
|
||||
|
||||
metadataParts := strings.Split(metadata, ",")
|
||||
if len(metadataParts) < 3 {
|
||||
return Message{}, fmt.Errorf("invalid kmsg: must contain at least 3 ',' separated pieces at the start")
|
||||
}
|
||||
|
||||
priority, sequence, timestamp := metadataParts[0], metadataParts[1], metadataParts[2]
|
||||
|
||||
prioNum, err := strconv.Atoi(priority)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as priority: %v", priority, err)
|
||||
}
|
||||
|
||||
sequenceNum, err := strconv.Atoi(sequence)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as sequence number: %v", priority, err)
|
||||
}
|
||||
|
||||
timestampUsFromBoot, err := strconv.ParseInt(timestamp, 10, 64)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as timestamp: %v", priority, err)
|
||||
}
|
||||
// timestamp is offset in microsecond from boottime.
|
||||
msgTime := p.bootTime.Add(time.Duration(timestampUsFromBoot) * time.Microsecond)
|
||||
|
||||
return Message{
|
||||
Priority: prioNum,
|
||||
SequenceNumber: sequenceNum,
|
||||
Timestamp: msgTime,
|
||||
Message: message,
|
||||
}, nil
|
||||
}
|
||||
-200
@@ -1,200 +0,0 @@
|
||||
/*
|
||||
Copyright 2016 Euan Kemp
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
// Package kmsgparser implements a parser for the Linux `/dev/kmsg` format.
|
||||
// More information about this format may be found here:
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
// Some parts of it are slightly inspired by rsyslog's contrib module:
|
||||
// https://github.com/rsyslog/rsyslog/blob/v8.22.0/contrib/imkmsg/kmsg.c
|
||||
package kmsgparser
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Parser is a parser for the kernel ring buffer found at /dev/kmsg
|
||||
type Parser interface {
|
||||
// SeekEnd moves the parser to the end of the kmsg queue.
|
||||
SeekEnd() error
|
||||
// Parse provides a channel of messages read from the kernel ring buffer.
|
||||
// When first called, it will read the existing ringbuffer, after which it will emit new messages as they occur.
|
||||
Parse() <-chan Message
|
||||
// SetLogger sets the logger that will be used to report malformed kernel
|
||||
// ringbuffer lines or unexpected kmsg read errors.
|
||||
SetLogger(Logger)
|
||||
// Close closes the underlying kmsg reader for this parser
|
||||
Close() error
|
||||
}
|
||||
|
||||
// Message represents a given kmsg logline, including its timestamp (as
|
||||
// calculated based on offset from boot time), its possibly multi-line body,
|
||||
// and so on. More information about these mssages may be found here:
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
type Message struct {
|
||||
Priority int
|
||||
SequenceNumber int
|
||||
Timestamp time.Time
|
||||
Message string
|
||||
}
|
||||
|
||||
func NewParser() (Parser, error) {
|
||||
f, err := os.Open("/dev/kmsg")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
bootTime, err := getBootTime()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &parser{
|
||||
log: &StandardLogger{nil},
|
||||
kmsgReader: f,
|
||||
bootTime: bootTime,
|
||||
}, nil
|
||||
}
|
||||
|
||||
type ReadSeekCloser interface {
|
||||
io.ReadCloser
|
||||
io.Seeker
|
||||
}
|
||||
|
||||
type parser struct {
|
||||
log Logger
|
||||
kmsgReader ReadSeekCloser
|
||||
bootTime time.Time
|
||||
}
|
||||
|
||||
func getBootTime() (time.Time, error) {
|
||||
var sysinfo syscall.Sysinfo_t
|
||||
err := syscall.Sysinfo(&sysinfo)
|
||||
if err != nil {
|
||||
return time.Time{}, fmt.Errorf("could not get boot time: %v", err)
|
||||
}
|
||||
// sysinfo only has seconds
|
||||
return time.Now().Add(-1 * (time.Duration(sysinfo.Uptime) * time.Second)), nil
|
||||
}
|
||||
|
||||
func (p *parser) SetLogger(log Logger) {
|
||||
p.log = log
|
||||
}
|
||||
|
||||
func (p *parser) Close() error {
|
||||
return p.kmsgReader.Close()
|
||||
}
|
||||
|
||||
func (p *parser) SeekEnd() error {
|
||||
_, err := p.kmsgReader.Seek(0, os.SEEK_END)
|
||||
return err
|
||||
}
|
||||
|
||||
// Parse will read from the provided reader and provide a channel of messages
|
||||
// parsed.
|
||||
// If the provided reader *is not* a proper Linux kmsg device, Parse might not
|
||||
// behave correctly since it relies on specific behavior of `/dev/kmsg`
|
||||
//
|
||||
// A goroutine is created to process the provided reader. The goroutine will
|
||||
// exit when the given reader is closed.
|
||||
// Closing the passed in reader will cause the goroutine to exit.
|
||||
func (p *parser) Parse() <-chan Message {
|
||||
|
||||
output := make(chan Message, 1)
|
||||
|
||||
go func() {
|
||||
defer close(output)
|
||||
msg := make([]byte, 8192)
|
||||
for {
|
||||
// Each read call gives us one full message.
|
||||
// https://www.kernel.org/doc/Documentation/ABI/testing/dev-kmsg
|
||||
n, err := p.kmsgReader.Read(msg)
|
||||
if err != nil {
|
||||
if err == syscall.EPIPE {
|
||||
p.log.Warningf("short read from kmsg; skipping")
|
||||
continue
|
||||
}
|
||||
|
||||
if err == io.EOF {
|
||||
p.log.Infof("kmsg reader closed, shutting down")
|
||||
return
|
||||
}
|
||||
|
||||
p.log.Errorf("error reading /dev/kmsg: %v", err)
|
||||
return
|
||||
}
|
||||
|
||||
msgStr := string(msg[:n])
|
||||
|
||||
message, err := p.parseMessage(msgStr)
|
||||
if err != nil {
|
||||
p.log.Warningf("unable to parse kmsg message %q: %v", msgStr, err)
|
||||
continue
|
||||
}
|
||||
|
||||
output <- message
|
||||
}
|
||||
}()
|
||||
|
||||
return output
|
||||
}
|
||||
|
||||
func (p *parser) parseMessage(input string) (Message, error) {
|
||||
// Format:
|
||||
// PRIORITY,SEQUENCE_NUM,TIMESTAMP,-;MESSAGE
|
||||
parts := strings.SplitN(input, ";", 2)
|
||||
if len(parts) != 2 {
|
||||
return Message{}, fmt.Errorf("invalid kmsg; must contain a ';'")
|
||||
}
|
||||
|
||||
metadata, message := parts[0], parts[1]
|
||||
|
||||
metadataParts := strings.Split(metadata, ",")
|
||||
if len(metadataParts) < 3 {
|
||||
return Message{}, fmt.Errorf("invalid kmsg: must contain at least 3 ',' separated pieces at the start")
|
||||
}
|
||||
|
||||
priority, sequence, timestamp := metadataParts[0], metadataParts[1], metadataParts[2]
|
||||
|
||||
prioNum, err := strconv.Atoi(priority)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as priority: %v", priority, err)
|
||||
}
|
||||
|
||||
sequenceNum, err := strconv.Atoi(sequence)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as sequence number: %v", priority, err)
|
||||
}
|
||||
|
||||
timestampUsFromBoot, err := strconv.ParseInt(timestamp, 10, 64)
|
||||
if err != nil {
|
||||
return Message{}, fmt.Errorf("could not parse %q as timestamp: %v", priority, err)
|
||||
}
|
||||
// timestamp is offset in microsecond from boottime.
|
||||
msgTime := p.bootTime.Add(time.Duration(timestampUsFromBoot) * time.Microsecond)
|
||||
|
||||
return Message{
|
||||
Priority: prioNum,
|
||||
SequenceNumber: sequenceNum,
|
||||
Timestamp: msgTime,
|
||||
Message: message,
|
||||
}, nil
|
||||
}
|
||||
-55
@@ -1,55 +0,0 @@
|
||||
/*
|
||||
Copyright 2016 Euan Kemp
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
*/
|
||||
|
||||
package kmsgparser
|
||||
|
||||
import stdlog "log"
|
||||
|
||||
// Logger is a glog compatible logging interface
|
||||
// The StandardLogger struct can be used to wrap a log.Logger from the golang
|
||||
// "log" package to create a standard a logger fulfilling this interface as
|
||||
// well.
|
||||
type Logger interface {
|
||||
Warningf(string, ...interface{})
|
||||
Infof(string, ...interface{})
|
||||
Errorf(string, ...interface{})
|
||||
}
|
||||
|
||||
// StandardLogger adapts the "log" package's Logger interface to be a Logger
|
||||
type StandardLogger struct {
|
||||
*stdlog.Logger
|
||||
}
|
||||
|
||||
func (s *StandardLogger) Warningf(fmt string, args ...interface{}) {
|
||||
if s.Logger == nil {
|
||||
return
|
||||
}
|
||||
s.Logger.Printf("[WARNING] "+fmt, args)
|
||||
}
|
||||
|
||||
func (s *StandardLogger) Infof(fmt string, args ...interface{}) {
|
||||
if s.Logger == nil {
|
||||
return
|
||||
}
|
||||
s.Logger.Printf("[INFO] "+fmt, args)
|
||||
}
|
||||
|
||||
func (s *StandardLogger) Errorf(fmt string, args ...interface{}) {
|
||||
if s.Logger == nil {
|
||||
return
|
||||
}
|
||||
s.Logger.Printf("[INFO] "+fmt, args)
|
||||
}
|
||||
-3
@@ -68,7 +68,6 @@ func (t *atomicTime) Time() time.Time {
|
||||
}
|
||||
|
||||
type containerData struct {
|
||||
oomEvents uint64
|
||||
handler container.ContainerHandler
|
||||
info containerInfo
|
||||
memoryCache *memory.InMemoryCache
|
||||
@@ -471,8 +470,6 @@ func (cd *containerData) updateStats() error {
|
||||
}
|
||||
}
|
||||
|
||||
stats.OOMEvents = atomic.LoadUint64(&cd.oomEvents)
|
||||
|
||||
if cd.summaryReader != nil {
|
||||
if err := cd.summaryReader.AddSample(*stats); err != nil {
|
||||
// Ignore summary errors for now.
|
||||
|
||||
+3
-56
@@ -25,7 +25,6 @@ import (
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/google/cadvisor/lib/cache/memory"
|
||||
@@ -35,7 +34,6 @@ import (
|
||||
"github.com/google/cadvisor/lib/machine"
|
||||
info "github.com/google/cadvisor/lib/model"
|
||||
"github.com/google/cadvisor/lib/stats"
|
||||
"github.com/google/cadvisor/lib/utils/oomparser"
|
||||
"github.com/google/cadvisor/lib/utils/sysfs"
|
||||
"github.com/google/cadvisor/lib/version"
|
||||
"github.com/google/cadvisor/lib/watcher"
|
||||
@@ -149,9 +147,9 @@ type Manager interface {
|
||||
// Get the list of processes running in a container.
|
||||
GetProcessList(containerName string, options info.RequestOptions) ([]info.ProcessInfo, error)
|
||||
|
||||
// SetEventSink wires the sink that receives container lifecycle and OOM
|
||||
// events. The full binary injects its events manager; the kubelet does not
|
||||
// call this (no events are emitted). See events.go.
|
||||
// SetEventSink wires the sink that receives container lifecycle events. The
|
||||
// full binary injects its events manager; the kubelet does not call this (no
|
||||
// events are emitted). See events.go.
|
||||
SetEventSink(sink EventSink)
|
||||
}
|
||||
|
||||
@@ -339,11 +337,6 @@ func (m *manager) Start() error {
|
||||
}
|
||||
m.containerWatchers = append(m.containerWatchers, rawWatcher)
|
||||
|
||||
// Watch for OOMs.
|
||||
if err := m.watchForNewOoms(); err != nil {
|
||||
klog.Warningf("Could not configure a source for OOM detection, disabling OOM events: %v", err)
|
||||
}
|
||||
|
||||
// If there are no factories, don't start any housekeeping and serve the information we do have.
|
||||
if !container.HasFactories() {
|
||||
return nil
|
||||
@@ -646,52 +639,6 @@ func (m *manager) GetRequestedContainersInfo(containerName string, options info.
|
||||
return containersMap, errs.OrNil()
|
||||
}
|
||||
|
||||
// watchForNewOoms feeds a per-container OOM-kill counter from the kernel log via
|
||||
// oomparser. It is an engine-level async router (NOT a per-tick StatsAugmenter):
|
||||
// OOM events arrive keyed by container name and are matched to the registry here.
|
||||
// No event stream — only the counter that backs container_oom_events_total.
|
||||
func (m *manager) watchForNewOoms() error {
|
||||
outStream := make(chan *oomparser.OomInstance, 10)
|
||||
oomLog, err := oomparser.New()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
go oomLog.StreamOoms(outStream)
|
||||
|
||||
go func() {
|
||||
for oomInstance := range outStream {
|
||||
// Surface OOM and OOM-kill events to the sink (no-op if unset).
|
||||
m.addEvent(&info.Event{
|
||||
ContainerName: oomInstance.ContainerName,
|
||||
Timestamp: oomInstance.TimeOfDeath,
|
||||
EventType: info.EventOom,
|
||||
})
|
||||
m.addEvent(&info.Event{
|
||||
ContainerName: oomInstance.VictimContainerName,
|
||||
Timestamp: oomInstance.TimeOfDeath,
|
||||
EventType: info.EventOomKill,
|
||||
EventData: info.EventData{
|
||||
OomKill: &info.OomKillEventData{
|
||||
Pid: oomInstance.Pid,
|
||||
ProcessName: oomInstance.ProcessName,
|
||||
Constraint: oomInstance.Constraint,
|
||||
},
|
||||
},
|
||||
})
|
||||
|
||||
// Count OOM events for later collection by prometheus.
|
||||
conts, err := m.getRequestedContainers(oomInstance.ContainerName, info.RequestOptions{IdType: info.TypeName, Count: 1})
|
||||
if err != nil || len(conts) != 1 {
|
||||
continue
|
||||
}
|
||||
for _, cont := range conts {
|
||||
atomic.AddUint64(&cont.oomEvents, 1)
|
||||
}
|
||||
}
|
||||
}()
|
||||
return nil
|
||||
}
|
||||
|
||||
func (m *manager) getRequestedContainers(containerName string, options info.RequestOptions) (map[string]*containerData, error) {
|
||||
containersMap := make(map[string]*containerData)
|
||||
switch options.IdType {
|
||||
|
||||
-176
@@ -1,176 +0,0 @@
|
||||
// Copyright 2014 Google Inc. All Rights Reserved.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//go:build linux
|
||||
|
||||
package oomparser
|
||||
|
||||
import (
|
||||
"path"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"github.com/euank/go-kmsg-parser/kmsgparser"
|
||||
|
||||
"k8s.io/klog/v2"
|
||||
)
|
||||
|
||||
var (
|
||||
legacyContainerRegexp = regexp.MustCompile(`Task in (.*) killed as a result of limit of (.*)`)
|
||||
// Starting in 5.0 linux kernels, the OOM message changed
|
||||
containerRegexp = regexp.MustCompile(`oom-kill:constraint=(.*),nodemask=(.*),cpuset=(.*),mems_allowed=(.*),oom_memcg=(.*),task_memcg=(.*),task=(.*),pid=(.*),uid=(.*)`)
|
||||
lastLineRegexp = regexp.MustCompile(`Killed process ([0-9]+) \((.+)\)`)
|
||||
firstLineRegexp = regexp.MustCompile(`invoked oom-killer:`)
|
||||
)
|
||||
|
||||
// OomParser wraps a kmsgparser in order to extract OOM events from the
|
||||
// individual kernel ring buffer messages.
|
||||
type OomParser struct {
|
||||
parser kmsgparser.Parser
|
||||
}
|
||||
|
||||
// struct that contains information related to an OOM kill instance
|
||||
type OomInstance struct {
|
||||
// process id of the killed process
|
||||
Pid int
|
||||
// the name of the killed process
|
||||
ProcessName string
|
||||
// the time that the process was reported to be killed,
|
||||
// accurate to the minute
|
||||
TimeOfDeath time.Time
|
||||
// the absolute name of the container that OOMed
|
||||
ContainerName string
|
||||
// the absolute name of the container that was killed
|
||||
// due to the OOM.
|
||||
VictimContainerName string
|
||||
// the constraint that triggered the OOM. One of CONSTRAINT_NONE,
|
||||
// CONSTRAINT_CPUSET, CONSTRAINT_MEMORY_POLICY, CONSTRAINT_MEMCG
|
||||
Constraint string
|
||||
}
|
||||
|
||||
// gets the container name from a line and adds it to the oomInstance.
|
||||
func getLegacyContainerName(line string, currentOomInstance *OomInstance) error {
|
||||
parsedLine := legacyContainerRegexp.FindStringSubmatch(line)
|
||||
if parsedLine == nil {
|
||||
return nil
|
||||
}
|
||||
currentOomInstance.ContainerName = path.Join("/", parsedLine[1])
|
||||
currentOomInstance.VictimContainerName = path.Join("/", parsedLine[2])
|
||||
return nil
|
||||
}
|
||||
|
||||
// gets the container name from a line and adds it to the oomInstance.
|
||||
func getContainerName(line string, currentOomInstance *OomInstance) (bool, error) {
|
||||
parsedLine := containerRegexp.FindStringSubmatch(line)
|
||||
if parsedLine == nil {
|
||||
// Fall back to the legacy format if it isn't found here.
|
||||
return false, getLegacyContainerName(line, currentOomInstance)
|
||||
}
|
||||
currentOomInstance.ContainerName = parsedLine[6]
|
||||
currentOomInstance.VictimContainerName = parsedLine[5]
|
||||
currentOomInstance.Constraint = parsedLine[1]
|
||||
pid, err := strconv.Atoi(parsedLine[8])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
currentOomInstance.Pid = pid
|
||||
currentOomInstance.ProcessName = parsedLine[7]
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// gets the pid, name, and date from a line and adds it to oomInstance
|
||||
func getProcessNamePid(line string, currentOomInstance *OomInstance) (bool, error) {
|
||||
reList := lastLineRegexp.FindStringSubmatch(line)
|
||||
|
||||
if reList == nil {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
pid, err := strconv.Atoi(reList[1])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
currentOomInstance.Pid = pid
|
||||
currentOomInstance.ProcessName = reList[2]
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// uses regex to see if line is the start of a kernel oom log
|
||||
func checkIfStartOfOomMessages(line string) bool {
|
||||
potentialOomStart := firstLineRegexp.MatchString(line)
|
||||
return potentialOomStart
|
||||
}
|
||||
|
||||
// StreamOoms writes to a provided a stream of OomInstance objects representing
|
||||
// OOM events that are found in the logs.
|
||||
// It will block and should be called from a goroutine.
|
||||
func (p *OomParser) StreamOoms(outStream chan<- *OomInstance) {
|
||||
kmsgEntries := p.parser.Parse()
|
||||
defer p.parser.Close()
|
||||
|
||||
for msg := range kmsgEntries {
|
||||
isOomMessage := checkIfStartOfOomMessages(msg.Message)
|
||||
if isOomMessage {
|
||||
oomCurrentInstance := &OomInstance{
|
||||
ContainerName: "/",
|
||||
VictimContainerName: "/",
|
||||
TimeOfDeath: msg.Timestamp,
|
||||
}
|
||||
for msg := range kmsgEntries {
|
||||
finished, err := getContainerName(msg.Message, oomCurrentInstance)
|
||||
if err != nil {
|
||||
klog.Errorf("%v", err)
|
||||
}
|
||||
if !finished {
|
||||
finished, err = getProcessNamePid(msg.Message, oomCurrentInstance)
|
||||
if err != nil {
|
||||
klog.Errorf("%v", err)
|
||||
}
|
||||
}
|
||||
if finished {
|
||||
oomCurrentInstance.TimeOfDeath = msg.Timestamp
|
||||
break
|
||||
}
|
||||
}
|
||||
outStream <- oomCurrentInstance
|
||||
}
|
||||
}
|
||||
// Should not happen
|
||||
klog.Errorf("exiting analyzeLines. OOM events will not be reported.")
|
||||
}
|
||||
|
||||
// initializes an OomParser object. Returns an OomParser object and an error.
|
||||
func New() (*OomParser, error) {
|
||||
parser, err := kmsgparser.NewParser()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
parser.SetLogger(glogAdapter{})
|
||||
return &OomParser{parser: parser}, nil
|
||||
}
|
||||
|
||||
type glogAdapter struct{}
|
||||
|
||||
var _ kmsgparser.Logger = glogAdapter{}
|
||||
|
||||
func (glogAdapter) Infof(format string, args ...interface{}) {
|
||||
klog.V(4).Infof(format, args...)
|
||||
}
|
||||
func (glogAdapter) Warningf(format string, args ...interface{}) {
|
||||
klog.V(2).Infof(format, args...)
|
||||
}
|
||||
func (glogAdapter) Errorf(format string, args ...interface{}) {
|
||||
klog.Warningf(format, args...)
|
||||
}
|
||||
Vendored
+1
-5
@@ -136,9 +136,6 @@ github.com/dustin/go-humanize
|
||||
## explicit; go 1.13
|
||||
github.com/emicklei/go-restful/v3
|
||||
github.com/emicklei/go-restful/v3/log
|
||||
# github.com/euank/go-kmsg-parser v2.0.0+incompatible
|
||||
## explicit
|
||||
github.com/euank/go-kmsg-parser/kmsgparser
|
||||
# github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f
|
||||
## explicit; go 1.15
|
||||
github.com/exponent-io/jsonpath
|
||||
@@ -261,7 +258,7 @@ github.com/golang/protobuf/ptypes/wrappers
|
||||
# github.com/google/btree v1.1.3
|
||||
## explicit; go 1.18
|
||||
github.com/google/btree
|
||||
# github.com/google/cadvisor/lib v0.60.2
|
||||
# github.com/google/cadvisor/lib v0.60.3
|
||||
## explicit; go 1.24.0
|
||||
github.com/google/cadvisor/lib/cache/memory
|
||||
github.com/google/cadvisor/lib/cadvisorflags
|
||||
@@ -299,7 +296,6 @@ github.com/google/cadvisor/lib/model
|
||||
github.com/google/cadvisor/lib/stats
|
||||
github.com/google/cadvisor/lib/storage
|
||||
github.com/google/cadvisor/lib/utils
|
||||
github.com/google/cadvisor/lib/utils/oomparser
|
||||
github.com/google/cadvisor/lib/utils/sysfs
|
||||
github.com/google/cadvisor/lib/utils/sysinfo
|
||||
github.com/google/cadvisor/lib/version
|
||||
|
||||
Reference in new issue
Block a user