Apache SkyWalking – Logging

Blog: Meet Horizon UI · 7/17: The Log Explorer

Tue, 23 Jun 2026 00:00:00 +0000

This is the seventh post in the Meet Horizon UI series. Part 6 was one request’s spans; this one is the log lines around it. Horizon surfaces logs through two distinct tabs, because there are really two different questions: “what did this service log over the last half hour?” and “what is this pod printing to stdout right now?”

The Logs tab queries the logs SkyWalking has already collected and stored — indexed, filterable, correlated with traces.
The Pod Logs tab live-tails a Kubernetes pod’s container logs on demand — these aren’t stored logs at all: OAP reads them straight from the Kubernetes API server (the kubectl logs path), Horizon shows the window, and it’s discarded. Nothing is persisted, and SkyWalking’s log storage is never involved.

Which tabs a layer shows is up to its template: the Logs tab appears on layers that enable it (General, Mesh, Nginx, the Envoy AI Gateway, the mobile and mini-program layers); the Pod Logs tab appears only on the Kubernetes-aware layers (Kubernetes Service, Mesh, Mesh data plane). Browser JavaScript errors are a different thing again — not service logs but client-side error events the browser agent reports, with their own categories and their own source-map de-obfuscation (turning a minified app.min.js:1:… frame back into your original file:line). That’s a separate tab on the Browser layer, and its own post later in the series.

The stored log stream

Open a layer that has a Logs tab, pick a service in the header, and its stored log stream loads newest-first. Like the trace explorer, this tab owns its own time range — the global topbar picker is paused while you’re here, so auto-refresh can’t shift the window out from under an investigation. Pick a rolling preset (last 15 minutes through 24 hours, default 30) or a custom absolute window; queries run at second precision so the most recent lines are never rounded off the minute.

A conditions bar narrows the stream, and every filter is optional and AND-joined:

Instance — restrict to one service instance (labelled Sidecar on a sidecar layer).
Endpoint — type to search the service’s endpoints, click to pin, × to clear.
Trace ID — show only the lines correlated with one trace. This is also how a log lands when you arrive from a trace: the field pre-fills and the stream is already scoped.
Tags — a single key=value field with autocomplete; start a key to see suggestions, type = to switch to known values, Enter to commit. Committed tags ride along as removable chips.
Level — the Levels strip above the stream doubles as a filter: click error, warn, info, or debug to keep only that level, click again to clear.

There’s no log query language here — no LogQL box to learn. The conditions above are the whole surface, and edits refresh the stream as you make them; Run query is just the explicit “I’m done editing, refresh now” button that resets to the first page.

Reading the stream

The point of a log view isn’t to list lines, it’s to find the shape in them — so the stream comes with two pieces of orientation above it.

A density histogram plots log count over time, each bar stacked by level in the legend’s colors; hover a bar for that bucket’s time range and per-level counts. It’s drawn from the page currently on screen, so it shows the shape of what you’re looking at. And the Levels strip carries a running count per level — sampled across the window, not just the visible page, so the error/warn/info mix reflects the whole window you’re querying.

Each row then shows the timestamp, the level (the row is color-keyed to it), the service, an ↗ trace link when the line is trace-correlated, a JSON / YAML / TEXT format chip, and a one-line preview of the content. Horizon decides that chip by what the payload actually is: OAP labels a body JSON or plain text, and on top of that Horizon sniffs for JSON and YAML structure, so an unlabelled-but-structured line still gets the right treatment — JSON flattened to one line in the preview, YAML keeping its keys, plain text whitespace-collapsed.

Figure 1: A service’s stored log stream — a level histogram and level counts over the window, then the rows, each tagged JSON / YAML / TEXT and linked to its trace.

Into a single line

Click a row and the full payload opens in a popout: the complete content with format-aware pretty-printing — JSON and YAML laid out properly, plain text given the whole canvas instead of a cramped strip — plus a Copy button, the service / instance / endpoint / trace context, and a table of every tag on the line. When the line is trace-correlated, an ↗ trace button opens the related trace’s waterfall in an overlay without leaving the stream — and it passes the row’s timestamp along, so the trace is found even when it has aged into a colder storage tier. Escape or a backdrop click closes it.

Figure 2: One line in full — its payload pretty-printed by format, with its context and every tag laid out beside it.

Pod Logs: tailing what’s printing right now

The Pod Logs tab answers the other question, and it’s a fundamentally different source: not SkyWalking’s stored logs, but the pod’s container output read live from the Kubernetes API server through OAP — the exact thing kubectl logs -f reads. There’s no stored history to page through; each refresh pulls the trailing window, shows it, and throws it away.

Starting a tail is a few picks: choose a Pod (one service instance, pinned), a Container (Horizon lists the pod’s containers and selects the first), a look-back Window (last 30s, 1m, 5m, 15m, or 30m — how far back each poll reaches), and a poll Interval (2s, 5s, 10s, or 30s — how often it re-fetches). Press Start and the window streams into a read-only viewer that keeps the newest line in view and re-polls until you Pause; a header strip shows the container, the line count, a live indicator, and how long ago it last updated. Two Include / Exclude filter rows narrow what you see — each chip is a full-line regular expression (.*error.*) evaluated by OAP, so they match the whole line rather than a substring, and they stack.

One thing to know going in: on-demand pod logs are disabled by default on OAP, because container output can carry secrets. When the feature is off — or when the pod you picked has been rolled or scaled away — OAP returns a reason, and Horizon shows it in a banner rather than an empty pane, so you can tell “turn this on” apart from “that pod is gone.”

Figure 3: A live tail of one pod’s container — windowed, interval-polled, regex-filterable, never persisted.

Where to go next

Both tabs — the stored queries, the tag and container autocomplete, and the live tail — are gated by a single logs:read permission, so granting “can read logs” is one switch. For the field reference — every condition, the histogram, the Pod Logs windows and filters — see the Logs docs.

Next up: Browser & RUM monitoring — the browser agent’s own error stream, and de-obfuscating a minified stack with source maps.

Blog: Meet Horizon UI · 8/17: Browser Errors & Source Maps

Tue, 23 Jun 2026 00:00:00 +0000

This is the eighth post in the Meet Horizon UI series. Part 7 was your services’ logs; this one is your users’ errors — the JavaScript exceptions the browser agent reports — and the one capability that turns them from noise into something you can act on.

A production JavaScript stack is unreadable. Your code shipped minified and bundled, so the browser reports an error at app.min.js:1:98412 — a position into machine-generated soup that tells you nothing. The point of this feature is to walk that stack back to your source: the original file, line, column, symbol name, and a snippet of the code around it — frame by frame — by pointing the error at the right source map.

The browser-error feed

On the BROWSER layer, the Browser Logs tab (the on-screen label — it’s specifically the JavaScript-error feed) lists what your browser agent reports. The BROWSER layer renames its slots to match its world — services become Applications, instances Versions, endpoints Pages — and the feed reads like the Log Explorer: a clickable category legend with counts and a density histogram over a stream of rows. Each row carries the time, the category, the page, the app version, and the error message — with the minified line:col shown as a chip when there is one.

You scope it with the same triage instincts as the trace and log tabs: it owns its own Time range (the global topbar is paused), and you narrow by Version, Page, or Category and hit Run query — there’s no background polling to shift the view under you, and no query language to learn, just structured controls. Click a row and it expands inline, right there in the stream.

Figure 1: The browser agent’s error feed — categorized, charted, and scoped to one app’s version and pages.

That minified line:col is the whole problem in miniature. It’s a real position — but into your built bundle, not your source. Which is where the rest of this post comes in.

From a minified stack back to your source

Expand an error and the panel splits in two: on the left, the raw stack exactly as the browser reported it (the gibberish); on the right, where you resolve it. Pick a source map from the dropdown and click Resolve, and Horizon parses the stack and maps every frame through that map:

each frame’s original file:line:column,
the original symbol name (when the map carries it), and
a few lines of the original source around the offending line, with the hit line highlighted (when the map embeds sourcesContent).

A frame the map doesn’t cover is shown honestly as unmapped. So a stack whose top frame read app.min.js:1:45 resolves to computeCartTotal at checkout.ts:2:20, with the lines of checkout.ts around it — the cart.items.reduce(...) that actually threw — sitting right there, the whole stack top to bottom, not just the first frame.

It’s careful about the details that make this either trustworthy or quietly wrong: browser stacks count columns from 1 while source maps count from 0, so the resolver shifts before each lookup — and that path is tested against real bundler output, not a hand-made fixture.

Figure 2: The hero — point a minified stack at the right map and read it back in your own source, frame by frame.

Which errors carry a stack to resolve

Not every category has something to translate. JS, PROMISE, and VUE are real JavaScript errors whose stack points into your bundle — these resolve. AJAX and RESOURCE are network and load failures; their “stack” is an HTTP status or a failed URL, not code, so there’s simply nothing for a source map to map (Horizon doesn’t block them — there’s just no JavaScript there to walk back). Frames from code with no source map, or from eval/inline scripts, stay unmapped too. (JS is also the only category the browser reports a top-level line:col for; the others carry their position inside the stack string, which the resolver parses.)

Getting maps in: upload, or mount

A map has to be available before you can resolve against it, and there are two ways to provide one — deliberately different in durability:

Upload a .map straight from the tab. It’s held in the server’s memory only — there’s no backend storage — and it’s temporary by design: it counts against a memory budget, is evicted least-recently-used under pressure, is lost when the server restarts, and (in a multi-instance deployment) lives only on the instance that received it. This is the fast path for ad-hoc triage: drag a map in, resolve, move on.
Mount .map files into the server’s source-map directory (/app/sourcemaps in the container image, via HORIZON_SOURCEMAPS_DIR). These are validated as Source Map v3 at boot, read from disk on demand (so they never sit in the memory budget), survive restarts, reload on their own, and can’t be deleted from the UI. This is the durable, production path — bake your builds’ maps into the image and they’re always there.

The manager shows each map’s origin (an uploaded · temporary map vs a mounted · durable one) and the live memory usage against the budget; budgets (a per-file cap and a total resident-upload cap, 64 MiB and 512 MiB by default) live in a sourceMaps block in horizon.yaml.

Figure 3: Two ways to provide a map — upload for a quick triage, mount for the durable, production set.

You pick the map — on purpose

One thing Horizon deliberately does not do is guess. The browser agent reports an app version but no exact build fingerprint, so there’s no safe way to auto-match an error to a map — and applying a map from the wrong build gives you confidently wrong line numbers, which is worse than no answer. So the choice is yours: pick the map that matches the error’s build, and keep your maps labelled by version. (One caution worth stating plainly: a source map’s sourcesContent embeds your original source code, so treat the maps you upload or mount as sensitive, and provision them only on servers you trust.)

That manual-by-design choice also draws a clean permission line. Viewing the errors, listing the maps, and resolving a stack are all reads, gated by browser-errors:read; uploading or removing a map is a write, gated by source-map:write. So a read-only viewer can de-obfuscate stacks all day without ever being able to change what maps are loaded — reading is reading, mutating the map store is a write.

Where to go next

For the field reference — the categories, the two provisioning paths, the budgets and the matching-maps-to-builds guidance — see the Browser Logs & Source Maps docs.

Next up: Profiling — five profilers (trace, async, eBPF, Go pprof, network) rendered through one flame graph.

Zh: 认识 Horizon UI · 7/17：日志探索器

Tue, 23 Jun 2026 00:00:00 +0000

译自英文原文：Meet Horizon UI · 7/17: The Log Explorer。

这是 Meet Horizon UI 系列的第七篇。第六篇讲的是一个请求的 spans；这一篇讲它周围的日志行。Horizon 用 两个标签页 展示日志，对应两类排查问题：“这个服务过去半小时打了什么日志？” 以及 “这个 pod 现在正在向 stdout 打什么？”

Logs 标签页查询 SkyWalking 已经 采集并存储 的日志：已索引、可过滤、可与 Trace 关联。
Pod Logs 标签页按需 实时 tail Kubernetes pod 的容器日志。这类日志不走 SkyWalking 的日志存储：OAP 直接从 Kubernetes API server 读取它们（也就是 kubectl logs 那条路径），Horizon 只展示当前窗口，然后丢弃，不会持久化。

某个 Layer 展示哪些标签页由模板决定：启用日志的 Layer（General、Mesh、Nginx、Envoy AI Gateway、mobile 和 mini-program Layer）会显示 Logs 标签页；只有感知 Kubernetes 的 Layer（Kubernetes Service、Mesh、Mesh data plane）会显示 Pod Logs 标签页。Browser JavaScript 错误又是另一类数据：它不是服务日志，而是浏览器端 agent 上报的客户端错误事件，有自己的分类，也可以借助 source map 把压缩后的 app.min.js:1:... frame 还原到原始 file:line。这是 Browser Layer 上的独立标签页，会在这个系列的另一篇文章里讲。

查询已存储日志

打开一个有 Logs 标签页的 Layer，先在顶部选择服务，日志流会按新到旧加载。和 Trace 探索器一样，这个标签页使用 独立的时间范围。当你在这里排查时，全局顶栏时间选择器会暂停，自动刷新不会把你正在看的窗口不断往前推。可以选择滚动预设（最近 15 分钟到 24 小时，默认 30 分钟），也可以选择自定义绝对窗口；查询按 秒级精度 执行，所以最新日志不会被分钟取整吞掉。

条件栏用来收窄日志流，每个过滤条件都是可选的，多个条件按 AND 连接：

Instance：限制到某个服务实例。在 sidecar Layer 上标签显示为 Sidecar。
Endpoint：输入搜索服务 endpoints，点击固定，按 × 清除。
Trace ID：只显示和某条 Trace 关联的日志行。从 Trace 跳转过来时，也会预填这个字段并直接限定日志流。
Tags：单个 key=value 字段，带 autocomplete；输入 key 可看建议，输入 = 后切换到已知 value，按 Enter 提交。提交后的 tags 以可删除标签形式保留。
Level：日志流上方的 Levels 条也可以当过滤器用。点击 error、warn、info 或 debug 只保留该级别，再点一次清除。

这里不提供单独的日志查询语言，也不需要学习 LogQL。上面的条件就是完整界面，并且 编辑时会立即刷新日志流；Run query 只是显式告诉系统“我改完了，现在刷新”，同时回到第一页。

用直方图和级别计数定位日志

日志视图的目标不只是列出行，还要帮你看出分布和异常，所以日志流上方有两类辅助信息。

Density histogram 按时间展示日志量，每个柱按 legend 颜色 堆叠 level；hover 柱子可以看到该 bucket 的时间范围和每个 level 的计数。它基于当前页面上可见数据绘制，所以展示的是你正在看的日志分布。Levels 条则保留每个 level 在窗口内的累计计数。这个计数跨整个查询窗口采样，而不只是当前可见页，所以 error/warn/info 比例反映的是整个窗口。

日志行会展示 timestamp、level（行颜色跟随 level）、service、存在 Trace 关联时的 ↗ trace 链接、JSON / YAML / TEXT 格式标记，以及内容的一行预览。Horizon 按日志内容本身决定这个标记：OAP 会标注日志 body 是 JSON 还是 plain text，在此基础上 Horizon 还会识别 JSON 和 YAML 结构，所以即使一行没有被标注但内容是结构化的，也会得到正确处理。JSON 在预览里压平成一行，YAML 保留 key，plain text 会折叠空白。

图 1：某个服务的存储日志流：窗口上的 level histogram 和 level 计数，下面是每行日志，带 JSON / YAML / TEXT 标记并可跳到 Trace。

查看单行日志详情

点击一行后，完整日志内容会在弹层里打开：内容会按格式排版，JSON 和 YAML 正常缩进，plain text 则展开成完整内容，而不是被挤在一条窄条里。面板还提供 Copy 按钮、service / instance / endpoint / trace 上下文，以及该行所有 tag 的表格。日志行与 Trace 关联时，↗ trace 按钮会在 overlay 中打开相关 Trace 瀑布图，不离开日志流；它还会把这行日志的 timestamp 传过去，所以 Trace 即使已经进入更冷的存储 tier，也仍然能找到。按 Escape 或点击背景遮罩即可关闭。

图 2：完整查看一行日志：内容按格式排版，旁边展示上下文和所有 tags。

Pod Logs：实时查看容器输出

Pod Logs 标签页回答另一个问题，它的数据源也完全不同：不是 SkyWalking 存储的日志，而是通过 OAP 从 Kubernetes API server 实时读取 pod 的容器输出，也就是 kubectl logs -f 读取的同一类内容。这里没有可翻页的存储历史；每次刷新拉取尾部窗口，展示出来，然后丢弃。

开始 tail 只需要几个选择：选一个 Pod（固定的服务实例）、一个 Container（Horizon 会列出 pod 的容器并默认选第一个）、一个回看 Window（last 30s、1m、5m、15m 或 30m，决定每次轮询向前取多远），以及轮询 Interval（2s、5s、10s 或 30s，决定多久重新取一次）。按 Start 后，日志会流入只读查看器，保持最新行可见，并持续轮询直到你按 Pause。顶部状态条显示 container、行数、live 状态，以及上次更新距今多久。两行 Include / Exclude 过滤器用来收窄可见内容；每个标签都是一个 整行正则表达式（.*error.*），由 OAP 执行，所以它匹配整行而不是子串，并且可以叠加。

使用前需要知道一点：按需读取 pod logs 在 OAP 上 默认关闭，因为容器输出可能包含敏感信息。功能关闭时，或者你选择的 pod 已经滚动或缩容消失时，OAP 会返回一个原因，Horizon 会把它显示成横幅，而不是给你一个空面板。这样你能区分“需要打开这个功能”和“那个 pod 已经不存在”。

图 3：一个 pod 容器的实时 tail：窗口化、按间隔轮询、可用正则过滤、永不持久化。

后续阅读

两个标签页，包括存储查询、tag/container 自动补全和 live tail，都由同一个 logs:read 权限控制。所以授予“可以读日志”就是一个开关。字段参考，包括每个条件、histogram、Pod Logs 的窗口和过滤器，可以看 Logs 文档。

下一篇看 Browser/RUM：浏览器端错误如何上报，又如何用 source map 把压缩后的 stack 还原到源码位置。

Zh: 认识 Horizon UI · 8/17：浏览器错误与 Source Map

Tue, 23 Jun 2026 00:00:00 +0000

译自英文原文：Meet Horizon UI · 8/17: Browser Errors & Source Maps。

这是 Meet Horizon UI 系列的第八篇。第七篇讲的是服务日志；这一篇讲用户遇到的错误，也就是浏览器端 agent 上报的 JavaScript 异常，以及把这些错误定位到源码的关键一步。

生产环境 JavaScript stack 基本不可读。代码经过压缩和打包后发布，浏览器只会报告错误出现在 app.min.js:1:98412，也就是一段机器生成代码里的位置，几乎不给你任何线索。Horizon 要做的是找到正确的 source map，把 stack 里的每一帧映射回源码：原始文件、行、列、符号名，以及出错位置附近的代码片段。

浏览器端错误流

在 BROWSER Layer 上，Browser Logs 标签页（屏幕上的标签名，专指 JavaScript 错误流）会列出浏览器端 agent 上报的内容。BROWSER Layer 会把槽位重命名成自己的语义：services 变成 Applications，instances 变成 Versions，endpoints 变成 Pages。这个列表的阅读方式类似 Log Explorer：可点击的 category legend 带计数，density histogram 位于日志流之上。每一行都有时间、category、page、app version 和错误消息；如果带有压缩后的 line:col，也会显示成标记。

排查方式和 trace/log 标签页一致：它使用独立的 Time range（全局顶栏暂停），你可以按 Version、Page 或 Category 收窄，然后点击 Run query。没有后台轮询把视图不断往前推，也没有要学习的查询语言，只有结构化控件。点击一行后，它会在日志流里原地展开。

图 1：浏览器端 agent 上报的错误流：按 category 组织、带图表，并限定到某个 app 版本和页面。

压缩后的 line:col 就是最典型的问题。它是真实位置，但位置在 构建后 的 bundle 里，不在你的源码里。后面的解析流程就是为了解决这个落差。

从压缩后的 stack 定位到源码

展开一个错误后，面板分成两侧：左边是浏览器原样报告的 raw stack，也就是那段很难读的生产栈；右边是解析结果区域。选择一个 source map，点击 Resolve，Horizon 会解析 stack，并通过这份 map 映射 每一帧：

每帧原始 file:line:column；
原始 symbol name（如果 map 携带了）；
出错行附近几行 原始源码，命中行会高亮（如果 map 内嵌 sourcesContent）。

map 覆盖不到的 frame 会明确标为 unmapped。所以一条顶层 frame 为 app.min.js:1:45 的 stack，可以还原成 checkout.ts:2:20 上的 computeCartTotal，并把 checkout.ts 附近几行显示出来。真正抛错的 cart.items.reduce(...) 就在面板里，不只是还原第一帧，而是从上到下还原整条 stack。

这里有一些细节决定结果是否可信：浏览器 stack 的列号从 1 开始计数，source map 从 0 开始计数，所以解析器每次查找前都会做偏移；这条路径用真实 bundler 输出测试，而不是手写 fixture。

图 2：核心流程：把压缩后的 stack 匹配到正确 map，然后逐帧还原到你自己的源码。

哪些错误可以用 source map 解析

不是每类错误都有可还原内容。JS、PROMISE 和 VUE 是真实 JavaScript 错误，它们的 stack 指向 bundle，可以解析。AJAX 和 RESOURCE 是网络和加载失败；它们的“stack”是 HTTP status 或失败 URL，不是代码，所以 source map 没有东西可映射（Horizon 不会阻止它们，只是没有可映射的 JavaScript 位置）。没有 source map 的代码、eval 或 inline scripts 里的 frame，也会保持 unmapped。（JS 也是唯一由浏览器上报顶层 line:col 的 category；其他 category 的位置在 stack 字符串内部，由解析器提取。）

提供 source map：上传或挂载

解析前必须让 map 可用。Horizon 提供两种方式，持久化策略有意不同：

Upload 一个 .map，直接从标签页上传。它只保存在服务端内存里，没有后端存储，并且临时性是设计目标：它占用内存预算，在压力下按 least-recently-used 淘汰，服务重启后丢失；多实例部署时，它也只存在于接收上传的那一个实例。这个路径适合临时排查：拖一个 map 进来，解析，处理完离开。
Mount .map 文件到服务端 source-map 目录（容器镜像中是 /app/sourcemaps，可通过 HORIZON_SOURCEMAPS_DIR 指定）。这些文件在启动时按 Source Map v3 校验，按需从磁盘读取（所以不占内存预算），重启后仍然存在，会自动重新加载，并且 不能从 UI 删除。这是生产路径：把构建产物里的 map 文件放进镜像或挂载目录，它们就一直可用。

管理器会显示每个 map 的来源（uploaded · temporary 还是 mounted · durable），以及当前内存预算使用量。预算配置，包括单文件上限和常驻上传总量上限，默认 64 MiB 和 512 MiB，位于 horizon.yaml 的 sourceMaps 块。

图 3：两种提供 map 的方式：upload 用于快速排查，mount 用于生产环境长期使用。

map 必须手动选择

Horizon 刻意 不猜测。浏览器端 agent 会上报 app version，但不会上报精确的构建指纹，所以没有安全方法自动把一个错误匹配到某份 map。用错构建的 map 会给出非常自信、但完全错误的行号，比没有答案更糟。所以这里由你选择：挑选和这次错误对应构建的 map，并按版本给 map 清晰命名。（还有一个必须直说的注意点：source map 的 sourcesContent 会包含你的原始源码，所以无论上传还是挂载，都要把 map 当成敏感内容，只放在可信服务器上。）

手动选择 map 这件事，也划清了权限边界。查看错误、列出 maps、解析 stack 都是读操作，由 browser-errors:read 控制；上传或删除 map 是写操作，由 source-map:write 控制。所以只读用户可以反复解析 stack，但没有权限改变已加载的 map 集合。读就是读，修改 map 存储才是写。

后续阅读

字段参考，包括 categories、两种提供路径、预算，以及如何按构建版本匹配 map，可以看 Browser Logs & Source Maps 文档。

下一篇进入 Profiling：五种 profiler（trace、async、eBPF、Go pprof、network）如何共用一套火焰图视图，以及为什么 network profiling 是例外。

Blog: Use of SkyWalking Go Toolkit Log and Metrics

Sun, 20 Oct 2024 00:00:00 +0000

Intro

Apache SkyWalking Go is an observability tool specifically designed for Golang applications, aimed at monolithic services, microservices, cloud-native architectures, and containerized applications. It is the Go language implementation of the Apache SkyWalking probe project, providing comprehensive capabilities for service tracing, performance metrics analysis, and application topology analysis.

SkyWalking Go leverages Go’s concurrency features to achieve efficient data collection and analysis. By inserting a minimal amount of probe code into the application during compilation using AST (Abstract Syntax Tree), it can capture service request and response data, as well as system runtime status information. By reporting this collected data, SkyWalking Go can generate detailed service call chain diagrams, helping developers understand the dependencies between services and the performance status of each service.

SkyWalking Go currently provides the following three capabilities for users to manually report relevant information:

Trace
Metrics
Log

This document aims to guide users on how to manually report log entries and metrics using the toolkit. For information on how to report trace linkage information with the toolkit, please refer to the SkyWalking Go Toolkit Trace Detailed Explanation.

Before diving deeper, you may want to check the SkyWalking Go Agent Quick Start Guide to learn how to use the SkyWalking Go Agent.

The following sections will introduce how to use these interfaces in specific scenarios.

Import Trace Toolkit

First, execute the following command in the root directory of the project:

go get github.com/apache/skywalking-go/toolkit

Manually report logs

In traceability, logs play a crucial role. They record detailed information about each request in the system, including timestamps, processing nodes, error messages, etc., which helps developers and operations teams quickly identify performance bottlenecks and the root causes of failures. By comparing logs from different requests, teams can analyze the flow of requests, optimize system architecture, and improve service response speed and stability.

In the SkyWalking Go toolkit, manually reported logs will be attached to the current context Span, allowing us to associate specific log information with particular spans.

First, we need to import the toolkit log package:

"github.com/apache/skywalking-go/toolkit/logging"

We can build a simple web service that determines the validity of a username based on the request parameters. When the userName parameter is invalid, we log an error using the logging.Error API. This log will be attached to the currently active Span in the context.

When recording logs, we can also append keyValues to the log information using variadic parameters, making the log entries more descriptive.

For detailed usage documentation, please refer to SkyWalking Go toolkit-logging.

package main

import (
	"log"
	"net/http"

	_ "github.com/apache/skywalking-go"
	"github.com/apache/skywalking-go/toolkit/logging"
)

func main() {
	http.HandleFunc("/user", func(w http.ResponseWriter, r *http.Request) {
		userName := r.URL.Query().Get("userName")
		if len(userName) == 0 || userName != "root" {
			// Log an entry, which will be attached to the currently active Span in the context.
			// We can append log tags using variadic parameters.
			logging.Error("deny user login", "userName", userName)
			w.WriteHeader(http.StatusUnauthorized)
			return
		}

		w.WriteHeader(http.StatusAccepted)
	})

	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalln("server running by err:", err)
	}
}

Then we enhance it using the SkyWalking Go Agent:

go build -toolexec="/path/go-agent" -a -o demo .

Manually reporting metrics information

Metrics are extremely important in traceability, as they provide quantitative analysis of system performance. By monitoring metrics such as request latency, throughput, and error rates, teams can identify performance bottlenecks and potential issues, allowing them to optimize system architecture and resource allocation. Combined with traceability, metrics can reveal the flow of requests between services, helping teams gain deeper insights into the health status and usage patterns of the system, ensuring high availability of services and a positive user experience, ultimately supporting business objectives effectively.

The current toolkit metrics support the following types of metrics:

Counter
Gauge
Histogram

First, execute the following command in the root directory of the project:

"github.com/apache/skywalking-go/toolkit/metric"

We can build a simple Echo service that creates a Counter type metric to record the number of requests. At the same time, we can use metric.WithLabels to add additional labels to the metric.

package main

import (
	"log"
	"net/http"

	_ "github.com/apache/skywalking-go"
	"github.com/apache/skywalking-go/toolkit/metric"
)

func main() {
	// Create a Counter type metric
	// We also set labels for this Counter
	counter := metric.NewCounter(
		"http_request_count",
		metric.WithLabels("path", "/ping"),
	)

	http.HandleFunc("/ping", func(w http.ResponseWriter, r *http.Request) {
		// Increment the count by one for each incoming request
		counter.Inc(1)

		w.WriteHeader(http.StatusOK)
	})

	if err := http.ListenAndServe(":8080", nil); err != nil {
		log.Fatalln("server running by err:", err)
	}
}

Then we enhance it using the SkyWalking Go Agent:

go build -toolexec="/path/go-agent" -a -o demo .

We can display the metric information in the SkyWalking Custom Dashboard.

Summarize

This document discusses the basic usage of the Log APIs and Metrics APIs in SkyWalking Go. It provides users with the functionality to customize and report log entries and metric information.

The SkyWalking Go toolkit was designed with simplicity in mind, aiming to shorten the distance between users and the product.

For more information, please refer to SkyWalking Go.

Zh: 使用 LAL 收集并分析 Nginx access log

Sun, 29 Oct 2023 00:00:00 +0000

背景介绍

Nginx access log 中包含了丰富的信息，例如：日志时间、状态码、响应时间、body 大小等。通过收集并分析 access log，我们可以实现对 Nginx 中接口状态的监控。

在本案例中，将由 fluent-bit 收集 access log，并通过 HTTP 将日志信息发送给 SkyWalking OAP Server 进行进一步的分析。

环境准备

实验需要的 Nginx 及 Fluent-bit 相关配置文件都被上传到了Github，有需要的读者可以自行 git clone 并通过 docker compose 启动，本文中将介绍配置文件中几个关键点。

Nginx日志格式配置

LAL 目前支持 JSON、YAML 及 REGEX 日志解析，为了方便获取到日志中的指标字段，我们将 Nginx 的日志格式定义为 JSON.

http {
    ...
    ...

    log_format  main  '{"remote_addr": "$remote_addr",'
            '"remote_user": "$remote_user",'
            '"request": "$request",'
            '"time": "$time_iso8601",'
            '"status": "$status",'
            '"request_time":"$request_time",'
            '"body_bytes_sent": "$body_bytes_sent",'
            '"http_referer": "$http_referer",'
            '"http_user_agent": "$http_user_agent",'
            '"http_x_forwarded_for": "$http_x_forwarded_for"}';

    access_log  /var/log/nginx/access.log  main;
    
    ...
    ...
}

Fluent bit Filter

我们通过 Fluent bit 的 lua filter 进行日志格式的改写，将其调整为 SkyWalking 所需要的格式，record的各个字段含义如下：

body：日志内容体
service：服务名称
serviceInstance：实例名称

function rewrite_body(tag, timestamp, record)
    local newRecord = {}
    newRecord["body"] = { json = { json = record.log } }
    newRecord["service"] = "nginx::nginx"
    newRecord["serviceInstance"] = "localhost"
    return 1, timestamp, newRecord
end

OAP 日志分析

LAL定义

在 filter 中，我们通过条件判断，只处理 service=nginx::nginx 的服务，其他服务依旧走默认逻辑：

第一步，使用 json 指令对日志进行解析，解析的结果会被存放到 parsed 字段中，通过 parsed 字段我们可以获取 json 日志中的字段信息。

第二步，使用 timestamp 指令解析 parsed.time 并将其赋值给日志的 timestamp 字段，这里的 time 就是access log json 中的 time。

第三步，使用 tag 指令给日志打上对应的标签，标签的值依然可以通过 parsed 字段获取。

第四步，使用 metrics 指令从日志中提取出指标信息，我们共提取了四个指标：

nginx_log_count：Nginx 每次请求都会生成一条 access log，该指标可以帮助我们统计 Nginx 当前的请求数。
nginx_request_time：access log 中会记录请求时间，该指标可以帮助我们统计上游接口的响应时长。
nginx_body_bytes_sent：body 大小指标可以帮助我们了解网关上的流量情况。
nginx_status_code：状态码指标可以实现对状态码的监控，如果出现异常上涨可以结合 alarm 进行告警。

rules:
  - name: default
    layer: GENERAL
    dsl: |
      filter {
        if (log.service == "nginx::nginx") {
          json {
            abortOnFailure true
          }
    
          extractor {
            timestamp parsed.time as String, "yyyy-MM-dd'T'HH:mm:ssXXX"
            tag status: parsed.status
            tag remote_addr: parsed.remote_addr
      
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_log_count"
              value 1
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_request_time"
              value parsed.request_time as Double
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance
              name "nginx_body_bytes_sent"
              value parsed.body_bytes_sent as Long
            }
            metrics {
              timestamp log.timestamp as Long
              labels service: log.service, instance: log.serviceInstance, status: parsed.status
              name "nginx_status_code"
              value 1
            }
          }
        }
      
        sink {
        }
      }

经过 LAL 处理后，我们已经可以在日志面板看到日志信息了，接下来我们将对 LAL 中提取的指标进行进一步分析：

MAL定义

在 MAL 中，我们可以对上一步 LAL 中提取的指标进行进一步的分析聚合，下面的例子里：

nginx_log_count、nginx_request_time、nginx_status_code 使用 sum 聚合函数处理，并使用 SUM 方式 downsampling，

nginx_request_time 使用 avg 聚合函数求平均值，默认使用 AVG 方式 downsampling。

完成聚合分析后，SkyWalking Meter System 会完成对上述指标的持久化。

expSuffix: service(['service'], Layer.GENERAL)
metricPrefix: nginx
metricsRules:
  - name: cpm
    exp: nginx_log_count.sum(['service']).downsampling(SUM)
  - name: avg_request_time
    exp: nginx_request_time.avg(['service'])
  - name: body_bytes_sent_count
    exp: nginx_body_bytes_sent.sum(['service']).downsampling(SUM)
  - name: status_code_count
    exp: nginx_status_code.sum(['service','status']).downsampling(SUM)

最后，我们便可以来到 SkyWalking UI 页面新建 Nginx 仪表板，使用刚刚 MAL 中定义的指标信息创建 Nginx Dashboard（也可以通过上文提到仓库中的 dashboard.json 直接导入测试）：

参考文档

Blog: New Features of SkyWalking Go 0.2.0

Mon, 31 Jul 2023 00:00:00 +0000

Announcing Apache SkyWalking Go 0.2.0

I’m excited to announce the release of Apache SkyWalking Go 0.2.0! This version packs several awesome new features that I’ll overview below.

Log Reporting

The log reporting feature allows the Go agent to automatically collect log content from supported logging frameworks like logrus and zap. The logs are organized and sent to the SkyWalking backend for visualization. You can see how the logs appear for each service in the SkyWalking UI:

Making Logs Searchable

You can configure certain log fields to make them searchable in SkyWalking. Set the SW_AGENT_LOG_REPORTER_LABEL_KEYS environment variable to include additional fields beyond the default log level.

For example, with logrus:

# define log with fields
logrus.WithField("module", "test-service").Info("test log")

Metrics Reporting

The agent can now collect and report custom metrics data from runtime/metrics to the backend. Supported metrics are documented here.

Automatic Instrumentation

In 0.1.0, you had to manually integrate the agent into your apps. Now, the new commands can automatically analyze and instrument projects at a specified path, no code changes needed! Try using the following command to import skywalking-go into your project:

# inject to project at current path
skywalking-go-agent -inject=./ -all

Or you can still use the original manual approach if preferred.

Get It Now!

Check out the CHANGELOG for the full list of additions and fixes. I encourage you to try out SkyWalking Go 0.2.0 today! Let me know if you have any feedback.

Blog: SourceMarker: Continuous Feedback for Developers

Tue, 16 Mar 2021 00:00:00 +0000

SourceMarker is an open-source continuous feedback IDE plugin built on top of Apache SkyWalking, a popular open-source APM system with monitoring, tracing, and diagnosing capabilities for distributed software systems. SkyWalking, a truly holistic system, provides the means for automatically producing, storing, and querying software operation metrics. It requires little to no code changes to implement and is lightweight enough to be used in production. By itself, SkyWalking is a formidable force in the realm of continuous monitoring technology.

SourceMarker, leveraging the continuous monitoring functionality provided by SkyWalking, creates continuous feedback technology by automatically linking software operation metrics to source code and displaying feedback directly inside of the IDE. While currently only supporting JetBrains-based IDEs and JVM-based programming languages, SourceMarker may be extended to support any number of programming languages and IDEs. Using SourceMarker, software developers can understand and validate software operation inside of their IDE. Instead of charts that indicate the health of the application, software developers can view the health of individual source code components and interpret software operation metrics from a much more familiar perspective. Such capabilities improve productivity as time spent continuously context switching from development to monitoring would be eliminated.

Logging

The benefits of continuous feedback technology are immediately apparent with the ability to view and search logs directly from source code. Instead of tailing log files or viewing logs through the browser, SourceMarker allows software developers to navigate production logs just as easily as they navigate source code. By using the source code as the primary perspective for navigating logs, SourceMarker allows software developers to view logs specific to any package, class, method, or line directly from the context of the source code which resulted in those logs.

Tracing

Furthermore, continuous feedback technology offers software developers a deeper understanding of software by explicitly tying the implicit software operation to source code. Instead of visualizing software traces as Gantt charts, SourceMarker allows software developers to step through trace stacks while automatically resolving trace tags and logs. With SourceMarker, software developers can navigate production software traces in much the same way one debugs local applications.

Alerting

Most importantly, continuous feedback technology keeps software developers aware of production software operation. Armed with an APM-powered IDE, every software developer can keep track of the behavior of any method, class, package, and even the entire application itself. Moreover, this allows for source code to be the medium through which production bugs are made evident, thereby creating the feasibility of source code with the ability to self-diagnose and convey its own health.

Download SourceMarker

SourceMarker aims to bridge the theoretical and empirical practices of software development through continuous feedback. The goal is to make developing software with empirical data feel natural and intuitive, creating more complete software developers that understand the entire software development cycle.

https://github.com/sourceplusplus/sourcemarker

This project is still early in its development, so if you think of any ways to improve SourceMarker, please let us know.

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Fri, 05 Feb 2021 00:00:00 +0000

Origin: Tetrate.io blog

The Apache SkyWalking team today announced the 8.4 release is generally available. This release fills the gap between all previous versions of SkyWalking and the logging domain area. The release also advances SkyWalking’s capabilities for infrastructure observability, starting with virtual machine monitoring.

Background

SkyWalking has historically focused on the tracing and metrics fields of observability. As its features for tracing, metrics and service level monitoring have become more and more powerful and stable, the SkyWalking team has started to explore new scenarios covered by observability. Because service performance is reflected in the logs, and is highly impacted by the infrastructure on which it runs, SkyWalking brings these two fields into the 8.4 release. This release blog briefly introduces the two new features as well as some other notable changes.

Logs

Metrics, tracing, and logging are considered the three pillars of observability [1]. SkyWalking had the full features of metrics and tracing prior to 8.4; today, as 8.4 is released, the last piece of the jigsaw is now in place.

Figure 1: Logs Collected By SkyWalking

Figure 2: Logs Collected By SkyWalking

The Java agent firstly provides SDKs to enhance the widely-used logging frameworks, log4j (1.x and 2.x) [2] and logback [3], and send the logs to the SkyWalking backend (OAP). The latter is able to collect logs from wherever the protocol is implemented. This is not a big deal, but when it comes to the correlation between logs and traces, the traditional solution is to print the trace IDs in the logs, and pick the IDs in the error logs to query the related traces. SkyWalking just simplifies the workflow by correlating the logs and traces natively. Navigating between traces and their related logs is as simple as clicking a button.

Figure 3: Correlation Between Logs and Traces

Infrastructure Monitoring

SkyWalking is known as an application performance monitoring tool. One of the most important factors that impacts the application’s performance is the infrastructure on which the application runs. In the 8.4 release, we added the monitoring metrics of virtual machines into the dashboard.

Figure 4: VM Metrics

Fundamental metrics such as CPU Used, Memory Used, Disk Read / Write and Network Usage are available on the dashboard. And as usual, those metrics are also available to be configured as alarm triggers when needed.

Dynamic Configurations at Agent Side

Dynamic configuration at the backend side has long existed in SkyWalking for several versions. Now, it finally comes to the agent side! Prior to 8.4, you’d have to restart the target services when you modify some configuration items of the agent – for instance, sampling rate (agent side), ignorable endpoint paths, etc. Now, say goodbye to rebooting. Modifying configurations is not the only usage of the dynamic configuration mechanism. The latter gives countless possibilities to the agent side in terms of dynamic behaviours, e.g. enabling / disabling plugins, enabling / disabling the whole agent, etc. Just imagine!

Grouped Service Topology

This enhancement is from the UI. SkyWalking backend supports grouping the services by user-defined dimensions. In a real world use case, the services are usually grouped by business group or department. When a developer opens the topology map, out of hundreds of services, he or she may just want to focus on the services in charge. The grouped service topology comes to the rescue: one can now choose to display only services belonging to a specified group.

Figure 5: Grouped Service Topology

Other Notable Enhancements

Agent: resolves domain names to look up backend service IP addresses.
Backend: meter receiver supports meter analysis language (MAL).
Backend: several CVE fixes.
Backend: supports Envoy {AccessLog,Metrics}Service API V3 and adopts MAL.

Additional Resources

Read more about the SkyWalking 8.4 release highlights.
Get more SkyWalking updates on Twitter.

Apache SkyWalking – Logging

Blog: Meet Horizon UI · 7/17: The Log Explorer

The stored log stream

Reading the stream

Into a single line

Pod Logs: tailing what’s printing right now

Where to go next

Blog: Meet Horizon UI · 8/17: Browser Errors & Source Maps

The browser-error feed

From a minified stack back to your source

Which errors carry a stack to resolve

Getting maps in: upload, or mount

You pick the map — on purpose

Where to go next

Zh: 认识 Horizon UI · 7/17：日志探索器

查询已存储日志

用直方图和级别计数定位日志

查看单行日志详情

Pod Logs：实时查看容器输出

后续阅读

Zh: 认识 Horizon UI · 8/17：浏览器错误与 Source Map

浏览器端错误流

从压缩后的 stack 定位到源码

哪些错误可以用 source map 解析

提供 source map：上传或挂载

map 必须手动选择

后续阅读

Blog: Use of SkyWalking Go Toolkit Log and Metrics

Intro

Import Trace Toolkit

Manually report logs

Manually reporting metrics information

Summarize

Zh: 使用 LAL 收集并分析 Nginx access log

背景介绍

环境准备

Nginx日志格式配置

Fluent bit Filter

OAP 日志分析

LAL定义

MAL定义

参考文档

Blog: New Features of SkyWalking Go 0.2.0

Announcing Apache SkyWalking Go 0.2.0

Log Reporting

Making Logs Searchable

Metrics Reporting

Automatic Instrumentation

Get It Now!

Blog: SourceMarker: Continuous Feedback for Developers

Logging

Tracing

Alerting

Download SourceMarker

Blog: Apache SkyWalking 8.4: Logs, VM Monitoring, and Dynamic Configurations at Agent Side

Background

Logs

Infrastructure Monitoring

Dynamic Configurations at Agent Side

Grouped Service Topology

Other Notable Enhancements

Links

Additional Resources