從系統診斷,到能落地的工程改善。From system diagnosis to engineering changes you can put into practice.
以下依問題類型說明處理範圍、判斷所需的證據,以及可交付的工程產出。技術棧會影響實作方式,但不是判斷問題的起點。Explore the problems I address, the evidence needed to assess them, and the engineering deliverables. The technology stack shapes implementation; diagnosis starts with system behavior.
實際處理的六類問題。The six types of problems I take on.
先把系統的實際樣子畫出來Start by drawing what the system actually is
多數團隊手上的架構圖,跟線上實際跑的已經不一樣。第一步是還原真正的呼叫關係、資料流、共用資源和隱含依賴,並標出哪些部分現在沒有人解釋得清楚。For most teams the architecture diagram no longer matches what runs in production. The first step is reconstructing the real call paths, data flows, shared resources, and implicit dependencies, and marking the parts nobody can currently explain.
用 trade-off 說明架構,不只給一張圖Explain architecture as trade-offs, not as a diagram
架構決策需要權衡收益與代價。我的工作是把選項、代價和撤退路線寫清楚,讓團隊自己覆核得了,而不是只能相信一個外部意見。Architectural decisions involve benefits, costs, and constraints. My job is to write down the options, the costs, and the way back, so the team can review the reasoning instead of having to trust an outside opinion.
從「哪些不應該被改」開始Start from what should not be touched
替每個模組決定 keep / fix / refactor / extract / replace / migrate / rebuild,並排出先後順序,以維持可發布、可驗證、可回滾為改造目標。Assign every module one of keep / fix / refactor / extract / replace / migrate / rebuild, then sequence them so the system stays shippable and reversible throughout.
找出最先飽和的那個資源Find the resource that saturates first
處理 tail latency、lock contention、connection pool 耗盡、queue 堆積、cache 失效風暴、stateful 瓶頸。優化前後用同一組指標比較,不靠感覺驗收。Tail latency, lock contention, connection pool exhaustion, queue build-up, cache stampedes, stateful bottlenecks. Before and after are compared on the same metrics rather than signed off on impression.
讓事故可被觀測、可被解釋Make incidents observable and explainable
處理間歇性故障、部署風險、回滾路徑、降級行為,以及依賴故障怎麼擴散開來。目標是下一次發生時抓得到,不只是這次修好。Intermittent failures, deployment risk, rollback paths, degradation behavior, and how a dependency failure spreads. The goal is catching it next time, not only fixing it this time.
在不能停機的前提下搬資料Move data without taking the system down
schema 演進、雙寫與影子流量比對、切換點設計、一致性邊界、失敗中止條件。migration 難的不是搬,是中途出錯時能停得下來。Schema evolution, dual writes and shadow-traffic comparison, cutover design, consistency boundaries, abort conditions. The hard part of a migration is not moving the data, it is being able to stop halfway when something goes wrong.
合作方式取決於問題的性質。How we work together depends on the problem.
短期、範圍明確。針對一個具體症狀或一次架構評估,給出書面分析和建議的執行順序。Short and clearly scoped. For one specific symptom or a single architecture review, you get a written analysis and a recommended order of work.
產出:分析報告 + 選項比較FOR: the problem is not located yet
OUTPUT: analysis + options compared
目標明確的改造或效能工程,我負責設計與實作,和內部團隊一起上線、一起驗證。A defined piece of remediation or performance engineering. I design and implement it, then ship and verify it together with the in-house team.
產出:可上線的改動 + 驗證結果FOR: the work is already known
OUTPUT: shippable changes + verification
固定時段參與架構決策、事故覆盤和技術審查,讓判斷留在團隊裡,不是留在我這裡。Regular time on architecture decisions, incident reviews, and technical review, so the judgment stays with the team rather than with me.
產出:決策紀錄 + 固定技術審查 + 改善追蹤FOR: a system that keeps evolving
OUTPUT: decision records + regular technical reviews + improvement tracking
診斷以縮小範圍、整理證據與提出修復選項為交付,不預設後續需要大型專案。專案另行確認範圍、實作責任、受控上線、驗收與回滾條件。A diagnostic delivers evidence, a narrower problem scope, and remediation options without committing you to a larger project. Project scope separately defines implementation, controlled rollout, acceptance, and rollback conditions.
問題不一定在程式碼裡。The problem is not always in the code.
從業務需求到 production 維運,每一層都在限制上一層能做的事;架構上的每個決定,最後也都要在 production 的現實裡兌現。所以我不會只看一層。From business requirement down to production operation, each layer constrains what the one above it can do, and every architectural decision eventually has to hold up against production reality. So I do not read a single layer in isolation.
同一個症狀,在不同層的解法完全不同。分析要做的不是列出所有可能,是把範圍縮到可以驗證。The same symptom has entirely different fixes depending on the layer. Analysis is not about listing every possibility, it is about narrowing the range until it can be verified.
Production 事故要靠證據,不能靠猜。Production incidents are settled by evidence, not guesses.
事故持續時,先降低影響並恢復服務,同時盡量保留現場證據。服務穩定後,再縮小問題範圍。重啟、加機器或改參數可能緩解症狀;若未釐清觸發條件與失敗路徑,仍有復發風險。During an incident, reduce impact and restore service while preserving evidence where possible. Once the service is stable, narrow the investigation. Restarts, extra machines or parameter changes may relieve symptoms; recurrence remains a risk until the triggers and failure path are understood.
Rebuild 只是七個選項之一。Rebuild is one option out of seven.
每個模組可依限制選擇不同的處理方式。真正的工作是比較各選項的代價,說明選擇理由,再排出執行順序。Each module may need a different approach depending on its constraints. The work is comparing the costs, explaining the choice and deciding the order of execution.
這七個選項是評估集合,不代表固定執行順序;依風險、可逆性、成本與驗證方式逐案判斷。These seven options are a decision set, not a fixed migration sequence. Assess risk, reversibility, cost, and verification for each module.
加硬體之前,先量出瓶頸在哪一層。Measure which layer is the bottleneck before adding hardware.
資源競爭與排隊是高併發下常見的效能問題,也需要檢查同步、狀態一致性、重試與故障傳播。先用證據定位限制,再判斷擴容或程式調整是否有效;若共享瓶頸未解除,增加節點可能加重下游負載。Contention and queuing are common performance problems under high concurrency. Synchronization, state consistency, retries and failure propagation also need examination. Locate the constraint with evidence before choosing scaling or code changes; if a shared bottleneck remains, more nodes may increase downstream load.
數值為用量/配置上限,CPU 以配額百分比表示;橫條呈現各自的使用比例。這些示意值不代表通用警戒門檻。Values show usage / configured limit; CPU is a percentage of its quota. Bars show each resource's utilization ratio. These illustrative values are not universal alert thresholds.
連線池接近上限,值得優先追查,但尚不能據此判定 DB 是瓶頸。需對照取得連線的等待時間、逾時率、查詢耗時與 DB 負載,再決定調整連線池、查詢或容量。A connection pool near its limit warrants investigation, but does not establish a database bottleneck. Compare connection acquisition wait, timeout rate, query duration and database load before changing the pool, queries or capacity.
技術棧決定實作方式,但不限制分析方法。Technology shapes implementation. Evidence guides diagnosis.
實際工作裡,語言和框架通常是既有系統決定的。真正要弄懂的是它在 production 的行為:runtime 特性、資源模型、會怎麼壞。In practice the language and framework are decided by the system that already exists. What matters is understanding its behavior in production: runtime characteristics, resource model, and how it breaks.
合作前會確認既有技術棧、可取得的證據,以及所需的除錯、程式審查、修改或維運範圍。能閱讀某種語言,不等於能承擔該系統的部署與營運責任。
Before an engagement, we establish the existing stack, available evidence, and the debugging, review, implementation, or operations work required. Reading a language does not by itself establish the ability to deploy and operate a system.