INDEPENDENT SYSTEMS ENGINEERING
中文ENGLISH
CAPABILITIES

從系統診斷,到能落地的工程改善。From system diagnosis to engineering changes you can put into practice.

以下依問題類型說明處理範圍、判斷所需的證據,以及可交付的工程產出。技術棧會影響實作方式,但不是判斷問題的起點。Explore the problems I address, the evidence needed to assess them, and the engineering deliverables. The technology stack shapes implementation; diagnosis starts with system behavior.

PRACTICES

實際處理的六類問題。The six types of problems I take on.

01 / SYSTEM ANALYSIS

先把系統的實際樣子畫出來Start by drawing what the system actually is

多數團隊手上的架構圖,跟線上實際跑的已經不一樣。第一步是還原真正的呼叫關係、資料流、共用資源和隱含依賴,並標出哪些部分現在沒有人解釋得清楚。For most teams the architecture diagram no longer matches what runs in production. The first step is reconstructing the real call paths, data flows, shared resources, and implicit dependencies, and marking the parts nobody can currently explain.

證據EVIDENCE程式碼、schema、設定、監控、部署流程、事故紀錄code, schema, configuration, monitoring, deployment process, incident records
交付DELIVERS系統現況地圖、風險清單、可改動性評估a map of the system as it is, a risk list, an assessment of what can be changed
02 / ARCHITECTURE

用 trade-off 說明架構,不只給一張圖Explain architecture as trade-offs, not as a diagram

架構決策需要權衡收益與代價。我的工作是把選項、代價和撤退路線寫清楚,讓團隊自己覆核得了,而不是只能相信一個外部意見。Architectural decisions involve benefits, costs, and constraints. My job is to write down the options, the costs, and the way back, so the team can review the reasoning instead of having to trust an outside opinion.

證據EVIDENCE業務成長預期、容量現況、團隊規模與交付節奏growth expectations, current capacity, team size and delivery cadence
交付DELIVERS決策紀錄(ADR)、邊界定義、演進順序decision records (ADRs), boundary definitions, an order of evolution
03 / LEGACY MODERNIZATION

從「哪些不應該被改」開始Start from what should not be touched

替每個模組決定 keep / fix / refactor / extract / replace / migrate / rebuild,並排出先後順序,以維持可發布、可驗證、可回滾為改造目標。Assign every module one of keep / fix / refactor / extract / replace / migrate / rebuild, then sequence them so the system stays shippable and reversible throughout.

證據EVIDENCE業務規則盤點、改動頻率、故障集中區、資料一致性需求an inventory of business rules, change frequency, where failures cluster, consistency requirements
交付DELIVERS模組決策表、分階段計畫、每階段的驗證與回滾方式a per-module decision table, a staged plan, verification and rollback for each stage
04 / HIGH CONCURRENCY & PERFORMANCE

找出最先飽和的那個資源Find the resource that saturates first

處理 tail latency、lock contention、connection pool 耗盡、queue 堆積、cache 失效風暴、stateful 瓶頸。優化前後用同一組指標比較,不靠感覺驗收。Tail latency, lock contention, connection pool exhaustion, queue build-up, cache stampedes, stateful bottlenecks. Before and after are compared on the same metrics rather than signed off on impression.

證據EVIDENCEpercentile 分佈、profiling、DB wait event、資源飽和度percentile distributions, profiling, DB wait events, resource saturation
交付DELIVERS瓶頸分析、改動清單與預期效果、容量模型a bottleneck analysis, a change list with expected effect, a capacity model
05 / PRODUCTION ENGINEERING

讓事故可被觀測、可被解釋Make incidents observable and explainable

處理間歇性故障、部署風險、回滾路徑、降級行為,以及依賴故障怎麼擴散開來。目標是下一次發生時抓得到,不只是這次修好。Intermittent failures, deployment risk, rollback paths, degradation behavior, and how a dependency failure spreads. The goal is catching it next time, not only fixing it this time.

證據EVIDENCE事故時間軸、logs / traces 覆蓋度、變更紀錄incident timelines, log and trace coverage, change records
交付DELIVERSobservability 缺口清單、事故處理流程、rollout 設計a list of observability gaps, an incident process, a rollout design
06 / DATA & MIGRATION

在不能停機的前提下搬資料Move data without taking the system down

schema 演進、雙寫與影子流量比對、切換點設計、一致性邊界、失敗中止條件。migration 難的不是搬,是中途出錯時能停得下來。Schema evolution, dual writes and shadow-traffic comparison, cutover design, consistency boundaries, abort conditions. The hard part of a migration is not moving the data, it is being able to stop halfway when something goes wrong.

證據EVIDENCE資料量與成長、讀寫比、正確性要求、可接受的落後時間volume and growth, read/write ratio, correctness requirements, acceptable lag
交付DELIVERS搬遷計畫、比對機制、切換與中止條件a migration plan, a comparison mechanism, cutover and abort conditions
HOW IT STARTS

合作方式取決於問題的性質。How we work together depends on the problem.

DIAGNOSTIC
系統診斷System diagnosis

短期、範圍明確。針對一個具體症狀或一次架構評估,給出書面分析和建議的執行順序。Short and clearly scoped. For one specific symptom or a single architecture review, you get a written analysis and a recommended order of work.

適用:不確定問題在哪
產出:分析報告 + 選項比較
FOR: the problem is not located yet
OUTPUT: analysis + options compared
PROJECT
專案執行Project delivery

目標明確的改造或效能工程,我負責設計與實作,和內部團隊一起上線、一起驗證。A defined piece of remediation or performance engineering. I design and implement it, then ship and verify it together with the in-house team.

適用:已知要做什麼
產出:可上線的改動 + 驗證結果
FOR: the work is already known
OUTPUT: shippable changes + verification
ONGOING
長期顧問Retained advisory

固定時段參與架構決策、事故覆盤和技術審查,讓判斷留在團隊裡,不是留在我這裡。Regular time on architecture decisions, incident reviews, and technical review, so the judgment stays with the team rather than with me.

適用:系統持續演進中
產出:決策紀錄 + 固定技術審查 + 改善追蹤
FOR: a system that keeps evolving
OUTPUT: decision records + regular technical reviews + improvement tracking

診斷以縮小範圍、整理證據與提出修復選項為交付,不預設後續需要大型專案。專案另行確認範圍、實作責任、受控上線、驗收與回滾條件。A diagnostic delivers evidence, a narrower problem scope, and remediation options without committing you to a larger project. Project scope separately defines implementation, controlled rollout, acceptance, and rollback conditions.

METHOD / HOW I LOOK AT A SYSTEM

問題不一定在程式碼裡。The problem is not always in the code.

從業務需求到 production 維運,每一層都在限制上一層能做的事;架構上的每個決定,最後也都要在 production 的現實裡兌現。所以我不會只看一層。From business requirement down to production operation, each layer constrains what the one above it can do, and every architectural decision eventually has to hold up against production reality. So I do not read a single layer in isolation.

01Business Requirements
02System Behavior
03Architecture
04Application
05Database
06Cache / Queue
07Network
08Infrastructure
09Deployment
10Observability
11Production Operations
每一層都用同一組維度看EVERY LAYER READ THROUGH THE SAME SET
Latency
平均值正常,不代表 tail latency 正常。A healthy average can hide high tail latency.
Throughput
辨認限制吞吐的資源與相依流程Identify resources and dependencies that limit throughput
Concurrency
併發模型會影響失敗與傳播方式The concurrency model shapes how it fails
Capacity
剩餘 headroom 應可量化,避免容量不足時才發現。Quantify remaining headroom before capacity runs out.
Failure Modes
故障時如何降級,以及如何限制影響Degradation behavior and containment of failures
Consistency
業務能接受的正確性邊界The correctness boundary the business accepts
Dependencies
別人的故障怎麼變成你的故障How someone else's outage becomes yours
Recovery
復原時間目標(RTO)與復原點目標(RPO)Recovery time objective (RTO) and recovery point objective (RPO)
Cost
加機器之前先確認瓶頸在哪Locate the bottleneck before buying capacity

同一個症狀,在不同層的解法完全不同。分析要做的不是列出所有可能,是把範圍縮到可以驗證。The same symptom has entirely different fixes depending on the layer. Analysis is not about listing every possibility, it is about narrowing the range until it can be verified.

PRODUCTION TROUBLESHOOTING

Production 事故要靠證據,不能靠猜。Production incidents are settled by evidence, not guesses.

事故持續時,先降低影響並恢復服務,同時盡量保留現場證據。服務穩定後,再縮小問題範圍。重啟、加機器或改參數可能緩解症狀;若未釐清觸發條件與失敗路徑,仍有復發風險。During an incident, reduce impact and restore service while preserving evidence where possible. Once the service is stable, narrow the investigation. Restarts, extra machines or parameter changes may relieve symptoms; recurrence remains a risk until the triggers and failure path are understood.

可能的 constraintPOSSIBLE CONSTRAINTS
Application Database Network Queue Lock contention Connection pool Storage GC / runtime Downstream dependency
DIAGNOSTIC LOOP
IN症狀:P99 latency 上升,錯誤率不變Symptom: P99 latency rising, error rate flat
01界定範圍 — 哪些 endpoint、哪個時段、哪個版本之後Scope it: which endpoints, which hours, after which release
02收集證據 — metrics、logs、traces、profiling、DB wait eventCollect evidence: metrics, logs, traces, profiling, DB wait events
03關聯信號 — 時間軸對齊,區分因果與巧合Correlate signals: align timelines, separate cause from coincidence
04隔離重現 — 讓問題可控地再發生一次Isolate and reproduce: make it happen again under control
05找出真正的 constraint,而不是最明顯的異常Identify the real constraint, not the most visible anomaly
06最小改動 — 可回滾、可觀測、影響面明確Smallest change: reversible, observable, with a known blast radius
07再測一次 — 用同一組指標證明改善,而不是感覺Measure again: prove the improvement on the same metrics
完成改動後,需用同一組指標複驗,才能確認改善是否成立。After the change, verify it against the same metrics to establish whether it improved the system.
LEGACY MODERNIZATION / DECISION SET

Rebuild 只是七個選項之一。Rebuild is one option out of seven.

每個模組可依限制選擇不同的處理方式。真正的工作是比較各選項的代價,說明選擇理由,再排出執行順序。Each module may need a different approach depending on its constraints. The work is comparing the costs, explaining the choice and deciding the order of execution.

決定CALL什麼時候用WHEN IT APPLIES需要的證據EVIDENCE NEEDED
KEEP能正確運作、改動風險大於收益的部分Works correctly, and changing it risks more than it returns先確認它為什麼是對的Confirm why it is right
FIX正在造成事故或阻擋演進的具體缺陷A specific defect causing incidents or blocking evolution可重現的失敗案例A reproducible failure case
REFACTOR行為不變,但讓後續改動變便宜Behavior unchanged, but future changes get cheaper行為等價的驗證方式A way to verify behavioral equivalence
EXTRACT邊界清楚的能力,先切開再處理A capability with clear boundaries: separate it first資料與呼叫關係盤點An inventory of data and call relationships
REPLACE同介面換實作,用流量逐步驗證Same interface, new implementation, verified gradually with traffic雙寫或影子流量比對Dual writes or shadow-traffic comparison
MIGRATE資料與流量的搬遷,含明確回滾路徑Moving data and traffic, with an explicit rollback path一致性與切換點設計Consistency and cutover design
REBUILD只在其他六個都被評估過之後Only after the other six have been assessed完整的業務規則盤點A complete inventory of business rules

這七個選項是評估集合,不代表固定執行順序;依風險、可逆性、成本與驗證方式逐案判斷。These seven options are a decision set, not a fixed migration sequence. Assess risk, reversibility, cost, and verification for each module.

WHERE IS THE ACTUAL CONSTRAINT?

加硬體之前,先量出瓶頸在哪一層。Measure which layer is the bottleneck before adding hardware.

資源競爭與排隊是高併發下常見的效能問題,也需要檢查同步、狀態一致性、重試與故障傳播。先用證據定位限制,再判斷擴容或程式調整是否有效;若共享瓶頸未解除,增加節點可能加重下游負載。Contention and queuing are common performance problems under high concurrency. Synchronization, state consistency, retries and failure propagation also need examination. Locate the constraint with evidence before choosing scaling or code changes; if a shared bottleneck remains, more nodes may increase downstream load.

Throughput Tail latency P95 / P99 Backpressure Connection management Lock contention Hot path Queue depth Database contention Cache strategy Horizontal scaling Stateful bottleneck Resource saturation Network RTT Capacity limit
RESOURCE UTILIZATION 示意 · ILLUSTRATIVEILLUSTRATIVE
CPU quota62%
Conn. pool97 / 100
Queue depth440 / 1,000 msgs
Network TX18 / 100 Mbps
Storage IOPS350 / 1,000

數值為用量/配置上限,CPU 以配額百分比表示;橫條呈現各自的使用比例。這些示意值不代表通用警戒門檻。Values show usage / configured limit; CPU is a percentage of its quota. Bars show each resource's utilization ratio. These illustrative values are not universal alert thresholds.

連線池接近上限,值得優先追查,但尚不能據此判定 DB 是瓶頸。需對照取得連線的等待時間、逾時率、查詢耗時與 DB 負載,再決定調整連線池、查詢或容量。A connection pool near its limit warrants investigation, but does not establish a database bottleneck. Compare connection acquisition wait, timeout rate, query duration and database load before changing the pool, queries or capacity.

TECHNOLOGY

技術棧決定實作方式,但不限制分析方法。Technology shapes implementation. Evidence guides diagnosis.

實際工作裡,語言和框架通常是既有系統決定的。真正要弄懂的是它在 production 的行為:runtime 特性、資源模型、會怎麼壞。In practice the language and framework are decided by the system that already exists. What matters is understanding its behavior in production: runtime characteristics, resource model, and how it breaks.

合作前會確認既有技術棧、可取得的證據,以及所需的除錯、程式審查、修改或維運範圍。能閱讀某種語言,不等於能承擔該系統的部署與營運責任。

Before an engagement, we establish the existing stack, available evidence, and the debugging, review, implementation, or operations work required. Reading a language does not by itself establish the ability to deploy and operate a system.

先談問題,再談範圍。Talk about the problem first, scope after.
了解合作方式Explore how we can work together