Before the incident

Health, a short list of signals worth paging on, and logs you can search across services. That is the operating picture. Logs on single machines are evidence you will assemble too late.

The stack is secondary. Alert only on what you would wake for. Put it in while the system is still small enough to explain.