03 — Logs, activity logs and settings (as built)

Repos. Paths without a prefix are in core = bigc-tms-core-v2 (worktree /Users/thanosorn/Works/Software/tms/wt-v2-prototype-core). Paths prefixed store: are in store-v2 (worktree /Users/thanosorn/Works/Software/tms/wt-v2-prototype-store). Line numbers are at the shas below.

1. Summary

The platform gives every module three things: settings (typed ops knobs declared in code, one Mongo record per area, read from an in-pod cache, saved whole with a revision check, secrets AES-256-GCM encrypted), a log base (one declaration creates the collection, its TTL + indexes, a retention setting log.<key> and an injectable writer, buffered or transactional) and activity_logs (the one shared business-history log, written inside the business transaction, with a unique event_key that turns a redelivered message into a no-op). A request context (AsyncLocalStorage) carries trace_id, actor and channel across HTTP, Kafka, job workers and lane ticks and stamps them on every log row and winston line. The same commit also cut 15 over-built pieces of the foundation ("foundation simplify", §2.3).

repo	branch	sha	feature commits	folders
core	main-v2-prototype (origin)	3d70475a	3dd1c5a (feature + simplify), d685533 (break settings → activity_log → log import loop)	src/settings, src/log, src/activity_log, src/common/context, src/common/utils/{mask,body_store,external_call,crypto,redis,transaction,logger}.ts, module *.settings.ts / *_log.schema.ts in src/{auth,google,kafka,script_runner,storage_setting}
store	main-v2-prototype	0851a3a	69c96e5	store:src/pages/setting/{authen,google,kafka}, store:src/services/setting, store:src/services/kafka, store:src/apis/{client,server}/setting, store:src/pages/api/settings/[code]

2. File map

2.1 Core

file	responsibility
src/settings/settings.types.ts	SettingRecord/SettingKey shapes, SETTING_SECRET_MASK '••••' (:1), pub/sub channel settings:changed (:2), reload interval (:4), name pattern ^[a-z][a-z0-9_]*$ (:5), SettingAudit hook type (:37)
src/settings/settings.registry.ts	defineSettingRecord (:6-8), defineSetting (:10-22), validateSettingValue (:25-36), SettingsRegistry with boot-time checks (:38-108), process-wide settingsRegistry (:111)
src/settings/settings.module.ts	@Global SettingsCoreModule (model, controller, service) (:10-24); SettingsModule.forFeature(refs) registers refs at module-definition time (:29-36)
src/settings/settings.service.ts	boot create-or-ignore, cache, typed get/getSecret, onChange, pub/sub + 60 s reload, save/reset with rev check and audit hook
src/settings/settings.controller.ts	GET/PUT /v1/settings[/:code], POST /v1/settings/:code/reset, admin only
src/settings/settings.schema.ts	collection _settings (:7), unique code (:39)
src/settings/settings.dto.ts	SaveSettingsInput, ResetSettingsInput, shared scalar schemas Flag / Text (:23-31)
src/settings/settings.fake-spec.ts	in-memory fakeSettings() for unit specs
src/log/base_log.schema.ts	BaseLog (at, expire_at, trace_id) every log schema extends
src/log/log.registry.ts	LogDefinition (:8-17), buildLogSchema (TTL + trace_id + own indexes, autoIndex off) (:19-27), ensureLogIndexes (:30-39), prepareLogRow (:42-46)
src/log/log.module.ts	LogModule.forFeature(defs) (:22-45), LogModule.forRoot() (:48-54)
src/log/log.decorators.ts	@InjectLog(key) → token LOG_WRITER:<key>
src/log/log.constants.ts	record log (:4), retention bounds 1–365 (:5-6, :13-18), buffer limits (:7-10), DAY_MS
src/log/expire_at.ts	defineLogRetention (:14-20), logRetention (:26-31), expireAt (:34-36)
src/log/buffered.writer.ts	BufferedLogWriter, apiLogDroppedTotal counter (:8)
src/log/transactional.writer.ts	TransactionalLogWriter.record(session, rows) / recordStandalone(rows)
src/activity_log/activity_log.schema.ts	ActivityLog + ACTIVITY_LOG_DEFINITION (transactional, 60 d, 3 indexes)
src/activity_log/activity_log.service.ts	record / recordStandalone, row shaping from context, registers itself as the settings audit (:33-42)
src/activity_log/activity_log.query.ts	cursor-paged reads for record history and actor search
src/activity_log/activity_log.controller.ts / .dto.ts / .constants.ts	GET /v1/activity_logs, GET /v1/activity_logs/:entity/:entity_id; DTOs; page size 100, search window 31 d
src/activity_log/changes.ts	diffChanges (:9-16), prepareChanges mask + 16 KB cap (:19-26)
src/activity_log/event_key.ts	eventKey(...parts), ActivityAlreadyApplied, isAlreadyApplied, isDuplicateKeyError
src/common/context/request_context.ts	AsyncLocalStorage store, W3C traceparent parse/emit
src/common/context/context.middleware.ts	Fastify onRequest hook opening an http context (registered src/main.ts:24)
src/common/context/actor.ts	actor type, systemActor() (:28), actorFromSession() (:33-45), setSystemUserId
src/common/guards/auth.guard.ts	sets the context actor after token validation (:44-45)
src/common/utils/mask.ts	log masking: CENTRAL_MASK_KEYS (:79), maskDeep (:96-104), maskUrl (:107-123), maskBody (:126-137)
src/common/utils/body_store.ts	64 KB body cap, cutLogBodies(row) (:42-53)
src/common/utils/external_call.ts	externalCall.http / .run: mandatory timeout + outbound traceparent; no logging
src/common/utils/crypto.ts	AES-256-GCM, k1: versioned ciphertext, env STORAGE_SECRET_ENC_KEY
src/common/utils/redis.ts	publish (:127-129), subscribe on a dedicated connection (:132-154)
src/common/utils/transaction.ts	inTransaction(conn, work) with afterCommit hooks
src/common/utils/logger.ts	winston logger; adds trace_id from context to every line (:4-8)
src/common/base/base.service.ts	optional { session } on create/update/delete (§2.3)
src/auth/auth.settings.ts	record authen
src/google/google.settings.ts, google_log.schema.ts, google_public.controller.ts, google.service.ts	record google, log google_logs, GET /v1/settings/public, call logging
src/kafka/kafka.settings.ts, kafka.config.ts, kafka_log.schema.ts, kafka.producer.ts, kafka.consumer.ts, kafka.service.ts, kafka.controller.ts	record kafka, log kafka_logs (in/out), reconnect on onChange, pod status in Redis
src/script_runner/script_setting.settings.ts, script_setting.service.ts	record script (sandbox limits)
src/storage_setting/storage_setting.settings.ts, storage_setting.service.ts, storage_setting.module.ts	record storage; S3 client rebuilt on change
src/delivery_order/delivery_order.service.ts	DO create/update/Kafka upsert write one activity row in the DO transaction
src/plan_history/plan_history.service.ts	plan history row + activity row in the plan transaction
src/user/user.service.ts	seeds the system user, sets systemActor id (:28-30)
src/jobs/side_effect.service.ts, fast_lane.worker.ts, long_lane.worker.ts, lossy.processor.ts	copy trace_id + actor onto job rows and reopen that context in the worker

2.2 Store-v2

file	responsibility
store:src/services/setting/index.ts	useSetting({name}) hook: load record, form mapping, save with rev, 409 reload
store:src/services/kafka/index.ts	kafka page hook: save + "Test connection"
store:src/apis/client/setting/index.ts	browser → Next API /settings/:name, /kafka/test-connection
store:src/apis/server/setting/index.ts	Next server → core /settings/:code, /settings/public
store:src/pages/api/settings/[code]/index.ts	Next API proxy GET/PUT
store:src/pages/setting/{authen,google,kafka}/index.tsx	hand-written setting pages
store:src/pages/_app.tsx	loads /settings/public into pageProps.settings (:130-131)
store:src/__tests__/pages/setting-google.test.tsx, store:src/services/setting/__tests__/setting.test.ts, store:jest.config.js	first jest + testing-library suite of store-v2

2.3 Foundation simplify (same commit) — what common/base and the platform look like after it

Cut list source: _docs/bigc-tms-core-v2/tms-v2-foundation-simplify-cut-list.md (why: D-406/D-424). Verified against the code at 3d70475a:

#	cut	state at 3d70475a
1	direct Kafka produce path + KafkaProducer facade	gone; src/kafka/producers/ deleted, every send goes KafkaTarget.send → KafkaService.send → KafkaProducerLifecycle.send (src/kafka/kafka.target.ts:79-83, src/kafka/kafka.producer.ts:55-70)
2	dead code (_reconcileProducer, isKafkaUnreachable, pollOnce, KafkaConsumed.error, Job.heartbeat_at …)	gone from src; heartbeat_at remains only on JobWorker where it is read (src/jobs/partition_lease.service.ts:81)
3	demo schedules in the prod module	src/schedules/demo.schedules.ts deleted
4	consume handler took an array	ConsumeHandler = (message) => Promise<string | undefined> (src/kafka/kafka.types.ts:14)
5	two producer connect paths	one: reconcile drops on signature change then ensure() (src/kafka/kafka.producer.ts:82-91)
6	12 unset env tuning knobs	literals (src/jobs/jobs.constants.ts:17-37, src/schedules/schedules.constants.ts:7-14)
7	duplicated Text/Flag setting schemas	one shared Flag/Text (src/settings/settings.dto.ts:23-31)
8	_bounds() metadata	gone; GET has no bounds
9	per-key onChange	record-level only (src/settings/settings.service.ts:93-98)
10	storage.allowed_image_extensions	gone
11	setLogContextProvider hook	prepareLogRow calls requestContext.traceId() directly (src/log/log.registry.ts:44)
12	LogModule.forRetention()	folded into forRoot() (src/log/log.module.ts:48-54)
13	job_stats collection + _stat/countDead	gone (src/jobs/job_stats.schema.ts deleted)
14	retiredIndexes + LOG_DROP_RETIRED_INDEXES	gone; boot never drops an index; old dev indexes are dropped by hand
15	one-time seed from the old prototype setting collections	gone; values are re-entered once in admin

Kept on purpose (D-407): Bull lossy lane, at_least_once via KAFKA_TARGET_TOPICS, 64-partition lease, schedules + long lane.

src/common/base after the commit:

file	shape
base.schema.ts	unchanged: BaseSchema (_translates, created_at/updated_at timestamps), BaseSchemaFactory.createForClass (toJSON: virtual id, no _id, Decimal128 → number). Log schemas and _settings also use the factory
base.service.ts	new WriteOptions { session? } (:11-13); create(data, {session}) (:205-211), update(id, data, {session}) (:213-225), delete(id, {session}) (:227-235) pass the session to Mongo; without one they behave as before. No logging, no hooks, no activity writes in the base — callers write activity rows themselves inside inTransaction

Also in the commit: ScriptEntitiesRegistry (src/script_runner/script_entities.registry.ts:13-33) — each domain module registers {name, service, controller} in its onModuleInit (e.g. src/delivery_order/delivery_order.module.ts:45), replacing script_runner's direct domain imports. Covered in the script/registry part, listed here only because it shipped in this commit.

3. Data model

3.1 _settings (model name settings, collection _settings)

Schema src/settings/settings.schema.ts:6-39. Written only through the raw collection (updateOne), so Mongoose defaults never run; the shape below is what _bootRecord (src/settings/settings.service.ts:253-271) and _write (:213-242) store.

field	type	required	default	validation	index	note
code	string	yes	—	^[a-z][a-z0-9_]*$ at registry (settings.registry.ts:71-73)	unique {code:1} (settings.schema.ts:39)	record id, e.g. kafka
owner	string	no	from defineSettingRecord	—	—	$setOnInsert only; never rewritten
name	string	no	name or code (settings.registry.ts:7)	—	—	$setOnInsert only; GET/list read the name from code, not from the doc
rev	number	no	1 on boot insert	—	—	optimistic-lock counter; every save/reset $inc 1
values	Mixed	no	{} on insert	each declared key validated by its schema on save and on load	—	one entry per key; an encrypted key holds k1:iv:tag:data; undeclared keys are kept untouched
created_at	Date	no	boot time	—	—	timestamps createdAt only (settings.schema.ts:9-12)
updated_by	{id?, name?}	no	—	—	—	from the HTTP profile (settings.controller.ts:44-47)
updated_at	Date	no	—	—	—	set on every save/reset

No default, declared, last_seen_at or bounds field (removed, D-414). The old settings collection of the prototype is left untouched (src/app.module.spec.ts:60).

3.2 Log collections (all extend BaseLog)

Common fields (src/log/base_log.schema.ts:4-8), written by prepareLogRow (src/log/log.registry.ts:42-46):

field	type	required	note
at	Date	yes	write time (pod clock)
expire_at	Date	no	at + log.<key>.days read at write time (src/log/expire_at.ts:34-36)
trace_id	string | null	no	requestContext.traceId(); null outside a context

Every log gets {expire_at:1} TTL expireAfterSeconds: 0 and, unless searchable:false, {trace_id:1} (src/log/log.registry.ts:23-24). autoIndex/autoCreate are off; ensureLogIndexes creates them one by one at provider creation (:30-39).

activity_logs — src/activity_log/activity_log.schema.ts:20-48, mode transactional, default 60 d, searchable:false.

field	type	required	note
entity	string	yes	e.g. delivery_order, plan, setting
entity_id	string	yes	stringified id (activity_log.service.ts:72); for settings = record code
action	string	yes	e.g. do-create, do-update, setting-change, plan actions
action_id	string	yes	context.action_id or a fresh uuid per record call; shared by every row of one call (:68)
actor	{type, id, role, name, meta, via}	no	item.actor → context actor → systemActor() (:69, :75)
channel	string	no	http/kafka/job/schedule, system without context (:76)
reason	string	no	settings: save/reset
changes	Mixed	no	{field: {from, to}}, masked, capped 16 KB (changes.ts:19-26)
changes_truncated	boolean	no	true when the masked JSON > 16 KB → every field becomes {truncated:true}
changes_bytes	number	no	size of masked JSON before cap
refs	Mixed	no	free refs (settings: {rev}; plan: event fields)
event_key	string	yes	business key or uuid; unique

Indexes (:43-47): {entity:1, entity_id:1, at:-1, _id:-1}, {'actor.id':1, at:-1, _id:-1}, {event_key:1} unique, + TTL. No trace_id index.

google_logs — src/google/google_log.schema.ts:10-37, buffered, default 30 d (google.constants.ts:20).

field	type	required	note
system	string	yes	always google
operation	string	yes	token | optimize
method, url	string	no	url masked by maskUrl
status	number	no	HTTP status
outcome	OK|ERROR|TIMEOUT	yes
duration_ms, attempt	number	no	attempt is always 1
req_headers, res_headers	Mixed	no	lower-cased keys, masked
req_body, res_body	Mixed	no	masked then cut to 64 KB
req_body_truncated, res_body_truncated	boolean	no
req_body_bytes, res_body_bytes	number	no	size before cut
error	{code, message}	no

Extra index {system:1, operation:1, at:-1}.

kafka_logs — src/kafka/kafka_log.schema.ts:8-32, buffered, default 60 d (kafka.constants.ts:27).

field	type	required	enum	note
direction	string	yes	in, out
topic, key, message_id	string	no		message_id parsed from the JSON value
partition	number, offset string	no		out: from the broker ack
req_body / res_body (+ _truncated, _bytes)	Mixed	no		out: req_body = masked value, res_body = {partition, offset}; in: req_body = masked payload (or raw text), res_body = {status:'done'} or {error}
status	string	yes	sent, consumed, failed
error	{message}	no

Extra index {at:-1}.

3.3 Redis keys

key / channel	client	value	TTL	writer → reader
channel settings:changed	redisCache pub/sub	{code, rev}	—	SettingsService._afterWrite (settings.service.ts:247) → every pod's _onPublished (:323-328)
kafka:status:<hostname>:<pid>	redisCache	{pod_id, status: connected|disconnected|disabled, last_connected_at}	30 s (kafka.constants.ts:8)	KafkaService._writeStatus every 10 s and on every reconcile (kafka.service.ts:178-189); deleted on shutdown (:74) → GET /v1/kafka/status

3.4 Relations

from	to	how
_settings record log	every log collection	key log.<collection> = {days}; one key per LogModule.forFeature def (src/log/log.module.ts:23)
activity_logs.entity/entity_id	delivery_order, plan (plan_code), _settings.code	string refs, no FK
activity_logs.actor.id	users._id (incl. seeded system user)	systemActor() id set at UserService.onModuleInit (src/user/user.service.ts:28-30)
plan_history.event_key	activity_logs.event_key	same key plan:<code>:<revision>:<action> (src/plan_history/plan_history.service.ts:29-31); plan_history is its own business collection without TTL
kafka_logs.message_id	kafka_consumeds.consume_id, DO event_key do:<delivery_no>:msg:<message_id>	by value
jobs.trace_id/actor	log rows of the work the job does	the worker reopens the context (src/jobs/fast_lane.worker.ts:42-47)

4. API

Global prefix /api, URI versioning → /api/v1/... (src/main.ts:32-37). Errors come from throwError as {statusCode, code, message: [..], additional_info, error, ...extra} (src/common/utils/error.ts:4-18). The guard rejects with Nest's default 403.

method + route	permission	request	response	errors	code
GET /v1/settings	role admin	—	[{code, owner, name, editable: true, updated_at}] one per registered record	403	settings.controller.ts:16-19, settings.service.ts:112-120
GET /v1/settings/:code	admin	—	{code, owner, name, rev, updated_by, updated_at, keys: [{key, value, code_default, encrypt}]}; encrypted value = •••• or null, code_default null for encrypted	404 not_found "Setting record not found" (undeclared code) or "…not loaded yet"	settings.controller.ts:21-24, settings.service.ts:122-143
PUT /v1/settings/:code	admin	body {rev: int ≥0, values: object}	{rev} (new rev)	400 DTO; 400 invalid_setting {key} "<key> is not declared" / schema message / "must be a string"; 404; 409 rev_conflict {rev: current}	settings.controller.ts:26-33, settings.service.ts:145-188
POST /v1/settings/:code/reset	admin	body {rev, keys: string[] non-empty}	{rev}	400 invalid_setting (undeclared, or "encrypted keys cannot be reset"); 404; 409	settings.controller.ts:35-42, settings.service.ts:190-211
GET /v1/settings/public	public	—	{google_map_api_key}	—	src/google/google_public.controller.ts:13-17
GET /v1/activity_logs	admin	query actor?, actor_role? (UserRole), entity?, entity_id?, entity_type?, action?, from, to (dates), cursor?, limit? 1–100	{rows: [{id, ...row}], next_cursor} newest first	400 invalid_input: no actor and no entity+entity_id, missing from/to, to < from, range > 31 d, bad cursor; DTO 400	activity_log.controller.ts:12-15, activity_log.query.ts:36-58
GET /v1/activity_logs/:entity/:entity_id	admin	query cursor?, limit?	{rows, next_cursor}	400 bad cursor	activity_log.controller.ts:17-24, activity_log.query.ts:32-34
GET /v1/kafka/status	any logged-in user (no @Roles)	—	[{pod_id, status, last_connected_at}] sorted by pod	403 without token	src/kafka/kafka.controller.ts:8-11, kafka.service.ts:128-132
POST /v1/kafka/test-connection	any logged-in user	{bootstrap_servers?, authen_type?, authen_setting?}; blank/•••• password or key → saved secret	{success, message}	never throws	kafka.controller.ts:14-19, kafka.service.ts:90-125
GET /v1/plan_history/plan/:code	admin, store_operator	limit? (default 100)	plan_history rows newest first	—	src/plan_history/plan_history.controller.ts:31-38

Cursor = base64url of <at ISO>|<_id>; next page filter at < c.at OR (at = c.at AND _id < c._id), sort {at:-1,_id:-1}, index hinted (activity_log.query.ts:13-22, 60-71). Rows from search have _id renamed to id.

There is no read API for google_logs or kafka_logs (query them in Mongo by trace_id).

5. Conditions

5.1 Settings

when …	the system must …	enforced at	test
a record code or key does not match ^[a-z][a-z0-9_]*$	fail boot naming it	settings.registry.ts:71-73, 87-89	settings.service.spec.ts:354
the same code is declared with another owner, or a key twice with a different schema/encrypt/default	fail boot	settings.registry.ts:75-77, 99-105	settings.service.spec.ts:359
a key defaults to null and is not encrypt	fail boot	settings.registry.ts:90-92	untested
a key's code default fails its own schema	fail boot	settings.registry.ts:93-96	untested
a pod boots and the record is missing	insert {code, owner, name, values:{}, rev:1}; an existing record is never touched	settings.service.ts:253-271	settings.service.spec.ts:115, 124, 260
a key has no stored value	read its code default	settings.service.ts:303-305	settings.service.spec.ts:401
a stored value fails its schema at load	serve the code default, warn once per new rev, never write the DB	settings.service.ts:279-288	settings.service.spec.ts:175
code calls get before the first load, or with a ref not registered through forFeature	throw	settings.service.ts:330-339	settings.service.spec.ts:364
get on an encrypted key / getSecret on a plain key	throw "use getSecret" / "use get"	settings.service.ts:76-78, 85-87	settings.service.spec.ts:304
getSecret on an unset secret	return null; otherwise decrypt at call time (cache holds only ciphertext)	settings.service.ts:88-90	settings.service.spec.ts:296
PUT/reset with a rev other than the stored one	409 rev_conflict with the current rev; checked before validating and again atomically in the update filter	settings.service.ts:349-356, 222-230	settings.service.spec.ts:206, 217
PUT carries a key the record does not declare	400 naming the key	settings.service.ts:151-155	untested
a plain key value (sent, or the stored one when not sent) fails its schema	400 with the first constraint message and {key}; nothing written	settings.service.ts:169-173	settings.service.spec.ts:134, 143
an encrypted key is sent as undefined or ••••	keep the stored ciphertext, no change row	settings.service.ts:163, 182	settings.service.spec.ts:287
an encrypted key is sent as null	clear it (store null)	settings.service.ts:183	untested
an encrypted key is sent as a non-string	400 "must be a string"	settings.service.ts:164-166	untested
an encrypted key is sent as a string	store k1: + AES-256-GCM ciphertext	settings.service.ts:183, crypto.ts:32-34	settings.service.spec.ts:270
reset names an encrypted key	400 "encrypted keys cannot be reset"	settings.service.ts:197-199	settings.service.spec.ts:369
reset	set each key to the serving pod's code default	settings.service.ts:206-209	settings.service.spec.ts:249
a save or reset commits	write one activity_logs row (entity: setting, action: setting-change, reason: save|reset, refs.rev) in the same transaction; encrypted keys recorded as {key, changed: true}	settings.service.ts:231-237, activity_log.service.ts:33-42	activity_log.service.spec.ts:186, 200
the audit write throws	the save fails, nothing published	settings.service.ts:231 (inside inTransaction)	settings.service.spec.ts:168
no audit is registered (ActivityLogModule not imported)	save succeeds with no history	settings.service.ts:231 (this._audit?.)	settings.service.spec.ts:161
the save committed	reload this pod's cache, then publish settings:changed; a publish failure only warns	settings.service.ts:244-251	settings.service.spec.ts:330
a cached record has a rev ≥ the incoming doc's rev	ignore the incoming doc (no notify)	settings.service.ts:308-311, 274	settings.service.spec.ts:421, 436, 449
a record's cached value is replaced (not first load)	call every onChange(record) handler; a throwing handler is logged	settings.service.ts:299, 313-321	settings.service.spec.ts:374
a non-admin calls any /settings admin route	403	settings.controller.ts:8, auth.guard.ts:34-36	settings.service.spec.ts:341

5.2 Logs

when …	the system must …	enforced at	test
a module injects a log key nobody declared	fail boot (no provider for LOG_WRITER:<key>)	log.decorators.ts:4-5	log.module.spec.ts:149
a log is declared	register log.<key> = {days: defaultRetentionDays} (1–365) and create TTL + trace_id + own indexes	log.module.ts:23-28, log.registry.ts:19-27	log.module.spec.ts:105, 202
an index cannot be created (spec clash)	log an error, keep booting, keep writing	log.registry.ts:33-37	log.module.spec.ts:136
a log is removed from code	nothing dropped; rows expire on their own expire_at	no code path deletes	log.module.spec.ts:158
a row is written	expire_at = now + current days; a retention change applies to new rows only	expire_at.ts:33-36	log.module.spec.ts:171, activity_log.service.spec.ts:109
retention set to 0 or 366	400 "1–365"	log.constants.ts:13-18 via settings.service.ts:170-173	settings.service.spec.ts:134, 143, 152
TransactionalLogWriter.record without a session	throw, write nothing	transactional.writer.ts:12	log.module.spec.ts:184
recordStandalone fails	log the error, never throw	transactional.writer.ts:18-26	untested
a buffered write() row fails prepareLogRow (e.g. cast error)	drop it, apiLogDroppedTotal[key]++, caller unaffected	buffered.writer.ts:43-48	buffered.writer.spec.ts:35
the buffer exceeds 1,000 rows	drop the oldest, count it	buffered.writer.ts:70-73	buffered.writer.spec.ts:45
100 rows are buffered	flush now; otherwise every 1 s	buffered.writer.ts:32, 50	buffered.writer.spec.ts:70
an insert fails with write errors	duplicates (11000) count as written; other rejected rows are counted and dropped, not retried	buffered.writer.ts:82-90	buffered.writer.spec.ts:79, 89, 106
an insert fails without write errors (Mongo down)	put the batch back at the front and stop until the next flush	buffered.writer.ts:91-93	buffered.writer.spec.ts:45
the app shuts down gracefully	flush the buffer (onModuleDestroy, shutdown hooks on)	buffered.writer.ts:62-65, main.ts:52	buffered.writer.spec.ts:61
a body (masked) is ≤ 65,536 bytes of UTF-8 JSON	store it as is, truncated:false	body_store.ts:31	body_store.spec.ts:9
a body is larger	store the first ≤ 65,536 bytes as a string cut on a character boundary, truncated:true, bytes = original	body_store.ts:32-35	body_store.spec.ts:15, 23, 30
a body is a Buffer, stream or multipart/*	store {binary:true, bytes, content_type} only	body_store.ts:25-28	body_store.spec.ts:41, 46
a log field name matches the central list (exact, after dropping -/_ and lower-casing) or ends in token/secret/password/apikey	replace a non-null value with ***, at any depth; URL query values the same	mask.ts:79-104, 107-123	mask.spec.ts:4-32
a body is a JSON string	parse it so its fields are masked, before cutting	mask.ts:126-137	mask.spec.ts:27
externalCall is called without timeoutMs > 0	throw	external_call.ts:26-28	external_call.spec.ts:55
an outbound HTTP call is made	add traceparent of the current trace; abort at timeoutMs (+50 ms signal)	external_call.ts:34-46	external_call.spec.ts:37, 43
externalCall.run exceeds timeoutMs	abort the signal and reject ExternalCallTimeoutError (ETIMEDOUT)	external_call.ts:49-64	external_call.spec.ts:59
a Google log write throws	count it, swallow, the caller gets its result	google.service.ts:69-72	google.service.spec.ts:114

5.3 Activity logs

when …	the system must …	enforced at	test
record(session, inputs)	write one row per input, one shared action_id, one insertMany in the caller's session	activity_log.service.ts:46-56, 65-82	activity_log.service.spec.ts:142
the event_key already exists	throw ActivityAlreadyApplied so the caller's transaction aborts	activity_log.service.ts:51-53	activity_log.service.spec.ts:155
the history insert fails	the business change and its job rows roll back, error surfaces	caller's inTransaction	activity_log.service.spec.ts:167, delivery_order.service.spec.ts:329
no request context	actor = system user (type user, role system_admin), channel system	activity_log.service.ts:69, 76, actor.ts:28	activity_log.service.spec.ts:208
a changed field name is a masked name	from/to become ***; other fields masked deep	changes.ts:20-22	activity_log.service.spec.ts:122
masked changes exceed 16 KB	keep only {truncated:true} per field	changes.ts:23-25	activity_log.service.spec.ts:132
a DO is created	one do-create row, event_key do:<delivery_no>:<changeKey|create>	delivery_order.service.ts:112-121	delivery_order.service.spec.ts:260
a DO is updated	one do-update row with diffChanges over the sent fields, event_key do:<delivery_no>:<changeKey|version>	delivery_order.service.ts:139-151	delivery_order.service.spec.ts:273
a Kafka DO message has no message_id	reject invalid_payload (DLQ, failed in-row), no DO write	delivery_order.service.ts:201-203	delivery_order.service.spec.ts:310
the same Kafka DO message arrives twice	the second is a no-op returning the current DO	delivery_order.service.ts:217-222	delivery_order.service.spec.ts:286
a plan action commits	plan_history row + activity row in the plan transaction, key plan:<code>:<revision>:<action>; failure of either aborts the plan save	plan_history.service.ts:28-33	plan_history.service.spec.ts:74, 109, 117, 207
an optimize call fails	standalone activity + plan_history row (uuid key)	plan_history.service.ts:36-39	untested
actor search	filter actor.id (+ actor.role), hint {actor.id, at, _id}	activity_log.query.ts:49-53	activity_log.query.spec.ts:108, 119, 128, 137
search entity_type differs from entity in a record search	return an empty page	activity_log.query.ts:56	untested

5.4 Request context

when …	the system must …	enforced at	test
an HTTP request has a valid W3C traceparent (non-zero trace id)	adopt its trace id; otherwise a new 32-hex id	request_context.ts:23-26, 35, context.middleware.ts:4-6	context.spec.ts:30, 36, 40
the token is valid and the session role is a known UserRole	set context actor {type:'user', id, role, name, meta:{username, device_id for drivers}}	auth.guard.ts:44-45, actor.ts:33-45	auth.guard.spec.ts:26-41
a Kafka message is handled	run its handler in a kafka context adopting the message's traceparent	kafka.consumer.ts:113	context.spec.ts:49
Kafka sends	add traceparent and plain trace_id headers	kafka.producer.ts:29-32, 62	context.spec.ts:73
a job is enqueued	copy trace_id (context or new) and context actor onto the row; a caller cannot pass its own	side_effect.service.ts:57	context.spec.ts:84, 109
a worker runs a job	reopen a job context with the row's trace_id, actor with via:'system', job {id, code}	fast_lane.worker.ts:42-47, 60, lossy.processor.ts:15	context.spec.ts:109, 129
a lane tick runs	open a fresh schedule context (new trace per tick)	fast_lane.worker.ts:227, long_lane.worker.ts:144	context.spec.ts:92
user create / profile update tries role system_admin	reject (system_admin is only the seeded system user)	src/common/enums/user.enum.ts:17, src/user/user.dto.ts:40	src/user/user.dto.spec.ts

6. States and flows

6.1 Setting record lifecycle

state	entered by	rev	values
absent	—	—	—
booted	first pod boot that declares the record ($setOnInsert)	1	{} — every key reads its code default
saved	PUT / reset	+1 per write	the whole effective map (stored + defaults) is written
hand-edited	Mongo edit outside the API	must $inc rev or no pod picks it up (D-369)	any; invalid values fall back to the default at load

Effect of the "whole map" write: once a record is saved, every key that existed at that moment is materialised in values, so a later code-default change no longer reaches it; only never-saved records (and keys added after the last save) follow code defaults (settings.service.ts:158, 355).

6.2 Boot

6.3 Save and propagate

6.4 Writing a log row and TTL

6.5 What is logged where

source	collection	written by	row per	context / trace
HTTP inbound request	none (access log line is not built, next round — D-388)	—	—	http context per request (context.middleware.ts:4-6); every winston line in it carries trace_id (logger.ts:4-8)
HTTP outbound to Google	google_logs	GoogleService.logged (google.service.ts:48-91)	call (token, optimize)	trace of the caller
Outbound S3	none — externalCall.run gives only the timeout (attachment.service.ts:49-51)	—	—	—
Kafka out	kafka_logs direction:out	KafkaProducerLifecycle._logOut (kafka.producer.ts:128-143)	send, sent or failed	traceparent + trace_id headers
Kafka in	kafka_logs direction:in	KafkaConsumerRunner._logIn (kafka.consumer.ts:228-241)	claimed message (duplicates skipped are not logged)	kafka context adopting the producer trace
Job worker run	none (job row itself in jobs)	—	—	job context, actor via:'system'
Lane / schedule tick	none	—	—	schedule context; the schedules service tick that enqueues schedule jobs has no context, so each enqueue gets a new trace (side_effect.service.ts:57)
Business change (DO, plan, setting)	activity_logs	ActivityLogService.record	changed record	context actor / channel
Plan screen history	plan_history (no TTL, not a LogModule log)	PlanHistoryService.record	plan revision + action	—

6.6 Kafka pod status

status	when
connected	producer connected and every wanted consumer running (kafka.service.ts:170-172)
disconnected	kafka.enabled true but not healthy; heartbeat retries reconcile every 10 s (:164-167)
disabled	kafka.enabled false

7. Jobs, schedules, queues, events

name	trigger	payload	retry	idempotency
settings reload	setInterval 60 s (settings.service.ts:63-66)	—	next tick	rev compare (:308-311)
settings:changed pub/sub	after commit of save/reset	{code, rev}	none (reload interval is the fallback)	rev compare (:323-328)
buffered log flush	setInterval 1 s, 100 rows, shutdown	rows with pre-set _id	failed batch re-queued at the front	duplicate _id = already written (buffered.writer.ts:16, 84)
Mongo TTL monitor	Mongo, ~60 s	—	—	—
Kafka heartbeat	setInterval 10 s (kafka.service.ts:66)	status row	reconcile when unhealthy	status key overwrite
Kafka reconcile	onChange(kafka), boot, heartbeat	—	pending flag reruns once more if a change lands mid-run (kafka.service.ts:139-161)	signature compare per producer/consumer

No schedule or job of this part writes or cleans logs; cleanup is TTL only. Jobs DEAD/job_stats TTL via expireAt is not built (D-318; job_stats was deleted instead).

8. Settings and env

8.1 Every setting key

get = SettingsService.get, secret = getSecret. All reads are cache reads (no Mongo).

area (record · owner)	key	type / validation	default	effect	read at
authen · auth	enable_universal_otp	boolean (Flag)	false	true → OTP verify also accepts the codes in universal_otp	auth.service.ts:46 → otp.ts:85-86 (from verifyOtp auth.service.ts:141)
authen · auth	universal_otp	encrypted string: JSON list text	null	universal OTP codes; non-JSON → warn, no code accepted	secret auth.service.ts:42-45
google · google	google_map_api_key	string (Text)	''	Maps JS key, served publicly to the browser	get google_public.controller.ts:16
google · google	route_optimization_service_account	encrypted string: service-account JSON (project_id, client_email, private_key required)	null	Route Optimization credentials; unset → 400 google_not_configured, bad → google_invalid_credentials; GoogleAuth rebuilt when the email/key hash changes	secret google.service.ts:94-123, 128-142
kafka · kafka	bootstrap_servers	string[], non-empty	KAFKA_BROKERS split on ,, else ['localhost:9092'] (at import)	brokers of producer and consumers; change → reconnect	get kafka.config.ts:22
kafka · kafka	authen_type	PLAINTEXT | SSL | SASL_PLAINTEXT | SASL_SSL	'PLAINTEXT'	kafkajs auth mode	get kafka.config.ts:23
kafka · kafka	authen_setting	{sasl_mechanism?, username?, ssl_ca_pem?, ssl_cert_pem?, ssl_reject_unauthorized?: boolean}	{}	plain half of auth	get kafka.config.ts:25
kafka · kafka	enabled	boolean	true	false → producer disconnected, every consumer stopped (≤ 2 s, fetch wait 1 s kafka.consumer.ts:20), status disabled	get kafka.config.ts:29 → kafka.service.ts:150-155, kafka.consumer.ts:66
kafka · kafka	events	[{event, event_type?, topic_name?, enabled?}]	one row per KAFKA_EVENTS (kafka.constants.ts:16-23) with its default topic, enabled:true	consume events: topic + on/off (kafka.consumer.ts:64-66); produce events: topic_name only (kafka.config.ts:35-39 → kafka.target.ts:81); produce enabled is ignored	get kafka.config.ts:30
kafka · kafka	sasl_password	encrypted string	null	SASL password (into authen_setting.password)	secret kafka.config.ts:26
kafka · kafka	ssl_key_pem	encrypted string	null	TLS client key (into authen_setting.ssl_key_pem)	secret kafka.config.ts:27
log · log	activity_logs	{days: int 1–365}	{days: 60}	expire_at of new activity rows	get expire_at.ts:30 via log.registry.ts:44
log · log	google_logs	{days: int 1–365}	{days: 30}	expire_at of new Google call rows	same
log · log	kafka_logs	{days: int 1–365}	{days: 60}	expire_at of new Kafka rows	same
script · script_runner	max_calls	int ≥ 1	20	sandbox DB calls per run	script_runner.sandbox.ts:66 (via script_setting.service.ts:13-15)
script · script_runner	max_docs	int ≥ 1	1000	documents a run may read	script_runner.sandbox.ts:92-97, 128
script · script_runner	wall_ms	int ≥ 1	2000	wall-clock budget	script_runner.sandbox.ts:199-212
script · script_runner	heap_mb	int ≥ 1	64	isolate memory limit	script_runner.sandbox.ts:171
script · script_runner	export_state_kb	int ≥ 1	64	none — declared, never read	—
script · script_runner	max_bulk_records	int ≥ 1	50	max records per bulk template render (400 bulk_limit_exceeded)	template.render.service.ts:42-50
storage · storage_setting	max_file_size_mb	int 1–100	10	upload cap = min(purpose cap, this)	attachment.service.ts:79
storage · storage_setting	allowed_extensions	string[]	[]	allow-list intersected with the purpose list; empty → every upload 400 EXTENSION_NOT_ALLOWED	attachment.service.ts:74-77
storage · storage_setting	provider	StorageProvider (aws_s3 only)	aws_s3	none — never read	—
storage · storage_setting	bucket	string	''	empty → 503 STORAGE_NOT_CONFIGURED on every attachment call	storage_setting.service.ts:32-34
storage · storage_setting	region	string	''	S3 client region	storage_setting.service.ts:46
storage · storage_setting	prefix	string	''	object key prefix	attachment.service.ts:99-100
storage · storage_setting	kms_key_id	string	''	non-empty → SSE-KMS with this key	attachment.service.ts:380-381
storage · storage_setting	credential_mode	iam_role | static	iam_role	static → client uses access_key_id + secret	storage_setting.service.ts:54-59
storage · storage_setting	access_key_id	string	''	static-mode key id	storage_setting.service.ts:56
storage · storage_setting	secret_access_key	encrypted string	null	static-mode secret	secret storage_setting.service.ts:30

onChange subscribers: kafka → reconcile (kafka.service.ts:62); storage → destroy the S3 client so the next call rebuilds it (storage_setting.service.ts:19, 64-69). Every other area is read per call, so a change applies on the next read.

8.2 Env and constants

key	default	effect	where read
STORAGE_SECRET_ENC_KEY	none (required to save/read any secret)	32-byte AES key, 64 hex or base64; missing/invalid → STORAGE_SECRET_ENC_KEY error	crypto.ts:4-22
CACHE_REDIS_URL	none (boot throws if empty)	pub/sub + kafka status	redis.ts:169
SETTINGS_RELOAD_INTERVAL_MS	60000	full cache reload period	settings.types.ts:4
LOG_BODY_CAP_BYTES	65536	log body cap	body_store.ts:4
ACTIVITY_SEARCH_MAX_DAYS	31	widest search window	activity_log.constants.ts:5
KAFKA_BROKERS	localhost:9092	code default of kafka.bootstrap_servers only	kafka.settings.ts:75
KAFKA_GROUP_PREFIX / KAFKA_CLIENT_ID	bigc-tms-core-v2	client id + consumer group prefix	kafka.constants.ts:4
KAFKA_TARGET_TOPICS	—	extra job-target topics with delivery mode	kafka.target.ts:64
S3_TIMEOUT_MS	10000	per S3 call timeout	attachment.constants.ts:16
constants	LOG_FLUSH_MS 1000, LOG_FLUSH_ROWS 100, LOG_CAP_ROWS 1000	buffer	log.constants.ts:7-10
constants	ACTIVITY_CHANGES_CAP_BYTES 16384, ACTIVITY_PAGE_SIZE/MAX 100	activity	changes.ts:6, activity_log.constants.ts:2-3
constants	STATUS_HEARTBEAT_MS 10000, STATUS_TTL_SECONDS 30	kafka status	kafka.constants.ts:7-8
constants	OPTIMIZE_TIMEOUT_MS 120000, TOKEN_TIMEOUT_MS 30000	Google call timeouts	google.constants.ts:8, 16

9. Concurrency and recovery

situation	what the code does	where
several pods boot on an empty _settings	each upserts with $setOnInsert; the E11000 loser retries the same update, which now matches	settings.service.ts:264-270
two admins save the same record	both may pass the read check; the conditional updateOne {code, rev} lets one win, the other gets 409 with the current rev	settings.service.ts:222-230
a stale reload result arrives after a newer save/publish	dropped by the rev compare; onChange fires once per new rev	settings.service.ts:274, 308-311
Redis down during save	the saving pod already reloaded after commit; other pods converge within the reload interval	settings.service.ts:244-251, 63-66
pod dies after commit, before publish	others converge on their 60 s reload	same
hand edit in Mongo without $inc rev	never picked up by a running pod (by design, D-369)	settings.service.ts:308-311
withTransaction retries the work on a transient error	afterCommit hooks reset per attempt; activity rows are re-prepared per attempt	transaction.ts:17-20
the same Kafka DO message on two pods	consume claim (kafka_consumeds) usually stops the second; if both run, the unique event_key makes the loser throw ActivityAlreadyApplied, its transaction aborts and it returns the current DO	kafka.consumer.ts:209-226, activity_log.service.ts:51-53, delivery_order.service.ts:217-222
two pods create the same new DO	loser hits DO unique E11000 and applies the message as an update	delivery_order.service.ts:210-216
same plan event saved twice	unique event_key on both plan_history and activity_logs → one row each	plan_history.schema.ts:72, plan_history.service.spec.ts:207
two pods create the same log index	createIndex is idempotent; a clashing spec is logged, boot continues	log.registry.ts:30-39
log Mongo down	callers return at once; rows re-queued; over 1,000 rows/log/pod the oldest are dropped and counted	buffered.writer.ts:67-74, 91-93
a flush batch partly written before an error	rows keep their _id, so the retry's duplicates count as written	buffered.writer.ts:16, 84
SIGTERM	enableShutdownHooks → onModuleDestroy flushes buffers; Kafka consumers stop before the producer disconnects	main.ts:52, buffered.writer.ts:62-65, kafka.service.ts:70-80
kill -9	up to 1,000 buffered rows per log per pod are lost (accepted)	—
a log row written before the settings cache loads	expireAt throws: buffered rows are dropped and counted, transactional record throws to the caller	expire_at.ts:26-31, settings.service.ts:332-334
pod dies	its Kafka status key expires after 30 s	kafka.constants.ts:8
a settings change lands while Kafka reconcile runs	pending makes it run once more	kafka.service.ts:139-161
two pods seed the system user	unique username; loser reads the winner's _id	user.service.ts:37-56

10. Web (store-v2)

route	page	API calls (browser → Next → core)	notes
/setting/authen	store:src/pages/setting/authen/index.tsx	GET/PUT /api/settings/authen → /v1/settings/authen	universal_otp as tag select; sent as one JSON string only when non-empty (store:src/services/setting/index.ts:39-43)
/setting/google	store:src/pages/setting/google/index.tsx	same for google	client-side JSON + required-field check of the service account (:9-27); secret shows •••• placeholder
/setting/kafka	store:src/pages/setting/kafka/index.tsx	GET/PUT /api/settings/kafka; POST /api/kafka/test-connection	brokers list, auth fields, sasl_password/ssl_key_pem as top-level secret fields, events table (topic + enabled)
all pages (layout)	store:src/pages/_app.tsx:130-131	GET /v1/settings/public server-side	feeds google_map_api_key to the plan map (store:src/components/business/plan/Map/index.tsx:28)
/setting/storage, /setting/script_settings	unchanged pages	still call /storage-settings and /script_settings (store:src/apis/server/storage_setting/index.ts:4-8, store:src/apis/server/script_setting/index.ts:4-8)	broken: core removed those endpoints (S-16)
logs	none	—	no web part for activity_logs, google_logs, kafka_logs, record log retention, or Kafka status

useSetting (store:src/services/setting/index.ts:49-104): loads the record, maps plain keys into the form and encrypted keys into a secrets map (set/unset); on save sends {rev, values} where untouched/blank/•••• secrets are left out (:30-47); on error toasts the message, re-reads the record and, when the rev moved, reloads the form (:69-76). No reset button (the Next proxy only has GET/PUT, store:src/pages/api/settings/[code]/index.ts:6-29).

Permission gating: the Setting menu has no roles (store:src/constants/menu.ts:67-120, filter store:src/utils/menu.ts:11), so every role sees it; core returns 403 for non-admin.

11. Tests

Core (jest, max 3 workers):

spec	covers
src/settings/settings.service.spec.ts	boot create-or-ignore, two-pod boot, 1–365 bounds, hand-edited invalid value, rev 409 incl. race, deploy/rollback adding keys, code_default, encrypted keys (mask, legacy 3-part ciphertext, byte-identical on ••••), Redis down, guard 403, registry errors, pub/sub + onChange, newest-rev cache, audit hook optional/failing
src/log/log.module.spec.ts	declare → key + indexes, two-pod boot, index clash, inject typo fails boot, log removed from code, retention change, session required, module defaults (google 30, kafka 60, activity 60)
src/log/log.registry.spec.ts	index rollout on old dbs, BaseLog schema shape, prepared row fields
src/log/buffered.writer.spec.ts	drop + count, Mongo down with 1,500 writes, SIGTERM flush, 100-row flush, duplicate/invalid rows in a batch
src/activity_log/activity_log.service.spec.ts	indexes, row shape, retention change, masking, 16 KB cap, batch action_id, redelivery, rollback, setting history rows and actors
src/activity_log/activity_log.query.spec.ts	paging without skip/repeat, at ties, IXSCAN plans, actor/role filters, index additions on old dbs, validation errors
src/common/context/context.spec.ts	traceparent adopt/ignore, Kafka in/out trace, enqueue trace/actor, tick contexts
src/common/utils/body_store.spec.ts, mask.spec.ts, external_call.spec.ts	cap edges and UTF-8 cut, binary; mask names; timeout + traceparent
src/google/google.service.spec.ts	one row per call, masking, truncation, ERROR outcome, swallowed log failure, 30 d
src/kafka/kafka_log.spec.ts	in/out rows sharing trace_id, store down, cut at 64 KB, failed handler rows, 60 d, masked secrets
src/kafka/kafka.service.spec.ts, kafka.consumer.it.spec.ts	config from settings, onChange disable/enable, producer reconnect on broker change, pod status + TTL + Redis restart, in-flight handler on disable
src/delivery_order/delivery_order.service.spec.ts	DO activity rows, Kafka redelivery, forced activity failure, topic override
src/plan_history/plan_history.service.spec.ts	plan_history + activity in one transaction, actors, failure aborts, duplicates
src/storage_setting/storage_setting.service.spec.ts	503 at defaults, client rebuilt on change
src/app.module.spec.ts	boot on dbs with old settings/app_settings, one doc per record in _settings
src/common/base/base.service.spec.ts	optional session on create/update/delete
src/common/guards/auth.guard.spec.ts, src/user/user.service.spec.ts, src/user/user.dto.spec.ts	actor per role, system user seed, system_admin not assignable
src/attachment/attachment.service.spec.ts, attachment.sweep.processor.spec.ts	S3 calls time out through externalCall.run

Store (npm test, jest + jsdom, store:jest.config.js):

spec	covers
store:src/services/setting/__tests__/setting.test.ts	form ↔ record mapping, secrets left out unless typed, OTP tags as JSON, stale-rev reload
store:src/__tests__/pages/setting-google.test.tsx	map key shown after reload, empty key, invalid service account blocked client-side

12. Gaps and known issues

#	gap	detail
1	store storage + script pages broken	they call /storage-settings and /script_settings, removed from core in this commit; ops cannot set storage.bucket/allowed_extensions or script limits from the UI, and with defaults every upload fails (503 / EXTENSION_NOT_ALLOWED). Design D-329/D-343 required all 5 records rewired
2	no UI for retention, activity history, Kafka status	record log has no page (store:src/enums/setting.ts lists only authen/google/kafka); activity_logs and /kafka/status have APIs only
3	HTTP inbound access log not built	deferred to next round (D-388); outbound S3 calls are not logged either
4	/kafka/test-connection and /kafka/status lack @Roles	any logged-in role (driver included) can make the pod connect to an arbitrary broker list (kafka.controller.ts:4-19)
5	first save freezes defaults	save writes the whole effective map (settings.service.ts:158, 355), so code-default changes stop reaching that record; design (A3/D-287) meant only touched keys to stay
6	an invalid stored value blocks the whole record	PUT validates untouched stored values too (settings.service.ts:169-173), so a hand-edited bad value must be fixed by sending a valid value for that key
7	GET /settings/:code shows the stored value, not the effective one	getRecord returns cached.values (settings.service.ts:130) while reads use effective
8	history is optional	after d685533 the audit is a hook (settings.service.ts:231); a deployment without ActivityLogModule saves settings with no history, silently (D-414 C3 wanted it mandatory)
9	dead keys	script.export_state_kb and storage.provider are declared, validated and shown but never read; kafka.events[].enabled is ignored for produce events
10	design items not built	per-record editPermission / setting.manage (admin role only, D-319); declared.last_seen_at / "not declared since" (removed D-414); 8 MB buffer byte cap (row cap only, D-414); S3 overflow body store; /metrics for apiLogDroppedTotal (in-process counter only, D-344); key rotation script; jobs DEAD TTL via expireAt (D-318); owner/name rewrite on boot (D-334 says "rewritten each boot", code does $setOnInsert)
11	schedule enqueue tick has no context	src/schedules never calls requestContext.run; each schedule job gets a fresh trace id rather than one per tick
12	old indexes are never dropped	retired {actor:1, at:-1} and pre-_id search indexes stay on any db that had them until dropped by hand (cut-list row 14)
13	context.action_id is never set	every record call gets its own uuid; one HTTP action writing through two services gets two action_ids (activity_log.service.ts:68)
14	settings cache must load before any log write	a log write in onModuleInit or before onApplicationBootstrap of SettingsCoreModule throws in expireAt
15	apiLogDroppedTotal is per process and unread	visible only in logs/specs

13. How to add the next one

13.1 A new setting area

1. Create src/<module>/<module>.settings.ts:
   • export const X_SETTINGS = defineSettingRecord({ code: 'x', owner: '<module>', name: 'X' }) (code and keys ^[a-z][a-z0-9_]*$).
   • One defineSetting(X_SETTINGS, { key, schema, default }) per key. Scalars validate as { value } — reuse Flag / Text from src/settings/settings.dto.ts:23-31 or write a small class with a value property and class-validator rules (copy src/storage_setting/storage_setting.settings.ts:6-27). Object values validate as the class itself (copy AuthenSetting in src/kafka/kafka.settings.ts:21-41).
   • Secrets: defineSetting<string>(..., { schema: Text, default: null, encrypt: true }) — the only keys allowed a null default; read with getSecret; cannot be reset.
2. In the module: imports: [SettingsModule.forFeature([X_SETTINGS, ...keys])] (copy src/storage_setting/storage_setting.module.ts:8-15). A ref not passed here throws on get.
3. Read with settings.get(REF) / settings.getSecret(REF) at call time (never in a constructor or onModuleInit — the cache loads in onApplicationBootstrap). If the module holds a client built from settings, subscribe settings.onChange(X_SETTINGS, rebuild) and unsubscribe in onModuleDestroy (copy src/storage_setting/storage_setting.service.ts:19-25, 64-69).
4. Specs: use fakeSettings() (src/settings/settings.fake-spec.ts) for unit specs; Mongo-backed specs follow src/settings/settings.service.spec.ts.
5. Web: add the code to store:src/enums/setting.ts, a page under store:src/pages/setting/<code>/index.tsx using useSetting({ name }) (copy the google page), and a menu entry in store:src/constants/menu.ts.
6. A Mongo hand edit or migration of _settings must $inc: { rev: 1 } in the same update.

13.2 A new log collection

1. Name it <module>_logs (one per owning module; there is no shared outbound log, D-406).
2. Schema src/<module>/<module>_log.schema.ts: @Schema({ strict: true }) class XLog extends BaseLog with the module's own fields (copy src/google/google_log.schema.ts). For bodies use req_body/res_body + _truncated + _bytes.
3. Definition: { key: '<module>_logs', schema: XLog, mode: 'buffered', defaultRetentionDays: N, indexes: [[{...}]] }. Use transactional only for rows that must commit with a business write.
4. Module: imports: [LogModule.forFeature([X_LOG_DEFINITION])] → creates log.<key> retention (1–365), TTL, trace_id index and own indexes.
5. Service: constructor(@InjectLog('<module>_logs') private log: BufferedLogWriter). Shape the row yourself: mask (maskDeep/maskBody/maskUrl from src/common/utils/mask.ts), then cutLogBodies(row) from src/common/utils/body_store.ts, then this.log.write(row). Wrap in try/catch and count drops like src/google/google.service.ts:54-73; never let a log failure reach the caller.
6. Outbound calls go through externalCall.http / externalCall.run with a timeoutMs (src/common/utils/external_call.ts); log around the call as GoogleService.logged does.
7. Specs: copy src/google/google.service.spec.ts (row shape, masking, truncation, expire_at) and add the default to the log.module.spec.ts C7 check.

13.3 Business history for a new entity

1. Import ActivityLogModule, inject ActivityLogService.
2. Do the business write and the history write in one inTransaction(this.model.db, async (tx) => ...), passing { session: tx.session } to BaseService.create/update/delete (copy src/delivery_order/delivery_order.service.ts:112-121, 139-151).
3. activity.record(tx.session, { entity, entity_id, action, changes: diffChanges(before, after, fields), event_key }). Set event_key from the business event (eventKey('<entity>', id, messageId)) whenever the same event can arrive twice; catch isAlreadyApplied(error) outside the transaction and treat it as done (copy delivery_order.service.ts:217-222). Pass mask: [...] for extra sensitive field names.
4. Outcomes with no business write (a failed external call) use recordStandalone.
5. Do not set actor unless it differs from the request context; the guard, Kafka context and job context already supply it.