06 · Media — storage settings, attachments, sweep, content link, data-script and template guards

As-built reference for TMS v2 phase-1 media. Paths without a prefix are in core = bigc-tms-core-v2 (worktree /Users/thanosorn/Works/Software/tms/wt-v2-prototype-core). Paths prefixed store: are in store-v2 (worktree /Users/thanosorn/Works/Software/tms/wt-v2-prototype-store). Line numbers are at the shas below.

Related siblings, not repeated here: 03-logs-and-settings.md (the _settings record mechanics, cache, rev check, secret encryption, externalCall), 04-jobs-engine.md (the durable job engine the sweep does not use), 02-auth.md (guard and roles).

1. Summary

An admin sets the S3 connection and the upload limits in the storage settings record. The browser asks core for a presigned PUT, uploads the bytes straight to S3 under <prefix>/v2/incoming/<uuid>, then calls confirm. Confirm moves the row pending → scanning with a guarded update, checks size (HeadObject), real type (4 KB ranged GET + magic bytes) and a malware verdict (stub, always clean), copies the object to <prefix>/v2/files/<uuid>, deletes the incoming copy and sets ready. A failure sets failed with a failed_reason and deletes the incoming object. A Bull repeatable job every 60 s fails pending rows older than 6 min and finishes scanning rows stuck for 5 min. Files are read through short-lived presigned GETs, or through the permanent link GET /attachments/:id/content that 302s to a fresh presign; the store proxies that link and sends a logged-out user to the login form. The same media rounds hardened the template engine (no file system, include/render/layout blocked) and the data-script filters (hidden/masked fields cannot be filtered, sorted, whole-value matched through a parent, or reached with $jsonSchema/$text).

repo	branch	sha	media commits	later refactors that changed media	folders
core	origin/main-v2-prototype	3d70475a	57f7e49 (storage + attachment + sweep + content link), b6972dd (AWS only, S3_ENDPOINT, no default checksum), 1bfe884 (template + data-script hardening, env/lockfile hygiene), 221052b (r5: $jsonSchema/$text, parent whole-value match)	38dad93 (jobs engine, no media change), 3dd1c5a (storage moved into the settings module, /storage-settings controller removed, S3 calls through externalCall.run with S3_TIMEOUT_MS)	src/attachment/, src/storage_setting/, src/common/enums/{attachment,storage_setting}.enum.ts, src/common/utils/{crypto,redis,external_call}.ts, src/script_runner/script_runner.service.ts, src/template_engine/template_engine.ts
store	main-v2-prototype	0851a3a	107e04f (pages, BFF, upload helper, preview, login redirect), 7d02252 (no Endpoint/Force path style, thumbnails on permanent link, logged-out redirect), 8d8e0a0 (lockfile/env hygiene)	none	store:src/pages/setting/{storage,attachments}/, store:src/pages/api/{attachments,storage-settings}/, store:src/apis/{client,server}/{attachment,storage_setting}/, store:src/services/{attachment,storage_setting}/, store:src/components/attachment/, store:src/components/ui/ImagePreview/

Answer to the auth writer's question: the store Storage page hits GET|PUT /api/storage-settings (store:src/apis/client/storage_setting/index.ts:4-10), whose BFF calls core GET|PUT <API_GATEWAY>/storage-settings (store:src/apis/server/storage_setting/index.ts:3-9). That controller existed at b6972dd (src/storage_setting/storage_setting.controller.ts, @Controller('storage-settings')) and was deleted in 3dd1c5a (commit note S-16). At 3d70475 the only storage endpoint is the generic GET|PUT /api/v1/settings/storage (src/settings/settings.controller.ts:21-33). The store page therefore gets 404 for both read and save (§12 G1).

2. File map

Core

file	responsibility
src/attachment/attachment.constants.ts	TTLs (PUT 300 s, GET 900 s), rate limits (20 pending, 30 URL/min), 4096 magic bytes, dirs v2/incoming and v2/files, sweep queue name/jobId/interval/batch, expiry (6 min pending, 5 min scanning), S3_TIMEOUT_MS
src/attachment/attachment.purposes.ts	Purpose registry in code; one purpose admin_file (private, jpg jpeg png webp pdf, 20 MB)
src/attachment/attachment.schema.ts	attachments collection and 3 indexes
src/attachment/attachment.dto.ts	AttachmentInput (create), AttachmentSearchQuery (list, limit ≤ 100)
src/attachment/attachment.mime.ts	detectMime (png, jpeg, webp, pdf by magic bytes) and isSameMime (image/jpg = image/jpeg, parameters ignored)
src/attachment/attachment.scanner.ts	MalwareScanner interface, MALWARE_SCANNER token, StubMalwareScanner (logs, returns clean)
src/attachment/attachment.service.ts	Create upload URL, confirm, verify, completeReady, fail, get, list, content URL, serialize, presign GET, S3 helpers; s3Run wraps every S3 call in externalCall.run
src/attachment/attachment.sweep.processor.ts	Bull processor attachment_sweep: registers the 60 s repeatable job, fails expired pending, resumes stale scanning
src/attachment/attachment.controller.ts	5 routes under /api/v1/attachments, class-level @Roles([admin])
src/attachment/attachment.module.ts	Model, Bull queue, StorageSettingModule, providers (service, processor, stub scanner)
src/common/enums/attachment.enum.ts	AttachmentStatus, AttachmentScanStatus, AttachmentVisibility
src/storage_setting/storage_setting.settings.ts	storage settings record and its 10 keys (one encrypted)
src/storage_setting/storage_setting.service.ts	Reads the keys into one object, 503 when no bucket, caches one S3Client per pod, destroys it on any storage change
src/storage_setting/storage_setting.module.ts	Registers the record and keys with SettingsModule.forFeature
src/storage_setting/storage_setting.constants.ts	STORAGE_SETTING_CODE, STORAGE_SECRET_ENC_KEY — used only by specs now
src/common/enums/storage_setting.enum.ts	StorageProvider (aws_s3), StorageCredentialMode (iam_role, static)
src/common/utils/crypto.ts	AES-256-GCM with k1: prefix, key from env STORAGE_SECRET_ENC_KEY (shared by every encrypted setting, see 03)
src/common/utils/redis.ts:156-162	incr(key, ttl) — sets TTL on the first hit only (fixed window)
src/common/utils/external_call.ts:49-64	externalCall.run: AbortSignal + timeout race, throws ExternalCallTimeoutError
src/script_runner/script_runner.service.ts	Data-script reader: filter allow-list, hidden/masked path checks, parent rules, sort check, projection and masking
src/script_runner/script_runner.constants.ts:21	FORBIDDEN_FILTER_OPERATORS ($where $function $accumulator $expr)
src/template_engine/template_engine.ts	Liquid engine with a no-disk fs, blocked file tags, render/parse/memory limits
.dockerignore, .gitignore:37-38	.env and .env.* kept out of git and images (!.env.example in .dockerignore)

Store

file	responsibility
store:src/pages/setting/storage/index.tsx	Storage form: limits, connection, credential mode, masked secret
store:src/services/storage_setting/index.ts, types.ts	Load/save hook; blank secret → keep current; mask ********
store:src/apis/{client,server}/storage_setting/index.ts, store:src/pages/api/storage-settings/index.ts	Browser → BFF → core /storage-settings (dead route, §12 G1)
store:src/pages/setting/attachments/index.tsx	Attachments grid/list, search, status filter, paging, upload modal, image preview
store:src/services/attachment/index.ts	List state (page, limit 10, search, status), upload with toast by error code
store:src/services/attachment/upload.ts	uploadFile: create → PUT to S3 → confirm
store:src/apis/client/attachment/index.ts	Browser calls; attachmentContentUrl(id) = /api/attachments/<id>/content
store:src/apis/server/attachment/index.ts	BFF → core; content call with redirect: 'manual'
store:src/pages/api/attachments/index.ts, [id]/index.ts, [id]/confirm.ts, [id]/content.ts	BFF routes; content passes the 302 through, 401/403 → 302 to login
store:src/components/attachment/UploadModal/index.tsx	Single-file picker with drag-drop
store:src/components/ui/ImagePreview/index.tsx	Shared preview modal: download (opens original), expand full screen, close
store:src/enums/attachment.ts	Web copies of statuses and error codes
store:src/hooks/session/index.ts:6-16	getLoginRedirectUrl: same-origin path only, no control chars or backslash
store:src/constants/menu.ts:93-100	Menu items Setting → Storage, Attachments

3. Data model

attachments (model attachment, attachment.schema.ts:4-62, strict, timestamps created_at/updated_at)

Every field is @Prop() without required or default; the service sets them.

field	type	set by	enum / validation	note
purpose	string	create	must be a key of ATTACHMENT_PURPOSES (attachment.service.ts:67-71)	only admin_file
visibility	string	create, from purpose	AttachmentVisibility = private only
source	string	create	constant v2 (attachment.constants.ts:6)	marks v2 rows apart from a future v1 migration
incoming_key	string	create	<prefix>/v2/incoming/<uuid v4> (attachment.service.ts:98-99,374-377)	unique index
key	string	create (planned), re-set at ready	<prefix>/v2/files/<same uuid> (:100, :245)	set at create, so a non-ready row already holds its future key
name	string	create	non-empty string (DTO)	original file name; never used in a key
size	number	create	int ≥ 1 (DTO); ≤ min(purpose cap, storage cap)	signed into the PUT as Content-Length
mime	string	create	any string (DTO)	client-declared; signed into the PUT as Content-Type
detected_mime	string	verify	detectMime result	written before the copy, so a resumed row can skip re-verify
scan_status	string	create pending, verify clean/infected	AttachmentScanStatus
uploader_id	string	create, @Profile('id')	—	index with status; never used to filter reads
status	string	every transition	AttachmentStatus: pending, scanning, ready, failed
failed_reason	string	fail / sweep	SIZE_MISMATCH, TYPE_MISMATCH, INFECTED, TIMEOUT, expired	serialized only when failed
scanning_started_at	Date	confirm claim, sweep claim	—	stale detection
confirmed_at	Date	completeReady	—
_translates	Mixed	—	—	from BaseSchema, unused

Indexes (attachment.schema.ts:60-62): {status:1, created_at:1} (sweep pending scan), {uploader_id:1, status:1} (pending cap count), {incoming_key:1} unique. There is no index for the stale-scanning query (status + scanning_started_at); the {status, created_at} prefix covers status only. The list sorts on created_at:-1 with an optional status and a name regex.

storage settings record (collection _settings, code storage, owner storage_setting, storage_setting.settings.ts:29-51)

Mechanics (rev, cache, pub/sub reload, masking, encryption) are in 03 §3.1, §6.3.

key	type	default	validation	effect
max_file_size_mb	int	10	1–100 (:6-11)	upload cap = min(purpose cap, this) (attachment.service.ts:79)
allowed_extensions	string[]	[]	array of strings (:13-17)	intersected with the purpose list; lower-cased at use (attachment.service.ts:74-75). Empty → every upload 400
provider	enum	aws_s3	StorageProvider	never read by code
bucket	string	''	string	empty → 503 STORAGE_NOT_CONFIGURED on every attachment call and the sweep skips (storage_setting.service.ts:32-34)
region	string	''	string	S3Client region (storage_setting.service.ts:46)
prefix	string	''	string	key prefix, / trimmed (attachment.service.ts:375)
kms_key_id	string	''	string	non-empty → SSE-KMS on presigned PUT and CopyObject (attachment.service.ts:379-391)
credential_mode	enum	iam_role	StorageCredentialMode	static → explicit credentials, else the AWS default chain (storage_setting.service.ts:54-59)
access_key_id	string	''	string	static mode key id
secret_access_key	encrypted string	null	string or null	static mode secret; getSecret decrypts (storage_setting.service.ts:30)

Gone since 3dd1c5a: the singleton storage_settings collection with version, and its keys allowed_image_extensions, endpoint, force_path_style (the last two already dropped in b6972dd). No seed copies an old storage_settings doc into _settings (the "one-time seed" in the 3dd1c5a message was removed by the same commit's simplify pass; grep finds no seed code).

Redis keys

key / channel	redis	value	TTL	where
attachment:upload_rate:<uploader_id>:<epoch minute>	CACHE_REDIS_URL	integer counter	60 s, set on the first INCR	attachment.service.ts:84-88, redis.ts:156-162
bull:attachment_sweep:*	QUEUE_REDIS_URL	Bull repeatable job, jobId attachment_sweep, every 60000 ms	job removed on complete and on fail	attachment.sweep.processor.ts:30-41, app.module.ts:40-42
channel settings:changed	CACHE_REDIS_URL	{code:'storage', rev}	pub/sub	see 03; triggers the S3 client reset

Relations

from	to	how
attachments.uploader_id	users._id (as string)	set from the session profile; no lookup
attachments	S3 objects	incoming_key while pending/scanning; key once ready
attachments	business records	none — no ref_type/ref_id (draft D-200 contract cut by D-202)
AttachmentService	storage record	through StorageSettingService on every call

4. API

Global prefix /api, URI version v1 (src/main.ts:32-37). Every attachment route has class-level @Roles([UserRole.admin]) (attachment.controller.ts:8): the guard compares session.user.role with admin (src/common/guards/auth.guard.ts:34-36), and a missing token, unknown token or other role returns Nest's default 403 {statusCode:403, message:'Forbidden resource', error:'Forbidden'} (auth.guard.ts:25-36,86-87). There are no permission keys at this sha (the setting.attachment.* keys in 02-auth §4 come from a later build). Error body from throwError = {statusCode, code, message:[..], additional_info, error:'', ...extra} (src/common/utils/error.ts:4-18). DTOs run through the global ValidationPipe with implicit conversion (main.ts:45-50), so a failed DTO is 400 with Nest's validation messages.

method + route	access	request	response	errors	code
POST /api/v1/attachments	role admin	body {purpose: string, name: non-empty string, size: int ≥ 1, mime: string}	201 {id, upload_url, method:'PUT', headers:{'Content-Type', 'Content-Length', +'x-amz-server-side-encryption':'aws:kms', +'x-amz-server-side-encryption-aws-kms-key-id' when kms_key_id}, expires_at (ISO, now+300 s)}	503 STORAGE_NOT_CONFIGURED; 400 PURPOSE_UNKNOWN; 400 EXTENSION_NOT_ALLOWED; 400 FILE_TOO_LARGE; 429 RATE_LIMITED "Too many upload requests" (31st/min); 429 RATE_LIMITED "Too many pending uploads" (≥ 20 pending); 400 DTO	attachment.controller.ts:16-23, attachment.service.ts:64-139
POST /api/v1/attachments/:id/confirm	admin	no body	200 serialized row (normally ready; scanning when another caller holds it)	503; 404 NOT_FOUND; 422 <reason> with extra {status:'failed', failed_reason} for SIZE_MISMATCH, TYPE_MISMATCH, INFECTED, TIMEOUT, expired; 500 for a non-ObjectId id (CastError, §12 G9); S3 errors other than 404 bubble as 500	attachment.controller.ts:25-29, attachment.service.ts:141-173,362-367
GET /api/v1/attachments	admin	query page? (positive), limit? (positive, ≤ 100, default 20), search? (name, case-insensitive, regex-escaped), status? (AttachmentStatus)	{items: [serialized], total, page, limit} sorted created_at desc	503; 400 DTO (e.g. limit=101)	attachment.controller.ts:31-34, attachment.service.ts:295-319
GET /api/v1/attachments/:id/content	admin	—	302, Location = new presigned GET (900 s), Cache-Control: no-store	503; 404 NOT_FOUND (unknown or non-ObjectId id, no Mongo query for the latter); 409 ATTACHMENT_NOT_READY	attachment.controller.ts:36-41, attachment.service.ts:282-293
GET /api/v1/attachments/:id	admin	—	serialized row	503; 404 NOT_FOUND; 500 for a non-ObjectId id	attachment.controller.ts:43-46, attachment.service.ts:273-280
GET /api/v1/settings/storage	admin	—	{code:'storage', owner, name:'Storage', rev, updated_by, updated_at, keys:[{key, value, code_default, encrypt}]}; secret value •••• or null	see 03 §4	src/settings/settings.controller.ts:21-24
PUT /api/v1/settings/storage	admin	{rev, values:{<key>: value}}; secret: omit or •••• = keep, null = clear, any other string = new secret	{rev}	400 invalid_setting {key} (undeclared key such as allowed_image_extensions, schema error such as max_file_size_mb 150); 409 rev_conflict	settings.controller.ts:26-33, settings.service.ts:145-188
POST /api/v1/settings/storage/reset	admin	{rev, keys[]} (not the secret)	{rev}	see 03	settings.controller.ts:35-42
GET|PUT /api/v1/storage-settings	—	—	not built at this sha (removed in 3dd1c5a); Nest 404	—	—

Serialized row (attachment.service.ts:321-345): {id, name, mime (declared), size, visibility, status, failed_reason (only when failed, else null), url, thumb_url, expires_at, created_at}. For ready rows url is a fresh presigned GET with ResponseContentType = detected_mime || mime and, for non-images, ResponseContentDisposition: attachment (:347-360); thumb_url = the same url for image/*, else null; expires_at = now + 900 s. Non-ready rows carry url, thumb_url, expires_at = null. List serializes rows one by one, so it presigns once per ready row (:313-316).

5. Conditions

when …	the system must …	enforced at	test
storage.bucket is empty	answer 503 STORAGE_NOT_CONFIGURED on create, confirm, get, list, content; skip the sweep with one info log	storage_setting.service.ts:27-36; callers attachment.service.ts:65,142,274,283,296; attachment.sweep.processor.ts:45-50	storage_setting.service.spec.ts:42
the purpose is not in ATTACHMENT_PURPOSES	400 PURPOSE_UNKNOWN, no row	attachment.service.ts:67-71	attachment.service.spec.ts:429
the file extension (text after the last ., lower-cased) is not in both the purpose list and storage.allowed_extensions	400 EXTENSION_NOT_ALLOWED, no row	attachment.service.ts:73-77,369-372	attachment.service.spec.ts:437 (svg)
size > min(purpose max_size_mb, storage.max_file_size_mb) × 1 MiB	400 FILE_TOO_LARGE, no row	attachment.service.ts:79-82	attachment.service.spec.ts:375 (purpose cap 20 wins over 100), :385 (storage cap 5 wins)
a user asks for the 31st upload URL within one clock minute	429 RATE_LIMITED, no row (the counter still counts refused calls)	attachment.service.ts:84-88	attachment.service.spec.ts:415
a user already has ≥ 20 pending rows	429 RATE_LIMITED, no row; soft cap, count-then-insert can overshoot under a burst (D-176)	attachment.service.ts:90-96	attachment.service.spec.ts:445
a create passes	insert a pending row first, then presign PUT for 300 s signing only content-type and content-length; keys are a new uuid, never the name	attachment.service.ts:98-126	attachment.service.spec.ts:140,191
kms_key_id is set	sign ServerSideEncryption: aws:kms + key id into the PUT, return both x-amz-server-side-encryption* headers, and pass them on CopyObject	attachment.service.ts:120,135,230,379-391	attachment.service.spec.ts:164,180,496,508
the client PUTs a body whose length differs from the signed Content-Length	S3 refuses with 403 (signature mismatch) — S3 enforces it, not core	presign attachment.service.ts:119,125	qa only (verify V-04)
confirm finds the row pending	claim it pending → scanning with findOneAndUpdate({_id, status:pending}); only one caller wins	attachment.service.ts:144-148	qa AC-08 (two pods, 10 confirms); no unit race test
confirm finds the row scanning or ready	return the current row, no S3 call	attachment.service.ts:155-157	attachment.service.spec.ts:279
confirm finds the row failed	422 with the stored failed_reason	attachment.service.ts:158-160	attachment.service.spec.ts:291
HeadObject is missing or its ContentLength ≠ size	fail SIZE_MISMATCH (covers confirm before upload)	attachment.service.ts:176-179,406-413	attachment.service.spec.ts:203,359
the first 4096 bytes are not png/jpeg/webp/pdf or differ from the declared mime	fail TYPE_MISMATCH	attachment.service.ts:182-193, attachment.mime.ts:1-34	attachment.service.spec.ts:222
the scanner says infected	set scan_status: infected, fail INFECTED	attachment.service.ts:195-206	attachment.service.spec.ts:256
verify passes	write detected_mime and scan_status: clean before the copy	attachment.service.ts:208-216	attachment.service.spec.ts:455
an S3 call exceeds S3_TIMEOUT_MS (body stream included)	abort it; in confirm, fail the row TIMEOUT (HTTP 422)	attachment.service.ts:49-51,170,182-189,427-429	attachment.service.spec.ts:302,320,338
a row is failed	write failed + reason (guard status: scanning) before a best-effort delete of the incoming object; a delete error is logged, never replaces the reason (D-181)	attachment.service.ts:258-271	attachment.service.spec.ts:236
completeReady finds the incoming object	CopyObject to key with CopySource encoded per path segment, then delete incoming	attachment.service.ts:223-232,393-395	attachment.service.spec.ts:455,516 (Thai, space, plus)
completeReady finds no incoming object	require key to exist (crash after delete), else fail SIZE_MISMATCH	attachment.service.ts:233-238	attachment.sweep.processor.spec.ts:245
the ready write runs	set ready, confirmed_at with guard status: scanning; if the guard misses, return the current row	attachment.service.ts:240-255	attachment.service.spec.ts:455
a ready row is read	presign GET 900 s, forced content type; non-images as attachment disposition; thumb_url only for images	attachment.service.ts:337-360	attachment.service.spec.ts:395,541
the content link is opened	presign anew on every call; 302 + no-store; 404 unknown/non-ObjectId; 409 not ready	attachment.service.ts:282-293, attachment.controller.ts:36-41	attachment.controller.spec.ts:45,56,65; attachment.service.spec.ts:555,578,582
the content link is opened without a token	403 from the global guard; the service never runs (D-206 accepts 403 over 401)	auth.guard.ts:25-27,86-87	attachment.controller.spec.ts:93
list limit > 100	400 DTO	attachment.dto.ts:34-37	attachment.service.spec.ts:594,598
a pending row is older than 6 min at sweep time	fail it expired (guarded on pending + cutoff), then delete incoming, errors swallowed	attachment.sweep.processor.ts:56-73	attachment.sweep.processor.spec.ts:108,124
a scanning row's scanning_started_at ≤ now − 5 min	claim it by moving scanning_started_at to now (guarded); re-verify unless detected_mime and scan_status: clean are already stored; then completeReady	attachment.sweep.processor.ts:75-102	attachment.sweep.processor.spec.ts:143,170,193,217,245
one sweep row throws (failure, S3 timeout)	log a warning and continue with the next row; a timed-out row stays scanning for a later tick	attachment.sweep.processor.ts:99-101	attachment.sweep.processor.spec.ts:315,334
S3_ENDPOINT env is set	build the client with that endpoint and virtual-hosted style; unset → AWS host	storage_setting.service.ts:48-52	storage_setting.client.spec.ts:66,76,86
a presigned PUT is signed	no default CRC32 checksum (requestChecksumCalculation: WHEN_REQUIRED)	storage_setting.service.ts:45-46	storage_setting.client.spec.ts:73
the storage record changes on any pod	destroy the cached S3Client so the next call builds one from the new values	storage_setting.service.ts:19,64-69	storage_setting.service.spec.ts:52
credential_mode = static and no secret is stored	not checked — the client is built with secretAccessKey: null and S3 calls fail at request time (the old SECRET_REQUIRED check died with /storage-settings)	storage_setting.service.ts:54-59	untested

Data-script and template guards (media rounds 4–5)

Data scripts read only registered entities (delivery_order, delivery_order_item, vehicle, plan, user, store; *.module.ts scriptEntities.register). The restricted paths of an entity are its controller's @HideFields plus @MaskFields (script_runner.service.ts:142-144); today user.password is hidden (src/user/user.controller.ts:22). attachments is not a script entity. Every refusal is ScriptRunError script_error (script_runner.error.ts:31-33); through POST /templates/render it comes back as HTTP 200 with result.error {code, message} (src/template/template.render.service.ts:74-82, D-257).

when a script filter or sort …	the system must …	enforced at	test
holds $where, $function, $accumulator or $expr at any depth	refuse <op> is not allowed in a filter; depth > 20 refused	script_runner.service.ts:18-25, script_runner.constants.ts:21	script_runner.service.spec.ts:69; :163
holds a top-level $ key other than $and, $or, $nor, $comment (allow-list)	refuse operator not allowed: <op>	script_runner.service.ts:27-29,66-67	script_runner.service.spec.ts:150,154,158,168
holds $jsonSchema or $text at any depth (inside a field, $not, $elemMatch)	refuse operator not allowed: <op>	script_runner.service.ts:31,66-67,70-71	script_runner.service.spec.ts:186,191
names a hidden/masked path, a child of it, or reaches it by dotted key, array index, nested object or $elemMatch	refuse field not allowed: <path> (numeric segments ignored)	script_runner.service.ts:36-41,72-75	script_runner.service.spec.ts:91,95,99,103-114 (C4 table),135,218
tests a parent of a hidden path	allow only $exists, $size, $type, $not of an allowed test, or $elemMatch that is a per-element field query; refuse any value comparison at any depth ($eq, $ne, $in, $nin, $all, literal object/array)	script_runner.service.ts:33-34,43-61,76	script_runner.service.spec.ts:196,200-207 (C4),209-216 (C5),223
$elemMatch value has only field keys and logical operators	check it as a document, so its field keys are re-checked as <parent>.<key>	script_runner.service.ts:44,70-71	script_runner.service.spec.ts:218
sorts on a hidden/masked path or on a parent of one	refuse field not allowed: <field>; sort must be a string of [A-Za-z0-9_.] terms	script_runner.service.ts:81-93	script_runner.service.spec.ts:116,125
asks for hidden fields in the projection	drop them from $project and from the output; mask masked fields	script_runner.service.ts:158-173	script_runner.service.spec.ts:50

when a template …	the system must …	enforced at	test
uses {% include %}, {% render %} or {% layout %} (static or dynamic name)	fail render_error "{% <tag> %} is not allowed in templates", never partial output or file content	template_engine.ts:38,53-60,75	template_engine.spec.ts:101-113
triggers any file lookup	the engine's fs throws "templates cannot read files"; relativeReference: false	template_engine.ts:40-51,64-65	template_engine.spec.ts:101-113, export columns :115-123
runs too long, parses too much or allocates too much	render_error from Liquid's renderLimit 2000 ms, parseLimit 200000, memoryLimit 1e7	template_engine.ts:68-70, template_engine.constants.ts:2-4	template_engine.spec.ts:82
reads an inherited property or calls an unknown filter	ownPropertyOnly, strictFilters	template_engine.ts:66-67	template_engine.spec.ts:76,95

6. States and flows

Attachment status

from	to	trigger	guard	code
—	pending	POST create	validation + rate limits pass	attachment.service.ts:101-113
pending	scanning	confirm	findOneAndUpdate {status: pending}	:144-148
pending	failed (expired)	sweep	{status: pending, created_at < now−6 min}	attachment.sweep.processor.ts:63-67
scanning	scanning (re-claimed)	sweep	{status: scanning, scanning_started_at ≤ now−5 min}	:82-90
scanning	failed (SIZE_MISMATCH/TYPE_MISMATCH/INFECTED/TIMEOUT)	verify, completeReady, confirm timeout	{status: scanning}	attachment.service.ts:258-263
scanning	ready	completeReady	{status: scanning}	:240-252
ready, failed	—	terminal; no delete API, no retention	—	—

scan_status: pending at create → clean or infected in verify (attachment.service.ts:110,201-215).

Upload (store → core → S3)

On any check failure core writes failed first, deletes incoming best effort, and answers 422 {code: <reason>, status:'failed', failed_reason}; the store maps the code to attachment_error_<code> (store:src/services/attachment/upload.ts:29-34, store:src/services/attachment/index.ts:55-58).

Sweep tick

Permanent content link

BFF code: store:src/pages/api/attachments/[id]/content.ts:6-28. Login return: store:src/pages/_app.tsx:47-50 sends logged-out page loads to /login?redirect_url=…; store:src/hooks/session/index.ts:6-16,29-33 keeps the form shown when a redirect_url is present and accepts only same-origin paths. Thumbnails, file links, preview image and the preview download all use this link, never the 15-min url (store:src/pages/setting/attachments/index.tsx:40,54,225-227).

7. Jobs, schedules, queues, events

name	kind	trigger	payload	retry	idempotency
attachment_sweep	Bull repeatable on QUEUE_REDIS_URL, queue attachment_sweep	onModuleInit of every pod adds {jobId:'attachment_sweep', repeat:{every:60000}, removeOnComplete, removeOnFail}; Bull keeps one schedule per repeat key; add errors are logged, boot continues (attachment.sweep.processor.ts:30-41)	{}	none; the next tick retries naturally. Each tick handles at most 100 expired pending + 100 stale scanning rows (attachment.constants.ts:14)	every row write is a guarded findOneAndUpdate; CopyObject to the same key and DeleteObject are idempotent; a claim moves scanning_started_at so a second worker misses the row (:63-68,82-91)

Not in the jobs engine: the sweep predates it (D-166) and was not moved by 38dad93; it is not a code-declared schedule and has no admin enable/disable (see 04). No Kafka event, no activity log row and no jobs row is written by any media path. S3 calls produce no external-call log rows (externalCall.run gives only the timeout; 03 §6.5).

8. Settings and env

key	default	effect	where read
storage.* record keys	§3 table	limits, bucket, region, prefix, SSE-KMS, credentials	storage_setting.service.ts:27-36
S3_ENDPOINT (env)	unset	local LocalStack only (http://s3.localhost.localstack.cloud:4566, D-222/D-223); sets endpoint, forcePathStyle: false. Must be unset in production	storage_setting.service.ts:49-52
S3_TIMEOUT_MS (env)	10000	per S3 call timeout (presign, head, get incl. body, copy, delete)	attachment.constants.ts:16, attachment.service.ts:49-51
STORAGE_SECRET_ENC_KEY (env)	none	32-byte key (64 hex or base64) to encrypt/decrypt secret_access_key; missing → error on read of a stored secret and on save of a new one	src/common/utils/crypto.ts:4-22
AWS default credential chain (env/instance role)	—	used when credential_mode = iam_role	storage_setting.service.ts:54
CACHE_REDIS_URL	—	upload rate counter, settings pub/sub	src/common/utils/redis.ts:169
QUEUE_REDIS_URL	—	Bull sweep queue	app.module.ts:40-42
TEMPLATE_RENDER_LIMIT_MS, TEMPLATE_PARSE_LIMIT, TEMPLATE_MEMORY_LIMIT	2000, 200000, 10000000	Liquid limits	template_engine.constants.ts:2-4
store API_GATEWAY	—	BFF base URL to core	store:src/apis/server/request.ts:20, store:src/apis/server/attachment/index.ts:35

Code constants (no setting): PUT TTL 300 s, GET TTL 900 s, 20 pending, 30 URL/min, 4096 magic bytes, 6/5 min expiry, 60 s tick, batch 100 (attachment.constants.ts:1-14); purpose caps (attachment.purposes.ts:3-9).

9. Concurrency and recovery

situation	what the code does	code
two confirms on one row (same or different pods)	one wins the pending → scanning claim; the loser reads the row and returns it (scanning or ready), no S3 work	attachment.service.ts:144-157
sweep and a slow confirm on the same scanning row	sweep only takes rows idle ≥ 5 min; confirm's S3 calls each time out at 10 s, so overlap needs a confirm stalled > 5 min. If both run, both copy to the same key (idempotent) and only one ready write matches; the other returns the current row	attachment.sweep.processor.ts:75-91, attachment.service.ts:240-255
two sweep ticks overlap (slow tick + next tick on another pod)	guarded updates let each row change once	attachment.sweep.processor.ts:63-68,82-91; test attachment.sweep.processor.spec.ts:276
pod dies after the claim, before verify	row stays scanning; after 5 min the sweep re-verifies and finishes	attachment.sweep.processor.ts:92-98
pod dies after verify wrote clean, before or after CopyObject	sweep skips re-verify, completeReady copies if incoming exists or accepts an existing key	attachment.service.ts:222-238; qa AC-09
pod dies after DeleteObject, before ready	incoming gone, key exists → ready	attachment.service.ts:233-237; test attachment.sweep.processor.spec.ts:245
pod dies between create's insert and the presign	row stays pending without an upload URL; sweep fails it expired after 6 min	attachment.service.ts:101-126
S3 stalls during confirm	TIMEOUT → row failed, best-effort delete; a copy already made under key is not removed (orphan object)	attachment.service.ts:169-171,258-271
S3 stalls during sweep	that row's error is logged, the row stays scanning, the next row continues	attachment.sweep.processor.ts:99-101; tests :315,334
incoming delete fails after failed	warning only; the object stays in v2/incoming/ forever (no S3 lifecycle rule built)	attachment.service.ts:264-269, attachment.sweep.processor.ts:105-111
storage settings saved on another pod	pub/sub (or the 60 s reload) refreshes the cache, onChange destroys the client; the next call builds a new one. An S3 call in flight keeps the old client object until it finishes	storage_setting.service.ts:19,38-42,64-69; 03 §6.3
many pods add the repeatable job at boot	same jobId + repeat options → one schedule	attachment.sweep.processor.ts:32-37
the rate counter key expires mid-burst	fixed window per clock minute; up to 60 calls across a minute boundary	attachment.service.ts:84-85

10. Web (store-v2)

page / route	components	API calls	gating
/setting/storage	Form, Input, Tag (extensions), Select (provider aws_s3, credential mode), InputPassword for key id and secret, shown only for static (store:src/pages/setting/storage/index.tsx:12-126)	GET and PUT /api/storage-settings → core /storage-settings (404 at this core sha) (store:src/services/storage_setting/index.ts:12-38)	logged-in check only (store:src/pages/_app.tsx:47-50); a 403 body "Forbidden resource" sends the browser to login (store:src/apis/client/request.ts:37-39)
/setting/attachments	PageHeader + Upload button, search Input (debounced, Enter searches at once), status Select (all/pending/scanning/ready/failed), grid/list toggle, AttachmentThumb (img on the content link when thumb_url set, else extension Badge), AttachmentName (non-images link to the content link in a new tab), AttachmentMeta (size, status badge, failed reason), UploadModal, ImagePreview, TableFooter (store:src/pages/setting/attachments/index.tsx:18-248)	GET /api/attachments?page&limit&search&status (store:src/services/attachment/index.ts:16-26); upload = POST /api/attachments → PUT upload_url with returned headers minus Content-Length → POST /api/attachments/:id/confirm (store:src/services/attachment/upload.ts:5-35); images and downloads via /api/attachments/:id/content	same
BFF /api/attachments (GET, POST), /api/attachments/[id] (GET), /api/attachments/[id]/confirm (POST)	—	forward cookie token as Authorization: Bearer, plus client IP and UA (store:src/apis/server/request.ts:51-73); list forwards only page limit search status, first value of each (store:src/apis/server/attachment/index.ts:4-16); on error pass status and body, for 403 pass message so the client sees "Forbidden resource"	—
BFF /api/attachments/[id]/content (GET)	—	fetch with redirect: 'manual'; 401/403 → 302 to login; a Location is passed through with the upstream status; otherwise status, content type and text body	—
BFF /api/storage-settings (GET, PUT)	—	forwards to the removed core route	—
menu	Setting → Storage (HardDrives), Attachments (Paperclip)	—	not gated (store:src/constants/menu.ts:93-100)

Upload modal: one file, drag-drop or click, Upload disabled while uploading, closes on success (store:src/components/attachment/UploadModal/index.tsx:14-100). The web never sends Content-Length itself because the browser sets it from the File, which equals the signed size (upload.ts:13-14). Error toasts: known codes in AttachmentErrorCode → t('attachment_error_<code>'), else the server message (store:src/services/attachment/index.ts:55-58, store:src/enums/attachment.ts:8-19). Labels come from t() with the key as fallback (store:src/utils/translate.ts:347-349); en strings at store:src/utils/translate.ts:166-186,296-344. There are no store tests for media.

11. Tests

spec	covers
src/attachment/attachment.service.spec.ts	create (row, 300 s URL, SSE headers with/without KMS, two uploads two keys, purpose/storage caps, unknown purpose, svg, 31st URL/min, 21st pending); confirm (size/type mismatch, cleanup denied keeps reason, infected, ready no S3, failed 422, HeadObject/GetObject-body timeouts → TIMEOUT, slow body inside timeout ok, confirm before upload); copy (incoming deleted, KMS on copy, no KMS, per-segment CopySource encoding); serialize failed_reason; GET pdf disposition; contentUrl (fresh 15-min sign per call, 404 unknown/non-ObjectId); DTO limit 100/101
src/attachment/attachment.sweep.processor.spec.ts	pending 5:59 untouched / 6:01 failed expired; stale scanning resume, size mismatch, PDF-as-png, infected, files key already present still verified; two concurrent sweeps act once; S3 timeout leaves row scanning, next row continues; ready row untouched; repeatable job registration
src/attachment/attachment.controller.spec.ts	content link 302 + no-store, 409, 404; no token → 403 from the real AuthGuard before the service
src/storage_setting/storage_setting.service.spec.ts	default record → 503; onChange rebuilds the client once; crypto round trip
src/storage_setting/storage_setting.client.spec.ts	AWS virtual-hosted host without S3_ENDPOINT, no CRC32 param; LocalStack host with S3_ENDPOINT; stray endpoint/force_path_style in stored values ignored
src/script_runner/script_runner.service.spec.ts	hidden fields never returned; id cast; server-code operators; hidden/masked filter and sort (C1–C7); document-operator allow-list; nested $jsonSchema/$text; parent value matches under $not/$elemMatch; $elemMatch field keys by full path; allowed shape tests
src/template_engine/template_engine.spec.ts	blocked include/render/layout incl. path traversal and dynamic name; export columns cannot include; normal tags still work; loop limit; strict filters; own properties only

Untested in unit: concurrent confirms (qa AC-08 only), the Bull schedule end to end, the settings-module save path for storage values, every store page and BFF route.

12. Gaps and known issues

#	gap	evidence	impact
G1	Store Storage page is dead. It calls /storage-settings, removed from core in 3dd1c5a; the generic /settings/storage has another shape ({rev, values} in, keys[] out)	store:src/apis/server/storage_setting/index.ts:3-9; src/settings/settings.controller.ts:21-33	Admin cannot view or save storage from the UI; GET fails silently (no onError), Save toasts a 404 message. Storage can be set only by calling PUT /api/v1/settings/storage directly
G2	Store secret mask ******** and allowed_image_extensions do not match core (••••, key removed). Pointing the page at /settings/storage unchanged would save the literal ******** as the new secret, and '' for iam_role, and 400 on allowed_image_extensions	store:src/services/storage_setting/types.ts:1,8, store:src/services/storage_setting/index.ts:31-36; settings.service.ts:163,182-183; settings.types.ts:1	Migration must use the settings-page pattern of store:src/pages/setting/{authen,google,kafka}
G3	No seed from the old storage_settings doc	grep: no seed code; commit 3dd1c5a note vs code	After upgrade every environment is bucket '' → 503 on every attachment call until re-entered
G4	allowed_extensions defaults to []	storage_setting.settings.ts:39, attachment.service.ts:74-75	Fresh storage config refuses every upload with EXTENSION_NOT_ALLOWED until the list is filled
G5	Static credentials without a secret are accepted (old SECRET_REQUIRED 400 gone)	storage_setting.service.ts:54-59	Misconfig shows up as S3 errors at upload time, not at save
G6	Malware scan is a stub that always returns clean (D-165, D-195: "file content check not supported")	attachment.scanner.ts:11-17	INFECTED is reachable only in tests
G7	No real thumbnails, no EXIF GPS strip, no pixel cap (D-172)	attachment.service.ts:342; store renders the full image as the thumbnail	Large images load full size in the grid
G8	Orphans not swept: incoming objects whose delete failed after failed; a files/ copy left by a confirm that timed out after CopyObject; no S3 lifecycle rule (infra, D-172)	§9 rows; attachment.sweep.processor.ts:56-102 handles only pending/scanning rows	Storage leak, no automatic cleanup
G9	GET /attachments/:id and POST /attachments/:id/confirm do not check the id format; a non-ObjectId raises a Mongoose CastError, not caught by MongoExceptionFilter (MongoError only)	attachment.service.ts:144,275; src/common/filters/mongo.filters.ts:5-16	500 instead of 404 (only /content checks, :284)
G10	TIMEOUT is final in confirm: a transient S3 stall fails a good upload; the store has no attachment_error_TIMEOUT or attachment_error_expired string	attachment.service.ts:170; store:src/enums/attachment.ts:8-19; store:src/utils/translate.ts:335-344	User must re-upload; list shows the raw key attachment_error_expired/attachment_error_TIMEOUT
G11	Access is role admin only; no permission keys, no store scope, no uploader filter; any admin lists and opens every file	attachment.controller.ts:8; 02-auth G19	02-auth's key table (setting.attachment.*) and "sweep wrapped in asSystem" describe a later build, not 3d70475 (no asSystem in attachment.sweep.processor.ts)
G12	Sweep is Bull, not the jobs engine; no admin visibility, no enable/disable, no metrics; batch 100 per tick	attachment.sweep.processor.ts:21-41	A backlog > 100/min drains slowly; failures only in logs
G13	provider key is never read; uploader_id and {uploader_id,status} index only serve the pending cap; no index for stale scanning	storage_setting.settings.ts:40; attachment.schema.ts:60-62	Dead config; stale query scans all scanning rows (small set in practice)
G14	Pending cap is soft and the URL counter counts refused calls	attachment.service.ts:84-96, D-176	A burst can exceed 20 pending by a few; a user who hits 429 keeps burning the minute
G15	Detection covers png, jpeg, webp, pdf only; a purpose adding another type fails every upload with TYPE_MISMATCH until detectMime learns it	attachment.mime.ts:1-25	Must be extended with each new purpose
G16	List presigns every ready row on each list call; url in list/get expires in 15 min	attachment.service.ts:313-316,340-343	Store avoids it by always using the content link; other clients must re-fetch
G17	Purposes pod, cod, public, ref_type/ref_id, public_base_url are not built (D-196..D-200 drafts, cut by D-202)	attachment.purposes.ts:3-9	Next phase adds them
G18	Template/script hardening covers Liquid file tags and script filters only; the rest of the template review (7 med, 12 low) is left to the human (D-252, D-259)	_docs/workspace review	Not tracked in code
G19	storage_setting.constants.ts is used only by specs; spec name "a record seeded from a doc" describes a seed that no longer exists	storage_setting.constants.ts:1-2, storage_setting.client.spec.ts:86	Misleading leftovers

13. How to add the next one

A new attachment purpose (e.g. pod)

1. Add the purpose to src/attachment/attachment.purposes.ts (visibility, extensions, max_size_mb). If it needs a new visibility, add it to AttachmentVisibility (src/common/enums/attachment.enum.ts) and branch serialize/_signGet (public rows would return a base URL instead of a presign, D-197).
2. If it accepts a type detectMime does not know, extend src/attachment/attachment.mime.ts and add a spec case.
3. If it binds to a record, add ref_type/ref_id to attachment.schema.ts + AttachmentInput, check the record exists in createUploadUrl, add a {ref_type, ref_id, status} index and list filter (draft contract D-200).
4. Keep every write guarded on the expected status; reuse verify → completeReady → fail so the sweep recovers it unchanged.
5. Add spec cases to attachment.service.spec.ts in the existing style (one case per refusal + happy path), and make sure the storage allowed_extensions includes the new extensions.
6. Store: add t('attachment_error_*') strings for any new failure code and call uploadFile(file, '<purpose>') (store:src/services/attachment/upload.ts).

A new media setting

1. Add defineSetting(STORAGE_SETTINGS, { key, schema, default }) to src/storage_setting/storage_setting.settings.ts and to STORAGE_KEYS (encrypted ones get encrypt: true, default null, and are read with getSecret).
2. Read it through StorageSettingService.getSettingsOrThrow(); if it changes the S3 client, nothing else is needed (the record onChange already resets the client).
3. Validation lives in the setting's class-validator schema; cross-key rules (like "static needs a secret") have no hook in the settings module — put them where the value is used, or add a record-level validator to settings.service.ts (see 03).
4. Store: build or fix the Storage page on the generic settings pattern (store:src/pages/setting/{authen,google,kafka}, store:src/services/setting, store:src/pages/api/settings/[code]), mask ••••, send {rev, values}.

A new background cleanup for media

Prefer a code-declared schedule on the jobs engine (src/schedules/, see 04 §7 and §13) over another Bull repeatable job; keep the same rules as the sweep: select a bounded batch, claim each row with a guarded findOneAndUpdate, make every S3 step idempotent, catch per row.

A new hidden field reachable from scripts

Put it in the read controller's @HideFields([...]) or @MaskFields([...]); ScriptRunnerService._reader picks both up for filters, sorts and projection (script_runner.service.ts:142-144). Add one refusal case to script_runner.service.spec.ts. A new entity becomes readable only by scriptEntities.register({ name, service, controller }) in its module's onModuleInit; never register an entity that holds secrets.