Internal planning document for keeping mica.wtf current across EU legislation, EU supervisory material, national competent authority material, and blog commentary. This document is not part of the GitBook navigation and should not be added to SUMMARY.md.

Freshness targets below are measured from the time an authoritative source publishes, updates, or republishes a relevant item.

scripts/update-checkers/
  sources/
    eurlex-mica.js
    eurlex-tofr.js
    esma-mica-docs.js
    esma-mica-qas.js
    esma-mica-registers.js
    eba-micar-docs.js
    eba-micar-qas.js
    nca-bafin.js
    nca-amf.js
    nca-cbi.js
    nca-cssf.js
    nca-dnb-afm.js
    nca-dfsa.js
    iosco-crypto-assets.js
  lib/
    fetch.js
    hash.js
    manifests.js
    classify.js
    markdown.js
    telegram.js
  run-checker.js
  run-all.js

.freshness/
  manifests/
  diffs/
  logs/
  dead-letter/
  generated/

Naming convention: checker files use <source>-<scope>.js; NCA checkers use nca-<authority>.js. Each checker exports sourceId, sourceName, sourceUrls, run(), and classify(item).

Run schedules in UTC through Hermes cron, not raw crontab. Staggering avoids simultaneous load on official sites and makes Telegram alerts easier to read. The checked-in wrapper is scripts/update-checkers/run-group.sh; production no-agent jobs use ~/.hermes/scripts/micawtf-*.sh launchers so Hermes can deliver script stdout directly to Telegram.

micawtf-esma          5 7 * * 1-5    ESMA Q&A, documents, and MiCA register
micawtf-eba           25 7 * * 1-5   EBA Q&A and publications
micawtf-nca-daily     5 8 * * 1-5    NCA high-priority pages
micawtf-nca-weekly    45 8 * * 2,5   Lower-volume NCAs and IOSCO
micawtf-legislation   15 9 * * 1     EUR-Lex legislative text checks
micawtf-blog-signals  15 12 * * 1-5  Blog topic discovery, queue only
micawtf-daily-audit   30 16 * * 1-5  Freshness audit and stale-source alert

See scripts/update-checkers/README.md for the exact hermes cron create commands.

Freshness automation changes the editorial order, not the publication controls. The default publishing runway is the issue #37 MiCA review consultation series in _drafts/blog-queue.md under Series fallback. Cron-driven findings enter Triggered or the appended checklist queue and preempt that series only when they are urgent, date-sensitive, or materially stronger than the next planned series part. If nothing fresh clears that bar, publish the next queued issue #37 series part. If both are blocked, use the evergreen queue.

Before drafting, run:

node scripts/editorial-status.js

The command reads _drafts/blog-queue.md and recent .freshness/logs/ events and prints the recommended lane. It is an editorial triage aid only; blog posts remain human-authored or human-reviewed and never auto-publish.

The weekly editorial clock is scripts/editorial-clock.js. In production the Hermes launcher runs it with --dispatch --commit, which creates the draft assignment, sets the review deadline, updates the queue, commits the dispatch, pushes it, and lets Hermes deliver stdout to Telegram. Daily review SLA reminders and auto-parking live in scripts/editorial-review-nudge.sh. Publishing remains gated by scripts/publish-draft.sh, which requires status: Approved before moving a draft into spaces/blog/.

Each checker should:

1. Fetch the configured source URLs with a consistent user agent, timeout, retry policy, and last-known ETag/Last-Modified headers where available.
2. Normalize the response into structured items. Use DOM parsing or CSV/JSON parsing where available; do not rely on brittle substring matching for primary detection.
3. Compute stable identifiers from official IDs first, then canonical URLs, then content hashes.
4. Compare against .freshness/manifests/<sourceId>.json.
5. Emit a JSON result and optional human-readable diff.
6. Classify findings as auto_publish_candidate, review_required, needs_classification, source_error, or no_change.

Signals:

Checker output should be a JSON event file, one per run:

{
  "run_id": "2026-06-06T07-05-00Z-esma-mica-qas",
  "source_id": "esma-mica-qas",
  "checked_at": "2026-06-06T07:05:00Z",
  "status": "changed",
  "source_urls": [
    "https://www.esma.europa.eu/esma-qa-search-page?field_qa_level1_target_id%5B0%5D=20011"
  ],
  "counts": {
    "seen": 42,
    "new": 2,
    "changed": 0,
    "errors": 0
  },
  "items": [
    {
      "source_item_id": "ESMA_QA_2671",
      "canonical_url": "https://www.esma.europa.eu/esma-qa-search-page?field_qa_level1_target_id%5B0%5D=20011",
      "title": "White paper obligations...",
      "type": "q-and-a",
      "authority": "ESMA",
      "published": "2026-06-01",
      "content_hash": "sha256:...",
      "classification": "auto_publish_candidate",
      "target_path": "eu-level/q-and-a/esma-qa-2671-white-paper-obligations.md",
      "review_reason": null
    }
  ]
}

Manifests live in .freshness/manifests/<sourceId>.json and contain canonical state only: source item IDs, URLs, hashes, known target files, last seen dates, and last successful check time. Diffs live in .freshness/diffs/<sourceId>/<runId>/.

checker
  -> structured JSON event
  -> manifest comparison
  -> classification
  -> generated Markdown in .freshness/generated/ or dead-letter entry
  -> quality gates
  -> move approved files into content tree
  -> node scripts/build-summary.js
  -> git diff validation
  -> git commit with source-specific message
  -> git push
  -> Telegram ping with result and links

build-summary.js should run only after content files are moved into the live tree. It should not run for queue-only or dead-letter-only detections.

All newly generated standards/NCA/blog files should include:

---
title: ""
shortTitle: ""
type: ""
authority: ""
jurisdiction: ""
id: ""
published: ""
updated: ""
status: ""
source_page: ""
source_pdf: ""
source_hash: ""
source_checked_at: ""
freshness_status: current
review_status: auto-generated
reviewed_by: ""
reviewed_at: ""
topics: []
related_articles: []
---

Legislative article files should not be regenerated automatically unless an amendment workflow is built. For amendment notes, use amendment_source, celex, oj_reference, effective_from, and affected_articles.

On timeout, DNS error, HTTP 5xx, or persistent HTTP 403:

1. Retry twice with exponential backoff.
2. Preserve the prior manifest.
3. Write .freshness/logs/<sourceId>/<runId>.json.
4. Set source status to degraded after one failed run and down after three consecutive failed runs.
5. Send Telegram alerts only when the source is high priority or has failed three consecutive runs.

HTTP 404 on a previously known canonical URL is a high-priority change, not a normal fetch failure. Queue it as source_removed_or_moved.

If a checker finds a changed hash, new URL, unknown document type, or non-English page that cannot be mapped with confidence:

.freshness/dead-letter/<sourceId>/<runId>/<sourceItemId>.json
.freshness/dead-letter/<sourceId>/<runId>/<sourceItemId>.html
.freshness/dead-letter/<sourceId>/<runId>/<sourceItemId>.md

The JSON file should include the raw metadata, classifier scores, fetch URL, content hash, proposed target section, and reason classification failed.

Use a concise severity prefix and one message per source run unless an urgent item is found.

[mica.wtf][INFO] ESMA MiCA Q&A: no change. checked=42 run=2026-06-06T07:05Z
[mica.wtf][NOTICE] EBA MiCAR Q&A: 2 new Q&As staged. auto=2 review=0 commit=abc123
[mica.wtf][REVIEW] AMF MiCA: 1 new guidance item requires review. queue=.freshness/dead-letter/nca-amf/...
[mica.wtf][URGENT] BaFin warning/enforcement item detected. No publish. Review source URL in dead-letter entry.
[mica.wtf][ERROR] EUR-Lex MiCA unreachable after 3 attempts. last_success=2026-06-01T09:15Z

Severity:

Safe to auto-commit after quality gates:

Human review required before publish:

If a source changes an existing document:

1. Preserve existing file.
2. Generate a diff file under .freshness/diffs/.
3. Add or update updated, source_hash, and source_checked_at only after review.
4. If the official source supersedes a prior document, mark the old page status: Superseded and add superseded_by.
5. If sources conflict, prefer the newest official primary source from the responsible authority. Do not reconcile by using law-firm summaries or media reports.

Use Git as the canonical version history. For user-facing pages, add a short ## Change history section only when a document has materially changed after publication:

## Change history

- 2026-06-06: Updated source metadata after ESMA republished the Q&A answer.
- 2026-06-01: Initial publication based on ESMA_QA_2671.

For legislative amendments, keep a dedicated internal amendment log in .freshness/diffs/legislation/ and publish a human-reviewed note before editing article text.

Build first:

• Replace or wrap scripts/check-updates.js with scripts/update-checkers/run-all.js while preserving existing ESMA/EBA Q&A behavior.
• Implement shared fetch, hash, manifest, and Telegram helpers.
• Implement ESMA Q&A, EBA Q&A, ESMA MiCA documents, EBA MiCAR documents, and EUR-Lex MiCA/ToFR checkers.
• Create JSON event output, manifests, diff directories, and dead-letter queues.
• Add quality gates for generated Q&A files.

Why first: ESMA/EBA Q&As and documents are high-value, already partially covered, and have the clearest source structure.

Build next:

• Add BaFin, AMF, CBI, CSSF, DNB/AFM, DFSA, and IOSCO checkers.
• Implement page-hash diffing with DOM normalization for pages that do not expose structured feeds.
• Add review queue summaries and Telegram REVIEW/URGENT alert routing.
• Add source-specific classification rules for guidance, forms, warnings, enforcement, and topic pages.
• Add freshness audit that flags stale source checks.

Why second: NCA material has higher classification risk and needs reliable review queue mechanics before any publishing.

Build last:

• Add generated Markdown templates for ESMA/EBA reports and NCA guidance.
• Add source PDF download, checksum, and optional text extraction for comparison.
• Add broken-link validation and relative-link validation before auto-commit.
• Add auto-generated branch/PR mode for review-required changes if Jacob prefers not to publish from the main branch.
• Add ESMA MiCA register CSV delta ingestion and optional register pages.
• Add a dashboard or report page summarizing last source checks, stale sources, and pending review items.

Why last: generation beyond Q&A requires stronger legal/status classification and more robust review tooling.

1. Should routine ESMA/EBA Q&As auto-publish to the live site after quality gates, or should they be staged for review first?
2. Should NCA updates always require human review, or can low-risk form/template changes auto-publish?
3. Should operational artifacts under .freshness/ be committed to Git, ignored, or stored externally?
4. Should Telegram alerts go to one channel, or should URGENT alerts route to a separate channel/chat?
5. Do we want a public MiCA register section for ESMA CSV deltas, or only internal monitoring?
6. Who is the named reviewer for reviewed_by values: Jacob only, or a broader reviewer list?
7. Should blog topic detection create draft Markdown files, issue-style queue entries, or Telegram-only prompts?
8. For non-English NCA pages, should the site publish English summaries only, or preserve translated excerpts with original-language labels?
9. Should source PDFs be stored locally for archival integrity, or should mica.wtf link only to official PDFs?
10. What is the acceptable same-day response window for urgent warnings/enforcement items: 4 business hours, end of day, or next morning?