Boardroom AI Monitor — methodology (version 0.5)

Version date: 2026-10-11 (v0.5; change log: §10) · Data edition: Monitor 2026 (FY2025, 20 WIG20 + 20 DAX) · Author: Andrzej Chądzyński

English translation of docs/METODYKA.md. The Polish text is the reference version; if the two differ, the Polish text prevails.

Boardroom AI Monitor measures how often, and in what context, the boards of listed companies talk publicly about artificial intelligence — using public corporate documents only. It is a monitor/signal, not an index or a benchmark: the sample is small and purposive, and the results should be read as a directional indicator, not as a ranking of management quality.

1. Sample (Monitor 2026: 20 WIG20 + 20 DAX companies)

Current edition (since 2026-10-11 the only one shown on the website): 20 WIG20 companies and the 20 largest DAX companies (companies-E2.yaml), FY2025 annual reports or management board reports, each company with an assigned sector group (sector_group) and — where possible — a European peer group (§5c). Earlier fiscal years of the same companies serve the time series only (§5b).

Pilot (July–September 2026, N=10) — a methodology archive, not a separate edition. Purposive selection: 5 companies from the DAX and 5 from the WIG20, with approximate DE↔PL sector pairing so that the country comparison is not purely an artefact of sector mix:

The pairing is approximate (SAP↔CD Projekt differ in scale and business model) and serves only to limit sector bias, not to support 1:1 comparisons.

2. Exclusions (conflict of interest)

Manufacturers of elevators and escalators are permanently excluded — in particular KONE, Otis, Schindler and TK Elevator, and any company with a material share of its business in that industry. Reason: a conflict of interest — the author works in that industry. The exclusion applies to all editions without exception and is stated in every publication of results.

3. Corpus

• Document type (E1): the annual report for fiscal year 2025 — for German companies the full Geschäftsbericht / integrated report, for Polish companies preferably the Management Board report on operations (the narrative part of the annual report), or the full annual report where the former is not published separately.
• Comparability: metrics are compared only within the same document type. Annual reports are never mixed with shareholder letters or transcripts in one metric.
• One language version per document (bilingual editions deduplicated), preferring the local language: German for German companies, Polish for Polish companies. Note: for Allegro.eu (incorporated in Luxembourg) the binding version is English; the official Polish translation published on the company's IR site is analysed, for language consistency within the PL cohort.
• Sources: official, public investor-relations websites only. No paid aggregators, no redistribution of full texts.
• Source period: fiscal year 2025. Siemens' fiscal year ends on 30 September 2025 (report published December 2025); the other companies report on calendar year 2025 (reports published February–April 2026). An edition published outside the reporting season states its source period explicitly.
• Earnings-call transcripts are not part of the E1 corpus (they will be considered only where a company publishes them officially and without a paywall).

4. Term dictionary and counting

• A multilingual (EN/DE/PL) dictionary in terms.yaml (versioned), covering among others: AI, artificial intelligence, KI, künstliche Intelligenz, sztuczna inteligencja (all inflections), machine learning, maschinelles Lernen, uczenie maszynowe, generative AI/KI, GenAI, LLM, large language model, duży model językowy, großes Sprachmodell, deep learning, foundation model.
• Matching on word boundaries; abbreviations (AI, KI, LLM, GenAI) are matched case-sensitively to avoid false positives.
• Overlapping hits count once — the longer match wins ("generative AI" = 1 mention, not 2).
• False-positive rules (dictionary v0.2, from the E1 audit): (a) "KI" is matched only in German-language documents — in Polish PDFs, decorative page headers extract as scattered letters (e.g. "WYNIKI" → "…KI WE"), which produced hundreds of spurious mentions; (b) "LLM" excludes the academic context (LLM is also the Master of Laws degree). Contextual exclusions are logged and auditable.
• Normalisation: mentions per 1,000 words of the document. Word count = whitespace-separated tokens after text extraction from PDF/HTML.
• Limitation: the whole document is analysed, including the financial section, table of contents and notes — this dilutes the frequency in long reports; the effect is, however, similar within the same document type.

5. Context classification of mentions

Each mention (context of ±2 sentences) is assigned to one class:

Decision rules (v0.4, the author's decisions from the manual review of 2026-09-19):

• R1/R2: MACRO and STRUCTURAL count towards frequency (§4 — whole document) but not towards the company's context mix or substance ratio; they are reported separately.
• R3: customer-service bots (chatbot, voicebot, assistant in a service channel) = COST; REVENUE only when AI is sold or monetised to customers.
• Class by purpose: the class follows the purpose AI serves in the passage (e.g. deploying a tool "to raise efficiency" = COST, even if regulatory compliance is mentioned alongside).

• R4: the substance flag (yes/no) is judged on the whole ±2-sentence context, not only on the sentence with the mention: does the context contain a number relating to AI, the name of an AI project/product/programme, or a concrete, verifiable result? Numbers unrelated to AI (e.g. GDP) do not count. The share of mentions "with substance" is the substance ratio.

• Classification is done by a language model (script classify.py); every mention keeps a short quote (≤25 words), the page number and the document URL — fully auditable in data/dataset.json.

• For documents with very many mentions a systematic sample (every k-th mention, at least 40 per document) may be classified; frequency is always computed on the full text, the class mix on the sample. Sampling is recorded in the data.
• Manual review: before publication a person checks a random ≥10% of the classified mentions; publication threshold: class agreement ≥80% (set on 2026-09-27) (file output/przeglad-probki-v<version>.md, with the full ±2-sentence context). When the methodology version changes, the sample is drawn with a new seed and without the mentions from the previous review, so it is an independent test of the new classification.
• In Edition 1 classification was done by Claude (Anthropic): v0.3 — 2026-07-10 (agreement with the author's decisions in the 2026-09-19 review: 36%, below the publication threshold); v0.4 — 2026-09-27, the same sample of 249 mentions, the 25 reviewed mentions taken as the author decided (data/przeglad-2026-09-19-decyzje.json), the other 224 classified again under rules R1–R4 (data/classifications-v04.json). The author's review of 2026-09-27 (a new sample of 25 mentions, seed 20260927): class agreement 92%, class and substance flag 88% (v0.3: 36%); 3 corrections applied (output/przeglad-probki-wynik-2026-09-27.md).

5a. Robustness test (mandatory for every edition)

With a small sample, a single company or the choice of estimator can shift a result several-fold. That is why every edition goes through a robustness test (sensitivity.py, published alongside the report as output/analiza-wrazliwosci-<edition>.md):

1. Estimator swap — the same ratio computed via the median, the mean, the geometric mean and pooled (mentions/words).
2. Leave-one-out — recomputed with each company removed in turn.
3. Stratification by document type — does the claim hold separately in each stratum?

Only a claim that passes these tests may become the headline number of an edition. In Edition 1 this demoted the DAX↔WIG20 comparison (a range of 1.5×–4.1×) and promoted the sector gradient, which holds in both document types. No report is produced unless the robustness test has been run (report.py stops).

From v0.5 (Monitor 2026, 40 companies): the test is run by robustness.py (output/analiza-odpornosci-2026.md, data/robustness.json): estimator swap, leave-one-out and a bootstrap over companies (2,000 draws, 95% interval) — also for the claims about the context mix and the lack of substance. sensitivity.py remains the test of the N=10 pilot.

5b. Time series — within-company change (since 2026-09-21)

Cross-sectional comparisons (company vs company, country vs country) carry sector and document-type confounders (§7). Comparing the same company with itself across years does not — which makes it the most robust metric of the Monitor, and it takes precedence as the headline number when it meets condition 4.

1. Series corpus (companies-timeseries.yaml): for each company, the documents for earlier fiscal years of the same type and in the same language as the edition's document (annual report ↔ annual report, management report ↔ management report). A company without a same-type document drops out of the series — types are never substituted. Every document has its official IR URL and SHA-256 checksum.
2. Metric: frequency only (mentions per 1,000 words), with the same dictionary (§4, same version) and the same code. No context classification — classification covers the edition's current year only.
3. Statistics (timeseries.py → output/seria-czasowa-<edition>.md, data/timeseries.json): direction of change per company (up/down/no change) between the first and last year; number of companies up and down at each year-on-year step; median of within-company ratios; ratio of medians and pooled ratio with leave-one-out.
4. Headline number (from v0.5 — order of precedence): (a) the direction of change (e.g. "10/10 companies up"), only when the direction is unanimous (0 declines) and N ≥ 5; (b) a claim about the context mix (e.g. "1 in 6 mentions is about revenue"), if the classification passed the manual review (class agreement ≥80%, §5) and the direction of the claim (e.g. revenue rarer than both cost and risk) holds in ≥95% of bootstrap draws over companies; the number is given as a point estimate with its leave-one-out and bootstrap ranges; (c) otherwise — the cross-sectional claims of §5a. A non-unanimous series direction (e.g. 31/39) is a supporting claim. The size of the increase is given as the median of within-company ratios, and the ratio of medians always with its leave-one-out range.
5. The "a jump, not a trend" reading appears in the report automatically only when the last step is unanimous and the previous one mixed (fewer than 75% of companies up) — it is a description of the step distribution, not a judgement.
6. Structural caveat and CSRD test: from FY2024, reports of companies covered by the CSRD include sustainability statements (ESRS) with their own AI paragraphs, so part of the increase could reflect a change in reporting convention rather than strategy. CSRD test (csrd_test.py, since 2026-09-27): pages of the ESRS statement are flagged heuristically (≥2 ESRS disclosure codes on the page, the statement heading, or a running header of the section present on ≥20 and <60% of pages; gaps of ≤3 pages are closed), and frequency is computed separately on the remaining text. XHTML/ESEF documents without page structure are excluded from the test. Result on 2026-09-27: E1 — 10/10 companies up outside the ESRS pages (median ×2.4); E2 — 30/38 (×2.3, after HTML extraction v2). The new sustainability statements do not explain the increase.

5c. European peer group and sector comparisons (since 2026-09-27)

A sector comparison of Polish/German companies needs peers from the same sector outside the company's own country — otherwise "sector" is confounded with "country" (the same confounding that demoted the DAX↔WIG20 comparison, §7).

1. Ring selection (companies-europe.yaml): companies from the EU, Switzerland and Norway selected for the sectors of the E2 companies (field sector_group), not from indices; official annual reports, English preferred. The elevator-industry exclusion (§2) applies here as well.
2. Threshold (sector_coverage.py): a company is compared within its sector only if it has ≥3 peers from ≥2 countries other than its own (counted per company, on documents actually processed: --counted). Below the threshold: frequency and within-company change only.
3. Fiscal year "FY2025" = the fiscal year covering most months of 2025; the source period is explicit in the data (period).
4. Metric: frequency only (mentions/1,000 words), same dictionary and code. Different report languages (EN/DE/PL) and document scopes (URD vs management report) affect normalisation — sector comparisons are a signal, not a ranking.

6. Auditability

• data/dataset.json: every mention with its quote, class, substance flag, page number and source URL; every document with its SHA-256 checksum. (The copy published on the website omits the ±2-sentence context and keeps quotes of ≤25 words.)
• Pipeline code, dictionary and methodology are versioned in a git repository — the history is the audit trail.
• The report's tone is descriptive, not evaluative: "mentions AI rarely" ≠ "is behind".

7. Limitations (read before citing)

1. N=10 — a signal, not a benchmark. Results do not generalise to whole indices or economies.
2. Purposive selection (largest/best-known companies) probably overstates AI visibility relative to the average index member.
3. Differences in reporting culture (length and conventions of DE vs PL reports) may affect normalisation per 1,000 words.
4. LLM classification can err; this is mitigated by the auditability of every mention and the ≥10% manual review.
5. Change over time (§5b) covers frequency only; context classification covers the edition's current year only.
6. Siemens' fiscal year is offset from the others (see §3).
7. In Edition 1, document type is almost identical with country (the German companies supplied full annual reports, most Polish companies management reports). Formally, the country effect cannot be separated from the document-type effect in E1, which is why the DAX↔WIG20 comparison is given only as a range with a caveat, not as a result. Stratified claims (the sector gradient) are not affected. Remedy in E2: for German companies, use the separate combined management report where it is published as a chapter, or report the metric only by document type.
8. Text-extraction quality. Some PDFs render bold type by doubling characters ("GGeesscchhääffttss") — such headings are not matched, and the word count rises by at most about 2% (measured on 2026-09-27: 3 lost mentions across 194 documents; left uncorrected so as not to change published figures). XHTML/ESEF documents are read with extraction v2 (since 2026-09-27), which does not split words at inline tags — the earlier version undercounted mentions (e.g. Erste and TAURON FY2023). No document has a material share of pages without text (scans/graphics).

8. Legal and ethical principles

• Fair use only: short quotes (≤25 words) with attribution; no redistribution of full document texts.
• Only documents that the companies themselves published on their IR websites are analysed.
• The report is not investment advice.

9. How to cite

Boardroom AI Monitor 2026, A. Chądzyński — an analysis of the FY2025 annual reports of 20 WIG20 and 20 DAX companies; methodology v0.5.

10. Versioning

Current version: 0.5. The pilot classification passed the manual review on 2026-09-27 (92% agreement), the Monitor 2026 classification on 2026-10-11 (98%); the ≥80% threshold is met. The website went public on 2026-09-27; since 2026-10-11 it shows a single, current edition.