Phase 2 — parsed_result Shape Reference

Single source of truth for parse_jobs.parsed_result JSONB shape across the v1-ssp legacy path (pre Phase 2 deploy in-flight jobs) and the v2-bundle new path (Phase 2 onward).

Used by:

  • Phase 2 Task 4 — SspImportConfirmService schema_version dispatch + bundle_restore
  • Phase 2 Task 7 — excel _write_all_data wire-up
  • Phase 2 Task 9 — docx confirm wire-up

Investigation scope (read-only):

  • app/oscal/service/excel_parser/types.pyParsedExcel.to_dict()
  • app/oscal/service/excel_parser/parser.py + sheet_handlers.py — what each sheet writes into the result dict
  • app/module_frame/excel_template/sheet_definitions.py — column-level ColumnDef spec (sheet templates v2.6.0)
  • app/oscal/service/ssp_excel_import_app_service.py_dict_to_parsed_* helpers (lines 1979–2166), _write_all_data (line ~689)
  • app/oscal/service/ssp_docx_import_app_service.py_parsed_docx_to_dict, _parsed_ssp_to_dict_overlay, _dict_to_parsed_parties
  • domain/oscal/parser/ssp_intermediate.py — v3 dataclasses (ParsedComponent, ParsedLeveragedAuthorization, ParsedInventoryItem, ParsedParty, ParsedSystemCharacteristic, ParsedExcelEntityBundle)
  • domain/oscal/parser/docx_intermediate.py — v1 ParsedDocx (matched_controls etc.)

1. v1-ssp (Legacy) Shape — pre Phase 2 deploy in-flight jobs

Authoritative writer: ParsedExcel.to_dict() (app/oscal/service/excel_parser/types.py:47-61).

Discriminator: absent or schema_version != "v2-bundle" is treated as v1-ssp legacy by Task 4 dispatcher.

1.1 Top-level keys

key type source notes
template_version str excel_parser/types.py:35 Excel template SemVer (e.g. v2.6.0). Absent on docx side.
metadata dict | None Excel 基本資料 sheet / docx _parsed_ssp_to_dict_overlay See §1.3
system_characteristic dict | None Excel 受評標的 sheet (v2.2.0+) See §1.4
parties_org list[dict] Excel 單位 sheet See §1.5
parties_person list[dict] Excel 參與人員 sheet See §1.6
devices list[dict] Excel 設備 sheet See §1.7
info_systems list[dict] Excel 資訊系統 sheet See §1.8
leveraged list[dict] Excel 外部利用服務 sheet See §1.9
controls_with_aos list[dict] Excel 控制項與AO sheet (父子 row 結構) See §1.10
ref_docs list[dict] Excel 程序書 sheet {doc_name, description}
validation_errors list[dict] excel_parser ValidationError.to_dict {sheet, row, field, code, message, blocking}

Docx-only extra top-level keys (ssp_docx_import_app_service.py:238-340):

key type notes
matched_controls list[dict] v1 ParsedDocx — {control_id, control_name, score, score_reason, parsed_implementation_description, current_implementation_description, current_objectives_descriptions, catalog_objective_descriptions, objectives:[{objective_key, parsed_description}]}
unmatched_paragraphs list[dict] {paragraph_idx, text, context, rule_score, rule_guess_control_id}
missing_baseline_controls list[str] control_id list
predicted_module_frame_controls list[str] mode=full only
summary dict parser-emitted summary stats
system_metadata dict | None v1 SspSystemMetadata serialized
baseline_controls list[dict] {control_id, control_name, catalog_control_uid, objective_keys, current_implementation_description}
parties list[dict] v2 adapter overlay — see §1.6 (docx merges org+person into single list)
system_characteristics dict v2 adapter overlay — ParsedSystemCharacteristics asdict
leveraged_services list[dict] v2 adapter overlay — ParsedLeveragedService asdict
information_types list[dict] v2 adapter overlay — ParsedInformationType asdict
framework_specific_props dict adapter pass-through
adapter_status dict[str,str] per-section ok / failed / not_run
adapter_errors dict[str,str] optional, only on adapter failure

Both Excel and docx legacy paths fall under v1-ssp from Task 4's perspective. The docx side never wrote devices / info_systems / leveraged legacy Excel keys — it skipped straight to v2 adapter overlay (parties, leveraged_services, etc.). Task 4 bundle_restore only needs to handle the Excel v1-ssp shape; docx in-flight legacy jobs already use the v2 overlay keys and will continue using _legacy_confirm for the docx-specific control diff workflow.

1.2 Row → dataclass mapping (v1-ssp helpers)

top-level key helper dataclass produced location
devices _dict_to_parsed_devices ParsedDevice ssp_excel_import_app_service.py:1979
info_systems _dict_to_parsed_info_systems ParsedInformationSystem line 1996
leveraged _dict_to_parsed_leveraged ParsedLeveraged line 2016
system_characteristic _dict_to_parsed_system_characteristic ParsedSystemCharacteristic line 2055
controls_with_aos _dict_to_parsed_controls ParsedControl + nested ParsedAssessmentObjective line 2084
parties_org + parties_person _dict_to_parsed_parties ParsedParty line 2121

1.3 metadata schema (Excel 基本資料 sheet)

Vertical layout single row. Keys from SHEET_METADATA columns (sheet_definitions.py:82-97):

key type required notes
name str yes MF name
group str yes MF group
version str yes MF version
description str | None no
framework_version_uid str | None no hidden col, internal UID
profile_uid str | None no hidden col, internal UID
frequency str yes
provider str | None no

1.4 system_characteristic schema (Excel 受評標的 sheet, v2.2.0+)

Vertical layout single row (SHEET_SYSTEM_CHARACTERISTIC columns, sheet_definitions.py:266-300). Maps to ParsedSystemCharacteristic.

key type required notes
system_name str yes
security_sensitivity_level str | None no enum low / moderate / high
status str | None no enum active / under-development / disposition / other
target_type str | None no
scope_description str | None no
owner_login_name str | None no lookup users (login_name)

1.5 parties_org[i] schema (Excel 單位 sheet)

SHEET_ORGS columns (sheet_definitions.py:100-121):

key type required notes
name str yes
short_name str | None no
matched_org_unit str | None no lookup org_units — reconciler stage 0 input
address str | None no v2.6.0
telephone str | None no v2.6.0
parent_org str | None no legacy v1.x_dict_to_parsed_parties reads as fallback to matched_org_unit
role str | None no legacy v1.x_dict_to_parsed_parties defaults to responsible-organization when missing (v2.0 sheet dropped role col)
email str | None no legacy v1.x_dict_to_parsed_parties reads as email_address

1.6 parties_person[i] schema (Excel 參與人員 sheet)

SHEET_PERSONS columns (sheet_definitions.py:124-153):

key type required notes
email str yes reconciler stage 0 input
name str yes
matched_user str | None no lookup users — reconciler stage 0 input
role str yes enum manager / reviewer / auditor / viewer
title str | None no v2.6.0
telephone str | None no v2.6.0 (mapped to ParsedParty.telephone_number)
address str | None no v2.6.0
org_unit str | None no legacy v1.x_dict_to_parsed_parties reads as matched_org_unit_label

Docx side parties[i] (merged list, top-level not split into org/person) is ParsedParty asdict — see §2.5 below for the merged shape.

1.7 devices[i] schema (Excel 設備 sheet)

SHEET_DEVICES columns (sheet_definitions.py:156-171) + legacy v1.x dropped columns still consumed by _dict_to_parsed_devices:

key type required notes
name str yes
ip str yes
matched_device str | None no lookup devices — reconciler stage 0 input
os str | None no legacy v1.x (dropped from v2.0 sheet, still read defensively)
device_type str | None no legacy v1.x
status str | None no legacy v1.x
purpose str | None no legacy v1.x

1.8 info_systems[i] schema (Excel 資訊系統 sheet)

SHEET_INFO_SYSTEMS columns (sheet_definitions.py:174-192) + legacy v1.x dropped columns still consumed by _dict_to_parsed_info_systems:

key type required notes
name str yes
abbreviation str | None no
matched_info_system str | None no lookup info_systems — reconciler stage 0 input
description str | None no legacy v1.x
component_type str | None no legacy v1.x
status str | None no legacy v1.x
system_owner str | None no legacy v1.x (mapped to ParsedInformationSystem.system_owner_label)

1.9 leveraged[i] schema (Excel 外部利用服務 sheet)

SHEET_LEVERAGED columns (sheet_definitions.py:195-252). _dict_to_parsed_leveraged skips rows where both service_name and provider are blank.

key type required notes
category str | None no enum external-service / interconnection / api / cli / leveraged-authorization / other
service_name str yes
provider str | None no not required as of v2.4.0
date_authorized str / date / datetime / None no ISO date string after to_dict(); helper coerces back to date
status str | None no enum operational / under-development / disposition / other
purpose str | None no
description str | None no v2.4.0
protocol str | None no v2.5.0
security_auth str | None no v2.5.0
nature_of_agreement str | None no v2.5.0
impact_level str | None no enum low / moderate / high / li-saas
data_types str | None no
authorized_users str | None no
fedramp_package_id str | None no
party str | None no legacy v1.x (mapped to ParsedLeveraged.party_label)

1.10 controls_with_aos[i] schema (Excel 控制項與AO sheet)

Parent-child row structure (sheet_handlers.py:194-238). Parent row (statement_id is None) contains nested objectives list of child rows (statement_id is the AO uid string).

Parent row:

key type required notes
statement_id None parent marker (always None for parent)
control_id str yes
control_name str | None no
objective_id str | None no
objective_name str | None no
impl_status str yes enum implemented / partial / planned / alternative / not_applicable
statement str yes
reference_doc str | None no lookup ref_docs_mf
include_in_profile str | None no enum TRUE / FALSE (v1.2.0+)
_target_in_profile bool yes parser-computed from include_in_profile (default True for legacy templates)
objectives list[dict] yes child rows (may be empty)

Child row (within objectives[]):

key type required notes
statement_id str yes AO uid (e.g. AC-1.a.1)
control_id str yes matches parent
objective_id str | None no
objective_name str | None no
impl_status str | None no
statement str | None no
reference_doc str | None no

Phase 2 does not change the controls path — controls_with_aos sticks with v1 shape in v2-bundle.


2. v2-bundle (New) Shape — Phase 2 onward

Authoritative writer: Phase 2 Task 7 (excel) + Task 9 (docx). Both pipelines output ParsedExcelEntityBundle v3 dataclasses serialized via asdict()-equivalent shape, then dispatched by Task 4 confirm_service on schema_version.

2.1 Top-level keys

key type notes
schema_version str discriminator. "v2-bundle" for new path.
template_version str Excel template SemVer (carried through for FE display)
components list[dict] aligned to ParsedComponent — see §2.2
leveraged_authorizations list[dict] aligned to ParsedLeveragedAuthorization — see §2.3
inventory_items list[dict] aligned to ParsedInventoryItem — see §2.4
parties list[dict] merged from parties_org + parties_person — see §2.5
controls_with_aos list[dict] unchanged from v1 shape — see §1.10
metadata dict | None unchanged from v1 shape — see §1.3
system_characteristic dict | None unchanged from v1 shape — see §1.4
ref_docs list[dict] unchanged from v1 shape — {doc_name, description}
import_warnings list[dict] new — see §2.6
validation_errors list[dict] unchanged from v1 shape

2.2 components[i] schema

Aligned to ParsedComponent (domain/oscal/parser/ssp_intermediate.py:222-249).

key type dataclass field notes
title str .title required
component_type str .component_type OSCAL 14 enum + otherthis-system / system / interconnection / software / hardware / service / policy / physical / process-procedure / plan / guidance / standard / validation / network / other
description str | None .description
purpose str | None .purpose
status str | None .status enum operational / under-development / under-major-modification / disposition / other
leveraged_authorization_ref str | None .leveraged_authorization_ref LA title string at parse time. Task 3 normalizer resolves to a real LA in the bundle (sets to None + emits warning if no match); reconciler later resolves to LA uid.
protocol str | None .protocol framework-specific prop
port_ranges str | None .port_ranges framework-specific prop
security_auth str | None .security_auth framework-specific prop
matched_party_uuid str | None .matched_party_uuid filled by reconciler
match_method str .match_method MatchMethod enum value as string (e.g. "unmatched" / "exact" / "normalized" / "fuzzy_high" / "fuzzy_low")
match_confidence float .match_confidence 0.0–1.0

2.3 leveraged_authorizations[i] schema

Aligned to ParsedLeveragedAuthorization (ssp_intermediate.py:252-268).

key type dataclass field notes
title str .title required. dedup key for Task 3 normalizer.
provider str | None .provider CSP / vendor name → party (CSP) at reconcile time
date_authorized str (ISO) | None .date_authorized date serialized to ISO string in JSONB
fedramp_package_id str | None .fedramp_package_id
impact_level str | None .impact_level enum low / moderate / high / li-saas
data_types str | None .data_types
nature_of_agreement str | None .nature_of_agreement
authorized_users str | None .authorized_users
remarks str | None .remarks

2.4 inventory_items[i] schema

Aligned to ParsedInventoryItem (ssp_intermediate.py:271-290).

key type dataclass field notes
description str .description OSCAL-required (NOT NULL)
asset_id str | None .asset_id
asset_tag str | None .asset_tag
ipv4_address str | None .ipv4_address
mac_address str | None .mac_address
fqdn str | None .fqdn
hostname str | None .hostname
software_name str | None .software_name
os_name str | None .os_name
implemented_component_refs list[str] .implemented_component_refs component title strings at parse time. Task 3 normalizer prunes unresolved refs (emits warning); reconciler later resolves to component uids.

2.5 parties[i] schema

Aligned to ParsedParty (ssp_intermediate.py:52-81). party_type discriminates person vs organization in a single list (replacing v1's parties_org / parties_person split).

key type dataclass field notes
name str .name required
party_type str .party_type "person" or "organization"
role str | None .role e.g. system-owner / security-officer / responsible-organization
title str | None .title OSCAL Party title (for person)
email_address str | None .email_address
telephone_number str | None .telephone_number
address str | None .address
short_name str | None .short_name (organization only)
remarks str | None .remarks
matched_user_id int | None .matched_user_id reconciler-filled (public.members.id)
matched_org_unit_id int | None .matched_org_unit_id reconciler-filled (public.org_units.id)
target_party_uid str | None .target_party_uid diff-service paired current party uid (write strategy precision target)
match_method str .match_method MatchMethod enum value as string
match_confidence float .match_confidence 0.0–1.0
matched_user_label str | None .matched_user_label for FE preview UI
matched_org_unit_label str | None .matched_org_unit_label for FE preview UI

2.6 import_warnings[i] schema

Per Phase 2 Task 2 ImportWarning dataclass (domain/oscal/import_pipeline/warnings.py, planned).

key type notes
ref str the offending value (e.g. LA title not found)
reason str warning message (e.g. "leveraged_authorization not found in bundle")
context dict additional context for FE display (e.g. {"component_title": "AcmeCorp Backup Service"})
source_table str | None optional — "components" / "leveraged_authorizations" / "inventory_items"

Emitted by Task 3 normalizer when:

  • duplicate (title, component_type) collapsed → source_table="components"
  • duplicate LA title collapsed → source_table="leveraged_authorizations"
  • ParsedComponent.leveraged_authorization_ref not found in bundle → set to None
  • ParsedInventoryItem.implemented_component_refs entry not found → pruned

3. v1 → v2 Conversion Rules (for bundle_restore reference)

Note: Phase 2 does NOT auto-migrate legacy v1-ssp parsed_result to v2-bundle. SspImportConfirmService raises LegacyConfirmRequired when it detects v1-ssp (no schema_version or schema_version != "v2-bundle"), and the caller dispatches to the existing _legacy_confirm path. This conversion table is informational reference only — useful for:

  • Future FE migration script (offer user to discard legacy parse jobs)
  • Retrospective debugging when comparing v1 vs v2 entity counts
  • Hand-written test fixtures translating v1 dicts to v2 dataclasses
v1 location v2 location mapping
devices[i] components[j] component_type='hardware', title=devices[i].name, description=devices[i].purpose (legacy field), port_ranges=None (no v1 source), other framework fields → None
info_systems[i] components[j] component_type='system', title=info_systems[i].name, description=info_systems[i].description (legacy field), purpose=None
leveraged[i] (Component side) components[j] component_type='service', title=leveraged[i].service_name, description=leveraged[i].description, purpose=leveraged[i].purpose, status=leveraged[i].status, protocol=leveraged[i].protocol, security_auth=leveraged[i].security_auth, leveraged_authorization_ref=leveraged[i].service_name (1:1 pairing)
leveraged[i] (LA side) leveraged_authorizations[k] title=leveraged[i].service_name, provider=leveraged[i].provider, date_authorized=leveraged[i].date_authorized, fedramp_package_id=leveraged[i].fedramp_package_id, impact_level=leveraged[i].impact_level, data_types=leveraged[i].data_types, nature_of_agreement=leveraged[i].nature_of_agreement, authorized_users=leveraged[i].authorized_users
parties_org[i] parties[j] party_type='organization', plus all OSCAL Party fields (name / short_name / email_address / address / telephone_number / matched_org_unit_label); role defaults to 'responsible-organization' if missing
parties_person[i] parties[j] party_type='person', plus all OSCAL Party fields (name / email_address / role / title / telephone_number / address / matched_user_label); name falls back to email when blank
parties_org[i].parent_org (legacy) parties[j].matched_org_unit_label fallback when matched_org_unit absent (pre-v2.0.1 templates)
controls_with_aos controls_with_aos unchanged — copied as-is
metadata metadata unchanged — copied as-is
system_characteristic system_characteristic unchanged — copied as-is
ref_docs ref_docs unchanged — copied as-is
n/a inventory_items no v1 source — v2-bundle inventory_items requires Excel 04_設備 → Component split where v1 conflated device-as-inventory + device-as-component into single devices rows. Migration would have to leave inventory_items=[] and let user re-upload.
n/a import_warnings no v1 source — initialized [] for converted bundle
n/a schema_version set to "v2-bundle"

3.1 Conflation pitfalls (why bundle_restore is informational only)

  1. Inventory items lost: v1 devices rows conflate "physical inventory" (ip / mac / hostname) with "component" (name / purpose / status). v2 splits these into separate inventory_items and components records — but a v1 row has no signal for "is this a physical asset I want tracked separately from the abstract component?". A retrofit must default to "component only" and lose the inventory side.
  2. LA title collision: v1 leveraged[i].service_name is reused as both the Component title and the LA title (single field). If a user accidentally created two leveraged rows with the same service_name but different provider, the v1 path used (service_name, provider) as the natural key while v2 normalizer dedups LAs by title alone — converter must surface a warning when collapsing.
  3. Party email loss: v1 Excel parties_org v2.x sheet dropped email / role columns. Existing in-flight jobs that have email filled in (from v1.x templates still defensive-read by _dict_to_parsed_parties) would round-trip cleanly, but the converter must read the legacy fields, not the stripped v2.x set.
  4. Docx legacy never had Excel v1 keys: docx-side parse jobs never wrote devices / info_systems / leveraged keys — they went straight to v2 adapter overlay (parties, leveraged_services, information_types). bundle_restore on docx in-flight jobs would need a separate path (out of Phase 2 scope; Task 9 handles docx parse v3 directly).

4. Key Observations & Open Questions

  1. template_version carry-through: v1-ssp Excel writes template_version at top level (ParsedExcel.to_dict()). v2-bundle plan does not explicitly list it. Recommendation: keep it in v2-bundle top-level for FE display continuity. Added to §2.1 table.

  2. Docx side has no Excel-equivalent legacy keys: docx in-flight jobs use a different key set (matched_controls, parties, leveraged_services, etc.). Task 4 dispatcher logic should:

    • check schema_version == "v2-bundle" → v2 path
    • else → legacy path (which internally already dispatches Excel-shape vs docx-shape via existing code)
    • bundle_restore only needs to handle the Excel v1-ssp shape (if at all)
  3. controls_with_aos parent/child structure not v3-aligned: Phase 2 plan intentionally keeps v1 shape (parent dict with nested objectives[]). The ParsedControl / ParsedAssessmentObjective dataclasses are unchanged from v2.x. This is correct per the plan ("Phase 2 not changing control path").

  4. MatchMethod serialization: dataclass field is MatchMethod enum (StrEnum); asdict() serializes to the string value. v2-bundle JSONB stores string, deserialization in bundle_restore must map back to enum (or callers accept str).

  5. date_authorized serialization: ParsedLeveragedAuthorization.date_authorized is date, JSONB stores ISO string. bundle_restore must date.fromisoformat() when reading back. Same pattern as legacy _dict_to_parsed_leveraged (line 2026-2036).

  6. role mandatory in parties_person legacy schema but free-form in v2-bundle?: v1 Excel SHEET_PERSONS.role enum is manager / reviewer / auditor / viewer. v3 ParsedParty.role is Optional[str] with OSCAL-style examples (system-owner / security-officer). Two different domains — v1 was project-participant role, v2 should be OSCAL responsible-party role. Open question for Task 7 (excel wire-up): does the new Excel template emit OSCAL-style roles or keep project-participant roles? Implicit answer in current code: kept as-is, role mapping done at write-strategy layer.

  7. ref_docs standalone: v2-bundle plan in implementation-plan-phase2.md §Task 1 sketch did not list ref_docs. Recommendation: keep ref_docs as top-level key in v2-bundle — added to §2.1 table — since it's a separate orthogonal concern (template ref doc list, not a v3 entity refactor target).