Skip to content

add windows event log xml parser to decoder - #1075

Open
ekneg54 wants to merge 1 commit into
mainfrom
dev-add-windows-eventlog-xml-parser
Open

ekneg54 wants to merge 1 commit into
mainfrom
dev-add-windows-eventlog-xml-parser

Conversation

@ekneg54

@ekneg54 ekneg54 commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Description

  • Add a windows_event_log source format to the decoder processor. It parses Windows Event Log XML from a string field into a flat dict: System values are mapped to im_msvistalog-style field names, EventData/UserData/ProcessingErrorData are flattened to the same level and the raw XML is kept in EventXML.
  • Normalize EventTime with TimeParser.from_string() from logprep.util.time to ISO 8601 UTC. Numeric system values become integers, values that cannot be converted stay strings.
  • Harden the XML parser against the risks listed in the Python XML security notes: a custom ElementTree target rejects DOCTYPE declarations (external entities, entity expansion) and documents nested deeper than MAX_XML_DEPTH. Malformed XML, undefined entities and DOCTYPE declarations become DecoderError and use the existing _decoder_failure handling.
  • Add the fields rule option for windows_event_log to configure values from any XML section, including target field name, section, name matching, XML attribute and conversion (str, int, timestamp). With fields set, only the configured values and EventXML are emitted. Invalid configurations are rejected when the rule is created.
  • Split decoders.py into a decoders package with one module per source format, shared helpers and a registry.
  • Document the format and its options in the legacy and ng processor docstrings, the rule configuration docs and CHANGELOG.md.
  • Add parser/configuration cases inherited by both the legacy and ng suites plus parametrized rule-validation cases, alongside the original Windows Event Log cases.

Assignee

  • The changes adhere to the contribution guidelines
  • I have performed a self-review of my code
  • My changes generate no new warnings (e.g. flake8/mypy/pytest/...) other than deprecations
  • Relevant changes are applied to non-ng and ng
  • Check xml security concerns https://docs.python.org/3/library/xml.html#xml-security and add tests (DOCTYPE, external entity, entity expansion, undefined entity, excessive nesting)
  • Make all XML fields configurable (fields rule option with validation and documentation)
  • restructure decoders module to single modules per decoder

Documentation

  • README.md is up-to-date (no changes required; it does not list decoder source formats)
  • CHANGELOG.md is up-to-date
  • Documentation (doc/source) is up-to-date
  • Preview for docs looks fine (local build with --fail-on-warning succeeded)
  • Examples are up-to-date and functional (including compose & k8s) (not affected)

Code Quality

  • Patch test coverage > 95% and does not decrease (whole decoders package: 100% statements and branches)
  • New code uses correct & specific type hints (mypy clean)

How did you verify that the changes work in practice?

  • Linux, Python 3.13, executed from the repository root:
    • uv run pytest -q tests/unit → 9494 passed, 26 skipped, 4 xfailed
    • uv run pytest -q tests/acceptance → 65 passed, 1 skipped
    • uv run pytest -q tests/unit/processor/decoder/test_decoder.py tests/unit/ng/processor/decoder/test_decoder.py → 239 passed
    • uv run pytest -q --cov=logprep.processor.decoder.decoders --cov-branch --cov-report=term-missing tests/unit/processor/decoder/test_decoder.py tests/unit/ng/processor/decoder/test_decoder.py → 100% (252 statements, 58 branches)
    • uv run mypy logprep/processor/decoder logprep/ng/processor/decoder → clean
    • uv run black --check --config ./pyproject.toml . → clean; uv run isort --check-only and uv run pylint --rcfile=./pyproject.toml on the changed files → clean (the whole-repo isort check reports pre-existing issues in unrelated files)
    • uv run pre-commit run --files <changed files> → all hooks passed
    • uv run make clean html SPHINXOPTS="--fail-on-warning" (in doc/) → build succeeded
  • A coverage run over the whole decoder package triggers 4 pre-existing responses/CookieJar failures in HTTP getter tests; without coverage all 239 decoder tests pass. The 100% figure above is for the new decoder modules.

Reviewer

@ekneg54 ekneg54 self-assigned this Nov 20, 2026
@codecov-commenter

codecov-commenter commented Nov 20, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.99%. Comparing base (146e267) to head (9e4d481).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1075      +/-   ##
==========================================
+ Coverage   93.92%   93.99%   +0.06%     
==========================================
  Files         243      257      +14     
  Lines       17890    18096     +206     
==========================================
+ Hits        16803    17009     +206     
  Misses       1087     1087              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@ekneg54
ekneg54 force-pushed the dev-add-windows-eventlog-xml-parser branch from 51ba04e to d39201e Compare November 20, 2026 10:57
@ekneg54
ekneg54 force-pushed the dev-add-windows-eventlog-xml-parser branch from d39201e to 9e4d481 Compare November 20, 2026 11:15
@ekneg54
ekneg54 requested a review from mhoff November 20, 2026 11:31

@krempel-4711 krempel-4711 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments.

Comment thread logprep/processor/decoder/decoders.py Outdated
DATA_SECTIONS = ("EventData", "UserData", "ProcessingErrorData")


def _parse_system(system: ET.Element, fields: dict[str, FieldValue]) -> None:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI: we introduced JsonObjectas an alias for dict[str,FieldValue]

Comment on lines +1021 to +1022
"message": "<!DOCTYPE Event [<!ENTITY x &#34;expanded&#34;>]>"
"<Event><System><EventID>&x;</EventID></System></Event>"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider testing for the acutal thing.

Suggested change
"message": "<!DOCTYPE Event [<!ENTITY x &#34;expanded&#34;>]>"
"<Event><System><EventID>&x;</EventID></System></Event>"
"message": """
<!DOCTYPE Event [
<!ENTITY a "BOOM">
<!ENTITY b "&a;&a;">
<!ENTITY c "&b;&b;">
<!ENTITY d "&c;&c;">
<!ENTITY e "&d;&d;">
<!ENTITY f "&e;&e;">
<!ENTITY g "&f;&f;">
<!ENTITY h "&g;&g;">
<!ENTITY i "&h;&h;">
<!ENTITY j "&i;&i;">
<!ENTITY k "&j;&j;">
<!ENTITY l "&k;&k;">
<!ENTITY m "&l;&l;">
<!ENTITY n "&m;&m;">
<!ENTITY o "&n;&n;">
<!ENTITY p "&o;&o;">
<!ENTITY q "&p;&p;">
<!ENTITY r "&q;&q;">
<!ENTITY s "&r;&r;">
<!ENTITY t "&s;&s;">
<!ENTITY u "&t;&t;">
<!ENTITY v "&u;&u;">
<!ENTITY w "&v;&v;">
<!ENTITY x "&w;&w;">
]>
<Event><System><EventID>&x;</EventID></System></Event>
"""

Comment thread logprep/processor/decoder/decoders.py Outdated
- Fields that are not part of the XML (for example rendered messages or
event categories) cannot be derived and are not added.
"""
if "<!doctype" in log_line.lower():

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Introducing this extra level of saftey is probably a good thing, though this heuristic will introduce a false-positive if the string <!doctype is part of the EventData (i.e. in an embedded xml) i am not familiar enough with the domain to judge if this might be a relevant edgecase.

On the other hand xml.ElementTree already checks for entity expansion and gives a Parsing error with the message:

limit on input amplification factor (from DTD and entities)

I am not opinionated if you should keep the current heuristic or rely on the library handeling that vulnerbility, as long as there is a meaningfull error message.

@krempel-4711

Copy link
Copy Markdown
Collaborator

Please set your PRs to Draft if they are not ready. My previous comments might be outdated...

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants