Repository navigation
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1075 +/- ##
==========================================
+ Coverage 93.92% 93.99% +0.06%
==========================================
Files 243 257 +14
Lines 17890 18096 +206
==========================================
+ Hits 16803 17009 +206
Misses 1087 1087 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
51ba04e to
d39201e
Compare
d39201e to
9e4d481
Compare
krempel-4711
left a comment
There was a problem hiding this comment.
I left some comments.
| DATA_SECTIONS = ("EventData", "UserData", "ProcessingErrorData") | ||
|
|
||
|
|
||
| def _parse_system(system: ET.Element, fields: dict[str, FieldValue]) -> None: |
There was a problem hiding this comment.
FYI: we introduced JsonObjectas an alias for dict[str,FieldValue]
| "message": "<!DOCTYPE Event [<!ENTITY x "expanded">]>" | ||
| "<Event><System><EventID>&x;</EventID></System></Event>" |
There was a problem hiding this comment.
Consider testing for the acutal thing.
| "message": "<!DOCTYPE Event [<!ENTITY x "expanded">]>" | |
| "<Event><System><EventID>&x;</EventID></System></Event>" | |
| "message": """ | |
| <!DOCTYPE Event [ | |
| <!ENTITY a "BOOM"> | |
| <!ENTITY b "&a;&a;"> | |
| <!ENTITY c "&b;&b;"> | |
| <!ENTITY d "&c;&c;"> | |
| <!ENTITY e "&d;&d;"> | |
| <!ENTITY f "&e;&e;"> | |
| <!ENTITY g "&f;&f;"> | |
| <!ENTITY h "&g;&g;"> | |
| <!ENTITY i "&h;&h;"> | |
| <!ENTITY j "&i;&i;"> | |
| <!ENTITY k "&j;&j;"> | |
| <!ENTITY l "&k;&k;"> | |
| <!ENTITY m "&l;&l;"> | |
| <!ENTITY n "&m;&m;"> | |
| <!ENTITY o "&n;&n;"> | |
| <!ENTITY p "&o;&o;"> | |
| <!ENTITY q "&p;&p;"> | |
| <!ENTITY r "&q;&q;"> | |
| <!ENTITY s "&r;&r;"> | |
| <!ENTITY t "&s;&s;"> | |
| <!ENTITY u "&t;&t;"> | |
| <!ENTITY v "&u;&u;"> | |
| <!ENTITY w "&v;&v;"> | |
| <!ENTITY x "&w;&w;"> | |
| ]> | |
| <Event><System><EventID>&x;</EventID></System></Event> | |
| """ |
| - Fields that are not part of the XML (for example rendered messages or | ||
| event categories) cannot be derived and are not added. | ||
| """ | ||
| if "<!doctype" in log_line.lower(): |
There was a problem hiding this comment.
Introducing this extra level of saftey is probably a good thing, though this heuristic will introduce a false-positive if the string <!doctype is part of the EventData (i.e. in an embedded xml) i am not familiar enough with the domain to judge if this might be a relevant edgecase.
On the other hand xml.ElementTree already checks for entity expansion and gives a Parsing error with the message:
limit on input amplification factor (from DTD and entities)
I am not opinionated if you should keep the current heuristic or rely on the library handeling that vulnerbility, as long as there is a meaningfull error message.
|
Please set your PRs to Draft if they are not ready. My previous comments might be outdated... |
Description
windows_event_logsource format to the decoder processor. It parses Windows Event Log XML from a string field into a flat dict:Systemvalues are mapped toim_msvistalog-style field names,EventData/UserData/ProcessingErrorDataare flattened to the same level and the raw XML is kept inEventXML.EventTimewithTimeParser.from_string()fromlogprep.util.timeto ISO 8601 UTC. Numeric system values become integers, values that cannot be converted stay strings.MAX_XML_DEPTH. Malformed XML, undefined entities and DOCTYPE declarations becomeDecoderErrorand use the existing_decoder_failurehandling.fieldsrule option forwindows_event_logto configure values from any XML section, including target field name,section,namematching, XML attribute and conversion (str,int,timestamp). Withfieldsset, only the configured values andEventXMLare emitted. Invalid configurations are rejected when the rule is created.decoders.pyinto adecoderspackage with one module per source format, shared helpers and a registry.CHANGELOG.md.Assignee
fieldsrule option with validation and documentation)Documentation
--fail-on-warningsucceeded)Code Quality
decoderspackage: 100% statements and branches)How did you verify that the changes work in practice?
uv run pytest -q tests/unit→ 9494 passed, 26 skipped, 4 xfaileduv run pytest -q tests/acceptance→ 65 passed, 1 skippeduv run pytest -q tests/unit/processor/decoder/test_decoder.py tests/unit/ng/processor/decoder/test_decoder.py→ 239 passeduv run pytest -q --cov=logprep.processor.decoder.decoders --cov-branch --cov-report=term-missing tests/unit/processor/decoder/test_decoder.py tests/unit/ng/processor/decoder/test_decoder.py→ 100% (252 statements, 58 branches)uv run mypy logprep/processor/decoder logprep/ng/processor/decoder→ cleanuv run black --check --config ./pyproject.toml .→ clean;uv run isort --check-onlyanduv run pylint --rcfile=./pyproject.tomlon the changed files → clean (the whole-repo isort check reports pre-existing issues in unrelated files)uv run pre-commit run --files <changed files>→ all hooks passeduv run make clean html SPHINXOPTS="--fail-on-warning"(indoc/) → build succeededresponses/CookieJar failures in HTTP getter tests; without coverage all 239 decoder tests pass. The 100% figure above is for the new decoder modules.Reviewer