Repository navigation
Closed
Description
Small follow-ups collected while reviewing #40 to #43, to be done in one pass once the stack is merged. Line numbers refer to master at 5398e5e.
Library and CLI
- Collision warning on the shipped DBs.
ApiScout.loadDbFilewarns above 1 % colliding exports (ApiScout.py:42,:102). With offsets applied, the shipped DBs exceed it: Win10 1803 has 4,482 of 115,781 (3.9 %), Server 2022 has 6,247 of 113,273 (5.5 %). Every scan with the defaultdbs/prints two warnings. Raise the threshold, or warn only when the offsets were ignored and log the applied-offset case at info. -
apiscout collectparses each DB twice: once inFingerprintCrawler.dbHasAslrOffsets(cli.py:407), again inloadDbFile. -
collect --use-aslr-dbs --ignore-aslr: given together,--ignore-aslrwins silently (cli.py:410,:418). Make them mutually exclusive. -
_getSortedAddressescache (from Scan about 25 times faster by searching candidate addresses with numpy #43): it is rebuilt only when the map object or its size changes, so replacing one entry ofapi_maps[...]in place scans with stale addresses. No caller does this today.
DB Builder
- ruff target for
db_builder/:requires-python >= 3.10makes ruff'sUP/Brules treat the builder as 3.10 code, but it must run on 3.8 (Windows 7).B905already needs anoqaatDatabaseBuilder.py:122. Oneruff check --fixcould break it without CI noticing. Use a per-pathtarget-version = "py38"or add a 3.8 job. - 32-bit module list on a System32-only copy: with no SysWOW64 folder, a 32-bit process's module list is applied to the 64-bit DLLs and every offset in the DB is wrong, without a warning. Addresses below 4 GB can't be rejected outright (Windows 7 x64 loads 64-bit ntdll/kernel32/user32 there), but a warning would help.
- ARM32 DLLs labelled 64-bit:
bitness = 32 if Machine == 0x14C else 64(DatabaseBuilder.py:143). With DBs built from any OS copy, ARM32 DLLs from SysArm32 come out as 64-bit. -
--os_versionchecked late: it names the default output file, so a/or\in it fails the write only after the whole crawl. Check it up front.
Ghidra
- scout.py path forgotten on any failure (
GhidraScout.java:214): check whether this carries over toghidra_scout.pyin Replace GhidraScout.java with a PyGhidra script #46.
smda (for danielplohmann/smda#435, after it is merged)
- The object-file limits (
MAX_OBJECT_SYMBOLS,MAX_OBJECT_RELOCATIONS,MAX_OBJECT_NAME_LENGTH) andObjectImage.allocate()'sMAX_IMAGE_SIZEare read from theSmdaConfigclass, so values set on a caller's config instance are ignored. -
BinaryInfoandObjectSymbolProviderimport the private_readable_namefromPeSymbolProvider. Done in fix(loaders): read object-file limits from the caller's config instance smda#437.
Metadata
Metadata
Assignees
Labels
No labels
Activity
r0ny123 commentedon Nov 18, 2026
Two questions before the stack goes in:
--min-confidence: should it default to 20? That filters out the hits that the confidence score marks as likely chance matches, but it changes what a plainapiscout scanprints compared with 2.x. The alternative is defaulting to 0 and documenting the flag.The Ghidra item can be ticked: it needs no change, since
ghidra_scout.pykeeps the saved DB folder after a failure.danielplohmann commentedon Nov 18, 2026
Thanks, answers to both:
--min-confidence: default to 20. The change in plainscanoutput is fine under a major version; please call it out in the CHANGELOG's breaking-changes notes.Also fine with the DB builder moving to Python 3.10 in #57, with Windows 7 DBs built offline from a copied DLL set via
--load_addresses. Ticking the Ghidra item.Merge pull request #57 from r0ny123/fix/review-followups
r0ny123 commentedon Nov 18, 2026
Thanks! Both done/noted:
--min-confidencedefaults to 20 in scan: drop hits with a confidence below 20 by default #59. Without the flag, a plainapiscout scannow drops the hits below 20;--min-confidence 0brings back the 2.x output. The CHANGELOG calls it out as (BREAKING) in the confidence entry, and the "Dropped N of M hits" line now tells you how to keep everything. The plugins andapiscout collectaren't touched, they still use every hit.One thing I'd like your take on:
collectstill builds collection vectors from all hits, whilescan --collectionnow matches with the filtered ones. Chance hits should be rare with the right DB so it shouldn't move scores much, but if you'd rather have both sides the same I can givecollectthe same default in a follow-up.Thanks for ticking the Ghidra item, and good to hear the Python 3.10 move for the builder is fine.
r0ny123 commentedon Nov 18, 2026
@danielplohmann here are the release notes for v3.0.0, for when #59 is in and you tag it. Feel free to cut or reword anything.
3.0.0 turns ApiScout into one
apiscoutcommand, adds new input formats, API hash detection and plugins for current IDA, Ghidra and Binary Ninja, and makes everything much faster.pip install apiscout, thenapiscout dbs downloadandapiscout scan dump_0x00400000 --db win7_prof-n_sp1_example.json.New input formats.
scanreads Windows minidumps (range by range, at their own addresses) and reads from standard input with-.--import-tableuses a parser of our own, so LIEF is no longer a dependency.DLL placement (ASLR). One API DB per Windows build now fits dumps of every boot.
--modules FILE, the module list of a minidump, or DLL images found in a flat dump place the DLLs at their real load addresses.Confidence and explain. Every hit gets a confidence from 0 to 100 against chance hits.
scandrops hits below 20 by default;--min-confidence Nsets the cut,--min-confidence 0keeps every hit.apiscout match A B --explainlists the APIs two vectors share, and those only one has, with their weight in the score.Dynamic API hashes.
--dynamicalso finds APIs a program resolves at run time: hashes of export names (ROR13, Metasploit block_api, CRC32, djb2, FNV-1a, sdbm), also with an XOR or ADD key of the sample, plus export names as strings. Without the flag, nothing changes.DBs.
apiscout dbs downloadfetches the example DBs, checked againstdbs/SHA256SUMS. New example DBs: Windows Server 2022 21H2 and Windows 11 Pro 26H2.DllDownloader.pybuilds the DB of a Windows build from Winbindex and Microsoft's symbol server, without a machine of that build.Plugins. IDA 9 (dockable view, batch mode,
ida-plugin.jsonfor hcli), Ghidra 12 with PyGhidra (ghidra_scout.pyreplacesGhidraScout.java), Binary Ninja 6 (binja_scout, new).Speed. Scanning a 62 MB dump with two DBs went from 73 s to 2.9 s, and a 65 MB dump with one DB from 2.0 s to 0.22 s.
--dynamicwith the key search on 65 MB: 5.3 s to 1.2 s. Matching against 20,000 vectors: 2.9 s to 0.08 s.scanwith all five example DBs: 1.9 s to 0.8 s. Building a DB of 439 DLLs: 80 s to 0.4 s. Same results as before.Also: Python 3.10+, packaging with
pyproject.toml, ruff and ty, CI on GitHub Actions, releases go to PyPI through trusted publishing. File headers now say BSD-2-Clause, likeLICENSE. The full list is in the CHANGELOG.Moving from 2.x
scout.py dump db.jsonapiscout scan dump --db db.jsonapiscout scan dump --db db.json --min-confidence 0scout.py -t/-i/-b/-f--import-table/--ignore-aslr/--base-addr/--filterscout.py -o out.jsonapiscout scan --json -o out.jsonmatch.py A -v B,-c FILEapiscout match A B,-c FILEexport.py -p x.png -w x.htmlapiscout export --png x.png --html x.htmlcollect.py ROOT -r _dump -u FILEapiscout collect ROOT --require _dump --update FILEupdate.pyapiscout updateGhidraScout.javaghidra_scout.py(PyGhidra)Breaking:
scout.py -owrote a list of rows.apiscout scan --jsonwrites an object with the same rows underapis, next tobase_address,api_dbs,apivector,apis_in_vectorand, with--collection,collection_matches. Scripts that read the list need["apis"]now. (There are more fields, likeplaced_dlls, anddynamic_apiswith--dynamic.)scout.py,match.py,export.py,collect.pyandupdate.pyare gone, and so aresetup.py,requirements.txtandtemplate.config.py.updatereads the Malpedia token from~/.config/apiscout/malpedia-token(or--token-file), not fromconfig.py.collectskips API DBs with ASLR offsets unless you pass--use-aslr-dbsor--ignore-aslr.scandrops hits with a confidence below 20 by default, so it can list fewer APIs thanscout.pydid.--min-confidence 0keeps every hit.fix(loaders): read object-file limits from the caller's config instan…
danielplohmann commentedon Nov 18, 2026
All items done: the apiscout ones in #57, the smda ones in danielplohmann/smda#437, and the Ghidra one needed no change. The
--min-confidencedefault of 20 follows in a separate PR before v3.0.0 is tagged.scan: drop hits with a confidence below 20 by default