Skip to content

Review follow-ups from #40 to #43 #55

Closed

Description

@danielplohmann

Small follow-ups collected while reviewing #40 to #43, to be done in one pass once the stack is merged. Line numbers refer to master at 5398e5e.

Library and CLI

  • Collision warning on the shipped DBs. ApiScout.loadDbFile warns above 1 % colliding exports (ApiScout.py:42, :102). With offsets applied, the shipped DBs exceed it: Win10 1803 has 4,482 of 115,781 (3.9 %), Server 2022 has 6,247 of 113,273 (5.5 %). Every scan with the default dbs/ prints two warnings. Raise the threshold, or warn only when the offsets were ignored and log the applied-offset case at info.
  • apiscout collect parses each DB twice: once in FingerprintCrawler.dbHasAslrOffsets (cli.py:407), again in loadDbFile.
  • collect --use-aslr-dbs --ignore-aslr: given together, --ignore-aslr wins silently (cli.py:410, :418). Make them mutually exclusive.
  • _getSortedAddresses cache (from Scan about 25 times faster by searching candidate addresses with numpy #43): it is rebuilt only when the map object or its size changes, so replacing one entry of api_maps[...] in place scans with stale addresses. No caller does this today.

DB Builder

  • ruff target for db_builder/: requires-python >= 3.10 makes ruff's UP/B rules treat the builder as 3.10 code, but it must run on 3.8 (Windows 7). B905 already needs a noqa at DatabaseBuilder.py:122. One ruff check --fix could break it without CI noticing. Use a per-path target-version = "py38" or add a 3.8 job.
  • 32-bit module list on a System32-only copy: with no SysWOW64 folder, a 32-bit process's module list is applied to the 64-bit DLLs and every offset in the DB is wrong, without a warning. Addresses below 4 GB can't be rejected outright (Windows 7 x64 loads 64-bit ntdll/kernel32/user32 there), but a warning would help.
  • ARM32 DLLs labelled 64-bit: bitness = 32 if Machine == 0x14C else 64 (DatabaseBuilder.py:143). With DBs built from any OS copy, ARM32 DLLs from SysArm32 come out as 64-bit.
  • --os_version checked late: it names the default output file, so a / or \ in it fails the write only after the whole crawl. Check it up front.

Ghidra

smda (for danielplohmann/smda#435, after it is merged)

  • The object-file limits (MAX_OBJECT_SYMBOLS, MAX_OBJECT_RELOCATIONS, MAX_OBJECT_NAME_LENGTH) and ObjectImage.allocate()'s MAX_IMAGE_SIZE are read from the SmdaConfig class, so values set on a caller's config instance are ignored.
  • BinaryInfo and ObjectSymbolProvider import the private _readable_name from PeSymbolProvider. Done in fix(loaders): read object-file limits from the caller's config instance smda#437.

Activity

  1. r0ny123 commented on Nov 18, 2026

    @r0ny123
    Contributor

    Two questions before the stack goes in:

    1. Release: should v3.0.0 be tagged only once the whole stack (Add apiscout dbs download for the example API DBs #44–DBs from the symbol server, keyed API hashes, and a lot faster everywhere #56) is merged, or would you rather tag earlier and follow up with a point release?
    2. --min-confidence: should it default to 20? That filters out the hits that the confidence score marks as likely chance matches, but it changes what a plain apiscout scan prints compared with 2.x. The alternative is defaulting to 0 and documenting the flag.

    The Ghidra item can be ticked: it needs no change, since ghidra_scout.py keeps the saved DB folder after a failure.

  2. danielplohmann commented on Nov 18, 2026

    @danielplohmann
    OwnerAuthor

    Thanks, answers to both:

    1. Release: tag v3.0.0 once the whole stack (Add apiscout dbs download for the example API DBs #44–IDA, Ghidra and Binary Ninja plugins: place the DLLs of an API DB by a module list #58) is merged, no intermediate tag.
    2. --min-confidence: default to 20. The change in plain scan output is fine under a major version; please call it out in the CHANGELOG's breaking-changes notes.

    Also fine with the DB builder moving to Python 3.10 in #57, with Windows 7 DBs built offline from a copied DLL set via --load_addresses. Ticking the Ghidra item.

  3. added a commit that references this issue on Nov 18, 2026
  4. r0ny123 commented on Nov 18, 2026

    @r0ny123
    Contributor

    Thanks! Both done/noted:

    1. v3.0.0 once everything's in. Since Add apiscout dbs download for the example API DBs #44–IDA, Ghidra and Binary Ninja plugins: place the DLLs of an API DB by a module list #58 are merged now, it's ready to tag whenever you are. I've got release notes with a 2.x → 3.0 migration table ready, I'll paste them here (or wherever you'd like them).
    2. --min-confidence defaults to 20 in scan: drop hits with a confidence below 20 by default #59. Without the flag, a plain apiscout scan now drops the hits below 20; --min-confidence 0 brings back the 2.x output. The CHANGELOG calls it out as (BREAKING) in the confidence entry, and the "Dropped N of M hits" line now tells you how to keep everything. The plugins and apiscout collect aren't touched, they still use every hit.

    One thing I'd like your take on: collect still builds collection vectors from all hits, while scan --collection now matches with the filtered ones. Chance hits should be rare with the right DB so it shouldn't move scores much, but if you'd rather have both sides the same I can give collect the same default in a follow-up.

    Thanks for ticking the Ghidra item, and good to hear the Python 3.10 move for the builder is fine.

  5. r0ny123 commented on Nov 18, 2026

    @r0ny123
    Contributor

    @danielplohmann here are the release notes for v3.0.0, for when #59 is in and you tag it. Feel free to cut or reword anything.


    3.0.0 turns ApiScout into one apiscout command, adds new input formats, API hash detection and plugins for current IDA, Ghidra and Binary Ninja, and makes everything much faster.

    pip install apiscout, then apiscout dbs download and apiscout scan dump_0x00400000 --db win7_prof-n_sp1_example.json.

    New input formats. scan reads Windows minidumps (range by range, at their own addresses) and reads from standard input with -. --import-table uses a parser of our own, so LIEF is no longer a dependency.

    DLL placement (ASLR). One API DB per Windows build now fits dumps of every boot. --modules FILE, the module list of a minidump, or DLL images found in a flat dump place the DLLs at their real load addresses.

    Confidence and explain. Every hit gets a confidence from 0 to 100 against chance hits. scan drops hits below 20 by default; --min-confidence N sets the cut, --min-confidence 0 keeps every hit. apiscout match A B --explain lists the APIs two vectors share, and those only one has, with their weight in the score.

    Dynamic API hashes. --dynamic also finds APIs a program resolves at run time: hashes of export names (ROR13, Metasploit block_api, CRC32, djb2, FNV-1a, sdbm), also with an XOR or ADD key of the sample, plus export names as strings. Without the flag, nothing changes.

    DBs. apiscout dbs download fetches the example DBs, checked against dbs/SHA256SUMS. New example DBs: Windows Server 2022 21H2 and Windows 11 Pro 26H2. DllDownloader.py builds the DB of a Windows build from Winbindex and Microsoft's symbol server, without a machine of that build.

    Plugins. IDA 9 (dockable view, batch mode, ida-plugin.json for hcli), Ghidra 12 with PyGhidra (ghidra_scout.py replaces GhidraScout.java), Binary Ninja 6 (binja_scout, new).

    Speed. Scanning a 62 MB dump with two DBs went from 73 s to 2.9 s, and a 65 MB dump with one DB from 2.0 s to 0.22 s. --dynamic with the key search on 65 MB: 5.3 s to 1.2 s. Matching against 20,000 vectors: 2.9 s to 0.08 s. scan with all five example DBs: 1.9 s to 0.8 s. Building a DB of 439 DLLs: 80 s to 0.4 s. Same results as before.

    Also: Python 3.10+, packaging with pyproject.toml, ruff and ty, CI on GitHub Actions, releases go to PyPI through trusted publishing. File headers now say BSD-2-Clause, like LICENSE. The full list is in the CHANGELOG.

    Moving from 2.x

    2.x 3.0.0
    scout.py dump db.json apiscout scan dump --db db.json
    every hit, chance hits included apiscout scan dump --db db.json --min-confidence 0
    scout.py -t / -i / -b / -f --import-table / --ignore-aslr / --base-addr / --filter
    scout.py -o out.json apiscout scan --json -o out.json
    match.py A -v B, -c FILE apiscout match A B, -c FILE
    export.py -p x.png -w x.html apiscout export --png x.png --html x.html
    collect.py ROOT -r _dump -u FILE apiscout collect ROOT --require _dump --update FILE
    update.py apiscout update
    GhidraScout.java ghidra_scout.py (PyGhidra)

    Breaking:

    • The JSON changed. scout.py -o wrote a list of rows. apiscout scan --json writes an object with the same rows under apis, next to base_address, api_dbs, apivector, apis_in_vector and, with --collection, collection_matches. Scripts that read the list need ["apis"] now. (There are more fields, like placed_dlls, and dynamic_apis with --dynamic.)
    • Python 2 and IDA before 9.0 are no longer supported. Python 3.10 or newer.
    • scout.py, match.py, export.py, collect.py and update.py are gone, and so are setup.py, requirements.txt and template.config.py.
    • update reads the Malpedia token from ~/.config/apiscout/malpedia-token (or --token-file), not from config.py.
    • collect skips API DBs with ASLR offsets unless you pass --use-aslr-dbs or --ignore-aslr.
    • scan drops hits with a confidence below 20 by default, so it can list fewer APIs than scout.py did. --min-confidence 0 keeps every hit.
  6. danielplohmann commented on Nov 18, 2026

    @danielplohmann
    OwnerAuthor

    All items done: the apiscout ones in #57, the smda ones in danielplohmann/smda#437, and the Ghidra one needed no change. The --min-confidence default of 20 follows in a separate PR before v3.0.0 is tagged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions