Skip to content

Unpacking optimizations - #1550

Draft
jstucke wants to merge 2 commits into
masterfrom
unpacking-optimizations
Draft

jstucke wants to merge 2 commits into
masterfrom
unpacking-optimizations

Conversation

@jstucke

@jstucke jstucke commented Sep 29, 2026

Copy link
Copy Markdown
Collaborator
  • replaces the sleep in the unpacking scheduler when no containers are available with a queue
    • for small files the scheduler sleeps while containers are long since ready
    • with the SimpleQueue no time is wasted sleeping (the scheduler only waits as long as necessary)
  • mounts the directory with firmware images and extracted files (firmware_file_storage_directory) into the extractor containers
  • also adds a check to make sure that docker_mount_base_dir and firmware_file_storage_directory are on the same file system (otherwise files get copied after extraction instead of a cheaper move/rename)

@jstucke jstucke self-assigned this Sep 29, 2026
@jstucke
jstucke force-pushed the unpacking-optimizations branch from 5a0491d to abbe1ac Compare September 29, 2026 14:30
* for small files the scheduler sleeps while containers are long since ready
* with the SimpleQueue no time is wasted sleeping (the scheduler only waits as long as necessary)
* mounts the directory with firmware images and extracted files (firmware_file_storage_directory) into the extractor containers
  * this means we don't need to copy the files before unpacking which reduces I/O overhead
* needs fkie-cad/fact_extractor#238
  * checks if the container image has the feature, and if not, aborts the start
* also adds a check to make sure that docker_mount_base_dir and firmware_file_storage_directory are on the same file system (otherwise files get copied after extraction instead of a cheaper move/rename)
@jstucke
jstucke force-pushed the unpacking-optimizations branch from c9a0541 to b8624d1 Compare November 16, 2026 13:25

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant