Skip to content

Commit 6c17f7c

Browse files
authored
Expose how much scratch a sandbox has allocated (#2019)
Adds `Sandbox::scratch_allocated_bytes()`, which returns how far the guest's scratch bump allocator has moved since the sandbox was last restored or snapshotted. Every page a guest writes after a restore is taken from scratch through that allocator, whether it's a copy-on-write copy of a snapshot page, a fresh page, or a page table. So this is how much memory the sandbox has dirtied. The host already reads the allocator on restore. This makes it public, so a host such as fiberd can charge each sandbox for what it wrote, the same way on KVM, mshv, WHP and HVF. - `SandboxMemoryManager::used_scratch_end_gpa()` reads the allocator, held to the transport arena since the guest controls it. `install_captured_snapshot` now uses it too. - `Sandbox::scratch_allocated_bytes()` returns the bytes between the first free scratch page and that end. It is 0 after a restore or snapshot and capped at the scratch the guest can allocate. - A unit test covers growth with writes, determinism across restores, and the reset at restore and at snapshot. - CHANGELOG entry under Added. --------- Signed-off-by: danbugs <danilochiarlone@gmail.com>
1 parent 3294ee3 commit 6c17f7c

3 files changed

Lines changed: 90 additions & 8 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
66

77
### Added
88
* `Sandbox::snapshot_with_options` takes a snapshot with `SnapshotOptions`. `SnapshotOptions::scratch_size` gives the sandbox, and later restores of the snapshot, a different scratch size. A guest can boot with the scratch its initialisation needs and run guest function calls from a snapshot with only the scratch they need, so each restore costs less. It can also give a guest more scratch than it booted with.
9+
* `Sandbox::scratch_allocated_bytes` returns how much scratch the guest has allocated since the sandbox was last restored or snapshotted. The guest controls the value it is read from.
910

1011
### Changed
1112
* Core dumps no longer record the guest binary path. The process name and command line are always `<unknown>`, as they already were for sandboxes created from snapshots by @ludfjig in https://github.com/hyperlight-dev/hyperlight/pull/1975

‎src/hyperlight_host/src/mem/mgr.rs‎

Lines changed: 22 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -987,22 +987,36 @@ impl SandboxMemoryManager<HostSharedMemory> {
987987
self.set_guest_requests(requests)?;
988988
return Ok(memory);
989989
}
990-
let allocator_offset = self.scratch_mem.mem_size()
991-
- hyperlight_common::layout::SCRATCH_TOP_ALLOCATOR_OFFSET as usize;
992-
let used_scratch_end = self.scratch_mem.read::<u64>(allocator_offset)?;
990+
let used_scratch_end = self.used_scratch_end_gpa()?;
993991
let gsnapshot = self.replace_snapshot_memory(snapshot)?;
994992
self.apply_snapshot_metadata(snapshot);
995993
self.update_snapshot_scratch_bookkeeping()?;
996994
self.zero_freed_scratch(used_scratch_end)?;
997995
Ok((gsnapshot, None))
998996
}
999997

1000-
/// Zeroes the scratch pages freed by resetting the allocator.
1001-
fn zero_freed_scratch(&mut self, used_end_gpa: u64) -> Result<()> {
998+
/// The end of the scratch the guest has allocated: its allocator's next
999+
/// free page. The guest controls that value, so it is held to the
1000+
/// transport arena, where the guest's allocator stops.
1001+
fn used_scratch_end_gpa(&self) -> Result<u64> {
1002+
let offset = self.scratch_mem.mem_size()
1003+
- hyperlight_common::layout::SCRATCH_TOP_ALLOCATOR_OFFSET as usize;
1004+
let end = self.scratch_mem.read::<u64>(offset)?;
1005+
Ok(end.min(self.layout.get_transport_arena().base_addr()))
1006+
}
1007+
1008+
/// Bytes of scratch the guest has allocated since the allocator was
1009+
/// last reset, at build, restore or snapshot.
1010+
pub(crate) fn scratch_allocated_bytes(&self) -> Result<u64> {
1011+
Ok(self
1012+
.used_scratch_end_gpa()?
1013+
.saturating_sub(self.first_free_scratch_gpa()))
1014+
}
1015+
1016+
/// Zeroes the scratch pages freed by resetting the allocator, up to
1017+
/// `end`, the allocator's end as `used_scratch_end_gpa` read it.
1018+
fn zero_freed_scratch(&mut self, end: u64) -> Result<()> {
10021019
let scratch_base = hyperlight_common::layout::scratch_base_gpa(self.scratch_mem.mem_size());
1003-
// The guest controls the allocator value. Its allocator stops at the
1004-
// transport arena.
1005-
let end = used_end_gpa.min(self.layout.get_transport_arena().base_addr());
10061020
let start = self.first_free_scratch_gpa();
10071021
if end <= start {
10081022
return Ok(());

‎src/hyperlight_host/src/sandbox/initialized.rs‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1279,6 +1279,35 @@ impl Sandbox {
12791279
pub fn status(&self) -> SandboxStatus {
12801280
self.status
12811281
}
1282+
1283+
/// Returns how many bytes of scratch memory the guest has allocated
1284+
/// since the sandbox was last restored or snapshotted.
1285+
/// Before the first restore or snapshot, this includes allocations
1286+
/// made during guest initialisation.
1287+
///
1288+
/// Host-reserved scratch, such as the transport and exception stack,
1289+
/// is excluded.
1290+
///
1291+
/// The guest controls this value. A malicious guest can make it
1292+
/// report less or more scratch than it actually allocated.
1293+
///
1294+
/// ```no_run
1295+
/// # use hyperlight_host::SandboxBuilder;
1296+
/// # fn example() -> Result<(), Box<dyn std::error::Error>> {
1297+
/// let mut sandbox = SandboxBuilder::from_file("guest.bin").build()?;
1298+
/// let snapshot = sandbox.snapshot()?;
1299+
/// sandbox.call::<()>("WriteSomething", ())?;
1300+
/// println!(
1301+
/// "guest-reported scratch allocation: {} bytes",
1302+
/// sandbox.scratch_allocated_bytes()?
1303+
/// );
1304+
/// sandbox.restore(snapshot)?;
1305+
/// # Ok(())
1306+
/// # }
1307+
/// ```
1308+
pub fn scratch_allocated_bytes(&self) -> Result<u64> {
1309+
self.mem_mgr.scratch_allocated_bytes()
1310+
}
12821311
}
12831312

12841313
impl Callable for Sandbox {
@@ -1673,6 +1702,35 @@ mod tests {
16731702
assert_eq!(echo, "recovered");
16741703
}
16751704

1705+
/// The scratch a guest has allocated covers every page it wrote, grows
1706+
/// as it writes more, and starts again from zero at a snapshot and at a
1707+
/// restore.
1708+
#[test]
1709+
fn scratch_allocated_bytes_counts_written_pages() {
1710+
let mut sbox = SandboxBuilder::from_file(simple_guest_as_pathbuf())
1711+
.heap_size(8 << 20)
1712+
.scratch_size(16 << 20)
1713+
.build()
1714+
.unwrap();
1715+
let snapshot = sbox.snapshot().unwrap();
1716+
assert_eq!(sbox.scratch_allocated_bytes().unwrap(), 0);
1717+
1718+
sbox.call::<()>("AllocAndWritePattern", 1u64 << 20).unwrap();
1719+
let one = sbox.scratch_allocated_bytes().unwrap();
1720+
assert!(one >= 1 << 20, "{one}");
1721+
// The new pattern is built before the old one is dropped.
1722+
sbox.call::<()>("AllocAndWritePattern", 2u64 << 20).unwrap();
1723+
let three = sbox.scratch_allocated_bytes().unwrap();
1724+
assert!(three >= 3 << 20, "{three}");
1725+
1726+
sbox.restore(snapshot).unwrap();
1727+
assert_eq!(sbox.scratch_allocated_bytes().unwrap(), 0);
1728+
sbox.call::<()>("AllocAndWritePattern", 1u64 << 20).unwrap();
1729+
assert_eq!(sbox.scratch_allocated_bytes().unwrap(), one);
1730+
sbox.snapshot().unwrap();
1731+
assert_eq!(sbox.scratch_allocated_bytes().unwrap(), 0);
1732+
}
1733+
16761734
/// A resized capture installs like any capture. The sandbox keeps its
16771735
/// state, folds mapped regions into the snapshot, and caches it.
16781736
#[test]
@@ -3121,6 +3179,15 @@ mod tests {
31213179
allocator,
31223180
)
31233181
.unwrap();
3182+
// The getter holds a corrupt allocator to the scratch the guest
3183+
// can allocate.
3184+
let arena = sbox.mem_mgr.layout.get_transport_arena().base_addr();
3185+
let allocated = if allocator == 0 {
3186+
0
3187+
} else {
3188+
arena - first_free
3189+
};
3190+
assert_eq!(sbox.scratch_allocated_bytes().unwrap(), allocated);
31243191

31253192
// `map_region` drops the cached snapshot without running the guest.
31263193
let map_mem = allocate_guest_memory();

0 commit comments

Comments
 (0)