Packages

The Tiger package format: file names, header, entry table, blocks, named tags, and how entries get loaded.

The game keeps its content in .pkg files, called packages. A package holds many entries. Each entry is one tag: a blob of bytes with a class. The bytes live in blocks, and a block can be compressed, encrypted or both.

Every package in build 86657 uses package format version 38. Field names are descriptive, not Bungie’s. For what a tag is and how the engine reads its bytes, see Tags, classes and definitions.

Files and patches

A package file name has this form:

w64_<family>_<package id>_<patch>.pkg
w64_sandbox_0197_5.pkg        family "sandbox", package 0x0197, patch 5
w64_globals_0377_en_2.pkg     an English locale package, package 0x0377, patch 2

part

meaning

w64

the platform, 64-bit Windows

<family>

a content group, such as sandbox, activities, investment or ui

<package id>

4 hex digits; the same value as header +0x04

_en

present on locale packages only

<patch>

a decimal patch number; the same value as header +0x20

One package id can have several files, one per patch. The install for build 86657 holds 2202 files for 528 package ids.

Three packages break the naming rule: w64_client_bootstrap_unp1_0.pkg, w64_ui_bootflow_unp1_0.pkg and w64_ui_startup_unp1_0.pkg. Their names have no hex id. Read the package id from header +0x04, not from the file name.

Picking the newest patch

A later patch replaces an earlier one. Use the file with the highest patch number for each package id. The client does the same: at load it keeps the highest patch id it sees per package id.

The newest file is not always self-contained. Each block record names the patch file that holds its bytes. So the block table of patch 5 can point into the file for patch 2. See Blocks.

A later patch can also reuse an entry index for a different tag. Data read from an older patch file can be wrong for the shipped game. Filter to the newest patch before you trust any entry.

The header

The header is the first 0x180 bytes of the file. All values are little-endian.

struct PackageHeader {              /* 0x180 bytes read */
    uint16_t version;               /* +0x00  always 38 */
    uint16_t platform;              /* +0x02  2 = w64 */
    uint16_t package_id;            /* +0x04  high bits of every tag in this file */
    uint16_t unknown_06;            /* +0x06  1 in every package; meaning unknown */
    uint64_t group_id;              /* +0x08  build signature; files of one build share it */
    uint64_t build_time;            /* +0x10  Unix time; 2017 to 2020 in this install */
    uint32_t content_build;         /* +0x18  highest value is 86657, the exe build */
    uint32_t content_revision;      /* +0x1C  0, 1 or 2; breaks ties in content_build */
    uint16_t patch_id;              /* +0x20  matches the file name; at most 0xFF */
    uint16_t language;              /* +0x22  matches the locale in the file name */
    char     tool_string[128];      /* +0x24  zero in every file */
    uint8_t  unknown_a4[0x0C];      /* +0xA4 */
    uint32_t signature_offset;      /* +0xB0  file offset of the signature; 0x800 in every file */
    uint32_t entry_count;           /* +0xB4  1 to 8192 */
    uint32_t entry_table_2017;      /* +0xB8  entry table offset, 2017 layout only */
    uint8_t  unknown_bc[0x14];      /* +0xBC */
    uint32_t block_count;           /* +0xD0 */
    uint32_t block_table_2017;      /* +0xD4  block table offset, 2017 layout only */
    uint8_t  unknown_d8[0x14];      /* +0xD8 */
    uint32_t unknown_ec;            /* +0xEC  zero in most packages; not the misc size */
    uint32_t misc_offset;           /* +0xF0  file offset of the misc block; 0 = none */
    uint32_t misc_length;           /* +0xF4  bytes the misc block hash covers */
    uint8_t  misc_sha1[20];         /* +0xF8  SHA-1 of the misc block */
    uint8_t  unknown_10c[4];        /* +0x10C */
    uint32_t entry_table_raw;       /* +0x110 table blob offset; entry table is at +96 */
    uint32_t table_blob_length;     /* +0x114 bytes the table blob hash covers */
    uint8_t  table_blob_sha1[20];   /* +0x118 SHA-1 of the table blob */
    uint8_t  unknown_12c[0x38];     /* +0x12C */
    uint32_t file_size;             /* +0x164 exact size of this file */
    uint32_t locale_check;          /* +0x168 a per-locale value the client checks at load */
    uint8_t  locale_id;             /* +0x16C 0, or 1 for the three unp1 packages */
};

Notes:

package_id is a 16-bit value. Read as 32 bits, it is wrong in almost every file.
entry_count is capped at 8192. The client rejects a larger value. 8192 is also the limit a tag can address, because a tag has 13 bits for the entry index.
entry_count and block_count are different fields. They are never equal in this install.
The package id range 0x100 to 0xA9F is not a validity check. A package outside it still loads. The loader only skips recording its content build.
locale_check has one value for every locale-0 package and another for the three locale-1 packages. It depends on the locale and on the content id the client holds, not on the package. A client with the wrong content id fails every locale-0 package at this check.
The unknown_* gaps are fields that are not identified.

The 256 bytes at file offset 0x800 are a signature, followed by zero bytes. It is RSA-2048 PSS with a 16-byte salt, over the SHA-256 of header bytes 0x000 to 0x16F. The client holds one public key for it. Every installed package verifies.

So any edit to the first 0x170 bytes breaks the signature. That range includes file_size and locale_check.

The last 0x800 bytes of every package are not a signature. They are 0xDEADBEEF followed by 2044 zero bytes. A tool that rewrites a package does not need to refresh them.

What the client checks at load

check

data

result on the install

header signature

the signature above

every package passes

misc block hash

SHA-1 of misc_length bytes at misc_offset, against misc_sha1

every package passes

table blob hash

SHA-1 of table_blob_length bytes at entry_table_raw, against table_blob_sha1

every later-layout package passes

The table blob check also bounds the blob’s own entry count to 8192. Packages in the 2017 layout have no table blob: all three fields are zero. Whether the client runs the table blob check on them is unverified.

Two header layouts

The install holds two layouts. Both are version 38, so the version does not tell them apart.

layout

built

entry table offset

block table offset

2017

entry_table_2017 at +0xB8

block_table_2017 at +0xD4

later (“pre-Beyond Light”)

2018 to 2020

entry_table_raw + 96, from +0x110

not in the header; see below

Byte +0x1A picks the layout: 0 for the 2017 layout, 1 for the later one. It is the third byte of content_build, so the later layout is every package with a content build of 0x10000 or more.

Every package id in this install resolves to a later-layout file at its newest patch. A reader that only reads the newest patch never meets the 2017 layout.

Typed arrays

The entry table, the block table and the tables inside the misc block are all engine arrays. Each array has a 20-byte prefix directly before its data:

struct ArrayPrefix {                /* 20 bytes, directly before the array data */
    uint32_t marker;                /* -0x14  always 0x80809FBD */
    uint64_t count;                 /* -0x10  element count of this array */
    uint32_t element_class;         /* -0x08  class of one element */
    uint32_t pad;                   /* -0x04 */
};

element class

array

element size

0x80809EF3

entry table

0x80809EEE

block table

0x80809EEC

named-tag table

0x80809D02

64-bit name hash table

A field inside a blob that points at an array is 16 bytes:

struct ArrayField {                 /* 16 bytes */
    uint64_t count;                 /* +0x00 */
    int64_t  relative_offset;       /* +0x08  from the address of this field */
};
/* prefix_count = field + 8 + relative_offset
   marker       = prefix_count - 4
   data         = prefix_count + 16 */

The entry table

The entry table has one 16-byte record per entry. The entry index is the record’s position.

struct EntryRecord {                /* 16 bytes */
    uint32_t reference;             /* +0x00  the tag's class; see below */
    uint32_t type_info;             /* +0x04  file type and subtype bits */
    uint64_t block_info;            /* +0x08  where the bytes are; three packed fields */
};

start_block  = block_info & 0x3FFF;                  /* index into the block table */
start_offset = ((block_info >> 14) & 0x3FFF) << 4;   /* byte offset in that block, 16-byte units */
size         = block_info >> 28;                     /* entry size in bytes */

type_info carries a file type in bits 9 to 15 and a subtype in bits 6 to 8. This split is not verified against the client. The other bits are unknown.

In the later layout, reference is the tag’s class. In the 2017 layout it is often a tag that points at a descriptor entry instead. Tags, classes and definitions explains how to resolve it.

The array prefix count can be larger than entry_count. The table is padded. Use the prefix count to find the end of the table, and entry_count to bound the indices you read.

Blocks

Entry bytes live in a block stream. A block holds up to 0x40000 bytes (256 KiB) once decoded. An entry starts at an offset inside one block and runs on through the blocks that follow it by index.

Finding the block table

In the 2017 layout the header gives the offset at +0xD4.

In the later layout no header field gives it. It follows the entry table:

entry_table  = header.entry_table_raw + 96;
entry_slots  = prefix_count(entry_table);        /* ArrayPrefix.count, not header.entry_count */
block_table  = entry_table + entry_slots * 16 + 32;
/* check: the prefix before block_table has marker 0x80809FBD and class 0x80809EEE */

The block record

struct BlockRecord {                /* 48 bytes */
    uint32_t offset;                /* +0x00  file offset of the stored block */
    uint32_t size;                  /* +0x04  stored size in bytes */
    uint16_t patch_id;              /* +0x08  which patch file holds the bytes */
    uint16_t flags;                 /* +0x0A  see below */
    uint8_t  hash[20];              /* +0x0C  Sunrise does not read it; meaning unverified */
    uint8_t  auth_tag[16];          /* +0x20  authentication tag for an encrypted block */
};

flag

meaning

0x1

the block is compressed with Oodle

0x2

the block is encrypted

0x4

the block uses the second of two keys

A block is stored as encrypt(compress(data)). Undo them in reverse order: decrypt first, then decompress.

The block bytes are in the file w64_<family>_<id>_<block.patch_id>.pkg. This can be an older patch than the file whose table you are reading.

The client reads every block of an entry from the patch file its first block names. It never reads the patch_id of the later blocks. The first block’s patch_id must also equal the patch the request is for, or the load fails.

So a tool that writes a patch must put each edited entry wholly into blocks of one patch file. An entry whose blocks are split across patch files reads the wrong bytes.

Encryption

Encrypted blocks use authenticated encryption (AES-GCM). The 16-byte tag in the record must check, or the block is rejected. The keys are not covered here.

You do not need a key to read the header, the entry table, the block table or the misc block. Only block contents are encrypted. So a scan for every tag of one class needs no decryption.

Compression

Compressed blocks use Oodle. The matching Oodle library ships with the game. A block can decode to less than 0x40000 bytes. Oodle accepts only output sizes that are a multiple of 0x4000. A reader that searches for the real size must step by 0x4000, not by halves. Halving skips real sizes and truncates the block silently.

Reading one entry

/* Returns the bytes of entry `index` of package `id`. */
bytes read_entry(uint16_t id, uint32_t index) {
    file   = newest_patch_file(id);            /* highest patch number */
    header = read(file, 0, 0x180);
    if (header.version != 38 || index >= header.entry_count) return FAIL;

    entry_table = (header[0x1A] == 0) ? header.entry_table_2017
                                      : header.entry_table_raw + 96;
    block_table = find_block_table(file, header, entry_table);

    e     = read_entry_record(file, entry_table + index * 16);
    start = e.block_info & 0x3FFF;
    skip  = ((e.block_info >> 14) & 0x3FFF) << 4;
    size  = e.block_info >> 28;

    out   = empty;
    first = read_block_record(file, block_table + start * 48);
    for (b = start; out.length < size; b++) {
        if (b >= header.block_count) return FAIL;
        rec  = read_block_record(file, block_table + b * 48);
        /* every block comes from the patch file the first block names */
        data = read(patch_file(id, first.patch_id), rec.offset, rec.size);
        if (rec.flags & 0x2) data = decrypt(data, key(rec.flags & 0x4), rec.auth_tag);
        if (rec.flags & 0x1) data = oodle_decompress(data);
        from = (b == start) ? skip : 0;
        take = min(data.length - from, size - out.length);
        if (take == 0) return FAIL;
        out.append(data[from .. from + take]);
    }
    return out;
}

Sunrise reads entries in src/middleware/content/packages/reader/package_entry_reader.cpp. It takes each block from the patch file its own record names. That gives the same bytes as long as an entry’s blocks sit in one patch file. It keeps decoded blocks in a cache, because neighboring entries often share a block.

The misc block and named tags

misc_offset at +0xF0 points at the misc block. A value of 0 means the package has none. The block always sits below the entry table, so bound it by the entry table offset. +0xEC is not its size. Whether misc_length always covers the whole block is unverified.

The misc block holds array fields. Their positions are not fixed. To find a table, walk the block’s array fields and keep each one whose prefix has marker 0x80809FBD and the element class you want.

The named-tag table

Element class 0x80809EEC. Each element names one tag.

struct NamedTag {                   /* 16 bytes */
    uint32_t tag;                   /* +0x00 */
    uint32_t class_id;              /* +0x04  the tag's class when the name was written */
    int64_t  name_offset;           /* +0x08  name is at &name_offset + name_offset */
};
/* The name is a zero-terminated string, such as "map:leviathan:root". */

Many names have a <subject>:<role> form, for example <activity>:scenario_client or map:<name>:root.

Rules for using these names:

Only some tags are named. A name missing from every table does not mean the tag is missing. Scan the entry tables by class before you conclude that.
Read names from the newest patch only. An old patch can name an entry index that a later patch reuses for a different tag.
Trust a name only when class_id equals the live class of the tag. In the install, some names point at tags whose live class differs from class_id.

The 64-bit name hash table

Element class 0x80809D02.

struct NameHash64 {                 /* 16 bytes */
    uint64_t name_hash;             /* +0x00  a 64-bit hash of a name */
    uint32_t tag;                   /* +0x08 */
    uint32_t class_id;              /* +0x0C */
};

Some content links by this 64-bit hash, not by a tag. A map root names its bubbles this way. A walk that follows only tag references stops at the map root. Which hash function makes name_hash is unverified.

The client’s own name lookup

The client does not read the named-tag tables to find a tag by name. It uses a separate index, in tags of class 0x80809780. Each of those tags carries a sorted array of:

struct NameIndexEntry {             /* 8 bytes, element class 0x80809252 */
    uint32_t name_hash;             /* FNV-1 of the name; the array is sorted by this */
    uint32_t tag;
};

The lookup binary-searches this array. So a name can be in this index and not in any named-tag table, and the other way round. For example, player_globals is only in the client’s index. Tags, classes and definitions gives the hash.

How an entry gets loaded

The client never loads a whole package. An entry loads in one of three ways:

A dependency record names it. Maps load this way.
A manifest names it. A manifest loads together with its root tag.
Code asks for its handle directly. Item, vendor and string language records load this way.

An entry that none of the three reaches is never requested. A reference to it from loaded content then points at nothing.

Dependency records

A package’s dependency records are load lists. A map loads the entries its records name, not every entry in its package. A record can name entries of any package.

The records sit in the misc block:

misc offset

content

+0x00

u64 header size; the record stream starts at align8(size)

+0x30

root record count; child record indices count from here

+0x40

array of record words

A record word holds three flag bits and a stream offset in its low 29 bits. The record body follows in flag order:

flag

section

bit 31

u16 group count, then per group {u16 package, u16 words} and that many u16 run words

bit 29

u16 count, then u16 child record indices

bit 30

u16 count, zero padding to 8 bytes, then u64 keys

A run word with any of its top three bits set marks word >> 13 entries starting at word & 0x1FFF. A word with none of them set gives a run of word + 1, and the next word gives the start.

239 of the 528 newest packages hold records. Every record table re-encodes to its original bytes.

A patched record can make a new package load, because a record can name any package. How Sunrise merges the records of several mods is on Mods.

Manifests

A manifest is a tag of class 0x80809EF9. The install holds 53,342. Each belongs to one root entry. A root has 0x2000 in the 0xF000 nibble of its entry type field. When the client requests such a root, it also requests the root’s manifest and marks every entry the manifest lists.

manifest offset

content

+0x00

u64 size of the tag

+0x08

u32 the manifest’s own tag

+0x0C

u32 its root tag

+0x10

group array: count, then a relative offset

+0x20

record key array, as in dependency records

A group is 40 bytes, and groups are sorted by package id:

group offset

content

+0

u16 package id; 0x100 to 0xCFF are accepted

+8

u32 bitmap word count, then at +16 the offset of a bitmap; bit n marks entry n

+24

u64 entry count, then at +32 the offset of a u16 entry list

An offset is relative to the field that holds it, and 0 means none. Every installed manifest lists both its root tag and its own tag.

The root-to-manifest map is not in the misc block. It is in the package metadata that header +0x110 locates: an array of {u32 root tag, u32 manifest tag} rows, element class 0x80809A13, sorted by root tag.

The manifests list 2,784,122 tags in all. For example, the manifest of investment_globals lists all 4,495 string containers, the string bank index, the investment root blob and the item table. No manifest lists item records, vendor records or string language data. Those load on request.

Streamed texture entries

An entry whose type field has 0x6000 in the 0xF000 nibble is a streamed texture buffer. Content may reference it, but it never belongs in a manifest. No installed manifest lists one: 15,478 such entries exist and none appears in any of the 1,191,098 manifest groups. The loader reads a streamed entry’s owner record without a null check, so listing one as resident is a crash risk. That a listed streamed entry reaches that read is unverified.

What the families hold

The largest families by file count are sandbox, environments, activities and audio. Others are per destination (edz, dreaming_city) or per PvP map (pvp_*).

A family name suggests what it holds, but the name is not proof. Find content by tag and class, not by family. For item and character data, see Items, characters and inventory. For activity content, see Activities and destinations.

The client carries only the client half of activity logic. Tags such as <activity>:scenario_client ship. The host-side scenario tags do not exist in the install. A server has to supply that logic.

Packages

Files and patches

Picking the newest patch

The header

The signature and the footer

What the client checks at load

Two header layouts

Typed arrays

The entry table

Blocks

Finding the block table

The block record

Encryption

Compression

Reading one entry

The misc block and named tags

The named-tag table

The 64-bit name hash table

The client’s own name lookup

How an entry gets loaded

Dependency records

Manifests

Streamed texture entries

What the families hold