Upload
charles-thornton
View
223
Download
6
Tags:
Embed Size (px)
Citation preview
Feeding the MonsterAdvanced Data Packaging for
Consoles
Bruno ChampouxNicolas Fleury
Outline
Next-Generation Loading Problems LIP: A Loading Solution Packaging C++ Objects Demo: LIP Data Viewer Questions
Some things never change…
RAM
Optical Disc
...since 1992!...since 1992!
Next-Gen Loading Problem
Processing power up by 10-40X Memory size up by 8-16X Optical drive performance up by 1-4X
Next-Gen Loading Problem
Xbox 360 12X dual-layer DVD drive Outer edge speed: 15 MB/s Average seek: 115 ms
PlayStation 3 Blu-ray performance still unknown 1.5X is the most likely choice CAV drive should give 6 to 16 MB/s Average seek time might be worse than
DVD
Next-Gen Loading Problem
Memory Size
Maximum Bandwidth
Time to fill
PS2 32 MB 4.4 MB/s 7.3 s
PS3 192 MB* 16 MB/s 12 s
Xbox 64 MB 6.9 MB/s 9.3 s
Xbox 360 480 MB** 15 MB/s 32 s
Next-Gen Loading Problem
In order to feed the next-gen data needs, loading will need to be more frequent
Hard drives are optional for PS3 and Xbox 360
Optical drive performance does not scale with the memory/CPU power increase
Conclusion: Loading performance must be optimal; any
processing other than raw disc transfers must be eliminated
Did I Hear “Loading Screen”? Disruptive Boring as hell Non-skippable cutscenes are not
better! Conclusion:
Loading screens must not survive the current generation
Background Loading
Game assets are loaded during gameplay
Player immersion is preserved Solution:
Use blocking I/O in a thread or another processor
Background Loading
Requirements: Cannot be much slower than a loading
screen Must have low CPU overhead Must not block other streams
Conclusion: Once again, loading performance must
be optimal; any processing other than raw disc transfers must be eliminated
Proposing A Solution
Requirements for a next-generation packaging and loading solution: Large amounts of assets must be
loaded at speeds nearing the hardware transfer limit
Background loading must be possible at little CPU cost
Data assets must be streamed in and phased out without causing memory fragmentation
Understanding Loading Times Freeing memory space
Unloading Defragmenting
Seek time Read time Allocations Parsing Relocation (pointers, hash ID lookups) Registration (e.g. physics system)
Reducing Loading Times
Always load compressed files Using N:1 compression will load N times
faster Double-buffering hides decompression
time Plenty of processing power available for
decompression on next-gen consoles
Reducing Loading Times
Compression algorithm choice Favor incremental approach Use an algorithm based on bytes, not
bits Lempel-Ziv family LZO
Reducing Loading Times
Take advantage of spatial and game flow coherence Batch related data together in one file
to save seek time Place related files next to each other on
the disc to minimize seek time
Reducing Loading Times
Take advantage of optical disc features Store frequently accessed data in the
outer section of the disc Store music streams in the middle
(prevents full seek) Store single-use data near the center
(videos, cutscenes, engine executable) Beware of layer switching (0.1 seconds
penalty)
Reducing Loading Times
Use the “flyweight” design pattern Geometry instancing Animation sharing
Favor procedural techniques Parametric surfaces Textures (fire, smoke, water)
Reducing Loading Times
Always prepare data offline Eliminate text or intermediate format
parsing in the engine Engine time spent converting or
interpreting data is wasted Load native hardware and
middleware formats Load C++ objects directly
Why Load C++ Objects?
More natural way to work with data Removes any need for parsing or
interpreting assets Creation is inexpensive
Pointer relocation Hash ID conversion Object registration
Loading C++ Objects
Requires a very smart packaging system Member pointers Virtual tables Base classes Alignment issues Endianness
Loading Non-C++ Objects
Must be in a format that is ready to use after being read to memory Texture/normal maps Havok structures Audio Script bytecode
Pretty straightforward
Load-In-Place (LIP)
Our solution for packaging and loading game assets
Framework for defining, storing and loading native C++ objects
Dynamic Storage: a self-defragmenting game asset container
Level Editor
LIPGenerator
Maya Exporter
LIP Packaging
Game Assets
Engine
Dynamic Storage
LIP Loading
Load-In-Place: “LIP Item”
1 LIP item 1 game asset 1 LIP item unique hash ID (64-bit)
32 bits for the type ID and properties 32 bits for the hashed asset name (CRC-
32) The smallest unit of data that can be
queried moved by defragmentation unloaded
Supports both C++ objects and binary blocks
Examples of LIP Items
Joint Animation Character Model Environment Model Section Collision Floor Section Game Object (hero, enemy, trigger, etc.) Script Particle Emitter Texture
C++-Based LIP Items
Can be made of any number of C++ objects and arrays
On the disc, all internal pointers are kept relative to the LIP item block Pointer relocation starts with a
placement new on a “relocation constructor”
Internal pointers are relocated automatically through “constructor chaining”
Placement “new” Operator
Syntax new(<address>) <type>;
Calls the constructor but does not allocate memory
Initializes the virtual table Called once for each LIP item on the
main class relocation constructor
Relocation Constructors
Required by all classes and structures that can get loaded by the LIP framework contain members that require relocation
3 constructors Loading relocation constructor Moving relocation constructor
(defragmentation) Dynamic constructor (optional, can be dummy)
No default constructor!
Object Members Relocation Internal pointer
Must point within the LIP item block Converted into absolute pointer
External reference (LIP items only) Stored as a LIP item hash ID Converted into a pointer in the global
asset table entry that points to the referenced LIP item
LIP framework provides wrapper classes with appropriate constructors for all pointer types
Relocation Exampleclass GameObject {public:GameObject(const LoadContext& ctx);GameObject(const MoveContext& ctx);GameObject(HASHID id, Script* pScript);
protected:lip::RelocPtr<Transfo> mpLocation;lip::LipItemPtr<Script> mpScript;
};
Relocation Example (cont’d)
GameObject::GameObject(const LoadContext& ctx) : mpLocation(ctx), mpScript(ctx) {}
GameObject::GameObject(const MoveContext& ctx) : mpLocation(ctx), mpScript(ctx) {}
GameObject::GameObject(HASHID id, Script* pScript) : mpLocation(new Transfo), mpScript(pScript) { SetHashId(id); }
Relocation Example (cont’d)template<typename LipItemT>
void PlacementNew(
lip::LoadContext& loadCtx)
{
new(loadCtx.pvBaseAddr)
LipItemT(loadCtx);
}
loadCtx.pvBaseAddr = pvLoadMemory;
PlacementNew<GameObject>(loadCtx);
Relocation Example (cont’d)
GameObject
Placement new
RelocPtr<Transfo> LipItemPtr<Script>
Transfo Script
Placement new hash ID lookup
Constructors
Constructors
Load-In-Place: “Load Unit” Group of LIP items The smallest unit of data that can be
loaded 1 load unit 1 load command
Number of files is minimized 1 language-independent file
Models, animations, scripts, environments, …
N language-dependent files Fonts, in-game text, some textures, audio, …
Load unit files are compressed
Load Unit Table
Each LIP item has an entry in the table Hash ID Offset to LIP Item
LIP itemsTable
Dynamic Storage
Loading process Load unit files are read and decompressed
to available storage memory Load unit table offsets are relocated Load unit table entries are merged in the
global asset table A placement new is called for each LIP item Some LIP item types may require a second
initialization pass (e.g. registration)
Dynamic Storage
Unloading process Each LIP item can be removed individually All LIP items of a load unit can be
removed together Destructors are called on C++ LIP items Dynamic storage algorithm will
defragment the new holes later Locking
LIP items can be locked Locked items cannot be moved or
unloaded
Platform-Specific Issues GameCube
Special ARAM load unit files Animations Collision floors
Small disc compression
Xbox/Xbox 360 Special LIP items for DirectX buffers
Vertex, index and texture buffers 4KB-aligned LIP items (binary blocks) Buffer headers in separate LIP items (C++
objects)
Load-In-Place: Other Uses
Network-based asset editing LIP items can be transferred from and
to our level editor during gameplay Changes in asset sizes do not matter
Used by Maya exporters to store our intermediate art assets LIP is much more efficient than parsing
XML!
Packaging C++ Objects
Nicolas Fleury
Our Previous Python-Based System class MyClass(LipObject):
x = Member(UInt32) y = Member(UInt32, default=1) p = Member(makeArrayType( makePtrType(SomeClass)))
Cool Things with this System Not too complex to implement. Python is easy to use. Introspection support. A lot of freedom in corresponding
C++ classes.
Problems with this System
Python and C++ structures must be synchronized.
Exporters must be written, at least partly, in Python.
Validations limited (unless you parse C++ code).
We just invented a Python/C++ hybrid.
C++-based system
class MyClass : public MyBaseCls { ... LIP_DECLARE_REGISTER(MyClass); uint32 x;};
// In .cppLIP_DEFINE_REGISTER(MyClass) { LIP_REGISTER_BASE_CLASS(MyBaseCls); LIP_REGISTER_MEMBER(x);}
Consequences
Exporters are now written in C++. Class content written twice, but
synchronization fully validated. Dummy engine .DLL must be compiled (not a
working engine, provides only reflection/introspection).
Need a good build system. We just added reflection/introspection to C+
+.
Member Registration Information Name Offset Type Special flags (exposed in level
editor, etc.)
(Non Empty) Base Class Registration Information Name Type Offset; calculated with:
(size_t)(BaseClassType*)(SubClassType*)1 - 1
Member Type Deduction
In IntrospectorBase class:
template < typename TypeT, typename MemberT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr));
Member Type Deduction (Arrays) In IntrospectorBase class:
template < typename TypeT, typename MemberTypeT, int sizeT>void RegisterMember( const char* name, MemberT(TypeT::*memberPtr)[sizeT]);
Needed Information in Tools Every class base class (to write their
members too). Every class member. Every base class offset (to detect
missing base class registration). Every member name, size, type and
special flags. For every type, the necessary
alignment and if it is polymorphic.
Introspection Classes (Members)
MemberInfoBase
MemberInfoTypedBase
TypeT
MemberInfo
TypeT, MemberTypeT
ArrayMemberInfo
TypeT, MemberTypeT, sizeT:int
IntrospectorBase
Introspector
TypeT
1 *
Introspection Classes (Base Classes)
BaseClassInfoBase
BaseClassTypedInfoBase
SubClassTypeT
BaseClassInfo
SubClassTypeT, BaseClassTypeT
IntrospectorBase
Introspector
TypeT
1
*
Result: Member Introspection Able to know all types and their
members. Can be used for both writing and
reading binary data. Same class used in tools to fill the
data as in engine.
LipViewer
Data of any platform, endianness, pointer size (binary files have a header with platform id).
Both for engine data and tools binary formats.
Hexadecimal viewer integration, edition support.
Excellent learning and debugging tool.
LipViewer Demo
Restrictions for Simplification of Implementation Polymorphic types must begin with a
vtable pointer (their first non-empty base class must be polymorphic).
Can’t inherit twice from same class indirectly (or offset trick doesn’t work).
No virtual base classes. All padding is explicit.
Explicit Padding
class MyClass {...LIP_PADDING(mP1, LIP_PS3(12));uint16 mSomeMember;lip::Padding<4> mP2;uint32 mSomeOtherMember;LIP_PADDING(mP3,LIP_PC(4) LIP_PS3(8));...
};
Particular Things to Handle
Endianness. 64 bits pointers (no more!). VTable padding. Type alignment.
VTable Padding
__declspec(align(16)) class Matrix {…}; class MyClass {
uint32 x, y, z; Matrix m;};
32 bytes on PS3, 48 bytes on PC.
vtable
xyz
m
vtablexyz
m
Automatic Versioning
Create a huge string with member names/types and member names/types of pointed classes.
In the case of polymorphic pointers, all sub-types must also be included.
Hash the huge string. Can be integrated in tools dependency
tree mechanism.
Needed Information in Engine A hash map of objects to do the
placement new of the appropriate type.
Smart pointers/arrays handle the rest.
Type Ids VTable pointers are replaced by a type id. LIP_DECLARE_TYPE_ID(MyClass, id) in .hpp.
Defines a compile-time mechanism to get id. Declares a global object.
LIP_DEFINE_TYPE_ID(MyClass) in .cpp. Defines the global object. Its constructor adds
itself as a hash node to a hash map. This object class is templated to make operations with the good type (example: placement new).
Hash Map Overview
+PolymorphicPlacementNew()+PolymorphicMovePlacementNew()+GetTypeSize()+StreamObject()+AddTypeConstructor()
TypeManager
+PlacementNew()+MovePlacementNew()+GetTypeSize()+StreamObject()
+mVtableId+mpNext : TypeConstructorBase
TypeConstructorBase
+PlacementNew()+MovePlacementNew()+GetTypeSize()+StreamObject()
TypeConstructorBase
TypeT
1 *
Pointers
Normal pointers like T* must be set to 0 when exporting an object.
For relocated pointers, smart pointer classes must be used.
Different types of smart pointers/arrays: Ownership of pointed data? Relocation of pointed data?
Smart Members
Classes can derive from a lip::SmartMember class to implement a custom writing/reading in tools.
Class only used as a tag, it doesn’t have any virtual function.
Classes deriving from lip::SmartMember are expected to implement a compile-time interface.
Useful for smart pointers (normal pointers cannot be load-in-placed).
Smart Members: Full Control Over Writing and Reading.class MySmartArray : lip::SmartMember {
public:
void Write(lip::LipWriter&) const;
void WriteExternalData( lip::LipWriter&)const;
void Stream(lip::LipReader&);
void StreamExternalData( lip::LipReader&);
};
WeakRelocPtr<Type>
mpPtr
Not owned pointed data
Parent class
RelocPtr<Type>
mpPtr
Parent class
Relocation assumes pointed data is not of a sub-type and does directly a placement new of Type.
RelocPolymorphicPtr<Type>
mpPtr
Parent class
Relocation looks in the hash map to do placement new of the good type (involves a search and a virtual function call).
RelocFixedArray<Type, size>
[1]
[2]
[0]
[3]
Parent class
WeakRelocArray
mpPtr
muiCount
Parent class
Owned pointed array (not needing relocation)
RelocArray<Type>
mpPtr
muiCount
Parent class
Owned pointed array (with relocation)
RelocPolymorphicArray<Type>
mppPtr
muiCount
Parent class
Owned array of pointers
Owned array of objects (can be of different sub-types)
RelocWeakPtrArray<Type>
mppPtr
muiCount
Owned array of pointers
Not owned pointed objects (can be of different sub-types)
Parent class
BinaryBlockPtr<alignment=4>
mpPtr
Parent class
Owned pointed data
Enums
Concept of exclusivity group masks to regroup values in mask in a radio button group in GUI.
LIP_REGISTER_ENUM(MyEnum) { LIP_DEFINE_ENUM_VALUE(eNO); LIP_DEFINE_ENUM_VALUE(eYES);}
Other Solutions
Parse debug info files. Compile as C++/CLI. Parse source code.
Questions?