Key to paper TrueTime exposes clock uncertainty, if the uncertainty is large, Spanner slows down to wait out the uncertainty. TS reflect serialization order, external consistency, or linearizability. Spanner zones Zone zonemaster assigns data to spanservers (1000) location proxies used by clients to locate spanservers Spanserver responsible for 1000 tablets Tablet (key: string, timestamp: int64) -> string more like multiversion kv store state store in a set of B-tree-like files and a write ahead log in DFS called colossus one Paxos state machine per tablet Paxos long time leader leases (10 sec) implements a consistently replicated bag of mappings writes initiates paxos protocol at leader reads access state directly from any replica set of replicas called Paxos group Logs: every ‘Paxos write’ goes to tablet log and to Paxos logic Replica leader every leader replica implements a lock table contains the state for 2-phase locking maps ranges of keys to lock states leaders are long lived to maintain this lock table performs poorly under optimistic concurrency control replicas in paxos group called participant slaves implements a transaction manager implements a participant leader used between other paxos groups for 2-phase commit Participant/Coordinator one participant leader (leader to a paxos group) is chosen as coordinator leader, other replica leader called coordinator slaves state of transaction manager store in underlying paxos group Directory and placement directory set of contiguous keys with common prefix unit of data placement -> same replica config unit of data movement between paxos groups paxos group/tablet contains multiple directories not necessarily lex.