Chefs:
 callback argument in setrootfh since it might not complete?

 check that READ returns EOF properly when the read is truncated.

Proxy re-encryption:

 RPC server
 Proxy re-enc

tool:
 Create admin key pair, user key pair
 Create delegation from admin PK to user PK (takes delegate PK, admin SK)

sfsrodb:
 Seal lockbox with proxy crypto to the admin's PK
  - Input admin PK

sfsrocd:
 Open lockbox with proxy crypto
  - Proxy re-encrypt with delegation key from admin PK -> user PK
  - Decrypt lockbox with user PK
  - input user SK, PK and delegation key

Kevin stuff:
r -d ~/src/proxy -s ~/src/build/sfs1-build/runinplace/sfs_host_key -o ~/src/build/sfs1-build/runinplace/test.rodb -g ~/src/build/sfs1-build/runinplace/1 -v -h 127.0.0.1
 implement some deterministic public key system. e.g., RSA without padding
 catch a USR1 signal to re-read the config file.
 suggestion from chris: why are people afraid of letting
  out ciphertext?  Figure out what is their threat model.
  What do they fear?  Brittleness?


Updating database

I. Generate an update list

Generate an "update list" of new pathnames and pathnames whose inodes
have changed.  Traverse the origin file system.  If an inode's mtime
from the origin file system is later than the start time in the SFSRO
database fsinfo, include the filepath in the update list.  Add all
parent directories of filepaths in the update list.  (e.g., if you see
/foo/bar/baz, also add /foo/bar, /foo, and /.)

This will later help us in pruning the traversal and recomputation of
the SFSRO database.  This will speed up the offline server computation
on small changes to databases containing deep directory structures.


II. Write updates to the database.

The idea is to traverse the current SFSRO database while
simultaneously traversing the updated origin file system.  Add an
extra sfs_hash argument to each traversal routine (recurse_path,
store_file, store_directory_block, store_inode, store_file_block).
During an update, the caller will set this argument to the SFSRO
handle of the original in-database structure.  This way, the traversal
routines can query for the original SFSRO structures for comparison.

1. Read fsinfo from SFSRO database.  Call recurse_path with
   the extra argument = fsinfo.rootfh

3. Re-recurse the origin file system.  Simultaneously recurse the
   public database.

 a. If the current pathname does not appear on the update
 list, do not modify the SFSRO database.  Simply
 set the pathname's SFSRO handle to the old handle.  Do not
 recurse further.

 b. If the current pathname appears on the update list, 
 we will at least have to recompute and store an updated inode.
 We now must decide what data blocks require recomputation.
 For each call to store_file_block (or store_directory_block), provide
 the old SFSRO handle (e.g., inode->direct[0]).  This allows the
 store_file_block routine to check if the plaintext has changed.
 If the plaintext changed, compute a new file key, store it in the lockbox,
 etc.
----
Database update:

Terminology:

origin file system = the directory where SFSRO originally obtained
			all the file data for the database

SFSRO database = the database containing mappings 
		{SFSRO handle} -> {SFSRO data block}

public SFSRO database = the historical SFSRO database with no access
		control

private SFSRO database = an SFSRO database whose contents are
		sealed with encryption.

auxilliary database = a database containing extra information to aid
		in updating a private SFSRO database.

-----------
Possible way to update group keys in SFSRO using the agent:

 Look at "agentcb_prog", in particular, these functions:
		sfsagent_auth_res
		AGENTCB_AUTHINIT (sfsagent_authinit_arg) = 1;

		sfsagent_auth_res
		AGENTCB_AUTHMORE (sfsagent_authmore_arg) = 2;

 It might get ugly: sfsrocd would call sfscd which would call the agent.
----
Here is more efficient code to marhsal with iovec:

  xdrsuio x;
  if (!xdr_sfs_hostinfo (&x, const_cast<sfs_hostinfo *> (&info))
      || !xdr_sfs_hostinfo (&x, const_cast<sfs_hostinfo *> (&info))) {
    warn << "sfs_mkhostid (" << info.hostname << ", "
	 << hexdump (id->base (), id->size ()) << "): XDR failed!\n";
    bzero (id->base (), id->size ());
    return false;
  }
  sha1_hashv (id->base (), x.iov (), x.iovcnt ());

Add fsstat data to the database.  Maybe
inside a special "fsstat" file handle.

High priority fixes:

 For incremental:
  Move the modification time into the directory so that incremental
  updates touch less of the database.  Embed these in each
  directory entry.

  Incremental updates should take an old database and any
  new file system.  Remove fhdb data before adding?  Must
  new data share data with old?  How do we know what data is
  old and can be removed?

  Incremental in that running time is proportional to
  size of update, not size of database?

  Get rid of "used" -- synthesize with size/blocksize.  The
  used concept does not have any meaning in our model, especially
  since like blocks can share the same space unknowningly.


 Proving non-existence of a file handle.

 The server will close all connections when updating a database.
 After the client's kernel asks for NFS stuff, the client attempts to
 reconnect with a call to GETFSINFO.  The worst an adversarial server
 can do is cause this reconnect to hang.  If GETFSINFO returns, the
 client verifies the signature. If the signature is good, the client
 returns from the callback.  If the signature is bad, the
 client keeps retrying (hangs) until a good signature results.
 We might allow a timeout with NMOPT_SOFT.  If the IV has changed,
 flush all caches because all the file handles will 

 Note, we serialize the call to GETFSINFO on each client with a lock.
 There should never be multiple outstanding GETFSINFO requets.

 If client calls GETDATA, but the server replies, "void, no such fh"
 we ask the server to prove that the fh does not exist.  The idea
 is that the client will do a binary search by iteratively calling
 GETDATA on a special tree of all the file handles.  The server
 maintains a special set of file handles in its database.  After
 the SFSRO database adds the data from the original file server, the
 sfsrodb program creates a lexically ordered list of all the file handles.
 Then it divides the list into chunks of approximately 256 
 file handles.  Each chunk is hashed to obtain a new file handle.
 The chunk is added to the SFSRO database.  We now group
 consecutive file handles of chunks.  We hash, add to the database, etc.
 We continue until we have only enough file handles to fill one
 chunk.  We hash this.  This hash is the fsinfo->fhdb.
 

 The protocol for a server to prove the non-existence of a file handle
 is as follows:

  The client calls GETDATA on the fsinfo->fhdb.  
  This returns an inner node of the n-ary tree.  Hash the node
  to verify the fhdb file handle matches.  If no match, then
  an adversary is playing with the client.  I guess we hang?
  Otherwise, the client compares the
  value of the disputed file handle to the keys in the node.  The
  client then follows the appropriate branch to a child node.  This
  process continues until the client reaches a leaf node.  A
  leaf node contains one of the "chunks" of lexigraphically ordered
  file handles.  

  The client verifies that two adjacent file handles in the chunk
  sandwich around the value of the disputed file handle.  Note,
  special case when the value of the disputed file handle
  falls between two different chunks.  Maybe we should duplicate
  the file handles on chunk boundaries so that we do not need
  to fetch multiple chunks?

  If the verification is good, the client returns a stale file handle
  error.  Otherwise...not sure.  Adverserial.

  Note, we shouldn't actually returns a stale file handle.  This should
  be transparent to the NFS client in the client machine's kernel.
  To fix this, we need a mapping from evolving SFSRO file handles to 
  static NFS file handles.  That is, NFS file handles should never
  change.  But we currently set the NFS file handle as the SFSRO
  file handle.  Doh.
  


Comments:

  We want to keep the server simple.  The server does NOT have a
  "Prove non-existence" RPC which takes as arugments the
  disputed file handle and returns a list of sibling hashes to
  the path of the disputed handle.  Instead we have the client perform the
  n-ary search.  This also makes it harder to mount a denial of
  service attack against the server.  Had the server returned
  a complete "proof," it would have to perform O(log n) comparisons
  where n is the number of file handles in the original database.
  By making the client do this, we prevent malicious clients from
  hosing the server with bogus "prove this doesn't exist" requests.


	Incremental updates.  Time them.

	Performance test comparing SFSRO to a GET of a web page.

	Performance test of fetching a page deep beneath
	a directory hierarchy.  Web will resolve names on the server
	in one call.  SFSRO will incrementally resolve names
	in several calls (one per directory) to the server.

done	remove code to special case the zero blocks.  make
	sure compression still works when many 8KB blocks of
	zeros though.

	fix sfsrocd code.  detect when the server closes tcp
	connection.. close, but do not flush cache.

	global structure mapping previously or currently mounted
	hostids to most durations,

	retry or reconnect if file handle does not match hash of data
	should check that not rolled back (start time).

	use duration time to flush caches if fsinfo different.

	pin down the root inode in the cache

done	env vars to disable caching, verification of hashse

	Limit directory entry blocks to 8KB.
	
	Selectively hide directory enties
	Set opaque directory type.  Allow client not to fetch
        more directory entries if directory marked as opaque.
	This improves the availability of central servers like
	Verisign.  if every client were to download ALL the self-
	certifying pathnames, the verisign server would get hosed.
	With the opaque option, the client will only download
	the directory structure.  Calls to ACCESS and READDIR must fail
	or return nothing but whats in the cache.  Calls to LOOKUP
	will enter directory info into the cache.

done	Add statistics to sfsrocd to determine effectiveness
	of cache (hit/miss ratio).  Add buffer cache.


Low priority fixes:

	Use a dummy root NFS file handle on the client because we
	cannot change this value after mounting.
	use the rootfh elsewhere.  because We cannot unmount/remount
	when the rootfh changes.

	Allow argument to specify update of database.  maintain old IV?
	
	Get rid of nlink in inode except for directories.
	The nlink from the original file system might be
	bogus w.r.t. the ROFS because hard links could point
	to inodes shared by other files on the orignal
	file system.  But we might not add an entire partition
	to the ROFS.  So other hard links shouldn't get counted.

	nlink is not useful since we perform no deletions or rmdirs.
	however, many OS's will fail if a directory's nlink is <= 2.
	Then it assumes the directory is empty.  So we might have
	to special case nlinks for directories by counting
	the number of directory entries.
	
	Create a special mapping from the internal file handle rep
	to the NFS3 file handle rep.  This way the SFSRO file handles
	can change (say, directory update), while the application
	level will see the same NFS FH.

	Allow targets of CAs to cache partial paths so that if
	verisign goes down, amazon and still prove authenticity.
	But this doesn't work because the hostid includes the 
	hostname.

	To avoid stale directory file handles, add a new
	tree of updated directory file handles to prove 
	whether a directory exists.

	we do everything in an integrated way.  verisign
	could issue new serverID certs every morning, but
	this is not a clean interface.  We provide
	system integrity in addition to file integrity.

Necessary measurements:

\item A single sfsrosd server can support many more short connections
than sfsrwsd. This is important for certificate authorities. 

	What is bottleneck in server performance? B-tree?  ARPC
	library?

	Test use as a certificate authority: make short connections.
	Compare CPS.  The SFSRO client simulator should get the
	root inode, readlink "sfsro.fooworld.org",   
	connect to sfsro.fooworld.org,.  Compare to the run time
	of connecting to https://snafu.fooworld.org with openssl
	and verifying the server certificate.  
	Make sure SSL cert and SFS sfs_host_key have similar key lengths.

	Test connection setup time

	Benchmark lookup of ..



A single sfsrosd server can support many more long downloads than
sfsrwsd. This is important for distributing large software packages.
Compare throughput of large reads from sfsrosd and sfsrwsd.  sfsrosd
should do better because encryption is disabled.

	Test connection throughput by:
		installing various software packages
		large compile (SFS)
		searching files for strings that do not exist
		sequential reads of large file
		random reads of large file
		followed by sequential reads of the same file (test cache)
		Total the run times.

		Also compile a large software package like the OpenBSD kernel?

	Test the same things when we disable the inode and/or block
	caches on the client.

	Test the same things when we disable FSINFO signature
	verification and SFSRO file handle verification.

	


Replicating sfsrosd servers improves throughput linearly.  (sfsrwsd
much harder to replicate; private key must be replicated too.)

	Test throughput of short connections.

	Test round-robin DNS.

Individual client performance is hardly effected by pushing work to
the client.  Microbenchmarks to test read performance. How expensive
is recomputing hashes? Test some a few large benchmarks to get
application numbers (large compile).


	??

Database can be updated relatively frequent.  This is important to
enable certificate authorities to add new certificates efficiently.
Measure how long it takes to create database.  (Incremental update
performance?)

	Measure how long it takes to create database and how fast to
	update.

Storage overhead over sfsrodb is acceptable. Database size is
acceptable (should be close to the file system size). Test how
efficient database is representing file system data.  Cool side thing:
how much compression do we get because blocks with the same data will
be stored once in the database.

	Measure database size with btree/sleepycat

	Measure how much compression do we get because blocks with the
	same data will be stored once in the database.






Error conditions to worry about:
* Callbacks in constructors?  What if we get a NFS call while
	we're still getting the root inode?  Might be error.
* What if there are two simultaneous requests for the same
  file handle, but they both manage to add cache entries or such?