append mode
 - No blocks are deleted except the fsinfo and conres.  Thus unexpired
 file systems will continue to work.  It's more like an archive of all
 history than a snapshot of the latest bits.

update mode
 - Remove files that no longer exist.  Add new files.  This is harder to
 implement because sfsrodb will have to parse the old directories, etc.

merge mode
 - Subordinate an entire directory tree as a sub directory tree.  Add
 files even if from a different directory hierarchy.
 E.g., merge /home/fubob/src and /home/kaashoek/src into one sfsro database.
 Conflicts can result

delete mode
 - Remove specific files from the archive without regenerating the database
 entirely.  Takes a list of files to delete.

compare mode
 - Compare an sfsro database against a living file system hierarchy. 
 Not very useful -- could just mount the database then compare.

concatenate
 - Concatenate two or more sfsro databases into one. 

write quick how-to document for sfsro:
  sfscd -l flag
  --with-db3

make sfsrodb an API so others can add a nice GUI

modify sfsrosd to use sfsrosd_config files of the form:

  Export /foo/bar/baz.rodb  # Serving original content
  Export port@hostname      # Replicating others' content
  Replica /foo2/bar.rodb    # Location to store cached replicated content

  Consider letting replicas replicate on any port?

sfsrodb incremental update mode.  Reduce wasted time
on recreating a database.  Allow incremental additions:

* Traverse  and unmarhsal the existing database and the original
   directory structure simultaneously.
* If the mtime differs, rehash and add.  Otherwise just return
  from recursive call with the old SFSRO fh
* Always rehash the directories.
* Mark things for deletion.
* Update fsinfo last.
* THen delete marked items.

For replicas, store replicated content in a disk cache
specified by the sfsrosd_config file.

make sfsrodb give some kind of status indication to show that
progress is being made during the creation process.  ncurses?
percent finished?  spinning \ | / ?  with ncurses, show current traversal.
report % files remaining, % bytes remaining, estimated time.
try hashing micobenchmark to estimate time.

client might not properly reconnect to a server.  Verify
that caches are not unnecessarily flushed.

If an sfsrodb is control-C'd, make it recoverable.  Don't re-hash
if done.  So make sfsrodb log-based or transaction-like.

Handle READDIRPLUS
> Date: Thu, 03 Jan 2002 11:06:03 -0500
> From: Kevin Fu <fubob@MIT.EDU>
>
> I'm having trouble getting my openbsd machine to generate READDIRPLUS
> RPCs to SFSRO.  The OS likes to issue READDIR and LOOKUPs instead.
> What's the easiest way to make the OS generate READDIRPLUS calls?
It's a separate mount flag, since READDIRPLUS is useless under the
BSDs (except for debugging your own code).  To get this, just set the
NMOPT_RDPLUS flag in the mntflags field of the reply to the
SFSCDPROC_MOUNT RPC.


Respect opaque directories and easily specify which are opaque.

fragment when NFS block size is 8KB but the SFSRO is 16KB?

Test pulldb, make sure easy to use.

Allow reconnection to any SFSRO server without losing cache contents. 

What if the sleepycat database is way too big.  Expect a 2x bloat
because of the btree and misaligned pages.  8KB might take 2 pages.
Sleepycat only works on files.  How to split database in pieces.
server multiple (and name) databases from one server.

fix sfscd such that the -l flag is not necessary for sfsrocd.

In sfsrodb, perhaps add an option to ignore files that result in
permission denied while creating databases.  E.g.:
 if (access (fqfn.cstr (), R_OK) >= 0) {
  recurse_path (fqfn, &fh);
 .. 
 } else {
  // Could not read the file, so skip for convenience.
  // This could be a race condition, but since the sys
  // admin intended to export the file, this should 
  // at worst not export a file that should be exported
  warnx << path << "/" << filename << " not readable. Skipping\n";
 }


partial replication.  What should a server do if it does not have
certain content replicated?  Forward to another replica?

policies.  if a user chooses to serve a replica of debian,
can the admin rate limit it?  fairness?

treat SFSRO server as a block I/O.  That is, getdata should  take
two arguments: a disk device and a block address.  These correspond
to the self-certifying path and SFSRO fh respectively.

hit/miss rate stats written to file?  Stats on most popular file handles?
Similar to web log?  Referrer log? :-)