Add progress bar while downloading.
add ftp upload builtin
add thumbdrive security.
consider moving sender.ser to distrib dir.
userid rxg pw mindprod
Plan:
GUAI-RMS.org
smear.biz
USERID: jas
PASSWORD: oxford
We have the old contents of the jars from a serialised ArrayList.
We span the tree and collect the files as they are now.
We can compare these two trees to find:
1. deadwood
2. new or updated files with recent dates
3. new files with old dates.
4. old files already in our zips.
We can assign the new files to new zips.
Now could either consolidate pairs, groups or just recreate entirely.
We want to avoid changing old jars, to avoid uploads, even if only a few will download them.
The upload problem.  We want to make sure all files are fully uploaded before there are any
references to them. We want to make sure any references to files are removed before the files
themselves are removed.
Zips go through stages of decay:
0. inuse
1. unused
2. shrunk to 0 bytes.
3. deleted entirely.
Can't progress to next stage without an upload.
Uploads, first upload all new zip files and a manifest file to them.
Once that is complete we update the manifest pointer file to point to the new manifest file.
The old manifest file goes through the same process,
unused
shrunk to 0 bytes
deleted entirely.
a manifest file
Mxxxx.ser
contains a serialised arraylist
of zd objects
each contain zip# last file in zip including timestamp in internal interned format.
sorted by date/file.
Only active zips included.
There is only one manifest file at a time active, though several unused ones may survive
in various states of decay.
How do we decide to retire old files?
We look at them in order.
Let's we track total uncompressed and deadwood in each zip.
From point a, we look seeing if subsequent zips could be collapsed.
If say we found a b c could be collapsed, we create a new zip,
and move all files into it, treating them like new files.  a b c then get put on the dying list.
Then we carry on at d.  Normally a won't collapse with anything, so you carry on with b.
You don't collapse recent files.  You collapse if the newly created file would be no bigger than
target + 50%.
Then you check aggregate files for being over the deadwood limit.  If they are, you create
all the old zips from scratch.
make sure filter out files too big.
Persistence:
What do we need to remember between incarnations:
1. highest zip number.
2. what files are currently in zips.
3. livewood, deadwood.
All the client needs to know from the server is the current list of active zip numbers.
This won't work. Since some new zip numbers could contain nothing but old files.
It must track the highest zip number it has successfully processed. It gets the file names and
dates from the zip files. It may unzip deadwood, but it will soon be replaced.
The client alwasy unzips the zip files in ascending order.
Ideally the date range of every zip file would be disjoint, but because of late arrivals with old dates,
there can be some overlap.  These late arrivals have to go in new zips.  We never change the contents
of a zip after creating it, only retire it.
Idea:
ZD contains only last elt : FD with filename and timestamp.
late arrivals leave true date on FD entry, but in ZD timestamp is treated
as cutoff, so that this zip will be considered more recent than it otherwise would be.
No need to store first elt.
To recycle deadwood in old files, set all files in that zip to zip # -1 rather than 0.
Then in clump, deadwood, latecomers and new files will all automatically get new zips
in one pass.
for merge, must sort by timestamp.
for clump, must sort by adjusted timestamp.
   Once we have built a zip, we never update it.  When it gets too full of
   deadwood, we may retire it.  We may possibly combine old files into new zips.
   Zips always contain files in a given date range that overlaps no other
   zip, though some may be deadwood.
       Object
worries
-- check that client.ser is always accessible.
-- compactor options
-- test https
-- later attempt to rebuild from existing zips & files found if state lost.
-- builtin ftp upload.