status of walkthru modules (for reference)

[Luka <luka@MIT.EDU>]

here's the report on what i did to /d9/projects/walkthru, for posterity,
including how to deal with possible special needs when i'm not around.

what i did, short form:

  exported former /d9/projects/walkthru head revision.

  moved aside the old repo to walkthru_2003-02-21_archive.

  imported the exported sources to a clean /d9/projects/walkthru.

  diffed stuff, manually fixed up some stuff, diffed more, and tested
  the hell out of at least the code that i'm familiar with now.

why you might care about the data above:

1. say you have an old working copy of walkthru, for example one you found
  lying around on a random crufty disk somewhere that looks like it may
  have very useful uncommitted mods in it.  to stand any chance of
  analyzing that tree, you need to find all CVS/Repository files in that
  working copy, which should contain the line 'walkthru' and replace it
  with a line directing it to the archival tree, which you now know from
  above is 'walkthru_2003-02-21_archive'.  (you can do that with one short
  line of shell -- exercise to the reader.)  after that's done, the
  working copy will be as sane with regard to cvs repo as it was before
  my export/import.  that isn't guaranteeing much since there were a lot
  of hairy changes done before i got here that may still make analysis
  tricky, but at least it gets you as close as i can put you to the old
  tree, as opposed to pointing at something new and completely unrelated.

  if you are so lucky as to discover that there *is* useful stuff in there
  which you want to merge into the new tree, you will need to generate a
  patch and apply it against a sandbox of the new tree and test it as new
  development.  ask someone clueful for help with this if you are not
  confident, and do it carefully -- much chance for slipups here.

2. say you are working on the 'new' walkthru tree, and find some anomoly
  that makes you go "hmmm..." (and you will), and you wonder what kind of
  history caused that anomoly to come into existence.  normally you'd do
  some 'cvs log's and similar stuff and try to deduce it out, but in this
  case the history you care about may have occurred prior to the export/
  import that you are now suddenly thinking i did just to make your life
  hard.  but never fear!  merely do a checkout of whatever relevant parts
  you want from the archive module (whose name you know by now, right?),
  and go poke around in that checkout to your heart's content... it will
  look exactly like everything did right when i did the export this past
  friday, and you can find all your old history under there just like
  always before (but none of the new history, clearly -- you look back at
  your new sandbox for that).

why you might think you care about this, but might just have a bad idea:

1. you have an old tree, you want to continue development in it (or already
  did!), and want to commit it back on the old tree.

    NO NO NO!

  no more development against the old tree.  that's what "archive" means
  in this context.  see (1) above for the *right* way to do this.

2. you are confused, have checked out an archive over a non-archive or
  vice versa while trying to look at old history (as per good-reason-2
  above, but done incorrectly), and you now have a mixed up sandbox pointed
  at two different repos, with overlapping files and mixed up versioning.
  VERY BAD.  best way out is to avoid getting in... by reading the above and
  doing it carefully the first time, if you need to go into the archival tree
  at all.  crossing the trees is bad, even worse than crossing the streams...
  if you don't know what that means, take a break, go watch ghostbusters, and
  by the time you're back i hope your head is clearer.

enough silliness, onward.

you might care about the data below, if:
- you are studious and like to learn from how i do things.
- you distrust me and want to see how i did things.
- you feel there may be something amiss and want to refer to how i think i
  did things after the fact and deduce whether that explains what's going on.
- you want to read about various anomolies in your cvs tree which are already
  known to me (discovered during the export/import), and i might fix, but
  might not due to finite time, and you should feel free to take off my plate
  and do the fixes yourself.

what i did, again, in moderately gory detail:

under /d9/workspace/temp, i checked out (non-pruned) and built a known-good
copy of walkthru, making sure nothing used parallel build opts and everything
got logged.  i build src, util/firewalk, and models/csb/huge this way (which
are the parts i'm most familiar with), ran it a bit, saw no trouble, and
moved it aside to walkthru_old_build_KG (KG == known good).  [correction,
reverse order but same effect:  renamed to walk...KG, and built in parallel
with the upcoming ops, on the assumption it would turn out OK.]

(the above was done on lime [octane2] over nfs to scan:/d9.  the remainder
down until actual build testing was all done locally on scan, for speed and
reliability.)

same temp dir, checked out with -P, applied a tag for export, via:
  cvs tag export_tag_2003-02-21
starting at 9:39 PM, friday night (2/21/2003).

moved that checked out tree aside to walkthru.dead (which is was about to
become), and at 9:46PM ran
  cvs export -r export_tag_2003-02-21 walkthru
to get a checkout of that export tag, sans CVS admin files, ready for import.

at about 10PM, checked cvs history to make sure no one had done any cvs ops since
i started, so my tag and export were still valid.  then moved the old tree aside:
  cd /d9/projects/
  mv walkthru walkthru_2003-02-21_archive
and ran the import:
  cd /d9/workspace/temp/walkthru
  cvs import -I \! walkthru mit2003 import2003

(the "-I !" is supposed to be an all-encompassing override to prevent ANY of the
so-called "cvsignore" rules from getting invoked.  they can come from many sources
and this turns them all off.  thereby preventing LOTS of things in our tree with
"ignored" suffixes from getting silently dropped during import.)

renamed walkthru (the exported tree i was importing from) to walkthru.export.

got a clean checkout of the brand new tree, renamed it walkthru.fixup and diffed
it against walkthru.export.  found following classes of diffs:
1. many thousand lines worth of affected RCS keywords (correct behavior).
2. a handful of dropped files
3. a handful of corrupted binaries that shouldn't be in the tree.
4. a handful of corrupted binaries that should be in the tree.

1. is just chaff, a lot of it, filtered out of the diff mainly through wicked
   bad grep wizardry.  hopefully i got it right and didn't lose any legit
   diffs.  (i'm very confident i got it right.)

2. was the set of 5 files walkthru/models/csb/csb1/test/*.dt
   apparently, even with '-I !' the 'cvs import' saw these in the dir to import
   and dropped them cold, presumably based on the suffix (there are no other
   .dt's checked into the tree, and nothing else to heuristically distinguish
   these from any other file).  look ma, cvs bug!  well, probably.  found no
   mention of this corner case in the manuals, and did not care enough to 
   read cvs source afterwards.

   added the 5 files manually.  add was nice and did not ignore them.  good.

3. whoops, i'd still missed two files even after i'd been cleaning this very
   directory the day before.  so that's why mit/src/terrain/painttest and
   sims/impulse/src/funcstrip (both executables, apparently corrupted even
   before i got to them), were removed after the import rather than before
   like they should have been.  if i didn't admit the mistake here, you would
   never have noticed.  so smile and nod and move on to....

4. corrupted binary files that [arguably] DO belong in the tree, and why handling
   this stuff sucked a bit:

        SYSTEM/E32M3/pkg/bin/wish8.0
        SYSTEM/E32M3/pkg/lib/libtk8.0.a
        SYSTEM/WIN32/pkg/bin/flex.exe
        SYSTEM/WIN32/pkg/bin/tk80.dll
        SYSTEM/WIN32/pkg/lib/tk80.dll
        win32/packages/flex/flex.exe

   ok, so, explanations first:

   A) this stuff is in the tree because we don't yet have a better way of
      dealing with binary lib/exe dependencies for the stuff at hand, notably
      since a lot of it is for windows, and i will just let it lie.  i don't
      do windows.  not touching it is my alternative to simply removing it and
      pretty certainly destroying any chance of the windows build surviving.

   B) NONE OF THE BINARIES WERE MARKED AS SUCH IN THE OLD TREE.
      hence, some of these may already have been corrupted even earlier; keep
      that in mind if you have trouble with them.  plus, i had no systematic
      way of identifying all bin files and marking them after the import.

   C) nit?  maybe.  read over that list of files again (A), and see if the
      redundancy is a bit suspicious to you.  furthermore, the pair of files
      named tk80.dll are not in fact identical.  sigh.  well, apply (B) above
      and maybe that accounts for it, or maybe not.  i dunno.  back to
      conclusion, i'm not touching them except to preserve them as they were,
      and some l33t w1ndoz3 haxx0r is free to fix them up for real someday.
      (oh yeah, and note there's more files like this in there, these are
      just the ones that showed up by virtue of getting corrupted in import.)

   anyway, cleanup process:

   i found these files by the process above (things which differed between the
   exported and newly-imported trees but not accounted for by earlier cases
   (1)-(3)), removed them, and re-added them by hand, supplying the -kb flag.
   they are now in the tree precisely as they were, a clean checkout of them
   (as well as everything else in the tree except the bits allowed for in (1)
   and (3)) precisely matches the old exported data, via exhaustive diff,
   postprocessed.  as for whether these items in (4) are actually "good" or not
   is still is still open for debate, but i assure you they are the same data
   they were before i touched them.  (now marked as binary so they won't get
   further corrupted in the future.)

mm, i think that's about it.  i built src, util/firewalk, models/csb/huge, as
i did for the old tree, ran it, manually tested it out a bit, and called it
good.

oh, and i fixed up perms on the new /d9/projects/walkthru via:
  chgrp -R city . ; chmod -R ug+w .
which i'm not sure is actually exactly what we want, but at least matches
what we had there before.  (specifically, i'm unsure about what should
really have the write bit set, if i were pedantic.  dirs need group write
since that's what provides cvs's concept of access control, but do file
bits reflect the real perms (and should those be ug+w?) or do they reflect
the RCS-style ug-w perms for ,v files?)  whatever, i'm tired, and this is
good enough... someone else feel free to overanalyze this further than i've
done. :)

that really is all.  good night.

-pl