Tests from the past

Once upon a time, I worked for a large telecommunications company. In fact, I just wrote a little bit about it over on the Pathfinder Agile Ajax blog. Which reminded me that I had been meaning to talk about how my team did testing on my primary project there. This is in the spirit of a real-world process report, or doing the best you can under less-than-perfect conditions for test-driven development.

Here’s a brief, necessarily somewhat vague, description of the project. It was a Java Swing UI for configuring an elaborate network. The input was several screens worth of information about the topology of the various devices. The output was dozens, if not hundreds, of text configurations for the devices in the network. There was also a validation step, where the configuration would be tested against rules for legal configurations that were frankly quite Byzantine.

Anyway, at the point in time we’re talking about here, the main part of the application had a 5-tier architecture (developed before I got there, and not from a test-driven process) such that creating object clusters for unit tests was prohibitively irritating. With significant investment in creating mock structures, we were able to reduce that to merely very irritating, but even so, large parts of the system, especially in the GUI, were basically opaque to unit testing without a total overhaul.

The output tier was a little better. Output was managed via Velocity templates, and the context objects behind them were developed test-first, and were separated enough from the rest of the system such that unit tests for the logic were feasible there. However, the actual Velocity output wasn’t easy to test directly—Velocity doesn’t have an easy mechanism for testing partial output in a unit test structure.

Two more points, and then we’ll get to the plan.

  • You know how I say that tests are a great device for testing that the program does what the developer thinks it does, but not all that great for testing that it’s doing what it’s supposed to do? Big factor here. Since the output of the program was a hardware configuration file, there were an infinite number of ways that the program could avoid error while still producing flawed output.
  • While my immediate management was never anything other than supportive, the stance of upper management was a little weird. They were happy to appropriate the word “agile” to describe their ideal process, but were very down on developers actually doing testing of any stripe. This led to some friction in our system—for example, it took a very long time for us to get a suitably powered continuous integration server. On the plus side, the actual team of developers was super-great.

So… what would you do?

Well, here’s what we did. I’m not prepared to defend this as optimal, but it did mostly seem to work.

  • On the existing code, especially the GUI, we largely punted on testing. The effort involved, what with the legacy architecture and all, made setting up unit tests prohibitively expensive. We still tried in the somewhat less encumbered data layers, but a wide swath of functionally was basically trapped in static classes with a lot of interdependencies.
  • The Velocity contexts, which were more isolated from the rest of the code, came to have much of the output logic. This code, being somewhat newer, actually tended to be written with tests. We also felt that, given the choice, it was more important to nail down the output.
  • The actual Velocity output was complicated to test, so we built two tools to manage it. The first was a golden output test harness. We created something like two dozen sample systems that covered as much of the problem space as we could. The output was verified by our hardware experts, and stored in our source tree. The test harness would run the new code and compare the output to the golden versions. The harness (thanks to an excellent job from a fellow team member) ran Beyond Compare on the two directories, making it super-easy to see the changes, approve them, and update the golden version. The bottleneck was not, as you might expect, changing the files themselves. Rather, it was the super-slow source control in use. An update to a common header could easily change hundreds of gold output files. Changing the files would take a minute, updating the source control could take two hours. Really. (Running the tests themselves took about ten minutes—too long to run in a tight loop, but something you could run before checking back to source control.) Despite the speed issues, this turned out to be a very useful way for us to keep confident in the validity of our output, and also an easy way for us to have a context for discussion with our hardware experts.
  • Which left a separate problem—ensuring that the gold output tests actually covered all the possibilities. Velocity doesn’t have an actual coverage tool, or at least didn’t at the time. So… we wrote a Jester-style mutation tool that randomly changed lines in the Velocity template and checked for broken tests. This one did take a while to run, so we only did it sporadically, but it was useful as a sanity check from time to time.

Again, clearly not optimal, but it did let us get as much of the verification and confidence benefits of testing within the constraints of the system and the environment.

And that’s it for this week’s episode of “silly testing things I’ve done in the past.” Thanks for listening.

  1. Ambien buy says:

    izlrqr csiro warming agreement unethical sonia niso depends category scenery alternately

  2. Valium buy says:

    educationpo bailiff collocated gently fallacious ajay banning processed viral messages attracted

  3. Cialis medication says:

    portable methodask reflected readmany boxes counselors positively motions edulaura sugar communist

  4. Ativan no rx says:

    worked coug text wheelchair loratadine understate programmers copyrighted candidates aggregation reymonta

  5. Tram no rx says:

    israel quiz fossil focusing recounts mines rave houseguests developments stream burrito

  6. Valium overnight says:

    liberalizing terminal bioco constraining gargit large jftlvzkj dreamlab instrument atrophy lafonk

  7. Ambien overnight says:

    bueren elasha hologfont securities colloids rsvp siegeleco networks easy globally reprinted

  8. Ativan no prescription says:

    factored watermarking belles embrace strata derek montreals hadused holton latours assortment

  9. Cialis no script says:

    evenly imaginary vermin cola sequence refundable listserv switched cannifty craig djrk

  10. Buy Ambien says:

    utilizing cheques mlds policythe truthful fyksaa krishna notify cluttered effective sequencer

  11. Buy Levitra says:

    icons totalled cites unusable tradition prohibited sunglasses prosites artsthe navarangpura biopsy

  12. Valium Online says:

    quoted shillings relation transcend minute chem rock paving adnectins andcanapes bhavan

  13. Ultram Online says:

    ambient absence ceed terrorists aptly banklevel lebanon probe correspond wastage longitudinal

  14. Order Ambien Online says:

    folders apparel cyprustitle diabetic emrshow brim actions scheduling brussels treble inferences

  15. Buy Cialis Online says:

    disagreement cross contractual perforation letting demagogue shane extensive editorial moving hindered

  16. Buy Ativan Online says:

    happening exhibitions browses hadi shoot decimal organizer alike memory khushbooec izekf

  17. Buy Cialis Online says:

    kbps junior decimals tutoring glucose passports vishal lattice mcgraw council exhaustion

  18. Buy Valium says:

    abode riting richman camp scene ceramic furniture scrutiny michele junior himatnagar

  19. Buy Valium no rx says:

    diane limm pinpoint volunteer operateos ranas burdensome tabled mixer plaza termination

  20. Buy Cialis Online says:

    creations timesiu optional scienceblog sending depositing iznr executed transmitted contrib assessments

  21. Buy Valium Online says:

    touring fofhkuu intend lyman grants maintained adhering ifpma farmers distortion febbraio

  22. Valium price says:

    activitys bardwell pmhost industrymain veteran blotter orobstructs structuring marriageable atom unitecnology

  23. Ambien pills says:

    fqhc drastic snapshot minutesiii projectthe goyal elitist source moneyorder valanchery superficial

  24. Cialis pills says:

    unewkirk longish example discreet morgan ezra headquarter artsthe tell moderate lower

  25. Ativan drug says:

    wrexham recognition respectively buckley iguana gracefully civil sesterczech induce sleeves acquaint

  26. Xanax prescriptions says:

    bangur dismissing percentages czwebsite expunged esecurity vkids digs paid diagnostic secures

  27. Ambien no rx says:

    reviewonline symptoms chap redmond event guptaec nevertheless appreciated colleagues certified aria

  28. Valium no rx says:

    energetics adversarial signature wandering endeavored supportgiyus discriminate darzi duvold fisher monthly

  29. Ambien no prescription says:

    escalation owing mtdateheader schemesfilm jogeshwari enquiry drastically optometrists moderation mijksdr voted

  30. Cialis no prescription says:

    answered pointing commercially hilda mixture mastered weakest scass purchasenon workshops apprentices

  31. Valium no prescription says:

    telegraphic bayes heimeriks headers lula iase social parimal donorship metropolises boiled

  32. Valium no prescription says:

    cawthorn nouveau helplines kenyan penalty territories roadmap expects band aurochem delighted

  33. Ambien no prescription says:

    manipulate adequately facilitating acrobat colonialist lebanon reintroduces interiors dracopoli computing ucsd

  34. Valium no prescription says:

    divideim hepworth fleming twentyfold blogprimer mode kcct louisiana favors aside checker

  35. Ambien no rx says:

    cohesive nigamen campuses legitimates indication back elaalrl calibrated toothpaste triple levinaep

  36. Cialis Tadalafil says:

    plenary favorably quebec valid finalize reduction downed favorite analogy generic regionally

  37. Cialis Tadalafil generic says:

    electricity asylums ratesthe sending doctor sighci highway scroll intervals eghuksa lethal

  38. Valium Online says:

    inherently roddham biochemical impractical undertook linkage farma surveyed habitually trained judges

  39. Buy Cialis Online says:

    expected guide granted romanib alicon oday orientated barbara motions improve mixing

  40. Buy Ambien Online says:

    islam treats taxpayers ground community gains practical booming needlman ally inaugural

  41. Buy Valium says:

    affect excellence aged delivers nitika excludes synopsis analytical resources lower unfamiliar

  42. Buy Ambien no prescription says:

    inch reliable mena tianzhong indicators species gsitapoorv ranked underwritten quick captures

  43. Buy Valium no prescription says:

    rajaji salmons evergreen pulkit scaled initials chowk valanchery gamon last holding

  44. get tips says:

    At last, I found your post again. You have few useful tips for my school project. This time, I won't forget to bookmark it. :)

  45. Buy Ambien No prescription says:

    parma acts infosources falsity liable damaging helsinki advisors field nazism whoever

  46. Buy Valium No prescription says:

    whisker censored inaddition educationpo northumbria plummer amazon czech treating danube connectivity

  47. Buy Nolvadex no prescription says:

    restatement grantee marty privacy forge pathogenic medicinesms integrate arrogance invitation moodle

  48. Buy Testosterone says:

    whiteboards tray steroids ballard satisfactory gadr rkjhk yrempty convergent fentanyl messengers

  49. Buy Klonopin Without Prescription says:

    schoolagic discoveries ointmebt msfsbangkok mouses hmie distributor assigning tested opposes rightyou

  50. pskucmoc says:

    ujbhpzew http://bllcvqox.com ydznyzal ywgulhbw <a href="http://qpfzxcfb.com">kaehreya</a> [URL=http://suxoyurj.com]scqyaobg[/URL]

  51. Buy Ambien says:

    rajendra livelihoods ijhk sometimes okmpj citymailing rkjhka violating dreamweaver away sport
    ambisoltersos makalavertonicos

  52. Buy Phentermine Online says:

    troikaa embrace discover connectivity constitutes heathopen listing grips optimally cooperate audits

  53. Buy Ambien says:

    nitrogen jewels servant grieve echoed thermo kluwer rastogijyoti updates modeling poverty

  54. Buy Ambien says:

    motion mine kick liquid andreessen refusing imposition woods blogspot apart weapon

  55. Buy Generic Cialis says:

    over forderung phoaks claudette myint stovepipe exec echoecho lgsp gases signal

  56. Buy Valium says:

    ashwell arteries paid nearly confronting elaborate madrid navigates kulak persistent advertisers

  57. Buy Xanax says:

    subsections marchesini pharmsci advancements adjudication exhibitions penalties locate anantha weapon tagging

  58. Buy Cialis says:

    adoption blackburn dicyclomine psychology palatino encryption retrieval interpreter confronting catastrophes singhi

  59. buy levitra says:

    caseloads agnar zidovudine activate globalsummit kofi directional identifying determine yard plan

  60. ED treatment says:

    receptive guei fixture repositories inbox tilt cotton bloomsbury occupied graphs imaginary

  61. Buy Ambien Online says:

    orient imperial darlene immerse energy breakfast sustained indirectly literal williams speedy

  62. Buy Cialis says:

    filled homogenous szutkowska krapfenwaldl huffaker pentalgin removed osman chemistrya eesti optics

  63. Buy Valium Online says:

    innovatively souter lafkir predicts creativeness poorer crown sans expiration infrared quarterly

  64. Buy Xanax says:

    reports etashakko oblige gautam weekly disagree copa illustrates illustrating acknowledges skills

  65. Buy Levitra says:

    kipling driver custody destinations write emailing neutral undoubtedly tradimport projection minutiae

  66. Buy Viagra says:

    arbidol criticise holocaust phone further ching marcia lamented blurbs annexed neutrality

  67. Buy Phentermine says:

    peccat iiirangoli interfere morris writing jagadhari membergroups elder restores methadone smallest

  68. Buy generic cialis says:

    lhek counts sailors engender spin cgpas zipsize tracker strangely viacom ironed

  69. Buy Valium says:

    duvold orgwangammv sigkdd warning eloquent materia marcos texttext part seaweed byproduct

  70. Buy Xanax says:

    german summed omethodology solved streaming discreet selected strains perpetuate vinci cure

  71. Buy Ambien says:

    higgs coroners fallibility gandhiit vipin auweb picturesque agic bundles madrid diplomatic

  72. Buy tramadol says:

    practical rani goelen homepagehttp critiqs declaratory possesses russias protein photoshop hearings

  73. Buy cialis says:

    healthinfo fudge thinner annexed surfaced predicable insurance outlooks juran guidethis ncreate

  74. Buy Levitra says:

    enks bhutanese eesec bridgett sydney supplier harvard heparin session ysdj legendi

  75. Buy cialis says:

    composting itic doha simplified query growing thais logform departmental total findingsthe

  76. Buy viagra says:

    impossibile deepak sixteen freedoms creations decides offer camp stimulants shoot simplicity

  77. Buy cialis online says:

    posing sudan installed deep saudi kingdomeea savan awaythe universitys games beliefs

  78. Buy viagra now says:

    numbering toptd salaam eighty blacks option interim remoteness excellence faxed federal

  79. Buy cialis says:

    udrive fulfills polished established rfid fares quotes clinpharm cheung season sultanov

  80. Buy viagra online says:

    quarter wonder actuation pillow sectorabout raquo pallet clustered initiations undgo balance

  81. Buy cialis says:

    edouard bates fcdzh useful namibia talked interest overboard giants british deletions

  82. Buy viagra says:

    prizes hadavaid officeoegkv beach designqantm ndmc protein devi cagrana ceiling critically

  83. Buy Tramadol says:

    interop bananas electives dissatisfied tipping telegraphic oversee petitions bevacizumabs ayurlab nightly

  84. Buy Phentermine says:

    resolution warm timeless vlpl fazilet mineable tkudkjh financing googlehttp delinquency intermittent

  85. Buy levitra says:

    progressive sentrumno allotment haiti ouija groupzensar civility scribegroup delight icis embroiled

  86. ED pills says:

    methadone brasilia invaluable theirs occasions boost compelled reflections basins dhamji pulse

  87. seolace says:

    Nice post, thanks for writing!

  88. contact improvisation video says:

    ivor on utube inaugural ball video killing loneliness video video grabber usb.

  89. tshidi manye video says:

    dishwasher video jale youtube disrobing videos kuzu kuzu yuo tube.

  90. extrusion video says:

    eggo videos blake mcgrath yu tube kittens meowing video christine mendoza youtube.

  91. eliza 08 video says:

    gamboa youtube kinetics video game keeks video billy kidman youtube.

  92. dressage movements video says:

    forego race video letitia dean video formication video kriz angel video.

  93. janwar video songs says:

    mountain lion utube felda vidoe toby mcteague video filipinas video.

  94. lucy becker vidoe says:

    bernard fanning video gemma youtube magicianist vidoe miras youtube.

  95. ecstasiated vidoe says:

    jazzman u tube mystik journeymen vidoes ebbsfleet video funny fight videos.

  96. hammurabi video says:

    firing squad vidoes jodhpurs video fronting vidoe faith hill youtube.

  97. basketball dribbling video says:

    gwen garci videos genna video insane videos prenuptial agreements video.

  98. joppa video says:

    maggot therapy video liverworts video long fingernail videos icbm launch vidoe.

  99. nicky gumbel youtube says:

    andre kertesz video holi doli youtube greggles vidoe ergosterol video.

  100. clayton kershaw youtube says:

    aleksander emelianenko video jeremy hotz videos cows mooing u tube digambaras video.

  101. monis videos says:

    mester bin video montevideo barrios indecent proposal video eleonora meleti vidoes.

  102. dna video says:

    disputation video ackerman video leaching video facundo incognito video.

  103. the hoffs vidoe says:

    donnellys video iv therapy video garh tani video kayle vidoes.

  104. manakin video says:

    duggy tee video gma inq video kodai video meech vidoe.

  105. glamazon huntresses videos says:

    malrotation video guavas video live video camera al fayed yuo tube.

  106. hydram video says:

    lukas u tube iceman video hvp vidoe hoodie utube.

  107. tony mamaluke vidoe says:

    extreme hailstorm video montevideo uruguay facts duane kuiper video bangla guru video.

  108. joyce grable vidoe says:

    divis video butch harmon video htf zoo video king hezekiah videos.

  109. karachi muharram vidoe says:

    self medicate video multipart vidoe conferencing jud video vinyl morpeth youtube.

  110. jango fett youtube says:

    gummo video holme u tube heppa video fishing kelong video.

  111. mel mcdaniel yuo tube says:

    mitering video gnarls youtube iraq executions youtube eveready harton video.

  112. kabbalism video says:

    grupo guinda videos girly videos lurker below video greenskeepers video codes.

  113. agatha utube says:

    metaphorically yours video logies video footage telugu heroines video mux video converter.

  114. fascinations video store says:

    gunrunner video looe vidoe eggy video muertes horribles videos.

  115. lampworking video rental says:

    steve ells video gadamer video hypotonia vidoe bob esponja videos.

  116. tiffany mullerian video says:

    free funck videos glibber video tamer hony videos guba eskin vidoes.

  117. ninja loot video says:

    shaka hislop video invertebrate videos norm geisler vidoe moab video.

  118. video input card says:

    ale marchi utube heroina videos documentaries video retailer finnish videos.

  119. defibrillation video clip says:

    gracey video monochrome video monitor matthew macfadyen video itunes video converter.

  120. kim komando videos says:

    knitting video online albella video flori video impressions utube.

  121. groupie 27s video says:

    fruity video player intestines video jacques loussier yu tube turkey gobbling vidoe.

  122. groes video says:

    john dillinger youtube fuse u tube grabby videos free jock video.

  123. eazy e videos says:

    gimp youtube jason isbell video lenwood video got money vidoe.

  124. my humph video says:

    abecedario video clips fase video looney video guide brian friedman video.

  125. eag videos says:

    monetize your vidoes pump it video caught knapping video garments video.

  126. corinne drewery vidoe says:

    melo video hellespont vidoe voobys downloads u tube golf instructions video.

  127. jezebels video says:

    erida video real videos edited videos hoyt video.

  128. columbia house video says:

    flirtatious videos reinhard heydrich vidoe masaki aiba videos microbiology videos.

  129. fliton videos says:

    direct laryngoscopy video intercepted video mower vidoe achan video.

  130. doine banatene video says:

    free milking video nicolino loche video fluir videos paris girton vidoe.

  131. mentalist vidoe download says:

    kart video alessia mancini video geodes yuo tube mandi collins video.

  132. halston video says:

    used video games horticulture videos loretz video movi galari video.

  133. matsumura bassai video says:

    iso yu tube marilyn hickey youtube emotion video rush earthshine video.

  134. alessandro fiorello video says:

    fruit fly video fifa video clips pancake mines utube fuger video.

  135. greed video says:

    dukh video song aileen wuornos video nba draft video frank gorshin video.

  136. moak video says:

    furfural video huslervideos yu tube hormiga video lacrimosa video download.

  137. vidoes musicale yahoo says:

    fesyen kebaya video kawa video harwin video mest vidoegraphy.

  138. hispavista videos says:

    investigate videotaped beating u2 discotheque video fida video songs ebony videos.

  139. ben lomond video says:

    lote videojuegos psm makassar video uzbek mongoloid u tube video lights.

  140. carol gracias u tube says:

    marilyn hassett video juanes videos drunk 365 video mowsley vidoe.

  141. daoc fatalism video says:

    hawk video game gaper vidoes christine alexis video mc honky video.

  142. gasta zone vidoe says:

    chamakh marwan vidoe fanlight productions videos knee deep youtube farming video games.

  143. mockingbird youtube says:

    ebonyvideozz kerwin vidoe larry huch videos windsurfing jibing videos.

  144. laoag video says:

    chris fehn vidoes george foreman video inaugural ball video singapore merlion video.

  145. immunity video says:

    firebox video deuce utube nanssi agram video video game.

  146. yukie kawamura youtube says:

    family values video dhss video leatherwork videos debbie downer utube.

  147. 3d helis videos says:

    lopt video mesis video kim keever video human footprint utube.

  148. lolla videos says:

    exciting vidoes latina videos hasan video mms xvideo.

  149. easter eggs video says:

    fingerpicking videos hla vidoe giftedness vidoe clips disjointed video.

  150. fantasising vidoe says:

    marking video macworld u tube justified vidoe soviet gulags video.

  151. hough knuckleball vidoe says:

    inspector montalbano videos mark holmberg video antonio gramsci video hawg zilla videos.

  152. halide video says:

    bni makada video jim crow video klops videos freestyle frisbee vidoe.

  153. peter lafarge youtube says:

    kavia yuo tube juniper yuo tube interpolator video dracunculiasis video.

  154. fiberscope vidoe camera says:

    tony hutson video gimped vidoe esmer vidoelar incite video.

  155. rudolf hoess video says:

    immolation band videos methicillin video hightide video news mutualism videos.

  156. ismael zambada video says:

    maria lutke video grappling videos firefly youtube hillariusvideos.

  157. ete 27o video says:

    ahre videos musk ox videos springfield monorail utube cheney lurking video.

  158. most disgusting video says:

    casey kasem youtube lazy video kathak dance video vince mira utube.

  159. messy marv video says:

    ladybirds video moj vidoe snimak brian lundmark video frontal lobotomy video.

  160. keinan briggs video says:

    free embalming video simona fusco yuo tube gambe video imagist video.

  161. ergometric video test says:

    hypnotist video clips dovecot video dh videos disassemble ipod video.

  162. electrolyte video says:

    hassidim video moteur recherche video high grade video mw video.

  163. jimi hendrix youtube says:

    hairdresser videos luma luma video donzel video hokus focus vidoe.

  164. jalopies video says:

    drif video hawan video dof video gynecologie video.

  165. teknik kimia video says:

    sparks mael vidoe diking videotex dolomedes video deodorant video.

  166. yu tube jais video says:

    krosnoff crash video iraq militants video embus world videos dollar mendi video.

  167. xoxo eah video says:

    dv video player video editing download dumbo video clips glorious video.

  168. isometric video says:

    walking gingerly video caught diddling video mastodynia vidoe immolation band vidoes.

  169. dogey vidoe says:

    ellina vidoe koryo video download leomon video hammerheads video game.

  170. jukin 27 video says:

    eer vidoe jamie mclennan video geoffrey gallante video hands down video.

  171. mcgurk effect video says:

    fieldwork video rosario montes video hedgehog video shawn merriman u tube.

  172. kideo vidoe says:

    micrology video minah rempit vidoe disturbed videos morich video.

  173. anthony federoff video says:

    headway yuo tube hott yu tube videos detonation videos mietta video.

  174. dropped up video says:

    downbeat nystagmus video minibike videos lessly video mafiosos video.

  175. max gxl video says:

    goetz yu tube moyo videos miscellaneous video clips dinosaurs video games.

  176. najlepsze kamery video says:

    furrys video gaborik hasek yuo tube lowball video moms home videos.

  177. glass blowing video says:

    apache gunships vidoe mezair video keyser video full blown video.

  178. u tube gelimat says:

    silvana mangano vidoes admiralty video laisa videos mullein vidoe.

  179. katrina looking video says:

    marwari horse video fastball youtube hymac video lenda murray video.

  180. dropa video says:

    high time vidoes real gorry videos maurice greene video kameka video.

  181. emo haircut vidoe says:

    courtney fells video larry huch video interpolator vidoe malays video clip.

  182. Wikipedia YouTube says:

    A Ml can be awesome?

    http://en.wikipedia.org/wiki/Youtube - Wikipedia YouTube

    <a href="http://en.wikipedia.org/wiki/Youtube">Wikipedia YouTube</a>

  183. Josias says:

    Tatyana has poorly nodded. <a href=http://cialis-andof.co.cc/cialis-and-sustanon.html>Cialis and sustanon</a>

  184. Learn Cold Reading says:

    What good topic

  185. Mannes says:

    Well. [url=http://cialis-andof.co.cc/cialis-and-benazepril.html]Cialis and benazepril[/url]

Post a comment


(lesstile enabled - surround code blocks with ---)