June 10th 2020
The spec is backed by fixture code which, when executed, will mark assertions in the spec as either passed or failed (html reporting showing green or red respectively). A quick search tells me nothing like this has been done in go yet so project candidate sorted.
My main takeaway was the idea of iterative testing and validation after every little step. I entertained the thought (and did some reading) of AST transforms using antlr but that got me nowhere.
Ideation and initial reading was from Saturday-Monday, May 9-11, around 12 hours effort.
Setup and preparation. Before I can get started on the first test there was the usual required project bootstrapping. I’ve got Intellij Idea for an IDE, decided on github as repo, and elected to just manage tasks in-project as a list of things to do in markdown. One of the main pre-requisites was to be able to parse markdown and convert it to xhtml.
This was over Tuesday-Wednesday, May 12-13, around 8 hours effort.
First test, running a fixture method that spits out “Hello World!”. Talk about biting off more than you can chew. First up was xml parsing the spec (now in the form of xhtml) into generic nodes, not standard but still pretty straightforward. What I didn’t foresee was that I’d need to be diving into Go reflection this early (specs contain only strings, the call to something like “HelloWorld()” needs to be deconstructed and resolved to the relevant fixture function via reflection).
With the first test under my belt it’s time to put the design hat on and think carefully about the commands that conthego will be supporting. These commands are the main interface for consumers using the tool so it pays to put enough upfront investment into them to make them simple and elegant to use. I was under no illusion I’ll get everything perfectly right but I was conscious that I needed to start with a ‘good set’.
An hour of writing and rewriting got me the command set below. Apart from the one using literal parameters, the rest have stood the test of time (a couple of weeks!). Early on I decided to stick close to what Concordion already offers. Deviations were mostly for ease of parsing.
Command prefixes were a single special character and (if required) appear only at the start of the command instruction string.
set [World](- "var1") echo (- "$var1") (- "$var1.prop2") exec (- "Hello()") (- "var1=Hello(TEXT)") (- "var2=Hello(var1)") (- "var1=Hello('literal')") assert equals or isTrue [World](- "?Hello()") [World](- "?var1")
The trick was to marshal my structs into json then unmarshal them back into a generic tree of json nodes, which go-dotnotation can navigate. Simple and sufficient for my usecase. At this point, passing structs and slices around, it took a fair bit of pointer wrangling to fix some gnarly bugs (I understood the issues then but am sure I’d need to keep re-learning pointer semantics). This lot was part Saturday and most of Sunday, the 17th, at least 8 hours effort.
Next up was execute-rows – executing a set of commands (specified on table headers) over each row of a table. Turning things around my head on how to make it work, I’ve come to appreciate the beauty of sequential processing of commands: top to bottom, left to right; a perfect match for depth-first traversal of the xhtml node tree.
This meant binning some future (complex) usecases that don’t fit the model but I was quite happy in exchange to keep functionality limited and simple. I introduced pre-processing which can transform commands in the xhtml nodes prior to the command collector collecting them for execution. In this case ‘transform’ means removing the command instructions from the table headers and distributing them to the actual data rows. This was Monday night, the 18th, 4 hours effort.
Verify-rows – there are two variants: first is verifying a simple list of strings, the second verifying a list of structs. This got me back to the drawing board (to be honest, I’m still not fully sold on the solution I came up with thus these commands are currently experimental). In the end I decided to constrain the implementation to support only unstructured lists for the first case, and a table for the second.
I needed to disambiguate the intent for list-processing from the regular commands. The top options were to introduce a new type of token in the command instruction string to signal list processing (ex. square brackets in “?listOfNames”) or to introduce a new directive command that would impact processing of the succeeding commands.
I opted for the latter for the reasons stated below. Implementation of verification for a list of strings was Tuesday night, 19th, 4 hours effort.
- to keep command instruction strings simple
- because there already was prior art – a directive command ‘!ExpectedToFail’ which impacted succeeding assertions
- I considered list-processing to be ‘advanced’ commands and merited ‘signaling’ via use of a directive
Verification for a list of structs is next, with support limited to being applied on a table. A major design decision here is to be able to choose for each column a specific struct property to assert on or echo. You can use echo exclusively (without asserts) for usecases like displaying a table of reference/config items. This was Wednesday night, 20th, 4 hours effort.
I wanted to encourage easy usage, in case anyone would be keen to try, Tidying up of the basic command examples and the README file was Thursday night 21st, 4 hours effort.
I’ve counted 60 hours above. I’m writing this on Sunday night, 24th, 4 hours effort. I’m a programmer, a serial optimist when it comes to estimates; and 64 is a nice round power of 2.
The decision to store the task backlog as a TODO list in Markdown has proven to be handy. It felt natural to keep maintaining it whilst implementing code, all in the IDE. This worked as it was only me working on the codebase. Add another person and I’d probably look for a lightweight board like Trello.
Being able to tick off items (or multiple) on something like a nightly basis builds up confidence and a sense of satisfaction.
I’ve found that living documentation in the form of executable specifications are extremely useful in understanding how a system works. I always make it a point to refer to these artifacts primarily as specs rather than tests. Specs should be able to stand on their own even without having been executed.
Crafting useful specs takes effort and time from the whole team with the main rewards being shared understanding, increased confidence that the system behaves as expected, and having something like a jumpstart tool for new team members.
The Concordion documentation is the best resource describing how to write good specs.
It’s been fun learning and creating something new, and there’s still a lot of TODOs to slowly nibble at. But now I feel I’ve been neglecting my books 🙂 Enjoy!
09874b0 Thu May 21 23:14:22 2020 +1200 fix README.md 77b491b Thu May 21 22:57:56 2020 +1200 Merge pull request #1 from fanovilla/feature/tidy-examples 78a09c7 Thu May 21 22:53:24 2020 +1200 better set and echo tests 290460f Thu May 21 22:31:13 2020 +1200 better assert-true test cadc82e Thu May 21 21:51:21 2020 +1200 better assert-equals test 5af7442 Wed May 20 22:28:45 2020 +1200 reorder backlog c2a03d6 Wed May 20 22:16:01 2020 +1200 refactor out spec pre-processing routines 1a3622a Wed May 20 22:13:04 2020 +1200 use concordion css 3dc272a Wed May 20 22:00:47 2020 +1200 implement verify table rows 83d58b7 Wed May 20 00:33:50 2020 +1200 implement verify rows for list of strings ea5cf4d Tue May 19 18:38:59 2020 +1200 add reporting of date time generated and directives c6fa62e Mon May 18 23:50:44 2020 +1200 implement execute rows df56ffd Sun May 17 23:17:14 2020 +1200 simplify run spec hook d3de448 Sun May 17 23:03:18 2020 +1200 update todos 12425cc Sun May 17 22:53:17 2020 +1200 fail on panic 0c83b22 Sun May 17 22:34:09 2020 +1200 implement return slice 620eee8 Sun May 17 22:08:05 2020 +1200 implement assert true 5ea8d27 Sun May 17 21:30:29 2020 +1200 Create LICENSE.txt 2ff8c59 Sun May 17 20:40:03 2020 +1200 implement map return values; failed assert fails test; ExpectedToFail directive fbf8cca Sun May 17 19:45:15 2020 +1200 implement struct return values; dot.navigation 613b76e Sun May 17 16:51:12 2020 +1200 implement TEXT substitution; assign execute result to var 8961de4 Sun May 17 14:47:20 2020 +1200 implement echo a953e87 Sun May 17 01:09:23 2020 +1200 rename ReflectUtils to MethodCaller 06d3a5f Sun May 17 01:03:33 2020 +1200 implement variable setting and parameter passing 5d22215 Sat May 16 23:13:56 2020 +1200 Initial commit