whatcanGOwrong

2024-09-19 21:38:24 -04:00
commit d0ae4d841d
17908 changed files with 4096831 additions and 0 deletions
@@ -0,0 +1 @@
+build_flags="-trimpath -ldflags '-w -s -X \"zgo.at/zli.version=$tag$commit_info\" -X \"zgo.at/zli.progname=toml-test\"' ./cmd/toml-test"
@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2018 TOML authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
@@ -0,0 +1,271 @@
+`toml-test` is a language-agnostic test suite to verify the correctness of
+[TOML][t] parsers and writers.
+
+Tests are divided into two groups: "invalid" and "valid". Decoders or encoders
+that reject "invalid" tests pass the tests, and decoders that accept "valid"
+tests and output precisely what is expected pass the tests. The output format is
+JSON, described below.
+
+Both encoders and decoders share valid tests, except an encoder accepts JSON and
+outputs TOML rather than the reverse. The TOML representations are read with a
+blessed decoder and is compared. Encoders have their own set of invalid tests in
+the invalid-encoder directory. The JSON given to a TOML encoder is in the same
+format as the JSON that a TOML decoder should output.
+
+Compatible with TOML version [v1.0.0][v1].
+
+[t]: https://toml.io
+[v1]: https://toml.io/en/v1.0.0
+
+Installation
+------------
+There are binaries on the [release page][r]; these are statically compiled and
+should run in most environments. It's recommended you use a binary, or a tagged
+release if you build from source especially in CI environments. This prevents
+your tests from breaking on changes to tests in this tool.
+
+To compile from source you will need Go 1.16 or newer (older versions will *not*
+work):
+
+    $ git clone https://github.com/BurntSushi/toml-test.git
+    $ cd toml-test
+    $ go build ./cmd/toml-test
+
+This will build a `./toml-test` binary.
+
+[r]: https://github.com/BurntSushi/toml-test/releases
+
+Usage
+-----
+`toml-test` accepts an encoder or decoder as the first positional argument, for
+example:
+
+    $ toml-test my-toml-decoder
+    $ toml-test my-toml-encoder -encoder
+
+The `-encoder` flag is used to signal that this is an encoder rather than a
+decoder.
+
+For example, to run the tests against the Go TOML library:
+
+    # Install my parser
+    $ go install github.com/BurntSushi/toml/cmd/toml-test-decoder@master
+    $ go install github.com/BurntSushi/toml/cmd/toml-test-encoder@master
+
+    $ toml-test toml-test-decoder
+    toml-test [toml-test-decoder]: using embeded tests: 278 passed
+
+    $ toml-test -encoder toml-test-encoder
+    toml-test [toml-test-encoder]: using embeded tests:  94 passed,  0 failed
+
+The default is to use the tests compiled in the binary; you can use `-testdir`
+to load tests from the filesystem. You can use `-run [name]` or `-skip [name]`
+to run or skip specific tests. Both flags can be given more than once and accept
+glob patterns: `-run 'valid/string/*'`.
+
+See `toml-test -help` for detailed usage.
+
+### Implementing a decoder
+For your decoder to be compatible with `toml-test` it **must** satisfy the
+expected interface:
+
+- Your decoder **must** accept TOML data on `stdin` until EOF.
+- If the TOML data is invalid, your decoder **must** return with a non-zero
+  exit, code indicating an error.
+- If the TOML data is valid, your decoder **must** output a JSON encoding of
+  that data on `stdout` and return with a zero exit code indicating success.
+
+An example in pseudocode:
+
+    toml_data = read_stdin()
+
+    parsed_toml = decode_toml(toml_data)
+
+    if error_parsing_toml():
+        print_error_to_stderr()
+        exit(1)
+
+    print_as_tagged_json(parsed_toml)
+    exit(0)
+
+Details on the tagged JSON is explained below in "JSON encoding".
+
+### Implementing an encoder
+For your encoder to be compatible with `toml-test`, it **must** satisfy the
+expected interface:
+
+- Your encoder **must** accept JSON data on `stdin` until EOF.
+- If the JSON data cannot be converted to a valid TOML representation, your
+  encoder **must** return with a non-zero exit code indicating an error.
+- If the JSON data can be converted to a valid TOML representation, your encoder
+  **must** output a TOML encoding of that data on `stdout` and return with a
+  zero exit code indicating success.
+
+An example in pseudocode:
+
+    json_data = read_stdin()
+
+    parsed_json_with_tags = decode_json(json_data)
+
+    if error_parsing_json():
+        print_error_to_stderr()
+        exit(1)
+
+    print_as_toml(parsed_json_with_tags)
+    exit(0)
+
+JSON encoding
+-------------
+The following JSON encoding applies equally to both encoders and decoders:
+
+- TOML tables correspond to JSON objects.
+- TOML table arrays correspond to JSON arrays.
+- TOML values correspond to a special JSON object of the form:
+  `{"type": "{TTYPE}", "value": {TVALUE}}`
+
+In the above, `TTYPE` may be one of:
+
+- string
+- integer
+- float
+- bool
+- datetime
+- datetime-local
+- date-local
+- time-local
+
+`TVALUE` is always a JSON string.
+
+Empty hashes correspond to empty JSON objects (`{}`) and empty arrays correspond
+to empty JSON arrays (`[]`).
+
+Offset datetimes should be encoded in RFC 3339; Local datetimes should be
+encoded following RFC 3339 without the offset part. Local dates should be
+encoded as the date part of RFC 3339 and Local times as the time part.
+
+Examples:
+
+    TOML                JSON
+
+    a = 42              {"type": "integer": "value": "42}
+
+---
+
+    [tbl]               {"tbl": {
+    a = 42                  "a": {"type": "integer": "value": "42}
+                        }}
+
+---
+
+    a = ["a", 2]        {"a": [
+                            {"type": "string", "value": "1"},
+                            {"type: "integer": "value": "2"}
+                        ]}
+
+Or a more complex example:
+
+```toml
+best-day-ever = 1987-07-05T17:45:00Z
+
+[numtheory]
+boring     = false
+perfection = [6, 28, 496]
+```
+
+And the JSON encoding expected by `toml-test` is:
+
+```json
+{
+  "best-day-ever": {"type": "datetime", "value": "1987-07-05T17:45:00Z"},
+  "numtheory": {
+    "boring": {"type": "bool", "value": "false"},
+    "perfection": [
+      {"type": "integer", "value": "6"},
+      {"type": "integer", "value": "28"},
+      {"type": "integer", "value": "496"}
+    ]
+  }
+}
+```
+
+Note that the only JSON values ever used are objects, arrays and strings.
+
+An example implementation can be found in the BurnSushi/toml:
+
+- [Add tags](https://github.com/BurntSushi/toml/blob/master/internal/tag/add.go)
+- [Remove tags](https://github.com/BurntSushi/toml/blob/master/internal/tag/rm.go)
+
+Implementation-defined behaviour
+--------------------------------
+This only tests behaviour that's should be true for every encoder implementing
+TOML; a few things are left up to implementations, and are not tested here.
+
+- Millisecond precision (4 digits) is required for datetimes and times, and
+  further precision is implementation-specific, and any greater precision than
+  is supported must be truncated (not rounded).
+
+  This tests only millisecond precision, and not any further precision or the
+  truncation of it.
+
+
+Assumptions of Truth
+--------------------
+The following are taken as ground truths by `toml-test`:
+
+- All tests classified as `invalid` **are** invalid.
+- All tests classified as `valid` **are** valid.
+- All expected outputs in `valid/test-name.json` are exactly correct.
+- The Go standard library package `encoding/json` decodes JSON correctly.
+- When testing encoders, the TOML decoder at
+  [BurntSushi/toml](https://github.com/BurntSushi/toml) is assumed to be 
+  correct. (Note that this assumption is not made when testing decoders!)
+
+Of particular note is that **no TOML decoder** is taken as ground truth when
+testing decoders. This means that most changes to the spec will only require an
+update of the tests in `toml-test`. (Bigger changes may require an adjustment of
+how two things are considered equal. Particularly if a new type of data is
+added.) Obviously, this advantage does not apply to testing TOML encoders since
+there must exist a TOML decoder that conforms to the specification in order to
+read the output of a TOML encoder.
+
+Adding tests
+------------
+`toml-test` was designed so that tests can be easily added and removed. As
+mentioned above, tests are split into two groups: invalid and valid tests. 
+
+Invalid tests **only check if a decoder rejects invalid TOML data**. Or, in the
+case of testing encoders, invalid tests **only check if an encoder rejects an
+invalid representation of TOML** (e.g., a hetergeneous array). Therefore, all
+invalid tests should try to **test one thing and one thing only**. Invalid tests
+should be named after the fault it is trying to expose. Invalid tests for
+decoders are in the `tests/invalid` directory while invalid tests for encoders
+are in the `tests/invalid-encoder` directory.
+
+Valid tests check that a decoder accepts valid TOML data **and** that the parser
+has the correct representation of the TOML data. Therefore, valid tests need a
+JSON encoding in addition to the TOML data. The tests should be small enough
+that writing the JSON encoding by hand will not give you brain damage. The exact
+reverse is true when testing encoders.
+
+A valid test without either a `.json` or `.toml` file will automatically fail.
+
+If you have tests that you'd like to add, please submit a pull request.
+
+Why JSON?
+---------
+In order for a language agnostic test suite to work, we need some kind of data
+exchange format. TOML cannot be used, as it would imply that a particular parser
+has a blessing of correctness.
+
+My decision to use JSON was not a careful one. It was based on expediency. The
+Go standard library has an excellent `encoding/json` package built in, which
+made it easy to compare JSON data.
+
+The problem with JSON is that the types in TOML are not in one-to-one
+correspondence with JSON. This is why every TOML value represented in JSON is
+tagged with a type annotation, as described above.
+
+YAML may be closer in correspondence with TOML, but I don't believe we should
+rely on that correspondence. Making things explicit with JSON means that writing
+tests is a little more cumbersome, but it also reduces the number of assumptions
+we need to make.
@@ -0,0 +1,258 @@
+//go:build go1.16
+// +build go1.16
+
+package tomltest
+
+import (
+	"strconv"
+	"strings"
+	"time"
+)
+
+// CompareJSON compares the given arguments.
+//
+// The returned value is a copy of Test with Failure set to a (human-readable)
+// description of the first element that is unequal. If both arguments are
+// equal, Test is returned unchanged.
+//
+// reflect.DeepEqual could work here, but it won't tell us how the two
+// structures are different.
+func (r Test) CompareJSON(want, have interface{}) Test {
+	switch w := want.(type) {
+	case map[string]interface{}:
+		return r.cmpJSONMaps(w, have)
+	case []interface{}:
+		return r.cmpJSONArrays(w, have)
+	default:
+		return r.fail(
+			"Key '%s' in expected output should be a map or a list of maps, but it's a %T",
+			r.Key, want)
+	}
+}
+
+func (r Test) cmpJSONMaps(want map[string]interface{}, have interface{}) Test {
+	haveMap, ok := have.(map[string]interface{})
+	if !ok {
+		return r.mismatch("table", want, haveMap)
+	}
+
+	// Check to make sure both or neither are values.
+	if isValue(want) && !isValue(haveMap) {
+		return r.fail(
+			"Key '%s' is supposed to be a value, but the parser reports it as a table",
+			r.Key)
+	}
+	if !isValue(want) && isValue(haveMap) {
+		return r.fail(
+			"Key '%s' is supposed to be a table, but the parser reports it as a value",
+			r.Key)
+	}
+	if isValue(want) && isValue(haveMap) {
+		return r.cmpJSONValues(want, haveMap)
+	}
+
+	// Check that the keys of each map are equivalent.
+	for k := range want {
+		if _, ok := haveMap[k]; !ok {
+			bunk := r.kjoin(k)
+			return bunk.fail("Could not find key '%s' in parser output.",
+				bunk.Key)
+		}
+	}
+	for k := range haveMap {
+		if _, ok := want[k]; !ok {
+			bunk := r.kjoin(k)
+			return bunk.fail("Could not find key '%s' in expected output.",
+				bunk.Key)
+		}
+	}
+
+	// Okay, now make sure that each value is equivalent.
+	for k := range want {
+		if sub := r.kjoin(k).CompareJSON(want[k], haveMap[k]); sub.Failed() {
+			return sub
+		}
+	}
+	return r
+}
+
+func (r Test) cmpJSONArrays(want, have interface{}) Test {
+	wantSlice, ok := want.([]interface{})
+	if !ok {
+		return r.bug("'value' should be a JSON array when 'type=array', but it is a %T", want)
+	}
+
+	haveSlice, ok := have.([]interface{})
+	if !ok {
+		return r.fail(
+			"Malformed output from your encoder: 'value' is not a JSON array: %T", have)
+	}
+
+	if len(wantSlice) != len(haveSlice) {
+		return r.fail("Array lengths differ for key '%s':\n"+
+			"  Expected:     %d\n"+
+			"  Your encoder: %d",
+			r.Key, len(wantSlice), len(haveSlice))
+	}
+	for i := 0; i < len(wantSlice); i++ {
+		if sub := r.CompareJSON(wantSlice[i], haveSlice[i]); sub.Failed() {
+			return sub
+		}
+	}
+	return r
+}
+
+func (r Test) cmpJSONValues(want, have map[string]interface{}) Test {
+	wantType, ok := want["type"].(string)
+	if !ok {
+		return r.bug("'type' should be a string, but it is a %T", want["type"])
+	}
+
+	haveType, ok := have["type"].(string)
+	if !ok {
+		return r.fail("Malformed output from your encoder: 'type' is not a string: %T", have["type"])
+	}
+
+	if wantType != haveType {
+		return r.valMismatch(wantType, haveType, want, have)
+	}
+
+	// If this is an array, then we've got to do some work to check equality.
+	if wantType == "array" {
+		return r.cmpJSONArrays(want, have)
+	}
+
+	// Atomic values are always strings
+	wantVal, ok := want["value"].(string)
+	if !ok {
+		return r.bug("'value' %v should be a string, but it is a %[1]T", want["value"])
+	}
+
+	haveVal, ok := have["value"].(string)
+	if !ok {
+		return r.fail("Malformed output from your encoder: %T is not a string", have["value"])
+	}
+
+	// Excepting floats and datetimes, other values can be compared as strings.
+	switch wantType {
+	case "float":
+		return r.cmpFloats(wantVal, haveVal)
+	case "datetime", "datetime-local", "date-local", "time-local":
+		return r.cmpAsDatetimes(wantType, wantVal, haveVal)
+	default:
+		return r.cmpAsStrings(wantVal, haveVal)
+	}
+}
+
+func (r Test) cmpAsStrings(want, have string) Test {
+	if want != have {
+		return r.fail("Values for key '%s' don't match:\n"+
+			"  Expected:     %s\n"+
+			"  Your encoder: %s",
+			r.Key, want, have)
+	}
+	return r
+}
+
+func (r Test) cmpFloats(want, have string) Test {
+	// Special case for NaN, since NaN != NaN.
+	if strings.HasSuffix(want, "nan") || strings.HasSuffix(have, "nan") {
+		if want != have {
+			return r.fail("Values for key '%s' don't match:\n"+
+				"  Expected:     %v\n"+
+				"  Your encoder: %v",
+				r.Key, want, have)
+		}
+		return r
+	}
+
+	wantF, err := strconv.ParseFloat(want, 64)
+	if err != nil {
+		return r.bug("Could not read '%s' as a float value for key '%s'", want, r.Key)
+	}
+
+	haveF, err := strconv.ParseFloat(have, 64)
+	if err != nil {
+		return r.fail("Malformed output from your encoder: key '%s' is not a float: '%s'", r.Key, have)
+	}
+
+	if wantF != haveF {
+		return r.fail("Values for key '%s' don't match:\n"+
+			"  Expected:     %v\n"+
+			"  Your encoder: %v",
+			r.Key, wantF, haveF)
+	}
+	return r
+}
+
+var datetimeRepl = strings.NewReplacer(
+	" ", "T",
+	"t", "T",
+	"z", "Z")
+
+var layouts = map[string]string{
+	"datetime":       time.RFC3339Nano,
+	"datetime-local": "2006-01-02T15:04:05.999999999",
+	"date-local":     "2006-01-02",
+	"time-local":     "15:04:05",
+}
+
+func (r Test) cmpAsDatetimes(kind, want, have string) Test {
+	layout, ok := layouts[kind]
+	if !ok {
+		panic("should never happen")
+	}
+
+	wantT, err := time.Parse(layout, datetimeRepl.Replace(want))
+	if err != nil {
+		return r.bug("Could not read '%s' as a datetime value for key '%s'", want, r.Key)
+	}
+
+	haveT, err := time.Parse(layout, datetimeRepl.Replace(want))
+	if err != nil {
+		return r.fail("Malformed output from your encoder: key '%s' is not a datetime: '%s'", r.Key, have)
+	}
+	if !wantT.Equal(haveT) {
+		return r.fail("Values for key '%s' don't match:\n"+
+			"  Expected:     %v\n"+
+			"  Your encoder: %v",
+			r.Key, wantT, haveT)
+	}
+	return r
+}
+
+func (r Test) kjoin(key string) Test {
+	if len(r.Key) == 0 {
+		r.Key = key
+	} else {
+		r.Key += "." + key
+	}
+	return r
+}
+
+func isValue(m map[string]interface{}) bool {
+	if len(m) != 2 {
+		return false
+	}
+	if _, ok := m["type"]; !ok {
+		return false
+	}
+	if _, ok := m["value"]; !ok {
+		return false
+	}
+	return true
+}
+
+func (r Test) mismatch(wantType string, want, have interface{}) Test {
+	return r.fail("Key '%s' is not an %s but %[4]T:\n"+
+		"  Expected:     %#[3]v\n"+
+		"  Your encoder: %#[4]v",
+		r.Key, wantType, want, have)
+}
+
+func (r Test) valMismatch(wantType, haveType string, want, have interface{}) Test {
+	return r.fail("Key '%s' is not an %s but %s:\n"+
+		"  Expected:     %#[3]v\n"+
+		"  Your encoder: %#[4]v",
+		r.Key, wantType, want, have)
+}
@@ -0,0 +1,427 @@
+//go:generate ./gen-multi.py
+
+//go:build go1.16
+// +build go1.16
+
+package tomltest
+
+import (
+	"bytes"
+	"embed"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"io/fs"
+	"os/exec"
+	"path/filepath"
+	"sort"
+	"strings"
+
+	"github.com/BurntSushi/toml"
+)
+
+type testType uint8
+
+const (
+	TypeValid testType = iota
+	TypeInvalid
+)
+
+//go:embed tests/*
+var embeddedTests embed.FS
+
+// EmbeddedTests are the tests embedded in toml-test, rooted to the "test/"
+// directory.
+func EmbeddedTests() fs.FS {
+	f, err := fs.Sub(embeddedTests, "tests")
+	if err != nil {
+		panic(err)
+	}
+	return f
+}
+
+// Runner runs a set of tests.
+//
+// The validity of the parameters is not checked extensively; the caller should
+// verify this if need be. See ./cmd/toml-test for an example.
+type Runner struct {
+	Files     fs.FS    // Test files.
+	Encoder   bool     // Are we testing an encoder?
+	RunTests  []string // Tests to run; run all if blank.
+	SkipTests []string // Tests to skip.
+	Parser    Parser   // Send data to a parser.
+	Version   string   // TOML version to run tests for.
+}
+
+// A Parser instance is used to call the TOML parser we test.
+//
+// By default this is done through an external command.
+type Parser interface {
+	// Encode a JSON string to TOML.
+	//
+	// The output is the TOML string; if outputIsError is true then it's assumed
+	// that an encoding error occurred.
+	//
+	// An error return should only be used in case an unrecoverable error
+	// occurred; failing to encode to TOML is not an error, but the encoder
+	// unexpectedly panicking is.
+	Encode(jsonInput string) (output string, outputIsError bool, err error)
+
+	// Decode a TOML string to JSON. The same semantics as Encode apply.
+	Decode(tomlInput string) (output string, outputIsError bool, err error)
+}
+
+// CommandParser calls an external command.
+type CommandParser struct {
+	fsys fs.FS
+	cmd  []string
+}
+
+// Tests are tests to run.
+type Tests struct {
+	Tests []Test
+
+	// Set when test are run.
+
+	Skipped, Passed, Failed int
+}
+
+// Result is the result of a single test.
+type Test struct {
+	Path string // Path of test, e.g. "valid/string-test"
+
+	// Set when a test is run.
+
+	Skipped          bool   // Skipped this test?
+	Failure          string // Failure message.
+	Key              string // TOML key the failure occured on; may be blank.
+	Encoder          bool   // Encoder test?
+	Input            string // The test case that we sent to the external program.
+	Output           string // Output from the external program.
+	Want             string // The output we want.
+	OutputFromStderr bool   // The Output came from stderr, not stdout.
+}
+
+// List all tests in Files for the current TOML version.
+func (r Runner) List() ([]string, error) {
+	if r.Version == "" {
+		r.Version = "1.0.0"
+	}
+	if _, ok := versions[r.Version]; !ok {
+		v := make([]string, 0, len(versions))
+		for k := range versions {
+			v = append(v, k)
+		}
+		sort.Strings(v)
+		return nil, fmt.Errorf("tomltest.Runner.Run: unknown version: %q (supported: \"%s\")",
+			r.Version, strings.Join(v, `", "`))
+	}
+
+	var (
+		v       = versions[r.Version]
+		exclude = make([]string, 0, 8)
+	)
+	for {
+		exclude = append(exclude, v.exclude...)
+		if v.inherit == "" {
+			break
+		}
+		v = versions[v.inherit]
+	}
+
+	ls := make([]string, 0, 256)
+	if err := r.findTOML("valid", &ls, exclude); err != nil {
+		return nil, fmt.Errorf("reading 'valid/' dir: %w", err)
+	}
+
+	d := "invalid" + map[bool]string{true: "-encoder", false: ""}[r.Encoder]
+	if err := r.findTOML(d, &ls, exclude); err != nil {
+		return nil, fmt.Errorf("reading %q dir: %w", d, err)
+	}
+
+	return ls, nil
+}
+
+// Run all tests listed in t.RunTests.
+//
+// TODO: give option to:
+// - Run all tests with \n replaced with \r\n
+// - Run all tests with EOL removed
+// - Run all tests with '# comment' appended to every line.
+func (r Runner) Run() (Tests, error) {
+	skipped, err := r.findTests()
+	if err != nil {
+		return Tests{}, fmt.Errorf("tomltest.Runner.Run: %w", err)
+	}
+
+	tests := Tests{Tests: make([]Test, 0, len(r.RunTests)), Skipped: skipped}
+	for _, p := range r.RunTests {
+		if r.hasSkip(p) {
+			tests.Skipped++
+			tests.Tests = append(tests.Tests, Test{Path: p, Skipped: true, Encoder: r.Encoder})
+			continue
+		}
+
+		t := Test{Path: p, Encoder: r.Encoder}.Run(r.Parser, r.Files)
+		tests.Tests = append(tests.Tests, t)
+
+		if t.Failed() {
+			tests.Failed++
+		} else {
+			tests.Passed++
+		}
+	}
+
+	return tests, nil
+}
+
+// find all TOML files in 'path' relative to the test directory.
+func (r Runner) findTOML(path string, appendTo *[]string, exclude []string) error {
+	// It's okay if the directory doesn't exist.
+	if _, err := fs.Stat(r.Files, path); errors.Is(err, fs.ErrNotExist) {
+		return nil
+	}
+
+	return fs.WalkDir(r.Files, path, func(path string, d fs.DirEntry, err error) error {
+		if err != nil {
+			return err
+		}
+		if d.IsDir() || !strings.HasSuffix(path, ".toml") {
+			return nil
+		}
+
+		path = strings.TrimSuffix(path, ".toml")
+		for _, e := range exclude {
+			if ok, _ := filepath.Match(e, path); ok {
+				return nil
+			}
+		}
+
+		*appendTo = append(*appendTo, path)
+		return nil
+	})
+}
+
+// Expand RunTest glob patterns, or return all tests if RunTests if empty.
+func (r *Runner) findTests() (int, error) {
+	ls, err := r.List()
+	if err != nil {
+		return 0, err
+	}
+
+	var skip int
+
+	if len(r.RunTests) == 0 {
+		r.RunTests = ls
+	} else {
+		run := make([]string, 0, len(r.RunTests))
+		for _, l := range ls {
+			for _, r := range r.RunTests {
+				if m, _ := filepath.Match(r, l); m {
+					run = append(run, l)
+					break
+				}
+			}
+		}
+		r.RunTests, skip = run, len(ls)-len(run)
+	}
+
+	// Expand invalid tests ending in ".multi.toml"
+	expanded := make([]string, 0, len(r.RunTests))
+	for _, path := range r.RunTests {
+		if !strings.HasSuffix(path, ".multi") {
+			expanded = append(expanded, path)
+			continue
+		}
+
+		d, err := fs.ReadFile(r.Files, path+".toml")
+		if err != nil {
+			return 0, err
+		}
+
+		fmt.Println(string(d))
+	}
+	r.RunTests = expanded
+
+	return skip, nil
+}
+
+func (r Runner) hasSkip(path string) bool {
+	for _, s := range r.SkipTests {
+		if m, _ := filepath.Match(s, path); m {
+			return true
+		}
+	}
+	return false
+}
+
+func (c CommandParser) Encode(input string) (output string, outputIsError bool, err error) {
+	stdout, stderr := new(bytes.Buffer), new(bytes.Buffer)
+	cmd := exec.Command(c.cmd[0])
+	cmd.Args = c.cmd
+	cmd.Stdin, cmd.Stdout, cmd.Stderr = strings.NewReader(input), stdout, stderr
+
+	err = cmd.Run()
+	if err != nil {
+		eErr := &exec.ExitError{}
+		if errors.As(err, &eErr) {
+			fmt.Fprintf(stderr, "\nExit %d\n", eErr.ProcessState.ExitCode())
+			err = nil
+		}
+	}
+
+	if stderr.Len() > 0 {
+		return strings.TrimSpace(stderr.String()) + "\n", true, err
+	}
+	return strings.TrimSpace(stdout.String()) + "\n", false, err
+}
+func NewCommandParser(fsys fs.FS, cmd []string) CommandParser     { return CommandParser{fsys, cmd} }
+func (c CommandParser) Decode(input string) (string, bool, error) { return c.Encode(input) }
+
+// Run this test.
+func (t Test) Run(p Parser, fsys fs.FS) Test {
+	if t.Type() == TypeInvalid {
+		return t.runInvalid(p, fsys)
+	}
+	return t.runValid(p, fsys)
+}
+
+func (t Test) runInvalid(p Parser, fsys fs.FS) Test {
+	var err error
+	_, t.Input, err = t.ReadInput(fsys)
+	if err != nil {
+		return t.bug(err.Error())
+	}
+
+	if t.Encoder {
+		t.Output, t.OutputFromStderr, err = p.Encode(t.Input)
+	} else {
+		t.Output, t.OutputFromStderr, err = p.Decode(t.Input)
+	}
+	if err != nil {
+		return t.fail(err.Error())
+	}
+	if !t.OutputFromStderr {
+		return t.fail("Expected an error, but no error was reported.")
+	}
+	return t
+}
+
+func (t Test) runValid(p Parser, fsys fs.FS) Test {
+	var err error
+	_, t.Input, err = t.ReadInput(fsys)
+	if err != nil {
+		return t.bug(err.Error())
+	}
+
+	if t.Encoder {
+		t.Output, t.OutputFromStderr, err = p.Encode(t.Input)
+	} else {
+		t.Output, t.OutputFromStderr, err = p.Decode(t.Input)
+	}
+	if err != nil {
+		return t.fail(err.Error())
+	}
+	if t.OutputFromStderr {
+		return t.fail(t.Output)
+	}
+	if t.Output == "" {
+		// Special case: we expect an empty output here.
+		if t.Path != "valid/empty-file" {
+			return t.fail("stdout is empty")
+		}
+	}
+
+	// Compare for encoder test
+	if t.Encoder {
+		want, err := t.ReadWantTOML(fsys)
+		if err != nil {
+			return t.bug(err.Error())
+		}
+		var have interface{}
+		if _, err := toml.Decode(t.Output, &have); err != nil {
+			//return t.fail("decode TOML from encoder %q:\n  %s", cmd, err)
+			return t.fail("decode TOML from encoder:\n  %s", err)
+		}
+		return t.CompareTOML(want, have)
+	}
+
+	// Compare for decoder test
+	want, err := t.ReadWantJSON(fsys)
+	if err != nil {
+		return t.fail(err.Error())
+	}
+
+	var have interface{}
+	if err := json.Unmarshal([]byte(t.Output), &have); err != nil {
+		return t.fail("decode JSON output from parser:\n  %s", err)
+	}
+
+	return t.CompareJSON(want, have)
+}
+
+// ReadInput reads the file sent to the encoder.
+func (t Test) ReadInput(fsys fs.FS) (path, data string, err error) {
+	path = t.Path + map[bool]string{true: ".json", false: ".toml"}[t.Encoder]
+	d, err := fs.ReadFile(fsys, path)
+	if err != nil {
+		return path, "", err
+	}
+	return path, string(d), nil
+}
+
+func (t Test) ReadWant(fsys fs.FS) (path, data string, err error) {
+	if t.Type() == TypeInvalid {
+		panic("testoml.Test.ReadWant: invalid tests do not have a 'correct' version")
+	}
+
+	path = t.Path + map[bool]string{true: ".toml", false: ".json"}[t.Encoder]
+	d, err := fs.ReadFile(fsys, path)
+	if err != nil {
+		return path, "", err
+	}
+	return path, string(d), nil
+}
+
+func (t *Test) ReadWantJSON(fsys fs.FS) (v interface{}, err error) {
+	var path string
+	path, t.Want, err = t.ReadWant(fsys)
+	if err != nil {
+		return nil, err
+	}
+
+	if err := json.Unmarshal([]byte(t.Want), &v); err != nil {
+		return nil, fmt.Errorf("decode JSON file %q:\n  %s", path, err)
+	}
+	return v, nil
+}
+func (t *Test) ReadWantTOML(fsys fs.FS) (v interface{}, err error) {
+	var path string
+	path, t.Want, err = t.ReadWant(fsys)
+	if err != nil {
+		return nil, err
+	}
+	_, err = toml.Decode(t.Want, &v)
+	if err != nil {
+		return nil, fmt.Errorf("could not decode TOML file %q:\n  %s", path, err)
+	}
+	return v, nil
+}
+
+// Test type: "valid", "invalid"
+func (t Test) Type() testType {
+	if strings.HasPrefix(t.Path, "invalid") {
+		return TypeInvalid
+	}
+	return TypeValid
+}
+
+func (t Test) fail(format string, v ...interface{}) Test {
+	t.Failure = fmt.Sprintf(format, v...)
+	return t
+}
+func (t Test) bug(format string, v ...interface{}) Test {
+	return t.fail("BUG IN TEST CASE: "+format, v...)
+}
+
+func (t Test) Failed() bool { return t.Failure != "" }
@@ -0,0 +1 @@
+*.toml  -text
@@ -0,0 +1 @@
+array = [1,,2]
@@ -0,0 +1,2 @@
+array = [1,2,,]
+
@@ -0,0 +1,6 @@
+a = [{ b = 1 }]
+
+# Cannot extend tables within static arrays
+# https://github.com/toml-lang/toml/issues/908
+[a.c]
+foo = 1
@@ -0,0 +1 @@
+wrong = [ 1 2 3 ]
@@ -0,0 +1 @@
+x = [42 #
@@ -0,0 +1 @@
+x = [{ key = 42 #
@@ -0,0 +1 @@
+x = [{ key = 42
@@ -0,0 +1 @@
+long_array = [ 1, 2, 3
@@ -0,0 +1,4 @@
+# INVALID TOML DOC
+fruit = []
+
+[[fruit]] # Not allowed
@@ -0,0 +1,10 @@
+# INVALID TOML DOC
+[[fruit]]
+  name = "apple"
+
+  [[fruit.variety]]
+    name = "red delicious"
+
+  # This table conflicts with the previous table
+  [fruit.variety]
+    name = "granny smith"
@@ -0,0 +1,4 @@
+array = [
+  "Is there life after an array separator?", No
+  "Entry"
+]
@@ -0,0 +1,4 @@
+array = [
+  "Is there life before an array separator?" No,
+  "Entry"
+]
@@ -0,0 +1,5 @@
+array = [
+  "Entry 1",
+  I don't belong,
+  "Entry 2",
+]
@@ -0,0 +1 @@
+a = falsify
@@ -0,0 +1 @@
+a = fals
@@ -0,0 +1 @@
+a = truthy
@@ -0,0 +1 @@
+a = tru
@@ -0,0 +1 @@
+a = f
@@ -0,0 +1 @@
+a = t
@@ -0,0 +1 @@
+valid = False
@@ -0,0 +1 @@
+a = falsey
@@ -0,0 +1 @@
+a = truer
@@ -0,0 +1 @@
+b = FALSE
@@ -0,0 +1 @@
+a = TRUE
@@ -0,0 +1,2 @@
+# The following line contains a single carriage return control character
+
@@ -0,0 +1 @@
+bare-formfeed = 
@@ -0,0 +1 @@
+bare-vertical-tab = 
@@ -0,0 +1 @@
+comment-cr = "Carriage return in comment" # 
@@ -0,0 +1 @@
+comment-del = "0x7f" # 
@@ -0,0 +1 @@
+comment-lf = "ctrl-P" # 
@@ -0,0 +1 @@
+comment-us = "ctrl-_" # 
@@ -0,0 +1,33 @@
+# "\x.." sequences are replaced with literal control characters.
+
+comment-null = "null" # \x00
+comment-lf = "ctrl-P" # \x10
+comment-us = "ctrl-_" # \x1f
+comment-del = "0x7f" # \x7f
+comment-cr = "Carriage return in comment" # \x0da=1
+
+string-null = "null\x00"
+string-lf = "null\x10"
+string-us = "null\x1f"
+string-del = "null\x7f"
+
+rawstring-null = 'null\x00'
+rawstring-lf = 'null\x10'
+rawstring-us = 'null\x1f'
+rawstring-del = 'null\x7f'
+
+multi-null = """null\x00"""
+multi-lf = """null\x10"""
+multi-us = """null\x1f"""
+multi-del = """null\x7f"""
+
+rawmulti-null = '''null\x00'''
+rawmulti-lf = '''null\x10'''
+rawmulti-us = '''null\x1f'''
+rawmulti-del = '''null\x7f'''
+
+string-bs = "backspace\x08"
+
+bare-null = "some value" \x00
+bare-formfeed = \x0c
+bare-vertical-tab = \x0b
@@ -0,0 +1 @@
+multi-del = """null"""
@@ -0,0 +1 @@
+multi-lf = """null"""
@@ -0,0 +1 @@
+multi-us = """null"""
@@ -0,0 +1 @@
+rawmulti-del = '''null'''
@@ -0,0 +1 @@
+rawmulti-lf = '''null'''
@@ -0,0 +1 @@
+rawmulti-us = '''null'''
@@ -0,0 +1 @@
+rawstring-del = 'null'
@@ -0,0 +1 @@
+rawstring-lf = 'null'
@@ -0,0 +1 @@
+rawstring-us = 'null'
@@ -0,0 +1 @@
+string-bs = "backspace"
@@ -0,0 +1 @@
+string-del = "null"
@@ -0,0 +1 @@
+string-lf = "null"
@@ -0,0 +1 @@
+string-us = "null"
@@ -0,0 +1,2 @@
+# time-hour       = 2DIGIT  ; 00-23
+d = 2006-01-01T24:00:00-00:00
@@ -0,0 +1,3 @@
+# date-mday       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on
+#                           ; month/year
+d = 2006-01-32T00:00:00-00:00
@@ -0,0 +1,3 @@
+# date-mday       = 2DIGIT  ; 01-28, 01-29, 01-30, 01-31 based on
+#                           ; month/year
+d = 2006-01-00T00:00:00-00:00
@@ -0,0 +1,2 @@
+# time-minute     = 2DIGIT  ; 00-59
+d = 2006-01-01T00:60:00-00:00
@@ -0,0 +1,2 @@
+# date-month      = 2DIGIT  ; 01-12
+d = 2006-13-01T00:00:00-00:00
@@ -0,0 +1,2 @@
+# date-month      = 2DIGIT  ; 01-12
+d = 2007-00-01T00:00:00-00:00
@@ -0,0 +1,2 @@
+# Day "5" instead of "05"; the leading zero is required.
+with-milli = 1987-07-5T17:45:00.12Z
@@ -0,0 +1,2 @@
+# Month "7" instead of "07"; the leading zero is required.
+no-leads = 1987-7-05T17:45:00Z
@@ -0,0 +1,2 @@
+# No seconds in time.
+no-secs = 1987-07-05T17:45Z
@@ -0,0 +1,2 @@
+# No "t" or "T" between the date and time.
+no-t = 1987-07-0517:45:00Z
@@ -0,0 +1,3 @@
+# time-second     = 2DIGIT  ; 00-58, 00-59, 00-60 based on leap second
+#                           ; rules
+d = 2006-01-01T00:00:61-00:00
@@ -0,0 +1,2 @@
+# Leading 0 is always required.
+d = 01:32:0
@@ -0,0 +1,2 @@
+# Leading 0 is always required.
+d = 1:32:00
@@ -0,0 +1,2 @@
+# Date cannot end with trailing T
+d = 2006-01-30T
@@ -0,0 +1,5 @@
+# There is a 0xda at after the quotes, and no EOL at the end of the file.
+#
+# This is a bit of an edge case: This indicates there should be two bytes
+# (0b1101_1010) but there is no byte to follow because it's the end of the file.
+x = """"""�
@@ -0,0 +1 @@
+# �
@@ -0,0 +1,2 @@
+# The following line contains an invalid UTF-8 sequence.
+bad = "Ã"
@@ -0,0 +1 @@
+bom-not-at-start ÿý
@@ -0,0 +1 @@
+bom-not-at-start= ÿý
@@ -0,0 +1 @@
+double-point-1 = 0..1
@@ -0,0 +1 @@
+double-point-2 = 0.1.2
@@ -0,0 +1 @@
+exp-double-e-1 = 1ee2
@@ -0,0 +1 @@
+exp-double-e-2 = 1e2e3
@@ -0,0 +1 @@
+exp-double-us = 1e__23
@@ -0,0 +1 @@
+exp-leading-us = 1e_23
@@ -0,0 +1 @@
+exp-point-1 = 1e2.3
@@ -0,0 +1 @@
+exp-point-2 = 1.e2
@@ -0,0 +1 @@
+exp-trailing-us = 1e_23_
@@ -0,0 +1,40 @@
+leading-zero = 03.14
+leading-zero-neg = -03.14
+leading-zero-plus = +03.14
+
+leading-point = .12345
+leading-point-neg = -.12345
+leading-point-plus = +.12345
+
+trailing-point = 1.
+trailing-point-min = -1.
+trailing-point-plus = +1.
+
+trailing-us = 1.2_
+leading-us = _1.2
+us-before-point = 1_.2
+us-after-point = 1._2
+
+double-point-1 = 0..1
+double-point-2 = 0.1.2
+
+exp-point-1 = 1e2.3
+exp-point-2 = 1.e2
+
+exp-double-e-1 = 1ee2
+exp-double-e-2 = 1e2e3
+
+exp-leading-us = 1e_23
+exp-trailing-us = 1e_23_
+exp-double-us = 1e__23
+
+inf-incomplete-1 = in
+inf-incomplete-2 = +in
+inf-incomplete-3 = -in
+
+nan-incomplete-1 = na
+nan-incomplete-2 = +na
+nan-incomplete-3 = -na
+
+nan_underscore = na_n
+inf_underscore = in_f
@@ -0,0 +1 @@
+inf-incomplete-1 = in
@@ -0,0 +1 @@
+inf-incomplete-2 = +in
@@ -0,0 +1 @@
+inf-incomplete-3 = -in
@@ -0,0 +1 @@
+inf_underscore = in_f
@@ -0,0 +1 @@
+leading-point-neg = -.12345
@@ -0,0 +1 @@
+leading-point-plus = +.12345
@@ -0,0 +1 @@
+leading-point = .12345
@@ -0,0 +1 @@
+leading-us = _1.2
@@ -0,0 +1 @@
+leading-zero-neg = -03.14
@@ -0,0 +1 @@
+leading-zero-plus = +03.14
@@ -0,0 +1 @@
+leading-zero = 03.14
@@ -0,0 +1 @@
+nan-incomplete-1 = na
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`build_flags="-trimpath -ldflags '-w -s -X \"zgo.at/zli.version=$tag$commit_info\" -X \"zgo.at/zli.progname=toml-test\"' ./cmd/toml-test"`
				`@@ -0,0 +1,2 @@`
				`# The following line contains a single carriage return control character`
				`@@ -0,0 +1 @@`
				`comment-cr = "Carriage return in comment" #`