Simple test framework

Split out test cases which should pass, from those which
should fail (one code per folder - hopefully we can continue
to maintain separate examples like this - does not yet cater
to a test case with multiple error codes expected.

Author: peterjc

.travis.yml

@@ -34,8 +34,8 @@ script:
  - echo "On this machine \$LANG=$LANG"
  - echo "Checking our own docstrings are valid RST"
  - flake8 --select RST setup.py flake8_rst_docstrings.py
- - echo "Checking we can parse our valid test cases"
- - flake8 --select RST --ignore RST303,RST304 tests/test_cases/*.py
+ - cd tests/
+ - ./run_tests.sh
 
 notifications:
   email: false

tests/test_cases/sphinx-needs.py tests/RST303/sphinx-directives.pytests/test_cases/sphinx-needs.py renamed to tests/RST303/sphinx-directives.py

@@ -1,6 +1,8 @@
 """Example reStructuredText from Sphinx-Needs project.
 
 From http://sphinxcontrib-needs.readthedocs.io/en/latest/
+but cut down and may not work in isolation. Intended just
+to trigger RST303.
 
 **Some data**
 
@@ -24,11 +26,6 @@
 
 **Some text**
 
-Wohooo, we have created :need:`req_001`,
-which is linked by :need_incoming:`req_001`.
-
-**A filter**
-
 .. needfilter::
    :tags: example
    :layout: table

tests/RST304/sphinx-roles.py

@@ -0,0 +1,14 @@
+"""Example reStructuredText from Sphinx-Needs project.
+
+From http://sphinxcontrib-needs.readthedocs.io/en/latest/
+but will not work in isolation - cut down just to trigger
+RST304.
+
+**Some text**
+
+Wohooo, we have created :need:`req_001`,
+which is linked by :need_incoming:`req_001`.
+
+"""
+
+print("sphinx-needs defines its own reStructuredText roles.")

tests/run_tests.sh

@@ -0,0 +1,27 @@
+#!/bin/bash
+IFS=$'\n\t'
+set -eu
+# Note not using "set -o pipefile" until after check error message with grep
+
+# Examples which should fail
+for code in RST??? ; do
+    echo "======"
+    echo $code
+    echo "======"
+    flake8 --select RST $code/ 2>&1 | grep $code
+    echo "Return code $? from $code tests"
+done
+# echo "Positive tests passed (RST errors reported as expected)."
+
+set -o pipefail
+
+# Examples which should pass
+echo "========="
+echo "Negatives"
+echo "========="
+flake8 --select RST test_cases/
+#echo "Negative tests passed (no RST errors reported as expected)."
+
+echo "============"
+echo "Tests passed"
+echo "============"