Skip to content

Commit bdd2151

Browse files
PatrickRobbIOLz46-dev
PatrickRobbIOL
and
z46-dev
authored
Feature/add testing framework (#59)
* Upload initial work to repository * Start working on workflow and bash scripting * Continue work on beginTests.sh script * Finish script to run tests * Add initial github actions tests.yml workflow * Add README * ocre-tests: add readme describing the ocre tests Adds a readme file which describes the Ocre testing framework, how to add new tests, and gives an example of valid config.json. Signed-off-by: Patrick Robb <probb@iol.unh.edu> * ocre-tests: add flash validation test cases Add flash validation test cases, and remove placeholder testcases. Signed-off-by: Patrick Robb <probb@iol.unh.edu> * ocre-tests: add test framework and flash validation test group Signed-off-by: Evan Parker <eparker@iol.unh.edu> Signed-off-by: Patrick Robb <probb@iol.unh.edu> * commit to be squashed * commit to be squashed --------- Signed-off-by: Patrick Robb <probb@iol.unh.edu> Signed-off-by: Evan Parker <eparker@iol.unh.edu> Co-authored-by: Evan Parker <eparker@iol.unh.edu>
1 parent 62cc42b commit bdd2151

9 files changed

Lines changed: 350 additions & 0 deletions

File tree

‎.github/workflows/tests.yml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
name: Tests
2+
on:
3+
push:
4+
branches:
5+
- main
6+
pull_request:
7+
branches:
8+
- main
9+
jobs:
10+
tests:
11+
runs-on: zephyr-xlarge-runner
12+
13+
steps:
14+
- name: Make the github actions runner recursively take ownership of the github workspace
15+
run: sudo chown -R $(whoami) ${{ github.workspace }}
16+
17+
- name: Checkout
18+
uses: actions/checkout@v4
19+
20+
- name: Flash Validation Tests
21+
run: |
22+
cd tests && bash beginTests.sh "flashValidation"
23+
24+
- name: Print Flash Validation Logs
25+
if: always()
26+
run: cat /tmp/flashValidation.log
27+
28+
- name: Upload log file as artifact
29+
if: always()
30+
uses: actions/upload-artifact@v4
31+
with:
32+
name: "FlashValidation.log"
33+
path: /tmp/flashValidation.log

‎tests/README.md‎

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
## Overview
2+
3+
In the Ocre testing framework, the Ocre execution is comprised of test groups, test suites, and test cases.
4+
5+
Test groups represent a group of test suites which have the same environmental prerequisites, and require a
6+
common setup and teardown process, e.g. the Flash Validation test group requires that the DUT board has been
7+
flashed with the Ocre runtime. Test groups will block each other on failure, meaning that if a test group
8+
fails, all subsequent test groups will be skipped.
9+
10+
Test suites act as a way to logically split up test cases. For instance, in a test group for container workloads,
11+
Test suite A might contain testcases for single container workloads, and test suite B might contain testcases for
12+
multi-container workloads. Test suites are non blocking on one another on failure.
13+
14+
Test cases are the individual tests, and are literally a python or bash script (or any other language) which runs
15+
some arbitrary code according to the test plan, and then exits / returns an exit code of 0 on success and 1 on
16+
failure.
17+
18+
## Adding Test Groups, Test Suites, and Test Cases.
19+
20+
To add a test group, you must create a new directory under the /tests/ dir, named after the new test group. Then,
21+
add a config.json file in the directory according to the format given below. Finally, add a step to the
22+
/.github/workflows/tests.yml file which runs the beginTests.sh against your new test group.
23+
24+
To add a new testcase, the only requirement is that the testcase script file be placed under a testgroup dir, and
25+
that it exits with 0 on success and 1 on failure.
26+
27+
28+
```json
29+
{
30+
"name": "Name of test group",
31+
"description": "Helpful description of the test group",
32+
"setup": [
33+
{
34+
"name": "Name of script",
35+
"exec": "execution command"
36+
}
37+
],
38+
"test_suites": [
39+
{
40+
"name": "Name of test suite",
41+
"description": "Quick description",
42+
"test_cases": [
43+
{
44+
"name": "Name of test case",
45+
"description": "Quick description",
46+
"exec": "execution command"
47+
}
48+
]
49+
}
50+
],
51+
"cleanup": [
52+
{
53+
"name": "Name of script",
54+
"exec": "execution command"
55+
}
56+
]
57+
}
58+
```
59+
60+
Execution commands are run with `bash -c "execution command"`.

‎tests/beginTests.sh‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
#!/bin/bash
2+
3+
TG=$1
4+
LOGFILE="/tmp/$TG.log"
5+
6+
clStr() {
7+
ARG=$1
8+
echo "${ARG:1:-1}"
9+
}
10+
11+
CONF=groups/$TG/config.json
12+
NAME=$(clStr "$(cat $CONF | jq .name)")
13+
DESCRIPTION=$(clStr "$(cat $CONF | jq .description)")
14+
15+
rm $LOGFILE > /dev/null 2>&1
16+
touch $LOGFILE
17+
echo "Beginning test group $NAME" >> $LOGFILE
18+
echo $DESCRIPTION >> $LOGFILE
19+
echo >> $LOGFILE
20+
21+
parseJSON() {
22+
echo $(cat $CONF | jq .$1)
23+
}
24+
25+
parseJSON2() {
26+
echo $(echo $1 | jq .$2)
27+
}
28+
29+
parseJSONArray() {
30+
echo $(cat $CONF | jq -r ".$1[] | @base64")
31+
}
32+
33+
parseJSONArray2() {
34+
echo $(echo $1 | jq -r ".$2[] | @base64")
35+
}
36+
37+
printHeader() {
38+
LEN=${#1}
39+
FORMAT=""
40+
for ((i = 1; i <= $LEN; i ++)); do
41+
FORMAT+="="
42+
done
43+
44+
echo $1 >> $LOGFILE
45+
echo $FORMAT >> $LOGFILE
46+
echo >> $LOGFILE
47+
}
48+
49+
CWD=$(pwd)
50+
51+
# Setup
52+
echo "Entering Setup..." >> $LOGFILE
53+
echo >> $LOGFILE
54+
for RAW_ROW in $(parseJSONArray "setup"); do
55+
ROW=$(echo $RAW_ROW | base64 -di)
56+
printHeader "Script: $(clStr "$(parseJSON2 "$ROW" "name")")"
57+
cd groups/$TG
58+
bash -c "bash -c $(parseJSON2 "$ROW" "exec")" &>> $LOGFILE
59+
cd $CWD
60+
echo >> $LOGFILE
61+
done
62+
63+
# Tests
64+
TEST_GROUP_RESULT=0
65+
echo "Entering Testing..." >> $LOGFILE
66+
echo >> $LOGFILE
67+
for RAW_ROW in $(parseJSONArray "test_suites"); do
68+
ROW=$(echo $RAW_ROW | base64 -di)
69+
70+
if [[ "$(parseJSON2 "$ROW" "test_cases")" != "null" ]]; then
71+
printHeader "Test Suite: $(clStr "$(parseJSON2 "$ROW" "name")")" >> $LOGFILE
72+
for RAW_TEST_ROW in $(parseJSONArray2 "$ROW" "test_cases"); do
73+
TEST_ROW=$(echo $RAW_TEST_ROW | base64 -di)
74+
75+
if [[ "$(parseJSON2 "$TEST_ROW" "exec")" != "null" ]]; then
76+
printHeader "Test Case: $(clStr "$(parseJSON2 "$TEST_ROW" "name")")" >> $LOGFILE
77+
cd groups/$TG
78+
bash -c "bash -c $(parseJSON2 "$TEST_ROW" "exec")" &>> $LOGFILE
79+
TESTCASE_RESULT=$?
80+
cd $CWD
81+
echo >> $LOGFILE
82+
if [ "$TESTCASE_RESULT" -eq 0 ]; then
83+
echo $'TEST PASSED\n' >> $LOGFILE
84+
else
85+
echo $'TEST FAILED\n' >> $LOGFILE
86+
TEST_GROUP_RESULT=1
87+
fi
88+
else
89+
echo "Could not figure out what to do with test block $(parseJSON2 "$TEST_ROW" "name")" >> $LOGFILE
90+
exit 1
91+
fi
92+
done
93+
else
94+
echo "Could not figure out what to do with test block $(parseJSON2 "$ROW" "name")" >> $LOGFILE
95+
exit 1
96+
fi
97+
done
98+
99+
# Setup
100+
echo "Entering Cleanup..." >> $LOGFILE
101+
echo >> $LOGFILE
102+
for RAW_ROW in $(parseJSONArray "cleanup"); do
103+
ROW=$(echo $RAW_ROW | base64 -di)
104+
printHeader "Script: $(clStr "$(parseJSON2 "$ROW" "name")")"
105+
cd groups/$TG
106+
bash -c "bash -c $(parseJSON2 "$ROW" "exec")" &>> $LOGFILE
107+
cd $CWD
108+
echo >> $LOGFILE
109+
done
110+
111+
echo "Test suite is complete" >> $LOGFILE
112+
113+
exit $TEST_GROUP_RESULT

‎tests/groups/flashValidation/clean.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
echo "Cleanup is complete"

‎tests/groups/flashValidation/config.json‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
{
2+
"name": "Flash Validation",
3+
"description": "Test the Ocre runtime after the flash is complete",
4+
"setup": [
5+
{
6+
"name": "Flash Validation Setup",
7+
"exec": "bash setup.sh"
8+
}
9+
],
10+
"test_suites": [
11+
{
12+
"name": "Runtime Validation Tests",
13+
"description": "Sends a break to the runtime and checks for the expected output",
14+
"test_cases": [
15+
{
16+
"name": "Check Runtime Hello World",
17+
"exec": "./flash_validation_hello_world.py"
18+
},
19+
{
20+
"name": "Check Runetime Error",
21+
"exec": "./flash_validation_error.py"
22+
},
23+
{
24+
"name": "Check Runtime Hello World With Error",
25+
"exec": "./flash_validation_hello_world_error.py"
26+
}
27+
]
28+
}
29+
],
30+
"cleanup": [
31+
{
32+
"name": "Flash Validation Cleanup",
33+
"exec": "bash clean.sh"
34+
}
35+
]
36+
}

‎tests/groups/flashValidation/flash_validation_error.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/usr/bin/env python3
2+
3+
import serial
4+
import time
5+
import sys
6+
7+
"""
8+
This testcase is to be used following the flashing of the Ocre runtime to a board.
9+
10+
The testcase forms a serial connection to the board, sends a break and checks that
11+
there are no instance of the error prefix "E: " in the output returned from the break.
12+
"""
13+
14+
def main():
15+
conn = serial.Serial('/dev/ttyACM0', 115200, timeout=1)
16+
conn.send_break(duration=1)
17+
18+
time.sleep(1)
19+
20+
print('**** READING RESPONSE FROM BREAK ****\n')
21+
22+
response = conn.read(1024).decode(errors='ignore')
23+
print(response)
24+
25+
print('**** CLOSING CONNECTION ****\n')
26+
27+
conn.close()
28+
29+
if "E:" not in response:
30+
sys.exit(0)
31+
sys.exit(1)
32+
33+
34+
if __name__ == "__main__":
35+
main()

‎tests/groups/flashValidation/flash_validation_hello_world.py‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
#!/usr/bin/env python3
2+
3+
import serial
4+
import time
5+
import sys
6+
7+
"""
8+
This testcase is to be used following the flashing of the Ocre runtime to a board.
9+
10+
The testcase forms a serial connection to the board, sends a break and checks that
11+
the string Hello World! appears in the output of that break command.
12+
"""
13+
14+
def main():
15+
conn = serial.Serial('/dev/ttyACM0', 115200, timeout=1)
16+
conn.send_break(duration=1)
17+
18+
time.sleep(1)
19+
20+
print('**** READING RESPONSE FROM BREAK ****\n')
21+
22+
response = conn.read(1024).decode(errors='ignore')
23+
print(response)
24+
25+
print('**** CLOSING CONNECTION ****\n')
26+
27+
conn.close()
28+
29+
if "Hello World!" in response:
30+
sys.exit(0)
31+
sys.exit(1)
32+
33+
34+
if __name__ == "__main__":
35+
main()

‎tests/groups/flashValidation/flash_validation_hello_world_error.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
#!/usr/bin/env python3
2+
3+
import serial
4+
import time
5+
import sys
6+
7+
"""
8+
This testcase is to be used following the flashing of the Ocre runtime to a board.
9+
10+
The testcase forms a serial connection to the board, sends a break and checks that
11+
the string Hello World! appears in the output of that break command, and without any
12+
instances of the error prefix "E: ".
13+
"""
14+
15+
def main():
16+
conn = serial.Serial('/dev/ttyACM0', 115200, timeout=1)
17+
conn.send_break(duration=1)
18+
19+
time.sleep(1)
20+
21+
print('**** READING RESPONSE FROM BREAK ****\n')
22+
23+
response = conn.read(1024).decode(errors='ignore')
24+
print(response)
25+
26+
print('**** CLOSING CONNECTION ****\n')
27+
28+
conn.close()
29+
30+
if "Hello World!" in response and "E:" not in response:
31+
sys.exit(0)
32+
sys.exit(1)
33+
34+
35+
if __name__ == "__main__":
36+
main()

‎tests/groups/flashValidation/setup.sh‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
echo "Setup is complete"

0 commit comments

Comments
 (0)