VHDL¶
A set of fixtures are provided which uses GHDL as a free open source simulator for VHDL and the Xilinx Vivado design software for to synthesise hardware designs written in VHDL for Xilinx FPGA devices which the students can then test in the laboratory. For the purpose of automatic marking we can test their designs using simulation testbenches (which can be the same as those we give the students to get feedback as they work) and the synthesis step o generate A binary “.bit” file that would the be used to program an FPGA device. I have found this to often be sufficient combined with having seen the students work in the laboratory without having to program the devices myself for every students design. I have used this both for intermediate digital desing courses using VHDL bu also for a digital systems hardware course where the students explore the tradeoffs in a microprocessor design using VHDL.
Directory structure¶
Below is the directory structure for these tests. Under the cohort directory we will have a directory for each student which will
contain their submitted designs in this case a VHDL entity and architecture for a trivial adder design based on a logic circuit in full_adder.vhd
and a constrainst file to map the ports onto hardware signals in file full_adder.xdc
- this is usually a first exercise I given the students to acclimitise them to using the software involved. They also design their
own test bench based on using a truth table for their full adder in file full_adder_testbench.vhd which we also want to check.
In the tests folder we have the python unit test script test_full_adder.py, our own testbench for the full adder full_adder_check.vhd
and two files to test the students testbench - one which implements a correct full adder full_adder_success and one which has errors
full_adder_fail to check that they detect a failed design:
root
├─ cohorts
│ ├─ cohort1
│ │ └─ student1
│ │ │ └─ full_adder.vhd
│ │ │ └─ full_adder_testbench.vhd
│ │ │ └─ full_adder.xdc
|
├─ tests
| ├─ cohort1
| │ └─ test_full_adder.py
| │ └─ full_adder_check.vhd
| │ └─ full_adder_fail.vhd
| │ └─ full_adder_success.vhd
Example 1 - Testing simulation and synthesis¶
In the first case we want to check that the students design pass a simulation and can be synthesised. They provide their implementation to a given entity specification and an associate constraints (xdc) file. I normally specify that the constraints file name must match that of the top level vhdl file and that these should match the entity name as a matter of good practice and the students work will fail if they do not follow this practice. It also makes automated marking easier to have a strict specification.
library ieee;
use ieee.std_logic_1164.all;
entity full_adder is
port (
x : in std_logic;
y : in std_logic;
cin : in std_logic;
s : out std_logic;
cout : out std_logic);
end;
architecture behavioral of full_adder is
begin
-- students implementation goes here
end;
The constraints file will be a modified version of a template provided that has all of the pins on their development board listed but commented out - they just have to comment out the lines for the appropriate switches, buttons, LEDs etc and make sure they connect to the correctly named VHDL ports.
My testbench for this is from the truth table - exactly the same as I expect the students to produce.
library ieee;
use ieee.std_logic_1164.all;
entity full_adder_check is
end full_adder_check;
architecture testbench of full_adder_check is
component full_adder is
port (
x : in std_logic;
y : in std_logic;
cin : in std_logic;
s : out std_logic;
cout : out std_logic);
end component;
signal input : std_logic_vector(2 downto 0);
signal output : std_logic_vector(1 downto 0);
begin
uut: full_adder port map (
x => input(0),
y => input(1),
cin => input(2),
s => output(0),
cout => output(1)
);
stim_proc: process
begin
input <= "000"; wait for 10 ns; assert output = "00" report "0+0+0 failed" ;
-- Other 7 tests here
wait;
end process;
end;
Below is the unit test file. Since I often want to test many different student entities I paramaterise the tests.
The first test is test_sim() which takes the students list of vhdl files that we need for the test
(just fill_adder.vhd in this case) and our provided testbench entity name (which is also used as the file name) full_adder_check. It calls the provided fixture pyam.fixtures.vhdl.vhdl_simulate() to analyse the student files,
together with our top level testbench, elaborate and run, throwing a pyam.fixtures.vhdl.VHDLSynthesisError if the test failed.
The second test we have test_full_adder_bit_file_present() simply tests to see if the students have synthesised their own design
- a reminder to check notes from observations in the laboratory. The third test test_synthesis() is also paramaterised as I may want to fdo this for several designs in a submission. It again takes the name of the top level entity (which also forms the name of the constraints file) and a full list of VHDL files needed. It calls the provided fixture pyam.fixtures.vhdl.vhdl_synthesise() which will return if the synthesis has completed and a bit file produced, otherwise raising an error.
@pytest.mark.parametrize(
"student_files,check",
((["full_adder.vhd"], "full_adder_check")))
def test_sim(check, student_files, vhdl_simulate):
vhdl_simulate(check, student_files)
def test_full_adder_bit_file_present(student):
assert (student.path/"full_adder.bit").exists()
@pytest.mark.slow
@pytest.mark.parametrize("top, files",
(("full_adder", ("full_adder.vhd")),))
def test_synthesis(top, files, vhdl_synthesise):
vhdl_synthesise(top, files)
Example 2 - Testing student test benches¶
Testing student test benches is a little more complex as we usually want to check if they detect both success (correct)
designs and failure (incorrect) designs. To accomplish this we provide a correct and incorrect design in our tests folder - full_adder_success.vhd and full_adder_fail.vhd. Below is the test function test_full_adder_testbench() for this case.
It makes use of the pyam.fixtures.vhdl.vhdl_simulate() fixture as before to test success.
To detect test bench failure we use the lower level pyam.fixtures.vhdl.ghdl() fixture provided to analyse our files and then run them, expecting a pyam.fixtures.vhdl.VHDLRunError indicating that the testbench has successfully produced an error. It is necessary to reanalyse the fles at this point as we are using a different file with the incorrect version of the top level entity in the test. In tis case we disable standard output capture (capsys) for the duration of the simulation as we don’t need it’s error output for the student (it is expected).
def test_full_adder_testbench(vhdl_simulate,capsys,ghdl, test_path, student):
#Expect student testbench to succeed for full_adder_successs provided
vhdl_simulate("full_adder_testbench", ["full_adder_testbench.vhd"],
["full_adder_success.vhd"])
#Expect student testbench to fail for full_adder_fail provided
with capsys.disabled():
ghdl("-a", test_path / "full_adder_fail.vhd")
ghdl("-a", student.path / "full_adder_testbench.vhd")
with pytest.raises(VHDLRunError):
ghdl("--elab-run", "full_adder_testbench", run_options=["--assert-level=error"])
Example 3 - Testing simulation and synthesis with VHDL only (without Python fixtures)¶
Directory structure¶
Below is the directory structure for these tests. Under the cohort directory we will have a directory for each student which will
contain their submitted code - in this case entity1.vhd and entity2.vhd. In our tests folder for the cohort we have the VHDL test bench files test_entity1.vhd, test_entity2.vhd etc. The
test files must start with “test_” and must contain some comments to be recognised as test files:
root
├─ cohorts
│ ├─ cohort1
│ │ └─ student1
│ │ │ └─ entity1.vhd
│ │ │ └─ entity2.vhd
|
├─ tests
│ └─ test_entity1.vhd
│ └─ test_entity2.vhd
The VHDL fixtures provided by pyAutoMark will set up the analysis to point to the appropriate test folder and student folder. It is assumed that the entity specification and architecture under test are both defined in the same students file.
To be collected the VHDL test filenames must start with “test_” and contain the following definition
- PYAM_TEST
A unix glob string used to find the students file (by name) under test.
Additional parameters may be specified
- PYAM_TIMEOUT
(Optional) A floating point value spcifying the timeout for these tests.
- PYAM_TEST_VALUE
(Optional) This may be specified multiple times to enable multiple test using the same testbench file. For each value given the simulaition will be called with a generic parameter “PYAM_TEST_VALUE” set to the corresponding value.
e.g.
-- PYAM_TEST "UUT.vhdl"
-- PYAM_TIMEOUT TIMEOUT
-- PYAM_TEST_VALUE VALUE1
-- PYAM_TEST_VALUE VALUE2
- UUT.vhdl¶
The filename glob to the entity and architecture definition file under test in the students directory
- TIMEOUT¶
The maximum run time for a test in seconds.
- VALUE1, VALUE2¶
This is the list of test values which will be passed as “PYAM_TEST_VALUE generic values to the simulation in turn