The $1,000,000 Compile
Hitting the compile and run button costs $1,000,000. How would such a cost affect your test strategy? As extreme as that sounds, its a harsh reality for ASIC designers. The start-up costs for an ASIC design are extreme, as a result ASIC verification is very important. Especially since you can’t just update a buggy ASIC. In software engineering, we design increasingly complex systems requiring more and more elegant test strategies. I propose ASIC verification is in a more mature state, and software engineers could learn a lot from the translation of these techniques.
SystemVerilog, the premier language in the ASIC verification space has an incredible feature I feel has a place in software verification: Random Constraints. Constrains allow the compiler to generate a set of test vectors that match some simple set of rules. The verification bench may then use these test vectors to test the unit under test. Conceptually, the random bench will test the device in ways the developer never thought of, either with a new test case, or a series of test cases totally unimaginable by the engineer. Find the bug before you tape-out => keep your job.
Hardware verification also teaches us that no language feature is a panacea. Random Constraints are a tool to help us find new test cases, but we need model the behavior of our system to get expected behavior. Hardware Verification uses a pattern similar to the block diagram presented here. Notice however that one requires a System Model in addition to the UUT (Unit Under Test. This System Model is ideally a “clean-room” designed block, entirely separate from the UUT developer’s group. This is difficult to do in smaller teams, but its important to understand that one is essentially designing the module twice. This is different from Mock objects however. Mock Objects are not pure system models. Mock Objects are standins that implement the interface, but with a listing of recalculated responses. However, one could use a higher level language for the Model such as python. Such a language would make the design easier, and make provide insight on how to constraint the test vectors.
So how does this apply to software? The generic pattern maps perfectly. Using metaprogramming techniques we can build a driver that generates random constraints. Drive the blocks and iterate through the tests. Publish a report. Compile & Run.