RTFSC - Test Individual Features in VexRiscv

RTFSC - Vexriscv: Test Individual Features

Background

When reading the source code of VexRiscv, I found one of the test code for regression tests introduces the functionality of randomly generating different CPU configurations, which could possibly be used in my current project.

In my understanding, this functionality is kind of tricky as configurable parameters may conflict with each other, leading either not practical RTL code or compiliation errors. Therefore, it’s very interesting that how the regression tests in VexRiscv manage to do that. So here today, let’s explore the source code of TestIndividualFeatures.scala in VexRiscv.

By the way, I’m still new to Scala so don’t mind if I make any mistake.

The App

First of all, let’s start with an object of TestIndividualExplore, which is the main function called when running regression tests.

object TestIndividualExplore extends App{
  val seeds = mutable.HashSet[Int]()
  val futures = mutable.ArrayBuffer[Future[Unit]]()
  implicit val ec = ExecutionContext.fromExecutorService(
    new ForkJoinPool(24, ForkJoinPool.defaultForkJoinWorkerThreadFactory, null, true)
  )
  for(i <- 0 until 1000){
    val seed = Random.nextInt(1000000) + 1
    futures += Future {
      if (!seeds.contains(seed)) {
        val cmd = s"make run REGRESSION_PATH=../../src/test/cpp/regression VEXRISCV_FILE=VexRiscv.v WITH_USER_IO=no REDO=10 TRACE=yes TRACE_START=100000000000ll FLOW_INFO=no STOP_ON_ERROR=no DHRYSTONE=yes COREMARK=yes THREAD_COUNT=1   IBUS=CACHED IBUS_DATA_WIDTH=128 COMPRESSED=yes DBUS=SIMPLE LRSC=yes  MUL=yes DIV=yes      FREERTOS=0 ZEPHYR=2 LINUX_REGRESSION=yes SUPERVISOR=yes  CONCURRENT_OS_EXECUTIONS=yes MMU=yes PMP=no SEED=$seed"
        val workspace = s"explor/seed_$seed"
        FileUtils.copyDirectory(new File("simWorkspace/ref"), new File(workspace))
        val str = DoCmd.doCmdWithLog(cmd, workspace)
        if(!str.contains("REGRESSION SUCCESS")){
          println(s"seed $seed FAILED with\n\n$str")
          sys.exit(1)
        }
        FileUtils.deleteDirectory(new File(workspace))
        println(s"seed $seed PASSED")
      }
    }
  }
  futures.foreach(Await.result(_, Duration.Inf))
}

There are not actually too much to talk about, the function simply copy the tests generated to a temporary directory, use make run to execute the tests parallelly, and then get the test result in PASS or FAIL. Maybe we can call it the backend of regression tests, it gives us a view of how the tests are run. The question comes to where do these tests come from, and the answer lies in the main class of TestIndividualFeatures.

The Main Class

Environment variables

The main class of TestIndividualFeatures starts with a bunch of environment variables:

val testCount = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_COUNT", "100").toInt
  val seed = sys.env.getOrElse("VEXRISCV_REGRESSION_SEED", Random.nextLong().toString).toLong
  val testId : Set[Int] =  sys.env.get("VEXRISCV_REGRESSION_TEST_ID") match {
    case Some(x) if x != "" => x.split(',').map(_.toInt).toSet
    case _ => (0 until testCount).toSet
  }
  val rvcRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_RVC_RATE", "0.5").toDouble
  val linuxRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_LINUX_RATE", "0.3").toDouble
  val machineOsRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_MACHINE_OS_RATE", "0.5").toDouble
  val secureRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_SECURE_RATE", "0.2").toDouble
  val linuxRegression = sys.env.getOrElse("VEXRISCV_REGRESSION_LINUX_REGRESSION", "yes")
  val coremarkRegression = sys.env.getOrElse("VEXRISCV_REGRESSION_COREMARK", "yes")
  val zephyrCount = sys.env.getOrElse("VEXRISCV_REGRESSION_ZEPHYR_COUNT", "4")
  val demwRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEMW_RATE", "0.6").toDouble
  val demRate = sys.env.getOrElse("VEXRISCV_REGRESSION_CONFIG_DEM_RATE", "0.5").toDouble
  val stopOnError = sys.env.getOrElse("VEXRISCV_REGRESSION_STOP_ON_ERROR", "no")
  val lock = new{}

Some of them are boolean and some are double, you can export these variables in advance to control the test process.

Universe and Dimensions

The next part is actually what I want to find, it setup a list of variables called dimensions :

val dimensions = List(
    new IBusDimension(rvcRate),
    new DBusDimension,
    new MulDivDimension,
    new ShiftDimension,
    new BranchDimension,
    new HazardDimension,
    new RegFileDimension,
    new SrcDimension,
    new CsrDimension(/*sys.env.getOrElse("VEXRISCV_REGRESSION_FREERTOS_COUNT", "1")*/ "0", zephyrCount, linuxRegression), //Freertos old port software is broken
    new DecoderDimension,
    new DebugDimension,
    new MmuPmpDimension
  )

In tests, the configurations of VexRiscv are divided into different dimensions as above. Theses dimensions are defined as classes originally extended from an abstract class called ConfigDimension :

abstract class ConfigUniverse

abstract class  ConfigDimension[T <: ConfigPosition[_]](val name: String) {
  def randomPosition(universes : Seq[ConfigUniverse], r : Random) : T = {
    val ret = randomPositionImpl(universes, r)
    ret.dimension = this
    ret
  }

  protected def randomPositionImpl(universes : Seq[ConfigUniverse], r : Random) : T
  protected def random[X](r : Random, positions : List[X]) : X = positions(r.nextInt(positions.length))
}

abstract class ConfigPosition[T](val name: String) {
  def applyOn(config: T): Unit
  var dimension : ConfigDimension[_] = null
  def isCompatibleWith(positions : Seq[ConfigPosition[T]]) : Boolean = true
}

ConfigDimension includes 3 functions randomPosition , randomPositionImpl and random , only first one is implemented.

What about ConfigUniverse and ConfigPosition? TODO

These classes are then extended to VexRiscvUniverse , VexRiscvDimension and VexRiscvPosition:

class VexRiscvUniverse extends ConfigUniverse

abstract class  VexRiscvDimension(name: String) extends ConfigDimension[VexRiscvPosition](name)

abstract class  VexRiscvPosition(name: String) extends ConfigPosition[VexRiscvConfig](name){
  def testParam : String = ""
}

Not much is added by this level of abstraction, I guess this is just for naming and readability.

Then comes to some configuration level definations:

object VexRiscvUniverse{
  val CACHE_ALL = new VexRiscvUniverse
  val CATCH_ALL = new VexRiscvUniverse
  val MMU = new VexRiscvUniverse
  val PMP = new VexRiscvUniverse
  val FORCE_MULDIV = new VexRiscvUniverse
  val SUPERVISOR = new VexRiscvUniverse
  val NO_WRITEBACK = new VexRiscvUniverse
  val NO_MEMORY = new VexRiscvUniverse
  val EXECUTE_RF = new VexRiscvUniverse
}

Some configurations which can not be divided are defined here in VexRiscvUniverse, such as the pipeline stage configurations - NO_WRITEBACK and NO_MEMORY.

After that, let’s see the first VexRiscvDimension we can find as an example, which is ShiftDimension:

class ShiftDimension extends VexRiscvDimension("Shift") {
  override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = {
    var l = List(
      new VexRiscvPosition("FullEarly") {
        override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new FullBarrelShifterPlugin(earlyInjection = true)
      },
      new VexRiscvPosition("Light") {
        override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new LightShifterPlugin
      }
    )

    if(!universes.contains(VexRiscvUniverse.NO_MEMORY)) l = new VexRiscvPosition("FullLate") {
      override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new FullBarrelShifterPlugin(earlyInjection = false)
    } :: l

    random(r, l)
  }
}

This is rather a simple one, only two or three kinds of configurations are involved here. The dimension overrides the randomPositionImpl , creating a list of different configurations in this dimension, and return a random one. The FullLate configuration fo shifter will only be included in the list if the NO_MEMORY in VexRiscvUniverse is set.

Other dimensions basically follow the same format, overriding the randomPositionImpl function. Some positions are constrained by the settings in universe, making some configurations unavailable.

Now let’s take a look at a more complex one - IBusDimension, it’s quite long so let’s read it separately:

class IBusDimension(rvcRate : Double) extends VexRiscvDimension("IBus") {

  override def randomPositionImpl(universes: Seq[ConfigUniverse], r: Random) = {
    val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL)
    val cacheAll = universes.contains(VexRiscvUniverse.CACHE_ALL)

First, we get some universal settings from VexRiscvUniverse.

if(r.nextDouble() < 0.5 && !cacheAll){
     val mmuConfig = if(universes.contains(VexRiscvUniverse.MMU)) MmuPortConfig( portTlbSize = 4) else null

     val latency = r.nextInt(5) + 1
     val compressed = r.nextDouble() < rvcRate
     val injectorStage = r.nextBoolean() || latency == 1
     val prediction = random(r, List(NONE, STATIC, DYNAMIC, DYNAMIC_TARGET))
     val catchAll = universes.contains(VexRiscvUniverse.CATCH_ALL)
     val cmdForkOnSecondStage = r.nextBoolean()
     val cmdForkPersistence = r.nextBoolean()
     new VexRiscvPosition("Simple" + latency + (if(cmdForkOnSecondStage) "S2" else "") + (if(cmdForkPersistence) "P" else "")  + (if(injectorStage) "InjStage" else "") + (if(compressed) "Rvc" else "") + prediction.getClass.getTypeName().replace("$","")) with InstructionAnticipatedPosition{
       override def testParam = "IBUS=SIMPLE" + (if(compressed) " COMPRESSED=yes" else "")
       override def applyOn(config: VexRiscvConfig): Unit = config.plugins += new IBusSimplePlugin(
         resetVector = 0x80000000l,
         cmdForkOnSecondStage = cmdForkOnSecondStage,
         cmdForkPersistence = cmdForkPersistence,
         prediction = prediction,
         catchAccessFault = catchAll,
         compressedGen = compressed,
         busLatencyMin = latency,
         injectorStage = injectorStage,
         memoryTranslatorPortConfig = mmuConfig
       )
       override def instructionAnticipatedOk() = injectorStage
     }

Then, we get a random double to generate a 50% possbility choice, like a coin toss. If coin lies on head and we don’t set the CACHE_ALL setting, we generate a latency number from 1 to 6, randomly decide the use of RV32C extension based on a rate setting, and a lot of configurations based on random booleans.

The else branch is pretty similar, mixed random types are used for mixed configurations.

Tests

Now let’s go back to the TestIndividualFeatures class, after creating the list of dimensions, the tests are going to start:

var clockCounter = 0l
 var startAt = System.currentTimeMillis()
 def doTest(positionsToApply : List[VexRiscvPosition], prefix : String = "", testSeed : Int, universes : mutable.HashSet[VexRiscvUniverse]): Unit ={
   val noMemory = universes.contains(VexRiscvUniverse.NO_MEMORY)
   val noWriteback = universes.contains(VexRiscvUniverse.NO_WRITEBACK)
   val name = (if(noMemory) "noMemoryStage_" else "") + (if(noWriteback) "noWritebackStage_" else "") + positionsToApply.map(d => d.dimension.name + "_" + d.name).mkString("_")
   val workspace = "simWorkspace"
   val project = s"$workspace/$prefix"
   def doCmd(cmd: String): String = {
     val stdOut = new StringBuilder()
     class Logger extends ProcessLogger {
       override def err(s: => String): Unit = {
         if (!s.startsWith("ar: creating ")) println(s)
       }
       override def out(s: => String): Unit = {
         println(s)
         stdOut ++= s
       }
       override def buffer[T](f: => T) = f
     }
     Process(cmd, new File(project)).!(new Logger)
     stdOut.toString()
   }

The beginning part of the doTest is basically creating the names, setting up the workspace and defining a command executor.

testMp(prefix + name) {
      println("START TEST " + prefix + name)

      //Cleanup
      FileUtils.deleteDirectory(new File(project))
      FileUtils.forceMkdir(new File(project))

      //Generate RTL
      FileUtils.deleteQuietly(new File("VexRiscv.v"))
      SpinalConfig(targetDirectory = project).generateVerilog{
        val config = VexRiscvConfig(
          withMemoryStage = !noMemory,
          withWriteBackStage = !noWriteback,
          plugins = List(
            new IntAluPlugin,
            new YamlPlugin("cpu0.yaml")
          )
        )
        for (positionToApply <- positionsToApply) positionToApply.applyOn(config)
        new VexRiscv(config)
      }

      //Setup test
      val files = List("main.cpp", "jtag.h", "encoding.h" ,"makefile", "dhrystoneO3.logRef", "dhrystoneO3C.logRef","dhrystoneO3MC.logRef","dhrystoneO3M.logRef")
      files.foreach(f => FileUtils.copyFileToDirectory(new File(s"src/test/cpp/regression/$f"), new File(project)))

      //Test RTL
      val debug = true
      val stdCmd = (s"make run REGRESSION_PATH=../../src/test/cpp/regression VEXRISCV_FILE=VexRiscv.v WITH_USER_IO=no REDO=10 TRACE=${if(debug) "yes" else "no"} TRACE_START=100000000000ll FLOW_INFO=no STOP_ON_ERROR=$stopOnError DHRYSTONE=yes COREMARK=${coremarkRegression} THREAD_COUNT=1 ") + s" SEED=${testSeed} "
      val testCmd = stdCmd + (positionsToApply).map(_.testParam).mkString(" ")
      println(testCmd)
      val str = doCmd(testCmd)
      assert(str.contains("REGRESSION SUCCESS") && !str.contains("Broken pipe"))
      val pattern = "Had simulate ([0-9]+)".r
      val hit = pattern.findFirstMatchIn(str)

      lock.synchronized(clockCounter += hit.get.group(1).toLong)
    }

The second part is the actual part where tests happen, it cleans up previous data, generates the RTL, setups tests files by copying from the template folder, and finally calling make to run the tests. The tests are executed parallelly using testMP .

Now, the test agenda is prepared, the regression tests are starting from here:

val rand = new Random(seed)

  testMp("Info"){
    println(s"MAIN_SEED=$seed")
  }
  println(s"Seed=$seed")
  for(i <- 0 until testCount){
    var positions : List[VexRiscvPosition] = null
    var universe = mutable.HashSet[VexRiscvUniverse]()
    if(rand.nextDouble() < 0.5) universe += VexRiscvUniverse.EXECUTE_RF
    if(linuxRate > rand.nextDouble()) {
      universe += VexRiscvUniverse.CATCH_ALL
      universe += VexRiscvUniverse.MMU
      universe += VexRiscvUniverse.FORCE_MULDIV
      universe += VexRiscvUniverse.SUPERVISOR
      if(demwRate < rand.nextDouble()){
        universe += VexRiscvUniverse.NO_WRITEBACK
      }
    } else if (secureRate > rand.nextDouble()) {
        universe += VexRiscvUniverse.CACHE_ALL
        universe += VexRiscvUniverse.CATCH_ALL
        universe += VexRiscvUniverse.PMP
        if(demwRate < rand.nextDouble()){
          universe += VexRiscvUniverse.NO_WRITEBACK
        }
    } else {
      if(machineOsRate > rand.nextDouble()) {
        universe += VexRiscvUniverse.CATCH_ALL
        if(demwRate < rand.nextDouble()){
          universe += VexRiscvUniverse.NO_WRITEBACK
        }
      }
      if(demwRate > rand.nextDouble()){
      }else if(demRate > rand.nextDouble()){
        universe += VexRiscvUniverse.NO_WRITEBACK
      } else {
        universe += VexRiscvUniverse.NO_WRITEBACK
        universe += VexRiscvUniverse.NO_MEMORY
      }
    }

    do{
      positions = dimensions.map(d => d.randomPosition(universe.toList, rand))
    }while(!positions.forall(_.isCompatibleWith(positions)))

    val testSeed = rand.nextInt()
    if(testId.contains(i))
      doTest(positions,"test_id_" + i + "_", testSeed, universe)
    Hack.dCounter += 1
  }
  testSingleThread("report"){
    val time = (System.currentTimeMillis() - startAt)*1e-3
    val clockPerSecond = (clockCounter/time*1e-3).toLong
    println(s"Duration=${(time/60).toInt}mn clocks=${(clockCounter*1e-6).toLong}M clockPerSecond=${clockPerSecond}K")
  }

Universal settings are all randomly generated based on the rate settings in environment variables. The dimensions are then generated from the universe and random. There is a compatibility check which I believe is used to check configuration conflicts, but there is only one implementation of this function, so I think dolu (the author of VexRiscv) is considering deprecating it. Apparently it’s more efficient when no conflicts are generated at all than conflicts are generated then aborted.

Finally, the doTest is called for each testCount, then the test report is generated.

Reference

https://github.com/SpinalHDL/VexRiscv/blob/master/src/test/scala/vexriscv/TestIndividualFeatures.scala