Text of releases.llvm.org › 2.6 › docs › TestingGuide.html as Styx crawled it on Nov 19, 2026. The page may have changed since. Only its words are kept, without pictures or layout, and only the first part of a long page.

Open the live page

LLVM Testing Infrastructure Guide

LLVM Testing Infrastructure Guide Overview Requirements LLVM testing infrastructure organization DejaGNU tests Test suite Quick start DejaGNU tests Test suite DejaGNU structure Writing new DejaGNU tests The FileCheck utility Variables and substitutions Other features Test suite structure Running the test suite Configuring External Tests Running different tests Generating test output Writing custom tests for llvm-test Running the nightly tester Written by John T.

Criswell, Reid Spencer , and Tanya Lattner Overview This document is the reference manual for the LLVM testing infrastructure. It documents the structure of the LLVM testing infrastructure, the tools needed to use it, and how to add and run tests. Requirements In order to use the LLVM testing infrastructure, you will need all of the software required to build LLVM, plus the following: DejaGNU The Feature and Regressions tests are organized and run by DejaGNU.

Expect Expect is required by DejaGNU. tcl Tcl is required by DejaGNU. LLVM testing infrastructure organization The LLVM testing infrastructure contains two major categories of tests: code fragments and whole programs. Code fragments are referred to as the "DejaGNU tests" and are in the llvm module in subversion under the llvm/test directory. The whole programs tests are referred to as the "Test suite" and are in the test-suite module in subversion.

DejaGNU tests Code fragments are small pieces of code that test a specific feature of LLVM or trigger a specific bug in LLVM. They are usually written in LLVM assembly language, but can be written in other languages if the test targets a particular language front end (and the appropriate --with-llvmgcc options were used at configure time of the llvm module). These tests are driven by the DejaGNU testing framework, which is hidden behind a few simple makefiles.

These code fragments are not complete programs. The code generated from them is never executed to determine correct behavior. These code fragment tests are located in the llvm/test directory. Typically when a bug is found in LLVM, a regression test containing just enough code to reproduce the problem should be written and placed somewhere underneath this directory. In most cases, this will be a small piece of LLVM assembly language code, often distilled from an actual application or benchmark.

Test suite The test suite contains whole programs, which are pieces of code which can be compiled and linked into a stand-alone program that can be executed. These programs are generally written in high level languages such as C or C++, but sometimes they are written straight in LLVM assembly. These programs are compiled and then executed using several different methods (native compiler, LLVM C backend, LLVM JIT, LLVM native code generation, etc).

The output of these programs is compared to ensure that LLVM is compiling the program correctly. In addition to compiling and executing programs, whole program tests serve as a way of benchmarking LLVM performance, both in terms of the efficiency of the programs generated as well as the speed with which LLVM compiles, optimizes, and generates code. The test-suite is located in the test-suite Subversion module. Quick start The tests are located in two separate Subversion modules.

The DejaGNU tests are in the main "llvm" module under the directory llvm/test (so you get these tests for free with the main llvm tree). The more comprehensive test suite that includes whole programs in C and C++ is in the test-suite module. This module should be checked out to the llvm/projects directory (don't use another name then the default "test-suite", for then the test suite will be run every time you run make in the main llvm directory).

When you configure the llvm module, the test-suite directory will be automatically configured. Alternatively, you can configure the test-suite module manually. DejaGNU tests To run all of the simple tests in LLVM using DejaGNU, use the master Makefile in the llvm/test directory: % gmake -C llvm/test or % gmake check To run only a subdirectory of tests in llvm/test using DejaGNU (ie. Transforms), just set the TESTSUITE variable to the path of the subdirectory (relative to llvm/test ): % gmake TESTSUITE=Transforms check Note: If you are running the tests with objdir != subdir , you must have run the complete testsuite before you can specify a subdirectory.

To run only a single test, set TESTONE to its path (relative to llvm/test ) and make the check-one target: % gmake TESTONE=Feature/basictest.ll check-one To run the tests with Valgrind (Memcheck by default), just append VG=1 to the commands above, e.g.: % gmake check VG=1 Test suite To run the comprehensive test suite (tests that compile and execute whole programs), first checkout and setup the test-suite module: % cd llvm/projects % svn co http://llvm.org/svn/llvm-project/test-suite/trunk test-suite % cd .. % ./configure --with-llvmgccdir=$LLVM_GCC_DIR where $LLVM_GCC_DIR is the directory where you installed llvm-gcc, not it's src or obj dir.

The --with-llvmgccdir option assumes that the llvm-gcc-4.2 module was configured with --program-prefix=llvm- , and therefore that the C and C++ compiler drivers are called llvm-gcc and llvm-g++ respectively. If this is not the case, use --with-llvmgcc / --with-llvmgxx to specify each executable's location. Then, run the entire test suite by running make in the test-suite directory: % cd projects/test-suite % gmake Usually, running the "nightly" set of tests is a good idea, and you can also let it generate a report by running: % cd projects/test-suite % gmake TEST=nightly report report.html Any of the above commands can also be run in a subdirectory of projects/test-suite to run the specified test only on the programs in that subdirectory.

DejaGNU structure The LLVM DejaGNU tests are driven by DejaGNU together with GNU Make and are located in the llvm/test directory. This directory contains a large array of small tests that exercise various features of LLVM and to ensure that regressions do not occur. The directory is broken into several sub-directories, each focused on a particular area of LLVM. A few of the important ones are: Analysis : checks Analysis passes.

Archive : checks the Archive library. Assembler : checks Assembly reader/writer functionality. Bitcode : checks Bitcode reader/writer functionality. CodeGen : checks code generation and each target. Features : checks various features of the LLVM language. Linker : tests bitcode linking. Transforms : tests each of the scalar, IPO, and utility transforms to ensure they make the right transformations. Verifier : tests the IR verifier.

Writing new DejaGNU tests The DejaGNU structure is very simple, but does require some information to be set. This information is gathered via configure and is written to a file, site.exp in llvm/test . The llvm/test Makefile does this work for you. In order for DejaGNU to work, each directory of tests must have a dg.exp file. DejaGNU looks for this file to determine how to run the tests. This file is just a Tcl script and it can do anything you want, but we've standardized it for the LLVM regression tests.

If you're adding a directory of tests, just copy dg.exp from another directory to get running. The standard dg.exp simply loads a Tcl library ( test/lib/llvm.exp ) and calls the llvm_runtests function defined in that library with a list of file names to run. The names are obtained by using Tcl's glob command. Any directory that contains only directories does not need the dg.exp file. The llvm-runtests function lookas at each file that is passed to it and gathers any lines together that match "RUN:".

This are the "RUN" lines that specify how the test is to be run. So, each test script must contain RUN lines if it is to do anything. If there are no RUN lines, the llvm-runtests function will issue an error and the test will fail. RUN lines are specified in the comments of the test program using the keyword RUN fo

The rest of this page is on the live site.