Intro to PFLOTRAN’s QA Test Suite

How To Run The Test Suite

To run the test suite, open a terminal and navigate to the qa_tests/ directory:

$ hg clone ssh://hg@bitbucket.org/pflotran/pflotran-doc-sandbox pflotran-doc-qa
$ cd pflotran-doc-qa/qa_tests

Run the python script from within the qa_tests/ directory, and use the -E flag to indicate which pflotran executable you want to use. To run all tests, you must include the -ALL argument flag also.

$ python run_qa_tests.py -E=$PFLOTRAN_DIR/src/pflotran/pflotran -ALL

The script also includes several options which allow you to run only part of the test suite. The following options are available. Several options can be chosen at a time (but -E must always be indicated).

Usage:

The -E flag is required and must indicate the path to the
PFLOTRAN executable which will run the QA tests.

  -E=path/to/PFLOTRAN/executable

If mpirun is not the MPI executable you desire, please the
correct path to the executable using the -MPI flag.

  -MPI=path/to/MPI/executable

At least one of the following flags must also be given:

  -ALL
  -1D             -2D             -3D
  -GENERAL_MODE   -RICHARDS_MODE  -TH_MODE
  -THERMAL        -FLOW           -GAS
  -STEADY         -TRANSIENT
  -NUM_TRIES=
  -REMOVE
  -SCREEN_ON
  -HELP

Inside The run_qa_tests.sh Script

Each test is executed via it’s unique python function, which must be called in the test’s directory location.

os.chdir('thermal/steady/1D/BC_1st_kind/th_mode'); cwd = os.getcwd()
qa.thermal_steady_1D_BC1stkind(cwd,'1D_steady_thermal_BC_1st_kind',remove,screen_on,pf_exe)
os.chdir('../../../../..')

The unique test scripts reside in the file qa_tests_engine.py. Within this script, each test generates the analytical solution, runs the PFLOTRAN simulation, reads the PFLOTRAN output, and compares the PFLOTRAN solution to the analytical solution both mathematically and visually. If the test does not pass, then the script will reduce the grid spacing by a factor of 2, and re-run the test. The test will be re-run for -NUM_TRIES number of times. For example, -NUM_TRIES=5. The default number of tries is 3.

The graph that visually compares the PFLOTRAN solution against the analytical solution will be created in the specific test directory called comparison_plot.png. To view it, you must navigate to the desired test directory and open the file.

Python Helper Functions

The unique, test-specific functions use several general helper functions, defined in the module qa_tests_helper in the file qa_tests_helper.py. The member functions are documented below.

# This module contains helper functions for the set of QA Tests.
#
# Author: Jennifer M. Frederick
# Date: 08/02/2016


import numpy as np
import math
import matplotlib.pyplot as plt
import h5py
import os

################################################################################
def calc_relative_error(s_benchmark,s_pflotran,ierr):
  """ This function calculates the relative error between the benchmark 
      solution and the PFLOTRAN solution.
      
      :param s_benchmark: benchmark solution array
      :param s_pflotran: PFLOTRAN solution array
      :returns: max_percent_error, the maximum relative error between the 
                benchmark and PFLOTRAN solution arrays
  """
  #shift solution to avoid dividing by zero, but note that
  #the relative error calculated will change depending on
  #the offset that is chosen here:
  offset = 0.01
  s_benchmark = s_benchmark + offset
  s_pflotran = s_pflotran + offset
  
  #limit how small a solution can get by truncating it when close to zero:
  trunc = 1.0e-10
  #s_pflotran[s_pflotran<trunc] = trunc
  #s_benchmark[s_benchmark<trunc] = trunc
  
  # calculate relative percent error
  percent_error = abs(100.*(s_pflotran-s_benchmark)/s_benchmark)
  max_percent_error = np.nanmax(percent_error)
  if ierr == 0:
    print('Relative Maximum Error: ',max_percent_error,'%')
  else:
    print('Relative Maximum Error: N/A')
  
  return max_percent_error


################################################################################
def does_pass(max_percent_error,try_count,num_tries):
  """ This function decides if a test passes or fails by comparing the maximum
      relative error to a passing criteria value.
      
      :param max_percent_error: the maximum relative error between the benchmark
                                and PFLOTRAN solution arrays
      :param try_count: the number of time the test has been tried
      :returns: true, false
  """
  passing_crit = 2.0  # [%]
  
  # Decide if test passes
  if abs(max_percent_error) > passing_crit:
    print('-- Test FAIL --\n')
    test_pass = False
  else:
    print('-- Test PASS --\n')
    test_pass = True
  if (try_count > (num_tries-1)) and (not test_pass):
    print('Simulation failed ' + str(try_count) + ' times. Aborting test!\n')
  return test_pass
  
  
################################################################################
def get_mode(path):
  """ This function gets the PFLOTRAN simulation mode from the path string.
      
      :param path: the path string of the qa test
      :returns: mode, the PFLOTRAN simulation mode string
  """
  k = 0
  for letter in reversed(path):
    if letter == '/':
      ind_reverse = k
      break
    k = k + 1
  ind = len(path) - ind_reverse
  mode = path[ind:len(path)]
  if mode == 'th_mode':
    mode = 'TH Mode'
  if mode == 'general_mode':
    mode = 'GENERAL Mode'
  if mode == 'richards_mode':
    mode = 'RICHARDS Mode'
    
  return mode


################################################################################
def check(solution):
  """ This function checks if the PFLOTRAN simulation output was read correctly.
      
      :param solution: the PFLOTRAN solution array
      :returns: ierr, integer error code
  """
  ierr = 0
  if len(np.shape(solution)) == 1:
    value = solution[0]
  elif len(np.shape(solution)) == 2:
    value = solution[0,0]
  else:
    value = solution[0,0,0]
    
  if value == -999:
    ierr = 1

  return ierr


################################################################################
def print_discretization(lxyz,nxyz,dxyz):
  """ This function prints the discretization information to screen.
      
      :param lxyz: domain size dimensions [m]
      :param nxyz: grid cell number [-]
      :param dxyz: grid cell spacing dimensions [m]
  """
  print('L = ' + str(lxyz) + ' [m]')
  print('n = ' + str(nxyz))
  print('d = ' + str(dxyz) + ' [m]')

  return


################################################################################
def record_error(error_analysis,nxyz_record,dxyz_record,max_percent_error,
                 nxyz,dxyz,try_count):
  """ This function records the error and grid spacing for each test try.
      
      :param error_analysis: array of the test error
      :param nxyz_record: record of the nx ny nz
      :param dxyz_record: record of the dx dy dz [m]
      :param max_percent_error: maximum relative error of the test
      :param dxyz: grid cell spacing dimensions [m]
  """
  error_analysis[try_count-1] = max_percent_error
  nxyz_record[try_count-1,:] = nxyz
  dxyz_record[try_count-1,:] = dxyz

  return


################################################################################
def plot_error(error_analysis,nxyz_record,dxyz_record,path,try_count,dims):
  
  # exit if only 1 try was made
  if try_count == 1:
    return;

  e_max = math.ceil(np.max(error_analysis))
  e_min = 0.
  d_max = math.ceil(np.max(dxyz_record))
  d_min = 0.
  pass_line = np.zeros(2) + 2.0
  
  mode = get_mode(path)
  note = mode + ' Error Analysis'

  # Plot the PFLOTRAN error analysis
  plt.figure(figsize=(6,5))
  plt.loglog(dxyz_record[0:try_count,0],error_analysis[0:try_count],'s-',
           dxyz_record[0:try_count,1],error_analysis[0:try_count],'o-',
           dxyz_record[0:try_count,2],error_analysis[0:try_count],'^-',
           [d_min,d_max],pass_line,'.-')
  plt.legend(('X-direction','Y-direction','Z-direction'),'best',numpoints=1)
  plt.xlabel('LOG Grid Spacing [m] ')
  plt.ylabel('LOG Max Rel. Error [%]')
  plt.xlim([d_min,d_max])
  plt.ylim([e_min,e_max])
  plt.annotate(note, xy=(.03, .990),xycoords='figure fraction',
               horizontalalignment='left',verticalalignment='top',fontsize=14)
  plt.savefig(path+'/error_plot.png')
  
  # Calculate the order of spatial convergence:
  conv_x = np.zeros(try_count-1)
  conv_y = np.zeros(try_count-1)
  conv_z = np.zeros(try_count-1)
  for k in range(try_count-1):
    # Change in error:
    dE = math.log(error_analysis[k]) - math.log(error_analysis[k+1])
    # Change in grid spacing:
    if dims == 1 or dims == 2 or dims == 3:
      dnx = math.log(nxyz_record[k,0]) - math.log(nxyz_record[k+1,0])
      conv_x[k] = dE/dnx
    if dims == 2 or dims == 3:
      dny = math.log(nxyz_record[k,1]) - math.log(nxyz_record[k+1,1])
      conv_y[k] = dE/dny
    if dims == 3:
      dnz = math.log(nxyz_record[k,2]) - math.log(nxyz_record[k+1,2])
      conv_z[k] = dE/dnz
  
  # Slope:  
  if dims == 1 or dims == 2 or dims == 3:
    print('Order of spatial convergence [x] = ' + str(np.mean(conv_x)))
  if dims == 2 or dims == 3:
    print('Order of spatial convergence [y] = ' + str(np.mean(conv_y)))
  if dims == 3:
    print('Order of spatial convergence [z] = ' + str(np.mean(conv_z)))
  print(' ')
  
  return;


################################################################################
def read_pflotran_output_1D(filename,index_string,remove):
  """ This function reads a 1D PFLOTRAN solution from an HDF5 file format.
      
      :param filename: the filename that contains the PFLOTRAN solution, which
                       must be in HDF5 output format
      :param index_string: the name of the path within the HDF5 file where the
                           data is contained
      :param remove: a Boolean that indicates if the HDF5 output file should be
                     removed after it is read
      :returns: solution, a 1D array which contains the PFLOTRAN solution
  """
  solution = -999
  try:
    f = h5py.File(filename,'r+')
    data = np.array(f[index_string][:],'=f8')
    solution = data[:,0,0]
    f.close()
  except IOError:
    print('PFLOTRAN HDF5 output is missing.')
    
  if remove:
    os.system('rm *.h5')
    
  return solution;


################################################################################
def read_pflotran_output_2D(filename,index_string,remove):
  """ This function reads a 2D PFLOTRAN solution from an HDF5 file format.
      
      :param filename: the filename that contains the PFLOTRAN solution, which
                       must be in HDF5 output format
      :param index_string: the name of the path within the HDF5 file where the
                           data is contained
      :param remove: a Boolean that indicates if the HDF5 output file should be
                     removed after it is read
      :returns: solution_array, a 2D array which contains the PFLOTRAN solution
  """
  solution_array = -999
  try:
    f = h5py.File(filename,'r+')
    data = np.array(f[index_string][:],'=f8')
    solution_array = data[:,:,0]
    f.close()
  except IOError:
    print('PFLOTRAN HDF5 output is missing.')
    
  if remove:
    os.system('rm *.h5')
    
  return solution_array


################################################################################
def read_pflotran_output_3D(filename,index_string,remove):
  """ This function reads a 3D PFLOTRAN solution from an HDF5 file format.
      
      :param filename: the filename that contains the PFLOTRAN solution, which
                       must be in HDF5 output format
      :param index_string: the name of the path within the HDF5 file where the
                           data is contained
      :param remove: a Boolean that indicates if the HDF5 output file should be
                     removed after it is read
      :returns: solution_array, a 3D array which contains the PFLOTRAN solution
  """
  solution_array = -999
  try:
    f = h5py.File(filename,'r+')
    data = np.array(f[index_string][:],'=f8')
    solution_array = data[:,:,:]
    f.close()
  except IOError:
    print('PFLOTRAN HDF5 output is missing.')
    
  if remove:
    os.system('rm *.h5')
    
  return solution_array


################################################################################
def plot_1D_steady(path,x_benchmark,s_benchmark,x_pflotran,s_pflotran,x_string,
                   s_string,e_string):

  s_max = math.ceil(np.max(s_benchmark*10.0))/10.0
  s_min = math.floor(np.min(s_benchmark*10.0))/10.0
  if s_min < 0:
    s_min = 0
  x_max = math.ceil(np.max(x_benchmark))
  x_min = math.floor(np.min(x_benchmark))
  
  mode = get_mode(path)
  note = mode + ' ' + e_string + '% error'

  # Plot the PFLOTRAN and analytical solutions
  plt.figure(figsize=(5,5))
  plt.plot(x_pflotran,s_pflotran,'o',x_benchmark,s_benchmark)
  plt.xlabel(x_string)
  plt.ylabel(s_string)
  plt.xlim([x_min,x_max])
  plt.ylim([s_min,s_max])
  plt.legend(('PFLOTRAN','analytical'),'best',numpoints=1)
  plt.annotate(note, xy=(.03, .990),xycoords='figure fraction',
               horizontalalignment='left',verticalalignment='top',fontsize=14)
  plt.savefig(path+'/comparison_plot.png')

  return;


################################################################################
def plot_2D_steady(path,x_array,y_array,s_benchmark,s_pflotran,xy_string,
                   s_string,e_string):
  s_max = math.ceil(np.max(s_benchmark*10.0))/10.0
  s_min = math.floor(np.min(s_benchmark*10.0))/10.0
  if s_min < 0:
    s_min = 0
  x_max = math.ceil(np.max(x_array))
  x_min = math.floor(np.min(x_array))
  y_max = math.ceil(np.max(y_array))
  y_min = math.floor(np.min(y_array))
  levels = np.linspace(s_min,s_max,11)
  X,Y = np.meshgrid(y_array,x_array)
  
  mode = get_mode(path)
  note = mode + ' ' + e_string + '% error'

  # Plot the PFLOTRAN and analytical solutions
  plt.figure(figsize=(7,5))
  plt.contourf(Y,X,s_benchmark,levels,alpha=0.75)
  C = plt.contour(Y,X,s_pflotran,levels,colors='black',linewidth=0.5)
  plt.clabel(C,inline=True,fontsize=10)
  plt.xlabel(xy_string)
  plt.ylabel(xy_string)
  plt.xlim([x_min,x_max])
  plt.ylim([y_min,y_max])
  plt.title(s_string)
  ann_title = 'Analytical (fill) vs. PFLOTRAN (contour)'+' '+note
  plt.annotate(ann_title,xy=(.03,.990),xycoords='figure fraction',
               horizontalalignment='left',verticalalignment='top',fontsize=14)
  plt.savefig(path+'/comparison_plot.png')

  return;


################################################################################
def plot_3D_steady(path,x_array,y_array,z_levels,s_benchmark,s_pflotran,
                   xy_string,s_string,e_string):
  plot_2D_transient(path,z_levels,' ind. slice',x_array,y_array,s_benchmark,
                    s_pflotran,xy_string,s_string,e_string)
  return;


################################################################################
def plot_1D_transient(path,t_array,t_unit,x_benchmark,s_benchmark,x_pflotran,
                      s_pflotran,x_string,s_string,e_string):
  s_max = math.ceil(np.max(s_benchmark*10.0))/10.0
  s_min = math.floor(np.min(s_benchmark*10.0))/10.0
  if s_min < 0:
    s_min = 0
  x_max = math.ceil(np.max(x_benchmark))
  x_min = math.floor(np.min(x_benchmark))
  
  mode = get_mode(path)
  note = mode + ' ' + e_string + '% error'

  # Plot the PFLOTRAN and analytical solutions
  plt.figure(figsize=(10,10))
  plt.subplot(221)
  plt.plot(x_pflotran,s_pflotran[0,:],'o',x_benchmark,s_benchmark[0,:])
  plt.xlabel(x_string)
  plt.ylabel(s_string)
  plt.xlim([x_min,x_max])
  plt.ylim([s_min,s_max])
  plt.title('t='+str(t_array[0])+t_unit)
  plt.legend(('PFLOTRAN','analytical'),'best',numpoints=1)
  plt.subplot(222)
  plt.plot(x_pflotran,s_pflotran[1,:],'o',x_benchmark,s_benchmark[1,:])
  plt.xlabel(x_string)
  plt.ylabel(s_string)
  plt.xlim([x_min,x_max])
  plt.ylim([s_min,s_max])
  plt.title('t='+str(t_array[1])+t_unit)
  plt.legend(('PFLOTRAN','analytical'),'best',numpoints=1)
  plt.subplot(223)
  plt.plot(x_pflotran,s_pflotran[2,:],'o',x_benchmark,s_benchmark[2,:])
  plt.xlabel(x_string)
  plt.ylabel(s_string)
  plt.xlim([x_min,x_max])
  plt.ylim([s_min,s_max])
  plt.title('t='+str(t_array[2])+t_unit)
  plt.legend(('PFLOTRAN','analytical'),'best',numpoints=1)
  plt.subplot(224)
  plt.plot(x_pflotran,s_pflotran[3,:],'o',x_benchmark,s_benchmark[3,:])
  plt.xlabel(x_string)
  plt.ylabel(s_string)
  plt.xlim([x_min,x_max])
  plt.ylim([s_min,s_max])
  plt.title('t='+str(t_array[3])+t_unit)
  plt.legend(('PFLOTRAN','analytical'),'best',numpoints=1)
  plt.annotate(note, xy=(.03, .990),xycoords='figure fraction',
               horizontalalignment='left',verticalalignment='top',fontsize=14)
  plt.savefig(path+'/comparison_plot.png')

  return;


################################################################################
def plot_2D_transient(path,t_array,t_unit,x_array,y_array,s_benchmark,
                      s_pflotran,xy_string,s_string,e_string):
  s_max = math.ceil(np.max(s_benchmark*10.0))/10.0
  s_min = math.floor(np.min(s_benchmark*10.0))/10.0
  if s_min < 0:
    s_min = 0
  x_max = math.ceil(np.max(x_array))
  x_min = math.floor(np.min(x_array))
  y_max = math.ceil(np.max(y_array))
  y_min = math.floor(np.min(y_array))
  levels = np.linspace(s_min,s_max,11)
  X,Y = np.meshgrid(y_array,x_array)
  
  mode = get_mode(path)
  note = mode + ' ' + e_string + '% error'

  # Plot the PFLOTRAN and analytical solutions
  plt.figure(figsize=(11,10))
  plt.subplot(221)
  plt.contourf(Y,X,s_benchmark[0,:,:],levels,alpha=0.75)
  C = plt.contour(Y,X,s_pflotran[0,:,:],levels,colors='black',linewidth=0.5)
  plt.clabel(C,inline=True,fontsize=10)
  plt.xlabel(xy_string)
  plt.ylabel(xy_string)
  plt.xlim([x_min,x_max])
  plt.ylim([y_min,y_max])
  plt.title(s_string+' '+str(t_array[0])+t_unit)
  plt.subplot(222)
  plt.contourf(Y,X,s_benchmark[1,:,:],levels,alpha=0.75)
  C = plt.contour(Y,X,s_pflotran[1,:,:],levels,colors='black',linewidth=0.5)
  plt.clabel(C,inline=True,fontsize=10)
  plt.xlabel(xy_string)
  plt.ylabel(xy_string)
  plt.xlim([x_min,x_max])
  plt.ylim([y_min,y_max])
  plt.title(s_string+' '+str(t_array[1])+t_unit)
  plt.subplot(223)
  plt.contourf(Y,X,s_benchmark[2,:,:],levels,alpha=0.75)
  C = plt.contour(Y,X,s_pflotran[2,:,:],levels,colors='black',linewidth=0.5)
  plt.clabel(C,inline=True,fontsize=10)
  plt.xlabel(xy_string)
  plt.ylabel(xy_string)
  plt.xlim([x_min,x_max])
  plt.ylim([y_min,y_max])
  plt.title(s_string+' '+str(t_array[2])+t_unit)
  plt.subplot(224)
  plt.contourf(Y,X,s_benchmark[3,:,:],levels,alpha=0.75)
  C = plt.contour(Y,X,s_pflotran[3,:,:],levels,colors='black',linewidth=0.5)
  plt.clabel(C,inline=True,fontsize=10)
  plt.xlabel(xy_string)
  plt.ylabel(xy_string)
  plt.xlim([x_min,x_max])
  plt.ylim([y_min,y_max])
  plt.title(s_string+' '+str(t_array[3])+t_unit)
  ann_title = 'Analytical (fill) vs. PFLOTRAN (contour)'+' '+note
  plt.annotate(ann_title,xy=(.03,.990),xycoords='figure fraction',
               horizontalalignment='left',verticalalignment='top',fontsize=14)
  plt.savefig(path+'/comparison_plot.png')

  return;


################################################################################
def add_to_report(path,outcome,error,ierr):
  # look for /qa_tests in path
  k = 0
  for letter in path:
    if letter == '/':
      ind = k
      if path[ind:ind+9] == '/qa_tests':
        ind_write = k
        filename = path[0:ind+9]+'/report.txt'
    k = k + 1
  target = open(filename, 'a')
  target.write(path[ind_write:len(path)])
  target.write("\n")
  if ierr == 0:
    target.write("maximum relative error: ")
    target.write(str(error))
    target.write("%\n")
    target.write("Test ")
    if outcome:
      target.write("PASS\n\n")
    else:
      target.write("FAIL\n\n")
  else:
    target.write("PFLOTRAN simulation error!\n")
    target.write("Test FAIL\n\n")

  return;


################################################################################
def print_report(path):
  filename = path+'/report.txt'
  f = open(filename,'r')
  count = 0
  pass_count = 0
  fail_count = 0
  for line in f:
    if line[0:4] == 'Test':
      count = count + 1
      if line[5:9] == 'PASS':
        pass_count = pass_count + 1
      if line[5:9] == 'FAIL':
        fail_count = fail_count + 1
  f.close
  
  grade = float(pass_count)/float(count)
  if (grade >= 0.90):
    letter_grade = 'A'
  if ((grade >= 0.80) & (grade < 0.90)):
    letter_grade = 'B' 
  if ((grade >= 0.70) & (grade < 0.80)):
    letter_grade = 'C'
  if ((grade >= 0.60) & (grade < 0.70)):
    letter_grade = 'D'
  if (grade < 0.60):
    letter_grade = 'F'

  print('See report.txt for summary of test outcomes.')
  print('Total number of tests: '+str(count))
  print('Passing: '+str(pass_count))
  print('Failing: '+str(fail_count))
  print('Grade: '+ letter_grade)
    
  target = open(filename, 'a')
  target.write('--------------------------')
  target.write("\n")
  target.write("Total number of tests: ")
  target.write(str(count))
  target.write("\n")
  target.write("Passing tests: ")
  target.write(str(pass_count))
  target.write("\n")
  target.write("Failing tests: ")
  target.write(str(fail_count))
  target.write("\n")
  target.write("Grade: " + letter_grade)
    
  return;


################################################################################
def check_fig_make_blank(path):
  
  return;

QA Tests Report Card

The most recent report card is included below:

/qa_tests/thermal/steady/1D/BC_1st_kind/th_mode
maximum relative error: 0.0%
Test PASS

/qa_tests/thermal/steady/1D/BC_1st_2nd_kind/th_mode
maximum relative error: 1.4465446574920607e-14%
Test PASS

/qa_tests/thermal/steady/2D/BC_1st_kind/th_mode
maximum relative error: 2.5669896523125002e-14%
Test PASS

/qa_tests/thermal/steady/2D/BC_1st_2nd_kind/th_mode
maximum relative error: 2.67523620391604e-14%
Test PASS

/qa_tests/thermal/steady/3D/BC_1st_kind/th_mode
maximum relative error: 1.063007659099679%
Test PASS

/qa_tests/thermal/steady/1D/BC_1st_kind/general_mode
maximum relative error: 5.297130302999594e-07%
Test PASS

/qa_tests/thermal/steady/1D/BC_1st_2nd_kind/general_mode
maximum relative error: 1.0706441922238881e-07%
Test PASS

/qa_tests/thermal/steady/2D/BC_1st_kind/general_mode
maximum relative error: 2.5669896523125002e-14%
Test PASS

/qa_tests/thermal/steady/2D/BC_1st_2nd_kind/general_mode
maximum relative error: 2.2657612747452175e-14%
Test PASS

/qa_tests/thermal/steady/3D/BC_1st_kind/general_mode
maximum relative error: 1.0594355530647085e-06%
Test PASS

/qa_tests/transport/transient/1D/IC/subsurface_transport
maximum relative error: 1.088132610492797%
Test PASS

/qa_tests/transport/transient/1D/IC_with_flow/subsurface_transport
maximum relative error: 1.7667534688063569%
Test PASS

/qa_tests/thermal/transient/1D/BC_1st_kind/th_mode
maximum relative error: 1.4254723242966036%
Test PASS

/qa_tests/thermal/transient/1D/BC_1st_2nd_kind/th_mode
maximum relative error: 3.142232075986244%
Test FAIL

/qa_tests/thermal/transient/1D/BC_2nd_kind/th_mode
maximum relative error: 0.6108446855260782%
Test PASS

/qa_tests/thermal/transient/2D/BC_1st_2nd_kind/th_mode
maximum relative error: 5.389362931492208%
Test FAIL

/qa_tests/thermal/transient/1D/BC_1st_kind/general_mode
maximum relative error: 1.4254723242966574%
Test PASS

/qa_tests/thermal/transient/1D/BC_1st_2nd_kind/general_mode
maximum relative error: 3.1448809497174697%
Test FAIL

/qa_tests/thermal/transient/1D/BC_2nd_kind/general_mode
maximum relative error: 0.6108446917463037%
Test PASS

/qa_tests/thermal/transient/2D/BC_1st_2nd_kind/general_mode
maximum relative error: 5.389362848863079%
Test FAIL

/qa_tests/flow/steady/1D/BC_1st_kind/richards_mode
maximum relative error: 1.005485003434104e-12%
Test PASS

/qa_tests/flow/steady/1D/BC_1st_2nd_kind/richards_mode
maximum relative error: 0.0%
Test PASS

/qa_tests/flow/steady/1D/hydrostatic/richards_mode
maximum relative error: 0.029197100931154137%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_kind/richards_mode
maximum relative error: 2.4434069317747596e-14%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_2nd_kind/richards_mode
maximum relative error: 1.991431434305214e-14%
Test PASS

/qa_tests/flow/steady/3D/BC_1st_kind/richards_mode
maximum relative error: 3.872871016134267e-14%
Test PASS

/qa_tests/flow/steady/1D/BC_1st_kind/th_mode
maximum relative error: 1.0264326076723144e-12%
Test PASS

/qa_tests/flow/steady/1D/BC_1st_2nd_kind/th_mode
maximum relative error: 0.0%
Test PASS

/qa_tests/flow/steady/1D/hydrostatic/th_mode
maximum relative error: 0.029197100931154137%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_kind/th_mode
maximum relative error: 2.4434069317747596e-14%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_2nd_kind/th_mode
maximum relative error: 1.991431434305214e-14%
Test PASS

/qa_tests/flow/steady/3D/BC_1st_kind/th_mode
maximum relative error: 3.872871016134267e-14%
Test PASS

/qa_tests/flow/steady/1D/BC_1st_kind/general_mode
maximum relative error: 0.0%
Test PASS

/qa_tests/flow/steady/1D/BC_1st_2nd_kind/general_mode
maximum relative error: 0.0%
Test PASS

/qa_tests/flow/steady/1D/hydrostatic/general_mode
maximum relative error: 0.029197100931154137%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_kind/general_mode
maximum relative error: 2.4434069317747596e-14%
Test PASS

/qa_tests/flow/steady/2D/BC_1st_2nd_kind/general_mode
maximum relative error: 1.991431434305214e-14%
Test PASS

/qa_tests/flow/steady/3D/BC_1st_kind/general_mode
maximum relative error: 3.872871016134267e-14%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_kind/richards_mode
maximum relative error: 1.0295216905978668%
Test PASS

/qa_tests/flow/transient/1D/BC_2nd_kind/richards_mode
maximum relative error: 0.8386376892269776%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_2nd_kind/richards_mode
maximum relative error: 0.6999341451387711%
Test PASS

/qa_tests/flow/transient/2D/BC_1st_2nd_kind/richards_mode
maximum relative error: 1.8128418463275258%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_kind/th_mode
maximum relative error: 1.0295216905946651%
Test PASS

/qa_tests/flow/transient/1D/BC_2nd_kind/th_mode
maximum relative error: 0.8386376894645762%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_2nd_kind/th_mode
maximum relative error: 0.6999341451724934%
Test PASS

/qa_tests/flow/transient/2D/BC_1st_2nd_kind/th_mode
maximum relative error: 1.8128418460459321%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_kind/general_mode
maximum relative error: 1.0317476605757487%
Test PASS

/qa_tests/flow/transient/1D/BC_2nd_kind/general_mode
maximum relative error: 0.838638899646684%
Test PASS

/qa_tests/flow/transient/1D/BC_1st_2nd_kind/general_mode
maximum relative error: 0.6999333569924733%
Test PASS

/qa_tests/flow/transient/2D/BC_1st_2nd_kind/general_mode
maximum relative error: 1.812874560251693%
Test PASS

/qa_tests/gas/steady/1D/BC_1st_kind/general_mode
maximum relative error: 0.18746245588855523%
Test PASS

/qa_tests/gas/steady/1D/BC_1st_2nd_kind/general_mode
maximum relative error: 0.5376624094986984%
Test PASS

/qa_tests/gas/steady/2D/BC_1st_2nd_kind/general_mode
maximum relative error: 0.6186144564031424%
Test PASS

/qa_tests/gas/steady/3D/BC_2nd_kind/general_mode
maximum relative error: 6.701136031513438%
Test FAIL

--------------------------
Total number of tests: 54
Passing tests: 49
Failing tests: 5
Grade: A