Testing#
Unit testing and integration testing are critical for the successful continuous integration (CI) and delivery of any library belonging to the PyAnsys project.
In 1993, Kent Beck developed Test Driven Development (TDD) as part of the Extreme Programming software development process. TDD is the practice of writing unit tests before writing production code. The benefit of this practice is that you know each new line of code is working as soon as it is written. It’s easier to track down problems because only a small amount of code has been implemented since the execution of the last test. Furthermore, all test cases do not have to be implemented at once but rather gradually as the code evolves.
You should follow TDD when developing your PyAnsys project. Examples and best practices for unit tests follow.
Test framework#
For consistency, PyAnsys tools and libraries should use either the pytest or
unittest framework
for unit testing. The pytest
framework is recommended unless a constraint
in your project prevents you from using it. As described in Required files,
you should place unit tests in The tests directory in the library’s
root directory.
Add testing dependencies#
Requirements for testing dependencies should be included in The setup.py file, The pyproject.toml file, or a
requirements_tests.txt
file. Only pytest
and pytest-cov
must be specified as third-party dependencies because unittest
is included
in The Python Standard Library.
[project.optional-dependencies]
test = [
"pytest",
"pytest-cov",
]
[tool.poetry.group.test.dependencies]
pytest = "*"
pytest-cov = "*"
setup(
name="ansys-<product>-<library>",
...,
extras_require={
"test": ["pytest", "pytest-cov"],
},
)
pytest
pytest-cov
You can use pip
to install these testing dependencies:
python -m pip install .[test]
python -m pip install -r requirements_tests.txt
Organize test files#
You must place your test files in The tests directory. To
guarantee that tests are run against the library source code, follow a src
layout as explained in The src directory rather than
having your Python library source located directly in the repository root directory.
This helps you to achieve these objectives:
Avoid testing the source of the repository rather than testing the installed package.
Catch errors caused by files that might be missed by the installer, including any C extensions or additional internal packages.
Test execution#
Once you have installed pytest
, you can execute the test suite with this command:
pytest -v tests/
Filter tests#
To run a subset of all available tests, you can taking advantage
of the keywords
and markers
flags:
Filter tests by keywords
pytest -k '<name pattern>'
pytest -k 'not <name pattern>'
Filter tests by markers
pytest -m slow
For more information about filtering tests, see Working with custom markers in the pytest
documentation.
Testing methodology#
You should consider three levels of testing for your PyAnsys library: unit, integration, and functional.
Unit testing validates your library at the lowest possible level, isolating individual classes and methods without any communication with other libraries or services.
Integration testing validates that your library works in the context of an app or software stack. For example, if your library extends or wraps the features of an external service, you must test that service in conjunction with your library. On GitHub, the ideal approach for this would be to start your service using Docker and then test accordingly. You should still be testing at the individual class or method level, but you can now test how multiple libraries or services interact. This is mandatory for testing APIs and is preferred over mocking the service.
Functional testing should be used for validating workflows or long-running examples. Assume that you have a library that wraps a CAD service. You would validate that you can create complex geometry while directly interfacing with the service. Functional tests are great at discovering edge cases that are not normally found at the unit or integration level. However, functional testing should be limited to only a handful of examples because these tend to be long running and difficult to validate.
Each PyAnsys project should have all three levels of testing implemented in its testing framework. Consider implementing functional tests as examples within your project’s documentation examples. This lets you write helpful user-facing tests while accomplishing functional testing.
Unit testing#
Unit testing tests at the lowest possible level, isolated from other applications or libraries. For Python tool libraries like ansys-tools-protoc-helper, unit testing is sufficient to get high coverage (> 80%) of your library while actually testing the library.
These tests should be written to test a single method in isolation. For example,
the following parse_chunks.py
file has a method that deserializes chunks. The
associated test_parse_chunks_py
file tests this method in isolation.
Note
This example assumes that you do not have a serialize_chunks
function in your
library. If you did, you could exclude it from the test_parse_chunks.py
file.
def parse_chunks(chunks):
"""Deserialize gRPC chunks into a Numpy array.
Parameters
----------
chunks : generator
Generator from gRPC. Each chunk contains a bytes payload.
dtype : np.dtype
Numpy data type to interpret chunks as.
Returns
-------
array : np.ndarray
Deserialized Numpy array.
"""
arrays = []
for chunk in chunks:
arrays.append(np.frombuffer(chunk.payload, ANSYS_VALUE_TYPE[chunk.value_type]))
return np.hstack(arrays)
from ansys.api.mapdl.v0 import ansys_kernel_pb2 as anskernel
import numpy as np
import pytest
from ansys.mapdl.core.common_grpc import parse_chunks
DEFAULT_CHUNKSIZE = 256*1024 # 256 kB
@pytest.fixture()
def sample_array():
"""Generate a non-trivial (n x 3) float array."""
sz = np.random.randint(100000, 200000)
array = np.random.random((sz, 3)).astype(np.float64)
assert array.nbytes > DEFAULT_CHUNKSIZE
return array
def serialize_chunks(array):
"""Serialize an array into chunks."""
# convert to raw
raw = array.tobytes()
value_type = 5 # float64
i = 0
while True:
piece = raw[i:i + DEFAULT_CHUNKSIZE]
i += DEFAULT_CHUNKSIZE
length = len(piece)
if length == 0:
break
yield anskernel.Chunk(payload=piece, size=length, value_type=value_type)
def test_deserialize_chunks(sample_array):
parsed_array = parse_chunks(serialize_chunks(sample_array))
parsed_array = parsed_array.reshape(-1, 3)
assert np.allclose(sample_array, parsed_array)
Integration testing#
This section explains Wrapped service methods and how to Test using remote method invocation.
Wrapped service methods#
Any PyAnsys library that provides features by wrapping a gRPC interface
should include tests of the gRPC methods exposed by the PROTO files and wrapped
by the Python library. They would not be expected to test the features of
the server but rather the APIs exposed by the server. For example, if testing
the GetNode
gRPC method, then your integration test would test the wrapped
Python function. If the Python library wraps this gRPC method with a
get_node
method, your test would be implemented within the
tests/test_nodes.py
file.
message Node
{
int32 id = 1;
double x = 2;
double y = 3;
double z = 4;
}
message NodeRequest {
int32 num = 1;
}
message NodeResponse {
Node node = 1;
}
service SomeService {
rpc GetNode(NodeRequest) returns (NodeResponse);
// other methods
}
from ansys.product.service.v0 import service_pb2
def get_node(self, index):
"""Get the coordinates of a node for a given index.
Parameters
----------
index : int
Index of the node.
Returns
-------
tuple
Coordinates of the node.
Examples
--------
>>> from ansys.product.service import SomeService
>>> srv = SomeService()
>>> srv.create_node(1, 4.5, 9.0, 3.2)
>>> node = srv.get_node(1)
>>> node
(4.5, 9.0, 3.2)
"""
resp = service_pb2.GetNode(index=index)
return resp.x, resp.y, resp.z
def test_get_node(srv):
srv.clear()
node_index = 1
node_coord = 0, 10, 20
srv.create_node(node_index, node_coord*)
assert srv.get_node(node_index) == node_coord
The goal of the unit test should be to test the API rather than the product or
service. The GetNode
gRPC method should have already been tested when
designing and developing the service.
Test using remote method invocation#
For a Remote Method Invocation (RMI)-like method, it is only necessary to test the method with a basic case and potentially with any edge cases. A RMI-like API might send and receive strings that are executed on the server using a custom API or language only available within the context of the service.
For example, if a method has a RMI service definition named SendCommand()
and
a Python wrapping named send_command
, your code and the example test would look
like this:
message SendCommand()
def send_command(command):
"""Run a command on the remote server.
Parameters
----------
command : str
Command to run on the remote server.
"""
def test_send_command(srv):
output = srv.send_command("CREATE,1")
assert "Created 1" in output
Note that this test only validates that the "CREATE,1"
command has been
received, executed, and sent back to the client. It does not validate all
commands. Running such a test is necessary only if there are edge cases, which
include characters that cannot be streamed or use long-running commands.
Functional testing#
Functional testing should test the Python library using scripts or examples
that are expected to be executed by the user. Unlike unit or integration
testing, functional tests are testing the library as a whole by calling
several methods to accomplish a task. You should run these tests only after unit
and integration testing is complete. Ideally, you should run them outside the
pytest
framework while building documentation with Sphinx-Gallery.
Note
Functional tests should not contribute to global library coverage. Testing should always be done on individual functions or methods.
Test code coverage#
Because Python is an interpreted language, syntax errors can only be caught during the almost trivial compile times. Thus, developers of Python libraries should aim to have high coverage for their libraries. Coverage is defined as parts of the executable and usable source that are tested by unit tests. You can use the pytest-cov library to view the coverage for your library.
Configure code coverage#
If you do not configure code coverage properly, the resulting report does not show the real scope covered by the test suite.
Assuming that a PyAnsys
project follows The src directory layout,
you must pass the following flag when executing tests:
pytest --cov=ansys.<product>.<library> --cov-report=term tests/
This command tells pytest-cov
to look for source code in the
src/ansys/<product>
directory and generate a terminal report for all tests
located in The tests directory.
While 100% coverage is ideal, the law of diminishing returns applies to
the coverage of a Python library. Consequently, achieving 80-90% coverage is
often sufficient. For parts of your library that are difficult or impossible
to test, consider using # pragma: no cover
at the end of the method
definition, branch, or line to denote that part of the code cannot be
reasonably tested. For example, if part of your module performs a simple
import
test of matplotlib
and raises an error when the library is not
installed, it is not reasonable to attempt to test this and assume full
coverage:
try:
import matplotlib
except ImportError: # pragma: no cover
raise ImportError("Install matplotlib to use this feature.")
You should only avoid coverage of parts of your library where you cannot
reasonably test without an extensive testing suite or setup. Most methods and
classes, including edge cases, can be reasonably tested. Even parts of your code
that raise errors like TypeError
or ValueError
when users input the
wrong data type or value can be reasonably tested.
Enforce code coverage#
One way of enforcing unit test coverage with a project on GitHub is to use
codecov.io
to enforce minimum patch (and optionally project) coverage. Because
this app is already available to the Ansys GitHub organization, you can simply
add a codecov.yml
file to the root directory of your repository. This example
file provides a sample configuration:
comment:
layout: "diff"
behavior: default
coverage:
status:
project:
default:
# basic
# target: 50%
threshold: 0%
# advanced
if_not_found: success
if_ci_failed: error
if_no_uploads: error
patch:
default:
# basic
target: 90%
if_not_found: success
if_ci_failed: error
if_no_uploads: error
Using a codecov.yml
file requires that each PR has a patch coverage of 90%, meaning that 90% of any
source added to the repository (unless ignored) must be covered by unit tests.
Test using GitHub Actions#
Effective CI/CD assumes that unit testing is developed during feature development or bug fixes. However, given the limited scope of the local development environment, it is often not possible to enforce testing on multiple platforms, or even to enforce unit testing in general. However, with the proper automated CI/CD, such testing can still occur and be enforced automatically.
GitHub Actions is the preferred automated CI/CD platform for running Python library unit tests for PyAnsys. It can be used immediately by cloning the project template.
tests:
name: "Test library"
runs-on: ${{ matrix.os }}
strategy:
matrix:
os: [ubuntu-latest, windows-latest]
python-version: ['3.9', '3.10', '3.11', '3.12']
steps:
- name: "Run pytest"
uses: ansys/actions/tests-pytest@v4
with:
python-version: ${{ matrix.python-version }}
pytest-markers: "-k 'mocked'"
pytest-extra-args: "--cov=ansys.<library> --cov-report=term --cov-report=xml:.cov/coverage.xml --cov-report=html:.cov/html"
- name: "Upload coverage results"
uses: actions/upload-artifact@v4
if: matrix.python-version == ${{ env.MAIN_PYTHON_VERSION }}
with:
name: coverage-html
path: .cov/html
retention-days: 7