.. _testing: Testing ======= Unit testing and integration testing are critical for the successful continuous integration (CI) and delivery of any library belonging to the PyAnsys project. In 1993, Kent Beck developed `Test Driven Development (TDD)`_ as part of the Extreme Programming software development process. TDD is the practice of writing unit tests before writing production code. The benefit of this practice is that you know each new line of code is working as soon as it is written. It's easier to track down problems because only a small amount of code has been implemented since the execution of the last test. Furthermore, all test cases do not have to be implemented at once but rather gradually as the code evolves. You should follow TDD when developing your PyAnsys project. Examples and best practices for unit tests follow. Test framework -------------- .. raw:: html
For consistency, PyAnsys tools and libraries should use either the `pytest`_ or `unittest `_ framework for unit testing. The ``pytest`` framework is recommended unless a constraint in your project prevents you from using it. As described in :ref:`Required files`, you should place unit tests in :ref:`The \`\`tests\`\` directory` in the library's root directory. Add testing dependencies ~~~~~~~~~~~~~~~~~~~~~~~~ Requirements for testing dependencies should be included in :ref:`The \`\`setup.py\`\` file`, :ref:`The \`\`pyproject.toml\`\` file`, or a ``requirements_tests.txt`` file. Only ``pytest`` and `pytest-cov`_ must be specified as third-party dependencies because ``unittest`` is included in `The Python Standard Library `_. .. tab-set:: .. tab-item:: Flit .. code-block:: toml [project.optional-dependencies] test = [ "pytest", "pytest-cov", ] .. tab-item:: Poetry .. code-block:: toml [tool.poetry.group.test.dependencies] pytest = "*" pytest-cov = "*" .. tab-item:: Setuptools .. code-block:: python setup( name="ansys--", ..., extras_require={ "test": ["pytest", "pytest-cov"], }, ) .. tab-item:: Requirements .. code-block:: text pytest pytest-cov You can use ``pip`` to install these testing dependencies: .. tab-set:: .. tab-item:: From setup.py or pyproject.toml .. code-block:: text python -m pip install .[test] .. tab-item:: From requirements_tests.txt .. code-block:: text python -m pip install -r requirements_tests.txt Organize test files ~~~~~~~~~~~~~~~~~~~ You must place your test files in :ref:`The \`\`tests\`\` directory`. To guarantee that tests are run against the library source code, follow a ``src`` layout as explained in :ref:`The \`\`src\`\` directory` rather than having your Python library source located directly in the repository root directory. This helps you to achieve these objectives: - Avoid testing the source of the repository rather than testing the installed package. - Catch errors caused by files that might be missed by the installer, including any C extensions or additional internal packages. Test execution -------------- Once you have installed ``pytest``, you can execute the test suite with this command: .. code-block:: text pytest -v tests/ Filter tests ~~~~~~~~~~~~ To run a subset of all available tests, you can taking advantage of the ``keywords`` and ``markers`` flags: **Filter tests by keywords** .. code-block:: text pytest -k '' pytest -k 'not ' **Filter tests by markers** .. code-block:: text pytest -m slow For more information about filtering tests, see `Working with custom markers `_ in the ``pytest`` documentation. Testing methodology ------------------- You should consider three levels of testing for your PyAnsys library: unit, integration, and functional. * :ref:`Unit testing` validates your library at the lowest possible level, isolating individual classes and methods without any communication with other libraries or services. * :ref:`Integration testing` validates that your library works in the context of an app or software stack. For example, if your library extends or wraps the features of an external service, you must test that service in conjunction with your library. On GitHub, the ideal approach for this would be to start your service using `Docker`_ and then test accordingly. You should still be testing at the individual class or method level, but you can now test how multiple libraries or services interact. This is mandatory for testing APIs and is preferred over mocking the service. * :ref:`Functional testing` should be used for validating workflows or long-running examples. Assume that you have a library that wraps a CAD service. You would validate that you can create complex geometry while directly interfacing with the service. Functional tests are great at discovering edge cases that are not normally found at the unit or integration level. However, functional testing should be limited to only a handful of examples because these tend to be long running and difficult to validate. Each PyAnsys project should have all three levels of testing implemented in its testing framework. Consider implementing functional tests as examples within your project's documentation examples. This lets you write helpful user-facing tests while accomplishing functional testing. Unit testing ~~~~~~~~~~~~ Unit testing tests at the lowest possible level, isolated from other applications or libraries. For Python tool libraries like `ansys-tools-protoc-helper`_, unit testing is sufficient to get high coverage (> 80%) of your library while actually testing the library. .. _ansys-tools-protoc-helper: https://github.com/ansys/ansys-tools-protoc-helper These tests should be written to test a single method in isolation. For example, the following ``parse_chunks.py`` file has a method that deserializes chunks. The associated ``test_parse_chunks_py`` file tests this method in isolation. .. note:: This example assumes that you do not have a ``serialize_chunks`` function in your library. If you did, you could exclude it from the ``test_parse_chunks.py`` file. .. tab-set:: .. tab-item:: parse_chunks.py .. code-block:: python def parse_chunks(chunks): """Deserialize gRPC chunks into a Numpy array. Parameters ---------- chunks : generator Generator from gRPC. Each chunk contains a bytes payload. dtype : np.dtype Numpy data type to interpret chunks as. Returns ------- array : np.ndarray Deserialized Numpy array. """ arrays = [] for chunk in chunks: arrays.append(np.frombuffer(chunk.payload, ANSYS_VALUE_TYPE[chunk.value_type])) return np.hstack(arrays) .. tab-item:: test_parse_chunks.py .. code-block:: python from ansys.api.mapdl.v0 import ansys_kernel_pb2 as anskernel import numpy as np import pytest from ansys.mapdl.core.common_grpc import parse_chunks DEFAULT_CHUNKSIZE = 256*1024 # 256 kB @pytest.fixture() def sample_array(): """Generate a non-trivial (n x 3) float array.""" sz = np.random.randint(100000, 200000) array = np.random.random((sz, 3)).astype(np.float64) assert array.nbytes > DEFAULT_CHUNKSIZE return array def serialize_chunks(array): """Serialize an array into chunks.""" # convert to raw raw = array.tobytes() value_type = 5 # float64 i = 0 while True: piece = raw[i:i + DEFAULT_CHUNKSIZE] i += DEFAULT_CHUNKSIZE length = len(piece) if length == 0: break yield anskernel.Chunk(payload=piece, size=length, value_type=value_type) def test_deserialize_chunks(sample_array): parsed_array = parse_chunks(serialize_chunks(sample_array)) parsed_array = parsed_array.reshape(-1, 3) assert np.allclose(sample_array, parsed_array) Integration testing ~~~~~~~~~~~~~~~~~~~ This section explains :ref:`Wrapped service methods` and how to :ref:`Test using remote method invocation`. Wrapped service methods +++++++++++++++++++++++ Any PyAnsys library that provides features by wrapping a gRPC interface should include tests of the gRPC methods exposed by the PROTO files and wrapped by the Python library. They would not be expected to test the features of the server but rather the APIs exposed by the server. For example, if testing the ``GetNode`` gRPC method, then your integration test would test the wrapped Python function. If the Python library wraps this gRPC method with a ``get_node`` method, your test would be implemented within the ``tests/test_nodes.py`` file. .. tab-set:: .. tab-item:: gRPC code .. code-block:: rust message Node { int32 id = 1; double x = 2; double y = 3; double z = 4; } message NodeRequest { int32 num = 1; } message NodeResponse { Node node = 1; } service SomeService { rpc GetNode(NodeRequest) returns (NodeResponse); // other methods } .. tab-item:: Python code .. code-block:: python from ansys.product.service.v0 import service_pb2 def get_node(self, index): """Get the coordinates of a node for a given index. Parameters ---------- index : int Index of the node. Returns ------- tuple Coordinates of the node. Examples -------- >>> from ansys.product.service import SomeService >>> srv = SomeService() >>> srv.create_node(1, 4.5, 9.0, 3.2) >>> node = srv.get_node(1) >>> node (4.5, 9.0, 3.2) """ resp = service_pb2.GetNode(index=index) return resp.x, resp.y, resp.z .. tab-item:: Unit test .. code-block:: python def test_get_node(srv): srv.clear() node_index = 1 node_coord = 0, 10, 20 srv.create_node(node_index, node_coord*) assert srv.get_node(node_index) == node_coord The goal of the unit test should be to test the API rather than the product or service. The ``GetNode`` gRPC method should have already been tested when designing and developing the service. Test using remote method invocation +++++++++++++++++++++++++++++++++++ For a Remote Method Invocation (RMI)-like method, it is only necessary to test the method with a basic case and potentially with any edge cases. A RMI-like API might send and receive strings that are executed on the server using a custom API or language only available within the context of the service. For example, if a method has a RMI service definition named ``SendCommand()`` and a Python wrapping named ``send_command``, your code and the example test would look like this: .. tab-set:: .. tab-item:: gRPC code .. code-block:: rust message SendCommand() .. tab-item:: Python code .. code-block:: python def send_command(command): """Run a command on the remote server. Parameters ---------- command : str Command to run on the remote server. """ .. tab-item:: Unit test .. code-block:: python def test_send_command(srv): output = srv.send_command("CREATE,1") assert "Created 1" in output Note that this test only validates that the ``"CREATE,1"`` command has been received, executed, and sent back to the client. It does not validate all commands. Running such a test is necessary only if there are edge cases, which include characters that cannot be streamed or use long-running commands. Functional testing ~~~~~~~~~~~~~~~~~~ Functional testing should test the Python library using scripts or examples that are expected to be executed by the user. Unlike unit or integration testing, functional tests are testing the library as a whole by calling several methods to accomplish a task. You should run these tests only after unit and integration testing is complete. Ideally, you should run them outside the ``pytest`` framework while building documentation with `Sphinx-Gallery `_. .. note:: Functional tests should not contribute to global library coverage. Testing should always be done on individual functions or methods. Test code coverage ------------------ Because Python is an interpreted language, syntax errors can only be caught during the almost trivial compile times. Thus, developers of Python libraries should aim to have high coverage for their libraries. Coverage is defined as parts of the executable and usable source that are tested by unit tests. You can use the `pytest-cov`_ library to view the coverage for your library. Configure code coverage ~~~~~~~~~~~~~~~~~~~~~~~ If you do not configure code coverage properly, the resulting report does not show the real scope covered by the test suite. Assuming that a ``PyAnsys`` project follows :ref:`The \`\`src\`\` directory` layout, you must pass the following flag when :ref:`executing tests `: .. code-block:: text pytest --cov=ansys.. --cov-report=term tests/ This command tells ``pytest-cov`` to look for source code in the ``src/ansys/`` directory and generate a terminal report for all tests located in :ref:`The \`\`tests\`\` directory`. While 100% coverage is ideal, the law of diminishing returns applies to the coverage of a Python library. Consequently, achieving 80-90% coverage is often sufficient. For parts of your library that are difficult or impossible to test, consider using ``# pragma: no cover`` at the end of the method definition, branch, or line to denote that part of the code cannot be reasonably tested. For example, if part of your module performs a simple ``import`` test of ``matplotlib`` and raises an error when the library is not installed, it is not reasonable to attempt to test this and assume full coverage: .. code:: python try: import matplotlib except ImportError: # pragma: no cover raise ImportError("Install matplotlib to use this feature.") You should only avoid coverage of parts of your library where you cannot reasonably test without an extensive testing suite or setup. Most methods and classes, including edge cases, can be reasonably tested. Even parts of your code that raise errors like ``TypeError`` or ``ValueError`` when users input the wrong data type or value can be reasonably tested. Enforce code coverage ~~~~~~~~~~~~~~~~~~~~~ One way of enforcing unit test coverage with a project on GitHub is to use ``codecov.io`` to enforce minimum patch (and optionally project) coverage. Because this app is already available to the `Ansys GitHub organization`_, you can simply add a ``codecov.yml`` file to the root directory of your repository. This example file provides a sample configuration: .. code:: yaml comment: layout: "diff" behavior: default coverage: status: project: default: # basic # target: 50% threshold: 0% # advanced if_not_found: success if_ci_failed: error if_no_uploads: error patch: default: # basic target: 90% if_not_found: success if_ci_failed: error if_no_uploads: error Using a ``codecov.yml`` file requires that each PR has a patch coverage of 90%, meaning that 90% of any source added to the repository (unless ignored) must be covered by unit tests. Test using GitHub Actions ------------------------- Effective CI/CD assumes that unit testing is developed during feature development or bug fixes. However, given the limited scope of the local development environment, it is often not possible to enforce testing on multiple platforms, or even to enforce unit testing in general. However, with the proper automated CI/CD, such testing can still occur and be enforced automatically. `GitHub Actions`_ is the preferred automated CI/CD platform for running Python library unit tests for PyAnsys. It can be used immediately by cloning the project `template `_. .. literalinclude:: code/tests.yml :language: yaml