Cicada: An integration testing framework for Docker and Kubernetes

Photo by Markus Winkler on Unsplash

It’s hard to be confident that the software you’ve just written will actually work. You can write unit tests, but these are for testing small, pure sections of code. There’s only so much certainty you can have if you have to mock connections to every single database and API your application will connect to. This is the problem integration testing tries to solve. As architectures become more complex, better tools for automatically testing services through multiple avenues are needed.

Some Background

I wrote Cicada because I needed to test a service connected to multiple APIs, queues, and databases. This is the process it would follow:

1. Receive a message from a queue containing an ID
2. Using an ID in the message, make an API call to get a thing
3. Do stuff with the thing received from the API to make a new thing
4. Save the new thing in a database
5. Send a message to another queue to tell another service to do stuff

Straightforward? Maybe. But there were a lot of external services in this flow that made it difficult to test. I’d have to send a message to it over JMS and verify that an outbound message was sent after it was done. I’d have to setup documents in the API to make sure the ID in the message would return something. I would also have to verify the correct rows were added in the database.

As complicated as it seemed to me, this was a solved problem. My team was responsible for a few other applications with similar flows, and had written different integration testing suites for each one. This brought me to creating Cicada. Cicada is an integration testing framework that abstracts many of the scenarios I saw re-implemented in different codebases. It also leverages Docker and Kubernetes to create environments where code can be tested.

Test Runners

One of my primary concerns with creating this framework was to keep the use cases extensible. In the previous example, I would need to test database connections, queues, and interact with a REST API. Cicada accomplishes this by moving all of the service connection logic to ‘Runners’. A Runner is effectively a gRPC server inside of a Docker container (and within a Kubernetes pod if applicable). As long as the image implements this gRPC schema, Cicada can start the container and tell it to run actions and asserts. Here is an example of a test using the ‘rest-runner’:

description: Example test
tests:
  - name: check-google
    description: Checks the status of the Google homepage
    runner: rest-runner
    asserts:
      - type: StatusCode
        params:
          method: GET
          actionParams:
            url: https://google.com
          expected: 200

In this example, Cicada will start the rest-runner container and tell it to run the StatusCode assert. This will make a GET request to https://google.com and verify it receives a 200 response code. If it doesn’t receive this, it will retry until it does or times out (15 seconds by default)

Currently, the following runners are available:

• rest-runner
• grpc-runner
• s3-runner
• kafka-runner
• sql-runner

Test State and Dependencies

Another problem I wanted to solve in Cicada were instances where the results of one test were needed in another (i.e. setup and teardown). In my use case, I would need to hit the REST API before starting the test, and use the ID returned in later steps. Cicada solves this through a shared state container, Jinja2 templating, and parallel runner execution.

For example, let’s say we were testing an API used to manage members for a website. First, we can use Cicada to hit API’s /members route to create a member:

description: Add and verify member test
tests:
  - name: add-member
    description: Creates a member
    runner: rest-runner
    actions:
      - type: POST
        params:
          url: "http://api:8080/members"
          body:
            name: Mr. Peanutbutter
            age: 47

After this test runs, it will save the data returned from the runner in the state container which will be available to subsequently ran tests. More on the structure of the state container here.

In another test, we can use the member that was just created to check if the new member can be retrieved from the API:

- name: check-member-added
  description: Checks that API returns new member
  dependencies:
    - add-member
  runner: rest-runner
  asserts:
    - type: JSON
      template: >
        params:
          method: GET
          actionParams:
            url: http://api:8080/members/{{ state['add-member']['actions']['POST0']['results'][0]['body']['id'] }}
          expected:
            name: Mr. Peanutbutter
            age: 47

Notice that there is a dependencies section and a template section. If a test has dependencies, the dependent test will run before it starts. Otherwise, steps without dependencies will run in parallel with other dependency-free steps.

If a template section is specified, Cicada will render it with access to the state container. In this case, it will get the first result of the POST0 action of add-member test and add the ID to the URL path. When rendered, the section might look like this:

- name: check-member-added
  description: Checks that API returns new member
  dependencies:
    - add-member
  runner: rest-runner
  asserts:
    - type: JSON
      params:
        method: GET
        actionParams:
          url: http://api:8080/members/123
        expected:
          name: Mr. Peanutbutter
          age: 47

Reports

Cicada would not be complete without report generation. At the moment, Cicada saves all of the test state after running each test and uses it to generate a test report in Markdown. Admittedly, this is still a work in progress in terms of integrations with CI/CD. An example of a generated report can be found here.

Conclusion

Cicada solves many of the problems I encountered with integration testing. However, it is still very much in development. Recently, I finished adding integrations to simplify the process for running it inside Kubernetes. Please feel free to give it a shot and let me know what you think!

Thanks for reading,

Jeremy