Alternatives to Postman for Exploratory and Automated API Testing

Prefer Open Source and Code Based Tooling

Update: In response to community feedback, I've enriched this newsletter with additional information to assist you and your teams in making informed decisions about API Testing tools..

Postman is a great tool for exploratory API Testing. However, it is not the best tool for automated API system testing and there will be a change in September which can result in higher costs for using Postman. In this blog post, I will explain why from my perspective code based open source tools are often better for automated API system testing than GUI based tools like Postman or Insomnia. I will also provide some recommendations which tool to use for which technology stack.

System Testing against the API layer is a vital part of software development, as it ensures that the APIs are functioning correctly and meeting the expected requirements. However, not all API testing tools are created equal. In this blog post, I will compare and contrast two types of API testing tools: code based open source tools and GUI based tools like Postman or Insomnia. While Postman and Insomnia have both value in the context of exploratory API Testing – I will argue that code based open source tools are better for automated API system testing, especially in terms of maintainability and integration into existing development infrastructures.

Postman change on the 15th of September 2023

On September 15th, Postman removed the legacy Scratch Pad component from its Postman Desktop App, but you can still continue using the same API testing, documentation and design functionality in the same app for free, in combination with a SaaS account that provides additional free functionality like monitors, mocking, real time collaboration and developer portal integration. Pricing is not changed, unless you have not paid for Postman before and need to adapt an Enterprise plan in order to use Postman’s SaaS offering in a company specific compliance manner. Still compliance requirements make it necessary for many companies to switch to an Enterprise plan.

Newman is OpenSource

Newman is an official open-source project associated with Postman. Its purpose is to enable the execution of any Postman collection test suite within a Continuous Integration / Continuous Delivery (CI/CD) pipeline. It can also be integrated natively into a JavaScript-based execution environment using an npm module. Moreover, Newman can successfully function with collections located within Postman’s SaaS platform, files on the local file system, and files within a version control system that follows the official collection format. In essence, collections can be run using this open-source tool. However, for convenient maintenance of collections and other artefacts, Postman is required.

Testing against the API layer on the Component Level

There are different levels of testing against the API. In addition to do API system testing through the whole stack, it is also possible to test against the API on the component level including isolation from fragile or slow dependencies (e.g. database or external services). Those are not covered by this document and usually require a different toolset fitting to the technology stack of the backend. For example, for Spring Boot there are several ways with different tradeoffs how this can be done.

Why Code Based Open Source Tools Are Better for Automated API System Testing compared to Postman, Insomnia and Bruno

System Testing against the API layer is a vital part of software development, as it ensures that the APIs are functioning correctly and meeting the expected requirements. However, not all API testing tools are created equal. In this blog post, I will compare and contrast two types of API testing tools: code based open source tools and GUI based tools like Postman, Insomnia and Bruno. While Bruno, Postman and Insomnia have value in the context of exploratory API Testing – I will argue that code based open source tools are better for automated API system testing, especially in terms of maintainability and integration into existing development infrastructures. And even for exploratory API testing many developers just use the tools for automated API system testing and do not need the special category.

What are code based open source tools for API system testing?

Code based open source tools are API testing tools that allow the user to write and execute test scripts using a programming language, such as Java, Python, JavaScript, etc. Some examples of code based open source tools are REST-Assured, Karate, supertest and PyTest.

What are GUI based tools like Postman, Insomnia or Bruno for exploratory API Testing?

GUI based tools like Postman, Insomnia or Bruno are API testing tools that provide a graphical user interface (GUI) for the user to create and run test requests using a point-and-click approach. These tools are popular among developers and testers, as they offer a user-friendly and intuitive interface. Some features of these tools include:

• Support for various authentication methods

• Ability to create and manage collections of requests

• Built-in support for API documentation

• Wide range of integrations and plugins

Due to cost and compliance issues Postman and Insomnia cannot be recommended anymore. Bruno is quite similar to Postman and Insomnia and available as OpenSource, free to use and does need a cloud account, collections can be stored in your version control system and Bruno includes cli tool for running collections. More details can be found in the comparison page.

What are the alternatives to Postman, Insomnia and Bruno?

There are a lot of tools available that are similar to Postman and Insomnia – for a listing of the most popular, see Awesome HTTP Clients. There is e.g. good support for JetBrains, a plugin for VS Code or the restclient package for Emacs, and a CLI, amongst others. Only a few of those are allowed to be used at SAP, more details about the SRI entry you can find at the end of the blog. An advantage is that this allows to store these request files in your repo, version control them like the rest of your (test) code and you are not reliant on yet another tool, but just use it from your IDE of choice.

Why are code based open source tools better for automated API system testing?

While GUI based tools like Postman or Insomnia have their advantages, they also have some limitations when it comes to automated API system testing. Automated API system testing is the process of running a set of test cases repeatedly and automatically to verify the functionality, performance, and security of the APIs. This process is essential for ensuring the quality and reliability of the software.

Code based open source tools are better for automated API system testing for the following reasons:

• Maintainability: Code based open source tools allow the user to write test scripts using a programming language, which gives them more control and flexibility over the test logic and flow. This makes it easier to maintain and update the test scripts as the APIs change or evolve. On the other hand, GUI based tools like Postman or Insomnia rely on a graphical interface, which can be cumbersome and prone to errors when dealing with complex or dynamic scenarios. Moreover, code based open source tools can leverage existing frameworks and libraries that provide reusable and modular code components, which can improve the readability and maintainability of the test scripts.

• Integration: Code based open source tools can be easily integrated with other tools and systems, such as continuous integration (CI), continuous delivery (CD), version control, reporting, etc. This can enhance the automation and collaboration capabilities of the testing process. GUI based tools like Postman or Insomnia have some extensibility features, but they are not as comprehensive or customizable as code based open source tools.

• Cost-effectiveness: Code based open source tools are free and open source, meaning that anyone can use them without paying any fees or licenses. This can save a lot of money and time for the users, as they do not have to worry about purchasing or renewing subscriptions or licenses. GUI based tools like Postman or Insomnia have free versions, but they also have paid plans that offer more features and functionalities. Sometimes these paid plans can be quite expensive or the license come with changes of terms and conditions that can be problematic for the users. Test code can be as important as productive code. You need to consider that for some products the code will need to be maintained for many decades. Sometimes the test code even outlives the productive code, since it contains the domain knowledge in the format of tests. This can remain relevant even if you rewrite the productive code. With 3rd party tools, there comes a risk and a dependency to the supplier including licensing issues. Therefore, for test code you should prefer open source tools over 3rd party tools.

Further benefits mentioned by teams after migrating to code-based tools for automation

The following further disadvantages of a Postman setup were mentioned by teams after migrating to code-based tools for automation:

• High complexity of test sequences and scripts in Postman / Newman.

• Poor maintainability, unstable execution

• Good Integration of Postman requires higher tier: Using newman and postman CLI, you can run Postman API tests from any Git based pipeline, also on the free plan. The higher tiers of Postman also provide a built-in GitHub, GitLab, ADO and BitBucket integration, including support for Pull Requests.

• Code Review with pull requests is not as straight forward, since collections are harder to review. The higher tiers of Postman provide better support.

• The common approach to store collections externally in the cloud can be incompliant with company policies. On the other hand, sharing code instead in a repository is transparent and compliant.

Recommendations: Alternative for Postman and Insomnia for Automated System Testing

Recommendations: Alternative for Postman and Insomnia for Automated System Testing

For Automated System Testing against the API layer, you can use the tool, which is fitting best for your technology stack. Below some code based Open Source recommendations for the respective technology stacks – the list is not exclusive and there are further similar Open Source alternatives.

• Java

  • Karate

  • RestAssured

• JavaScript

  • Mocha or Jest NodeJS test frameworks

  • Supertest HTTP client library

  • If you need to test against OData APIs you can also leverage the OData libraries SAP OData library. For QMate there is a small wrapper available, which eases to make API calls for preparing transactional data for UI testing.

• Python

  • Pytest Test framework

  • Requests HTTP client library

• ABAP

  • Either leveraging one of the above tools or writing system test with ABAP unit. Especially the tools from the JavaScript stack are used by teams in Procurement. It allows the additional benefit that it can be easily combined with JavaScript UI testing frameworks (like QMate)

Conclusion

In conclusion, code based open source tools are better for automated API system testing than GUI based tools like Postman or Insomnia, especially in terms of maintainability. Code based open source tools offer more control, maintainability, flexibility, extensibility, and cost-effectiveness than GUI based tools. Therefore, I recommend using code based open source tools for your automated API system testing needs. And tools like Bruno, Postman and Insomnia only for exploratory API testing. If Postman is becoming too expensive due to the changes in September and you prefer a similar tool, you can check out Bruno for the exploratory API system testing case.

Use CaseStackRecommended SolutionCommentExploratory TestingGeneralBruno or the IDE plugins Visual Studio Code REST Client or JetBrains IntelliJ HTTP clientChoice depends on preference for GUI support or better integration in IDETest AutomationJavaKarate or RestAssuredChoice depends on existing technology stackJavaScriptMocha or Jest, Supertest – HTTP client libraryChoice depends on existing technology stackABAPEither leveraging JavaScript based tools or writing system test with ABAP unit.Choice depends on existing technology stack

For exploratory API testing we can recommend to use Bruno or the mentioned plugins for VSCode. It should cover the most needed features for exploratory API Testing and allows to store the collections in your repository, version control them like the rest of your code and you do not need a cloud account to use it, which dependending on the context can increase compliance and security. And so far all colleagues mentioned that the migration was straight forward.

Appendix

Do I really need a tool like Postman, Insomnia or Bruno for exploratory API Testing?

Some developers prefer most of the time code based tools for exploration. Especially if everything is more like a continuous improvement of an existing system. But there are different roles and also sometimes more explorative ways (e.g. exploring unfamiliar APIs) where a exploratory testing tool is preferred. If you and all involved roles are fine with it, there is no need for exploratory API testing tools. Still there are contexts where such tools are needed.

What are benefits of Bruno compared to Postman and Insomnia?

Bruno is OpenSource and free to use. It allows to store the collection in your repository, version control them like the rest of your code. Besides, you do not need a cloud account to use it, which increases compliance and security. More details can be found in the comparision table.

Resources

• Testing Terminology: System Testing

• Visual Studio Code plus REST Client

• JetBrains PyCharm plus HTTP client

• Plugin for Postman to Karate migration

• Bruno

Related Blogs on Test Automation

You can find more information about test automation in the following blogs:

• Get Higher Quality and Productivity by tackling the Broken Window Effect

  • The hidden cost of flaky tests

  • Why a maintainable and readable test suite is important to stay agile

Subscription: If you want to get updates, you can subscribe to the free newsletter:

Mark as not spam: : When you subscribe to the newsletter please do not forget to check your spam / junk folder. Make sure to "mark as not spam" in your email client and move it to your Inbox. Add the publication's Substack email address to your contact list. All posts will be sent from this address: ecosystem4engineering@substack.com.

✉️ Subscribe to the newsletter 

❤️ Share it — The engineering ecosystem newsletter lives thanks to word of mouth. Share the article with someone to whom it might be useful! By forwarding the email or sharing it on social media.