This repository is a small journey through two real functional testing problems that did not fit into the comfortable "send request, check status code" box.
The goal is not to force Postman and JMeter into the same contest. They solve different problems in these examples. Instead, I use both cases to show where each tool feels natural, where it starts to resist you, and what happens when functional testing needs custom logic, data, parallel flows, WebSocket events, database setup, and even browser actions.
Let's dive into out-of-the-box solutions for non-standard cases, and see where the tools' limits and possibilities actually are.
Functional testing checks whether a feature behaves correctly from the user's or system's point of view: the right action is allowed, the wrong action is blocked, the right side effect happens, and the visible result matches the business rule.
Functional API testing is one important part of that. It verifies request/response behavior: status codes, response bodies, authorization, validation, and side effects. But real systems rarely stop at pure API calls. In the JMeter case here, the test also touches WebSocket events, database state through JDBC, and browser behavior through Selenium/WebDriver.
That is the reason this project exists. I wanted to show what happens when functional testing becomes a real engineering task:
- permissions must be validated across many roles and endpoints;
- WebSocket events must be received or suppressed depending on permissions;
- setup cannot be done only through public API calls;
- a listener must run while another flow triggers the event;
- the report must explain the scenario, not just say "passed" or "failed."
The examples are anonymized, but the problems are real.
| Capability | Postman | JMeter |
|---|---|---|
| First impression | Modern, approachable, fast to start | Older UI, more learning required, but very open-ended |
| Natural use case | REST/API workflows, collections, environment-driven checks | Complex test plans with threads, controllers, protocols, plugins, and scripts |
| Custom logic | JavaScript sandbox in pre-request and post-response scripts | JSR223 scripting, usually Groovy, with access to Java libraries and JMeter internals |
| Extensibility | You can script inside the sandbox, but you cannot freely add arbitrary modules or libraries | You can extend deeply: plugins, Java/Groovy code, JDBC, WebSocket clients, Selenium, custom reporting |
| Flow control | Collection order, folders, scripts, data iterations | Thread Groups, controllers, timers, properties, variables, test fragments, setup/teardown flows |
| Data and environment | Environments, collection variables, CSV iteration data | JMeter variables/properties, command-line -J overrides, JDBC, CSV when needed |
| Reporting | Newman reporters make API reports easy to publish | Reporting can be basic out of the box, or fully custom with scripts and external tools such as Allure |
| Ceiling | Excellent for API permission audits and request/response automation | Much higher ceiling when the test needs several moving parts at once |
My bias after working through these cases is simple:
- Postman is easier to enter and pleasant for API work.
- Postman cannot do everything JMeter can do.
- JMeter can do most things Postman can, but it asks for more practice and more patience.
- JMeter feels archaic at first, but once Java/Groovy and plugins enter the picture, the ceiling becomes much higher.
You need custom logic when the test is no longer a single assertion. That usually happens when the behavior depends on business rules, shared state, timing, roles, or another system layer.
In the Postman case, the problem was not "does this endpoint return 200?" The problem was: for every role and every endpoint, does the API return exactly the access that the permission policy allows? Writing hundreds of individual assertions would be noisy to write and fragile to maintain, so the collection uses a JavaScript permission matrix and generates the check at runtime.
In the JMeter case, the problem was even less linear. One thread triggers actions, another thread listens for WebSocket events, permissions are injected through the database, and some events can only be created through UI behavior. This is where JMeter's architecture matters: parallel Thread Groups, shared signals, JDBC samplers, WebSocket client logic, and Selenium/WebDriver samplers can all live inside one test plan.
That is the practical lesson behind this repository: functional testing is not only API testing, and complex functional testing is rarely solved by brute force. At some point it needs an engineering approach.
The tool matters because it decides how far you can go, what kinds of problems you can solve, and how much custom infrastructure you must build yourself. A more powerful tool is often rewarding, but it also raises a maintenance question: does the extra development time make the test stronger, or does the test become harder to maintain than the feature it protects?
-
Postman Case Study - role-based API permission testing across 100+ endpoints and 4 roles. Live sample report
-
JMeter Case Study - permission-gated WebSocket event verification using API calls, JDBC setup, Selenium/WebDriver actions, and a custom listener for WebSocket. Live sample report
Postman is a good choice when the problem is mainly API-shaped: authenticate, send many requests, reuse environments, iterate over role data, and publish a readable report without much ceremony. It was good in the Postman case because the permission matrix could live close to the collection and every generated assertion still stayed understandable.
Could the Postman case be solved in JMeter?
Yes. JMeter can send the same requests, load the same kind of data, and run custom logic around the responses. But for this specific case it would probably take longer. Postman is faster when the problem is still mostly request/response automation.
What happens if the number of endpoints, roles, and permissions keeps growing?
That is where I would start reconsidering the tool. Postman can still work, but a very large collection with many roles and permission rules may become harder to navigate. At that point JMeter or a code-based framework can become more attractive because the test logic can be organized with stronger structure.
JMeter shines when the test needs orchestration: background listeners, cross-thread coordination, database preparation, browser actions, protocol-level checks, or custom reporting. It was good in the JMeter case because the test was not one flow. It was a small system watching another system, with several technologies in between, and JMeter gave enough control to build that.
Could the JMeter case be solved in pure Java?
Yes. A Java framework could create users, update permissions through JDBC, open a WebSocket client, drive browser actions through Selenium, and generate a report. Technically, nothing in this case is impossible in pure code.
Would pure Java be faster to build?
For this case, no. The first version would require building a lot of supporting structure before the actual test logic becomes visible: configuration loading, scenario flow, logging, reporting, browser lifecycle, WebSocket waiting logic, and CI packaging. JMeter already gives a runnable test-plan shell for those moving parts.
Is JMeter more maintainable than pure Java?
For a single complex test plan like this one, yes. JMeter provides a middle ground between a GUI-driven tool and a full custom automation framework. It is not as clean as a well-designed Java project, but it is much cheaper to assemble, easier for non-Java-heavy QA teams to inspect, and powerful enough when the test needs more than API calls.
Both case studies come from real test suites. Hostnames, credentials, tokens, internal IPs, and identifying product/company references were anonymized before publication. The published reports are curated examples for portfolio review, not complete production test evidence.