Where do black box and grey box tests play in securing your cyber assets (web/mobile applications)?

This article is the last in a series of posts for the CISO and internal auditors to understand the types of VAPT (security assessment) for web and mobile assets.


First post: One Vulnerability Assessment Penetration Testing (VAPT) question asked by Internal Auditors and CISOs
Second post: Boxes, yet more boxes!
This post: Where do blackbox and grey box tests play in securing your cyber assets?


A competent and experienced security assessor will be able to detect and exploit security vulnerabilities in the web/mobile application or assets through black box and grey box tests. Although Completeness cannot be asserted, these tests and the security assessment report can give the reader a feel of the security controls or posture that had been built into the web or mobile asset.

What about this feel of security controls or security posture? How would you determine this? To do so, we can look into various indicators, listed below.


Here are some indicators for consideration:

1. Severity of detected vulnerabilities and number of detected vulnerabilities (using CVSS)

What is CVSS 3.1?

The Common Vulnerabilty Scoring System is an internationally recognised severity ranking framework. Vulnerabilities are ranked from 0.0 to 10.0, where 10.0 represents the highest severity.

In my opinion, relying on solely on CVSS is not an optimal solution as it does not speak the Board of Directors’ language in the Board Room. However it is an internationally recognised framework which is easy to use and understood by cyber risk professionals and is one which is recognised by MAS.


2. Root cause analysis of each finding

Example #1, Insecure direct object references (IDOR) findings suggest weakness in accepting arbitrary inputs from the end user, and was not picked up by existing test coverage or that could be weakness in reviewing test results.

What are IDOR findings?

Insecure direct object references (IDOR) are a class of vulnerability which arise when the application or program uses user-supplied input to access data, which were not authorised by the user’s role or privileges. IDOR findings are usually found in authenticated grey box tests and can lead to sensitive data leakage, such as customer account holdings, statements.

On the other end, IDOR findings can change other users’ sensitive information such as bank account number, bank account name, performing unauthorised transactions, leading to monetary losses and potential AML issues.


Let’s walk through the following real-life example.

Every month end, Sheryl checks her monthly bank statement by logging into the bank’s portal, which shows her account balance. The bank statement shows her name, address and account holdings.

Sheryl's Monthly Bank Statement
Sheryl’s Monthly Bank Statement

Through IDOR vulnerabilities in the web application, another user (Let’s use David here) can view Sheryl’s bank statement by changing the userid or user account parameter in the web or mobile application, to point to Sheryl’s userid or to run through the entire list of userids as most web/mobile assets tend to use numbers as unique identifiers.

I have found many of this issue while testing financial institutions’ web and mobile assets. There are other alternatives to using numbers (integers) as identifiers, depending on the underlying database used. Do reach out if you are interested to know more about the alternatives (including their pros and cons) that can help to make enumeration attacks more difficult for threat actors’.

The bank account number and balance sum presented in David’s bank statement, actually belong to Sheryl.

David's Monthly Bank Statement
David’s Monthly Bank Statement

In such cases, this usually means every account balance in the banking application can be leaked despite the account holder’s name and address remaining unchanged on the bank statement. This vulnerability can only be detected when testing for business logic weakness.


Example #2, Multifactor authentication bypass or redirection findings

What are Multifactor authentication bypass findings?

The presence of this vulnerability suggests severe weakness in software architecture, and design of the software architecture should be revisited.

This class of vulnerability allows an threat actor to sidestep the Multifactor authentication (such as SMS One-Time-Password or mobile token authentication) built into the web or mobile asset, allowing access into the web and mobile assets with using only username and password as credentials.

This vulnerability refers to the incorrect design and implementation of using SMS OTP as a 2nd Factor of Authentication by the developer, and is different from abusing SMS OTP from the Telcos (I will write about this and exploiting SMS/SS7 gateways in a separate post)


Example #3, Insufficient entropy in SMS One-Time-Password findings

What does Insufficient entropy mean?

Ans: Lack of Randomness

In a nutshell, it is possible to log into the web and/or mobile application by inputting a limited set of numbers, as the numbers generated by the SMS OTP or token module are not sufficiently randomised.

This meant the software code is generating a set of numbers which were not exactly random. Through testing the OTP numbers returned by the web service, a threat actor can determine the upper and lower limits of the OTP numbers, consequently yielding a OTP-protected account to the threat actor.

I’ve seen web services setting SMS OTPs to four digits. This is a practice I highly discourage, unless you have excruciating reasons for not using six digits or more.


3. Review the unit / integration tests performed by the software developers

Unit and integration tests must cover a wide variety of use cases and in various permutations. For example, a simple username/password authentication module has at minimum six categories of tests. Each category can have at least five individual subtests.

Note: These tests may not be accessible for review if the web and mobile assets are provided as part of Software-as-a-Service (SaaS) providers. But if your purchase is material to the SaaS provider, they may consider letting you take a look – subject to your negotiations.


4. Review the user acceptance tests performed by the end users.

End users should perform the user acceptance tests and sign off for the web and mobile assets. Again, these may not be accessible if the web and mobile assets are provided by third party SaaS providers.


I hope this article series can help the CISOs and internal auditors seeking to understand the differences in the VAPT assessment testing methodologies and their corresponding assurance levels, which impact web and mobile application risks, consequently give rise to business operational risks.

I will be addressing tips to get the most out of your VAPT assessments in the next few posts.

At the same time, DM me your questions.

Have a Great Week ahead and stay Cyber Strong!


Say Hong TAN

CISA, CRT, CA (Singapore)
Founder and Director
Dynafense Cybersecurity

Leave a Reply

Your email address will not be published. Required fields are marked *