4.1 Testing Process & Levels of Testing

What is software testing?

Software testing is the process of executing a program with the intent of finding errors (Glenford Myers, 1979, The Art of Software Testing).

Key insight: Testing is destructive by intent — a successful test is one that finds a defect. A successful test run that finds no defects only proves that the test cases did not detect the defects that exist.

Three definitions to memorise

Causal chain: Error → causes Defect → causes Failure

---

The Seven Testing Principles (ISTQB)

1. Testing shows the presence of defects, not their absence
2. Exhaustive testing is impossible — too many input combinations
3. Early testing saves time and money (Boehm's curve again)
4. Defects cluster together — Pareto: 80% of defects in 20% of modules
5. The "pesticide paradox" — repeated tests find fewer new bugs; tests need refreshing
6. Testing is context-dependent — gaming software tested differently from medical software
7. Absence-of-errors fallacy — finding no bugs ≠ correct software

Exam tip: "List the principles of software testing" is a 7.5-mark short answer staple. Memorise these seven.

---

Verification vs Validation (V&V)

Two terms often confused. Boehm's mnemonic:

Verification checks conformance to specification. Validation checks fitness for use.

Requirements ──► Design ──► Code ──► Tests ──► Product
     │              │         │                   │
     └─V─V─V────────V─V───────V                   V
       (Verification — internal)              (Validation — user)

PYQ classic: "Differentiate verification and validation with examples." — Verification: SRS review, code review (internal check). Validation: UAT, beta testing (user check).

---

The Testing Process

1. Test Planning

• Scope, objectives, schedule, resources
• Output: Test Plan (IEEE 829)

2. Test Design

• Identify test cases from requirements / design
• Output: Test cases, test data, expected results

3. Test Execution

• Run tests; compare actual vs expected
• Output: Test logs, defect reports

4. Defect Reporting

• Defect entered in tracker (Jira, Bugzilla, Azure DevOps)
• Fields: ID, summary, severity, priority, steps to reproduce, expected, actual, environment

5. Test Closure

• Test summary report
• Lessons learned
• Test-ware archived for re-use

---

Test Plan structure (IEEE 829)

1. Test plan identifier
2. Introduction
3. Test items
4. Features to be tested
5. Features not to be tested
6. Approach
7. Item pass/fail criteria
8. Suspension and resumption criteria
9. Test deliverables
10. Testing tasks
11. Environment / resources
12. Responsibilities
13. Staffing and training
14. Schedule
15. Risks and contingencies
16. Approvals

---

Levels of Testing — the IPU favourite

                  Acceptance Testing
                          │
                   System Testing
                          │
                 Integration Testing
                          │
                    Unit Testing
                          │
                       Coding

1. Unit Testing

• Scope: smallest testable units — functions, methods, classes
• Performed by: developer
• Goal: verify the unit works in isolation
• Tools: JUnit (Java), pytest (Python), NUnit (.NET), Jest (JS)
• Techniques: mostly white-box; uses stubs (callees) and drivers (callers) for dependencies

@Test
public void testInterestCalculation() {
    assertEquals(800, calc.simpleInterest(10000, 0.08, 1), 0.01);
}

2. Integration Testing

• Scope: combined units → interfaces between modules
• Performed by: developers or independent testers
• Goal: catch interface defects when modules are combined

Integration strategies

3. System Testing

• Scope: the entire integrated system, including hardware and externals
• Performed by: independent test team
• Goal: verify the system meets the SRS as a whole

Sub-types of system testing:

4. Acceptance Testing

• Scope: the system from user's perspective
• Performed by: customer / users
• Goal: confirm system is ready for production
• Output: acceptance / rejection

Sub-types

• Alpha testing — at developer site, by real users in controlled setting
• Beta testing — at user sites, in real environment
• User Acceptance Testing (UAT) — formal contractual sign-off
• Operational Acceptance Testing (OAT) — IT operations validates deployment

---

Quick comparison — 4 testing levels

---

Smoke and Sanity Testing — quick utility tests

---

Key Terms — Lesson 4.1

The vocabulary below defines the testing terminology you must use precisely in every Unit-IV PYQ.

Software Testing — Glenford Myers's classic 1979 definition: "the process of executing a program with the intent of finding errors." The key word is intent — testing is destructive by design; a successful test finds a defect.

Error / Mistake — A human action that produces an incorrect result — a developer's typo, a misunderstanding of requirements, a wrong design decision. Errors are the causes.

Defect / Fault / Bug — A flaw in the software — an incorrect line of code, a missing check, a wrong constant. Defects are the consequence of errors and the cause of failures. The terms defect, fault, and bug are used interchangeably in industry.

Failure — An observable deviation from expected behaviour — the application crashes, returns wrong output, or hangs. Failures are what users see. A defect that is never executed never causes a failure; this is why testing matters — to expose dormant defects.

Error → Defect → Failure — The standard causal chain in software quality. A human error introduces a defect; the defect, when executed, causes a failure. Testing discovers failures; debugging traces failures back to defects; process improvement reduces errors.

Verification — "Are we building the product right?" — checking that the current artefact (SRS, design, code) conforms to its specification. Achieved by reviews, inspections, walkthroughs, and static analysis. Verification happens throughout the SDLC.

Validation — "Are we building the right product?" — checking that the system meets the user's actual needs. Achieved by demos, user acceptance testing, beta testing. Validation typically happens later in the SDLC, with the user in the loop.

Static Testing vs Dynamic Testing — Static testing examines the artefact without executing the code — reviews, inspections, static analysis. Dynamic testing executes the code and observes behaviour — unit tests, integration tests, system tests, manual exploratory testing.

Test Plan (IEEE 829) — The master document describing the test approach for a project — scope, objectives, items to test, features in/out of scope, approach, pass/fail criteria, deliverables, schedule, responsibilities, risks. IEEE 829 defines the canonical structure.

Test Case — A specific scenario with inputs, execution conditions, and expected output designed to verify a specific aspect of the system. A good test case has a unique ID, clear preconditions, repeatable steps, defined expected results, and is traceable to a requirement.

Test Suite — A collection of related test cases, usually grouped by feature, module, or test level. Test suites are the unit of test execution and scheduling.

Test Data — The inputs used by test cases — valid values, invalid values, boundary values, large data sets. Test data design is part of test case design and often the harder half of the work.

Test Bed / Test Environment — The hardware, software, network, and configuration in which tests are executed. Ideally identical to production; in practice often smaller. Modern CI uses containerised test beds (Docker) for reproducibility.

Defect Report / Bug Report — The standardised record of a discovered defect — ID, summary, severity, priority, steps to reproduce, expected behaviour, actual behaviour, environment, attachments (logs, screenshots). Tools: Jira, Bugzilla, Azure DevOps, GitHub Issues.

Severity vs Priority — Two distinct attributes of a defect. Severity measures technical impact — does the system crash, lose data, expose security? Priority measures urgency to fix — based on business need, customer pressure, and release timing. A typo in the homepage banner is low severity but high priority; an obscure data-corruption bug is high severity but possibly low priority.

Unit Testing — Testing of the smallest testable units (functions, methods, classes) in isolation, typically by the developer who wrote the code. Tools: JUnit (Java), pytest (Python), Jest (JavaScript), NUnit (.NET). Unit tests are fast (milliseconds), are run on every commit, and form the foundation of the test pyramid.

Stub — A fake replacement for a callee during unit testing — a function that returns a hard-coded value instead of calling the real downstream service. Stubs let you test a unit in isolation when its dependencies are slow, unavailable, or hard to control.

Driver — A fake replacement for a caller during unit or bottom-up integration testing — code that invokes the unit under test with crafted inputs and captures its output. Drivers are what make it possible to test a low-level module before its upstream caller is built.

Mock — A more sophisticated stub that also records the calls made to it so the test can assert that the unit-under-test called its dependency correctly. Mocking frameworks: Mockito (Java), unittest.mock (Python), Jest mocks (JavaScript), Moq (.NET).

Integration Testing — Testing of the interfaces between modules after units pass. Catches defects in how modules pass data, propagate errors, and coordinate. Strategies: Big Bang, Top-Down, Bottom-Up, Sandwich/Hybrid, Incremental.

Big Bang Integration — A strategy where all units are integrated at once and tested as a whole. Simple but interface bugs are hard to localise because everything was integrated at the same time. Suitable only for very small systems.

Top-Down Integration — Start with the top-level module; replace lower-level modules with stubs; progressively replace stubs with real modules. Top-level defects surface early but many stubs must be written.

Bottom-Up Integration — Start with the lowest-level modules; provide drivers to invoke them; progressively integrate upward. Low-level modules get thoroughly tested but top-level defects surface late.

System Testing — Testing of the entire integrated system against its system-level requirements — functional behaviour, performance, security, usability, scalability, recovery. Performed by an independent test team in a production-like environment.

Functional Testing — A category of system testing that checks each function of the system works as specified — login works, search returns correct results, checkout completes. Black-box by nature.

Performance Testing — Measures system response time, throughput, and resource utilisation under defined load. Tools: JMeter, Gatling, k6, Locust, LoadRunner. Outputs include throughput (TPS), latency percentiles (p50, p95, p99), error rates.

Load Testing vs Stress Testing — Load testing validates the system at the expected peak (e.g., 10K users on Black Friday). Stress testing pushes the system beyond expected peak (50K, 100K) to find the breaking point and how it degrades.

Volume Testing — Validates system behaviour with large data sets — millions of records in the database, large file uploads, long histories. Catches algorithmic blow-ups that small test data hides.

Recovery Testing — Validates that the system can resume operation after a failure — power outage, server crash, network partition, database loss. Includes failover, replay, and roll-forward scenarios.

Security Testing — Validates the system's resistance to attacks — authentication bypass, injection (SQL, command), XSS, privilege escalation, cryptographic weaknesses. Often includes a penetration test by ethical hackers.

Compatibility Testing — Validates the system works across browsers, operating systems, devices, and screen sizes. Critical for consumer-facing web and mobile applications. Tools: BrowserStack, Sauce Labs.

Regression Testing — Re-running existing tests after a change to confirm that nothing previously working has broken. Automated regression suites are the safety net that makes continuous delivery possible.

Smoke Testing — A quick build-acceptance test: does the build start at all and do the most critical paths work? Smoke testing is the gate before fuller regression — if smoke fails, no further testing is worth doing on that build.

Sanity Testing — A quick check after a small change that the affected area still works. Narrower than regression; broader than smoke.

Acceptance Testing — Testing performed (typically by the customer) to determine whether the system satisfies the agreed requirements and is ready for production. The formal trigger for delivery sign-off and final payment.

Alpha Testing — Acceptance-style testing performed by real users at the developer's site, in a controlled environment. Catches usability issues before exposing the product to the wild.

Beta Testing — Acceptance-style testing performed by real users at their own sites, in real environments. Used to validate compatibility, scalability, and real-world usage patterns before general release.

User Acceptance Testing (UAT) — The formal customer-led test that determines whether the system meets the agreed requirements. Pass = contractual sign-off; fail = back to development. Sometimes split into Operational Acceptance Testing (OAT) for IT-ops readiness and Business Acceptance Testing (BAT) for business-process readiness.

Test Pyramid (Mike Cohn) — A widely-cited model for how to distribute automated tests — many unit tests at the base (fast, cheap, focused), fewer integration tests in the middle, few UI/E2E tests at the top (slow, expensive, flaky). Inverting the pyramid (many slow E2E tests, few unit tests) is the classic anti-pattern.

Pesticide Paradox — One of the seven testing principles: the same tests, run repeatedly, find fewer new bugs over time — bugs adapt around the regular tests. Tests must be periodically refreshed with new cases (different inputs, new scenarios, fresh exploratory testing) to remain effective.

ISTQB (International Software Testing Qualifications Board) — The leading international certification body for software testers, with a syllabus that defines the seven testing principles and the canonical testing vocabulary. ISTQB Foundation Level is a popular entry-level certification for QA engineers globally.

---

Study deep

1. Testing is the most expensive phase if done poorly, the cheapest if done well. Late-found defects cost 100× more than early ones (Boehm). Front-loaded test design — writing tests before or during coding — pays off hugely.

1. Exhaustive testing is impossible. A trivial function with 2 inputs of 32 bits each has 2⁶⁴ = 1.8 × 10¹⁹ combinations. Even at 1 µs per test, exhaustive testing would take 580,000 years. We sample the input space smartly.

1. The cost of a defect rises by phase. Indicative ratios (Boehm, IBM, Capers Jones):

1. The "Test Pyramid" (Mike Cohn). Modern best practice: many unit tests, fewer integration tests, even fewer end-to-end tests. Unit tests are fast and pinpoint; E2E tests are slow and flaky.

              /\
             /UI\        ← few, slow, expensive
            /----\
           / Int. \
          /--------\
         /  Unit    \    ← many, fast, cheap
        /------------\

1. CI/CD changes the math. When automated tests run on every commit (CI) and deployments happen automatically (CD), testing becomes a continuous activity, not a phase. Mature teams release multiple times per day with high confidence — built on a foundation of automated tests.

PYQ pattern: "Explain levels of software testing." — Diagram the 4-level pyramid; describe each level's scope, performer, technique, goal. End with sub-types of system testing.