Imagine you designed two classification algorithms, labeled A and B. The table below shows the accuracy that was observed for 10-fold cross-validation. Using a paired t-test, can you determine whether the performance differences are statistically similar? Extremely statistically similar? show all of your work.

Assume Alpha Level is 0.05 (5%)

Paired T-Test

• A paired t-test is also called a Correlated Paired t-test, a Paired-Samples t-test, or Dependent Samples t-test.
• Paired T-Test is used when two samples are somehow connected or dependent or perform tests on the same things.
• In the given scenario, two classification algorithms called A and B test the accuracy on the same 10-fold cross-validation samples.
• Therefore, here paired t-test is applicable to determine the performance differences of the two classification algorithms A and B are statistically similar or not.

Step 1 –

The null hypothesis in the paired t-test assumes the means are equal.

Therefore,

Null Hypothesis:

H₀ - There is no difference between classification algorithms A and B. The expected accuracies of classification algorithms A and B are the same.

$$H_0: µ_d = 0$$

Step 2 –

Find out Differences by subtracting the accuracies of algorithm B from the accuracies of algorithm A.

Step 3 –

Find out the square of all the differences generated in step 1.

Step 4 –

Find out the sum of differences and the sum of the square of differences.

Therefore,

Sum of Differences = ∑ D = - 55

Sum of Square of Differences = ∑ D² = 931

Step 5 –

Calculate the value of t using the formula of t-test.

Where,

Sample Size = N = 10

Sum of Differences ∑ D = - 55

Sum of Square of the Differences ∑ D² = 931

Therefore,

Now, we get the value of t = - 2.081

Step 6 –

Find out the Degree of Freedom (DF) = Sample Size (N) - 1

Therefore,

Degree of Freedom (DF) = Sample Size (N) - 1 = 10 - 1 = 9

Step 7 –

Find out the p-value in the t-table, using the Degrees of Freedom (DF) calculated in step 6.

Also the value of the Alpha Level is given as 0.05 (5%).

Therefore,

p-value = 2.262 .....(for DF = 9 and Alpha level = 0.05 using t-table)

Step 8 –

Compare the p-value = 2.262 with calculated t-value = - 2.081

The calculated t-value is less than the t-table value at an Alpha Level of 0.05 (5%)

The p-value is greater than the Alpha Level: p > 0.05.

Hence, We can ACCEPT the Null Hypothesis (H₀) that there is no difference between the classification Algorithm A and B.

That means Classification Algorithms A and B are Extremely Similar.