Improvements in Diagnostic Test Results Associated with the Adoption of the Science Bits Program

A new study suggests that the Science Bits program helps improve diagnostic test results in Science for ESO students (Middle School students, 12 -16 years old). Schools that adopted Science Bits from 2019 to 2023 have shown more significant progress in their students’ scientific skills.

27 de mayo de 2024

• The study compares the progress of 22 schools that adopted Science Bits in 2019 (experimental group) with 22 schools that did not (control group), based on the diagnostic tests for 10th grade from 2019 to 2023.
• Both groups of schools started with the same Science results in 2019 and include the same proportion of public and private schools, as well as a similar number of students.
• The schools in the experimental group experienced greater improvement in Science than the control group, while both groups showed the same changes in Mathematics, Catalan, Spanish, and English.
• If we rank the schools based on their Science results, both groups started from a very similar average position in 2019 (530 vs. 535). But in 2023, the experimental group moved up 35 positions, while the control group dropped more than 60 positions.
• Replicating the study with 22 more schools that adopted Science Bits in 2018 yields the same results.

At ISTF, our mission is to translate research on how people learn into education, with the goal of promoting educational practices that are informed by evidence. As part of this effort, we design educational materials based on research in didactics and learning psychology. Additionally, we regularly test and evaluate the impact of these materials in the classroom. To this end, we collaborate with researchers from institutions such as the University of Barcelona, the Autonomous University of Madrid, the University of Granada, the University of Zaragoza, the Rovira i Virgili University, Johns Hopkins University, and the American Institutes for Research.

In a previous article published in 2021 in the International Journal of Science Education, we presented a study providing evidence on the contribution of the Science Bits program (an educational project for teaching science in ESO—students from approximately ages 12 to 16, which corresponds to Middle School in the US—adopted by hundreds of schools in Spain) to improving conceptual learning (Garcia Grau et al., 2021). The study highlighted significant improvements in the conceptual learning of 3rd and 4th ESO students (Middle School students, 12-16 years old) in two schools that had introduced Science Bits for four years, while no significant changes occurred in the results of two other schools of the same socioeconomic level that did not introduce the program. Since no other significant changes were recorded in the participating schools during the four years of the study, this suggested that the introduction of Science Bits might have contributed to this improvement. In that study, we used assessment tests designed by the researchers, focusing on the ability to apply fundamental scientific ideas and identify common misconceptions. Although that study was informative and provided evidence on the impact of Science Bits, the small sample size and the fact that the assessment test was not standardized restricted the scope of its conclusions.

An Analysis Based on the Official Diagnostic Test Results

Today we present a study in which we analyzed the evolution of all schools in Catalonia that implemented Science Bits in the 2019-20 school year for all science subjects (Biology and Geology, Physics and Chemistry) across all ESO grades, and have continued using the program in these grades up to the present. We compared this to the evolution of a random sample of an equivalent number of schools that have never used Science Bits. This time, we compared the results obtained in the official diagnostic tests (competency tests) administered annually by the Department of Education of the Generalitat of Catalonia to 4th ESO students, specifically the results from 2019 and 2023. We specifically selected the schools that incorporated Science Bits in the 2019-20 school year to observe the differences in the results between students who had never used Science Bits (and took the diagnostic tests in April 2019) and students who used it throughout all four years of ESO (and took the diagnostic tests in April 2023). Using the results of the official diagnostic tests provides an external measure to the researchers, making the analysis of the program’s impact on learning more objective.

In total, 22 schools met the condition regarding the use of Science Bits (experimental group). Next, we randomly selected another 22 schools that have never used the program to establish a control group and proceeded to a comparative analysis of their diagnostic test results. The characteristics of both groups are detailed in the following table (Table 1):

First, we verified the randomness of the sample by performing a z-test, which confirmed that both the experimental (SB) and control (NO-SB) groups can be considered random samples of the population formed by all schools in Catalonia, regarding the 2019 Science diagnostic tests. Indeed, an independent samples t-test confirmed that there are no statistically significant differences between the results of both groups in these tests, meaning they started from identical results in 2019.

The average scores of each group in the diagnostic test results in 2019 and 2023 are as follows:

Next, we performed a paired samples t-test to compare the results of each school in 2019 and 2023 and find out if there were significant differences. The results are shown below (Table 3):

The Results Continue Supporting the Effectiveness of Science Bits

The data reveal that experimental schools show an improvement in Science results of 3.61 points, while control schools show an improvement of 1.32 points (Table 2). However, the statistical analysis suggests that only the difference in the experimental group is significant (it would not be explained by chance except in 0.02% of cases), while the difference in the control group is not: it could be due to chance (Table 3). In any case, it would not be surprising if both groups showed differences between 2019 and 2023 (as reflected in the results in other subjects). After all, the differences between the two years could be due to other factors, such as the fact that the diagnostic tests are different and the students as well. However, these differences would be common to both groups of schools, so the threefold greater improvement of the experimental group compared to the control group could not be attributed to these factors, but to a differential factor of the first group compared to the second. Since the only variable distinguishing all schools in the experimental group from those in the control group is the use of the Science Bits program, we can consider that these results suggest that the observed difference is related to the use of these materials.

Indeed, conducting a comparative analysis of results in other subjects provides valuable insights to dismiss the possibility that alternative factors, such as differences in the academic ability of students in 2019 and 2023, are behind the difference in Science results. In this regard, Table 3 shows that the differences in results in other subjects were very similar in both groups, both in languages (with very similar decreases in Catalan and Spanish, and no changes in English) and in Mathematics. Notably, in Mathematics, the difference in results between the two groups was practically the same: a decrease of around 3.6 points between 2019 and 2023. This further emphasizes a differential factor between the control group and the experimental group that would have caused the differences in the evolution of the Science subject specifically.

We can go even deeper into the analysis and statistically assess the differences between the control and experimental groups by applying an independent samples t-test to compare the changes experienced by the schools in their 2019 and 2023 results. The results are shown below:

As seen in Table 4, the differences observed in the two groups of schools between 2019 and 2023 can be deemed equivalent in Catalan, Spanish, Mathematics, and English. Regarding Science, the difference is almost statistically significant (based on the criteria we have adopted, where a p-value of less than 0.05 indicates significance). This means that there is a 94.5% probability that the evolution of the experimental group in Science is indeed different (and in this case, superior) to that of the control group. Moreover, the effect size—expressed by Cohen’s d—is clearly relevant, around d=0.596. This means that the difference between 2019 and 2023 achieved by the experimental schools is 0.596 standard deviations greater than that achieved by the control group schools. This difference can be considered very relevant in the field of educational interventions (Funder & Ozer, 2019).

Another very significant finding when comparing the experimental and control groups is the shift in their respective positions in the ranking of all schools based on Science competency test results (Table 5). If we rank all school in Catalonia offering ESO in descending order based on their Science results (with the top-ranking school having the best results), the average position of the experimental group schools in 2019 is 535^th, very close to that of the control group, 530^th. However, the position of the control group schools drops to 599^th in 2023, while the position of the experimental schools rises to 500^th.

The significant drop in ranking positions for the control group suggests it has fallen behind not only the current experimental group but also other schools that adopted the Science Bits program in previous or subsequent years. To investigate further, we conducted the same analysis on a second experimental group of 22 schools that implemented Science Bits in 2018 and have consistently used it across all grades since then. The characteristics of this group are shown in Table 6.

Regarding the results of this group in the competency tests of 2019 and 2023, the data is presented in Table 7:

The paired samples t-test comparing the results of these schools between 2019 and 2023 shows results similar to the previous experimental group, with Science scores improving twice as much as the control group, while no significant differences were found in other subjects, confirmed by an independent samples t-test:

Regarding their ranking, the schools in the second experimental group also improved, moving from 480^th to 454^th, confirming the upward trend of experimental schools and supporting the control group’s decline in positions in favor of the experimental groups.

In conclusion, this analysis provides strong evidence that the Science Bits program helps improve results in the official 4th ESO diagnostic tests. This aligns with previous research in science education, which shows that programs offering well-designed activities to address misconceptions and promote understanding-based learning (and not just explanatory content and exercises), along with effective use of digital media (quality multimedia materials, interactivity, feedback, and information for teachers about student achievement) often outperform conventional educational materials such as traditional textbooks (Cheung et al., 2017; Roblin et al., 2018).

Moreover, the 5E model—the teaching and learning model on which Science Bits is based—has ample evidence of its effectiveness in improving understanding of scientific ideas, reducing misconceptions, achieving academic success, and boosting students’ interest in science (including pursuing science careers), their confidence in learning science, and their attitudes towards science (Bybee et al., 2006; Bybee, 2015). While this analysis primarily examines the program’s impact on competency-based learning, it is also important to evaluate an educational material by how it contributes to promoting interest in science.

Would you like to know more about the 5E Model?