Evaluation of the omicron VLPs
The omicron VLPs with a concentration of 2.0 × 103, 1.0 × 103, 5.0 × 102, and 2.0 × 102 copies/mL were tested in duplicate by 15 commercial RT-PCR kits. All the RT-PCR kits reported correct qualitative results. Of them, 14 commercial RT-PCR kits can successfully detect the target genes. Zhijiang RT-PCR kit could only detect N and E genes but failed to detect ORF1ab because the VLPs didn’t contain the targeted sequence in the ORF1ab gene. According to the manufacturer’s instructions, the ORF1ab target failure did not influence the qualitative interpretation of the results.
Homogeneity evaluation showed no significant difference in Ct values among samples with the same concentration. The short-time stability study revealed that the EQA samples were stable under 2–8 °C for 10 days.
Performance of the laboratories for the qualitative interpretation of EQA data
A total of 8116 EQA panels were collected. All the panels were detected using commercial SARS-CoV-2 RT-PCR assays. For each EQA round, the proportion of competent laboratories ranged from 98.55 to 99.63%, PPA ranged from 99.31 to 99.91%, NPA ranged from 99.31 to 100%, and percent agreement ranged from 99.31 to 99.94% (Table 1). The overall proportion of competent laboratories, PPA, NPA, and percent agreement was 99.14% (1389/1401), 99.72% (5299/5314), 99.75% (2795/2802), and 99.73% (8094/8116), respectively.
This study noted that 22 incorrect EQA results, namely, 8 false negative results, 2 false positive results, and 12 invalidated results, were reported. Further analysis showed that incorrect data entry by the participants (leading to 12 invalidated reports and one false positive report), problems with the test system (leading to 6 false negative reports), and problems associated with techniques such as sample mixed up and improper handling of the sample (leading to two false negative reports and one false positive report) were the cause of the incorrect results.
Assessing the comparability in Ct values derived from RT-PCR
The EQA samples were tested by different extraction kits, RT-PCR kits, and PCR instruments. Ct values of the EQA samples with the same concentration were grouped to assess the comparability in Ct values. During the analysis, results containing clear outlier Ct values were excluded. As shown in Table 2, there was extreme variability in the Ct values for both ORF1ab and N. Regardless of the gene targets, the range of Ct values can be as large as 18 cycles. The IQR of the Ct values was 3 and 2 cycles for ORF1ab and N, respectively. There were 1404 results (39.24%) with absolute deviation from the respective median values by > 1 cycle, 681 results (19.03%) by > 2 cycles, 321 results (8.97%) by > 3 cycles, 141 results (3.94%) by > 4 cycles for ORF1ab. For N gene, 1482 results (41.66%) yielded absolute deviation from the respective median values by > 1 cycle, 704 results (19.79%) by > 2 cycles, 273 results (7.68%) by > 3 cycles, and 111 results (3.12%) by > 4 cycles.
Assessing the comparability of Ct values determined by different RT-PCR kits
There was a wide variation in Ct values obtained by different RT-PCR kits (Fig. 1a, Additional file 3: Table S1). The maximum SD was 2.55 cycles for ORF1ab and 2.93 cycles for N. There was a significant difference in Ct values among different RT-PCR kits. One should be noted that the comparison above did not consider the difference in nucleic acid extraction kits and PCR instruments.
The Ct values derived from EQA samples were detected by different RT-PCR kits. a The Ct values determined by different RT-PCR kits combined with different extraction kits and different PCR instruments, b the Ct values determined by different RT-PCR kits combined with the Tianlong nucleic extraction kit and different PCR instruments, c the Ct values determined by different RT-PCR kits combined with the DaAn nucleic extraction kit and different PCR instruments, d the Ct values determined by different RT-PCR kits combined with Tianlong nucleic extraction kit and ABI7500 PCR instrument
To diminish the variation in nucleic acid extraction, we focused on the results determined by the same extraction kit combined with different RT-PCR kits and PCR instruments. For samples extracted by Tianlong Nucleic Acid Extraction kit (Tianlong Technology Co., Ltd), the SD ranged from 0.5 to 2.55 cycles for ORF1ab and 0.89 to 3.54 cycles for N when results were grouped by RT-PCR kits (Fig. 1b, Additional file 3: Table S1). The variation in Ct values reached statistical significance for ORF1ab and N among different RT-PCR kits. For samples extracted by DaAn Nucleic Acid Extraction Kit (DaAn Gene Co., Ltd), a significant difference in Ct values for the N gene was observed (Fig. 1c, Additional file 3: Table S1). The significant difference in Ct values for ORF1ab was only observed for samples of 2.0 × 102 copies/mL. Noting that the statistical analysis did not consider the difference associated with PCR instruments and RT-PCR kits.
To avoid the diversity in extraction kits and PCR instruments, we focused on the samples detected by Tianlong Nucleic Acid Extraction kit and ABI7500 PCR instrument (Thermo Fisher Scientific, Waltham, MA, USA) (Fig. 1d, Additional file 3: Table S1). When grouped by RT-PCR kits, there was a significant difference in Ct values for ORF1ab for all concentrations except 5.0 × 102 copies/mL. A significant variance in Ct values for N was observed among different RT-PCR kits. These findings indicated that different RT-PCR kits yielded less comparable Ct values.
Assessing the comparability of Ct values obtained by different extraction methods
We analyzed the samples determined by the same RT-PCR kit but different extraction kits. For the samples tested by BioGerm RT-PCR kit, the SD ranged from 0 to 2.58 cycles for ORF1ab and 0 to 3.49 cycles for N gene across different extraction kits (Fig. 2a, Additional file 3: Table S2). There was a significant difference in Ct values for ORF1ab among different extraction kits. For N, the difference in Ct values reached significant for samples of 2.0 × 103 copies/mL and 5.0 × 102 copies/mL. The comparison above did not consider the variance in PCR instruments.
The Ct values derived from EQA samples were detected by different extraction kits. a The Ct values determined by different extraction kits combined with BioGerm RT-PCR kit and different PCR instruments, b the Ct values determined by different extraction kits combined with BioGerm RT-PCR kit and ABI7500 PCR instrument, c the Ct values determined by different extraction kits combined with BioGerm RT-PCR kit and SLAN PCR instrument
To avoid the effects of the diversity in PCR instruments, we compared the Ct values determined by test systems composed of BioGerm RT-PCR kit and ABI 7500 but different extraction kits (Fig. 2b, Additional file 3: Table S2). Among different extraction kits, a significant difference in Ct values was only observed for ORF1ab for the sample of 2.0 × 103 copies/mL, there was no significant difference for N. For EQA samples tested by BioGerm RT-PCR kit and SLAN-96S/96P Real-Time PCR System (referred as SLAN PCR, Shanghai Hongshi Medical Technology Co., Ltd, China), a significant difference was observed for ORF1ab for samples of 2.0 × 103, 1.0 × 103, and 5.0 × 102 copies/mL, and there is no significant difference for N when grouped by extraction kits (Fig. 2c, Additional file 3: Table S2). The findings indicated unlikely comparability in Ct values for ORF1ab among different extraction kits.
Assessing the comparability of Ct values determined by different PCR instruments
To assess the impact of the PCR instruments on the comparability of Ct values, we analyzed the results performed by the same extraction kits, the same RT-PCR kits but different PCR instruments. For samples tested by Tianlong Nucleic Acid Extraction kit and BioGerm RT-PCR kit, there was a significant difference in Ct values for ORF1ab for samples of 5.0 × 102 and 2.0 × 102 copies/mL among different PCR instruments (Fig. 3a, Additional file 3: Table S3). The difference in Ct values for N was statistically significant among different PCR instruments (Fig. 3a, Additional file 3: Table S3). Notably, the samples detected by Roche Light Cycler 480 Real-Time PCR System (referred as LC480, Roche Diagnostics, Mannheim, Germany) presented lower mean Ct values than those by other PCR instruments.
The Ct values derived from EQA samples were detected by different PCR instruments. a The Ct values determined by Tianlong nucleic extraction kit and BioGerm RT-PCR kit combined with different PCR instruments, b the Ct values determined by Zybio nucleic acid extraction kit and BioGerm RT-PCR kit combined with different PCR instruments
We also compared the Ct values by Zybio Nucleic Acid Extraction kit (Zybio lnc.) and BioGerm RT-PCR kit. Similarly, samples tested by LC480 presented the lowest mean Ct values. A significant difference in Ct values for ORF1ab for samples of 5.0 × 102 copies/mL and N gene for samples of 2.0 × 103, 5.0 × 102, and 2.0 × 102 copies/mL were observed (Fig. 3b, Additional file 3: Table S3). These results indicated that the difference in PCR instruments influenced the comparability of Ct values.
Assessing the comparability of Ct values among different laboratories using the same test systems
We then compared the Ct values among laboratories using the same test system. A laboratory offering less than 3 Ct values for EQA samples of the same concentration was excluded during the filtering stage. EQA results were determined by four different frequently used test systems, including Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kit & Gentier 48E/48R/96E/96R Real-Time PCR System PCR instrument (referred as Gentier PCR, Tianlong Technology Co., Ltd), Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kit & SLAN PCR instrument, Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kit & ABI7500 PCR instrument, and the BioGerm Nucleic Acid Extraction kit (Shanghai BioGerm Medical Technology Co., Ltd) & BioGerm RT-PCR kit & ABI7500 PCR instrument were used for analysis (Fig. 4, Additional file 3: Table S4). No significant difference in Ct values for ORF1ab and N among laboratories using the same test system was found.
The Ct values derived from EQA samples were detected by different laboratories using the same test system
Assessing the PCR efficiency through standard curves
To determine the suitable dilutions for standard curves, we used the Ct values for ORF1ab derived from the EQA samples with concentrations of 2.0 × 103, 1.0 × 103, 5.0 × 102, and 2.0 × 102 copies/mL. The EQA samples were determined by 12 different RT-PCR kits within a laboratory using the same extraction kit and PCR instrument (Table 3). r2 of the standard curves showed that all the RT-PCR kits except for BioGerm RT-PCR kit and NaGene RT-PCR kit had r2 > 0.94 when using four dilutions of 2.0 × 103, 1.0 × 103, 5.0 × 102, and 2.0 × 102 copies/mL (Table 3). When the samples of 2.0 × 102 copies/mL were excluded, the r2 values for the 12 RT-PCR kits were > 0.95. As a result, three dilutions of 2.0 × 103, 1.0 × 103, and 5.0 × 102 copies/mL were used for standard curves (Table 3). The amplification efficiency varied among different RT-PCR kits, with a minimum value of 61.55% and a maximum value of 128.24% (Table 3, Additional file 1: Figure S1).
We also assessed the PCR efficiency of the participating laboratories. The Ct values from EQA samples with concentrations of 2.0 × 103, 1.0 × 103, and 5.0 × 102 copies/mL were used. Only laboratories that reported at least three Ct values for EQA samples of the same concentration were included to avoid the random effects in each run [23]. The mean Ct values of the same concentrations versus the logarithm of the corresponding target concentrations were plotted onto the standard curves (Additional file 2: Figure S2). Four laboratories using Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kits & Gentier PCR instruments were included, with the PCR efficiency ranging from 44.71 to 116.02% for ORF1ab and 87.80% to 169.15% for N, respectively (Table 4). Two laboratories using Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kit & SLAN PCR instrument yielded efficiencies of 78.20% and 80.84% for ORF1ab and 98.62% and 100% for N (Table 4). For laboratories using Tianlong Nucleic Acid Extraction kit & BioGerm RT-PCR kit & ABI7500, the PCR efficiency varied with a minimum of 78.20% and a maximum of 217.47%, regardless of target genes (Table 4).