Oxford Nanopore Simplex Error Rate STILL >5%?

UPDATE: There’s a follow up to this analysis using a better dataset.

Yesterday I wrote a post about how I used the amazing and wonderful GenomeMiner.ai platform (which you should definitely try!) to analyzer some Oxford Nanopore data.

The dataset came from here and is tagged at r10.4.1. It includes the statement “data presented here should be representative of routine sequencing that can be performed by any lab using this latest release”.

The results prompted some discussion on the Discord and as I wasn’t 100% sure how the dataset I had was basecalled I decided to re-run the analysis against the fastqs provided. I grabbed the fastqs with the following¹:

aws s3 sync --no-sign-request s3://ont-open-data/giab_2023.05/flowcells/hg002/20230429_1600_2E_PAO83395_124388f5/basecalled/pass pass

And smashed them all together.

cd pass;cat *fastq > all.fastq

Then ran them against Dorado aligner and BEST on GenomeMiner. The results are pretty similar, except for some crazy weirdness at high Q:

combined_q is yesterday’s dataset. combinedfq_q is today’s (fastqs provided in the S3 bucket)

Whatever base caller is being used by ONT here is clearly happy calling out to higher Q scores. However the population at high Q is very low (10s). This is probably what causes all those crazy artifacts after Q60.

Looking at the distribution of Q scores, not much has changed:

If we look at the overall metrics, accuracy has improved slightly, but out error rate is still greater than 5%. The overall identity would put the error rate at 5.74%:

This is a long way off the accuracy claims on their website:

To me this looks like NumberWang!² And the results here are in a similar range to reports from other users using the hac model.

Based on this it would seem that Oxford Nanopore’s current accuracy is worse than first generation Illumina sequencers³.

More comments and suggestions are welcome on the Discord. BEST is pretty cool and it would be great if this could be used as a standard platform for assessing data quality, so if there are issues here it would be nice to resolve them.

In future post I’ll re-basecall this dataset with all models and see how they compare so… like and subscribe!

1. I’m only using the pass reads I assume ONT only quote metrics for passing reads and that seems fair (both results are for passing only). But this opens up the whole throughput metrics question.

2. ONT have in general reported modal per-read accuracy. Taking the mode here makes no sense to me (depending on the distribution the “most common” Q score bin could be anywhere and there are numerous ways this could be gamed to improve stats. In particular for ONT data this distribution doesn’t appear to be Gaussian so it’s not like the mode, mean and median sit on top of each other anyway…

3. Again this is fine, you’ve got lots of potential use cases for long reads and other features of the platform.