Normalization and Correction after hic2cool #724
Comments
|
Hi Matt, that's a tough one and there is no 'no fits all answer'.
Best, Joachim |
|
Hi Joachim, You mentioned normalizing first and then using hicCorrectMatrix - I am just a bit confused because this produces a pretty weird map compared to the other methods. Here's an uncorrected map: Here's a normalized and corrected map via hicNormalize and hicCorrectMatrix: And here's a map that was normalized with hicNormalize then used hicConvertMatrix with correction_name KR: The second map seems to be pretty much wiping my data out, which is odd. |
|
To which value range have you normalized? 0 - 1? because you have now 10^0 instead 10^5?! |
|
Hi Joachim, |
|
Hi Joachim, Another thought I had was perhaps my dataset isn't being corrected all the way through - I am using my laptop to correct data, but based on the hicexplorer website, it gives a timeframe for correction in hours with lots of RAM, while my correction command using hicCorrectMatrix on my laptop (16gb RAM) was on avg about 30 mins. Maybe this is why my values are so low in my corrected maps? As a reminder, I used hic2cool for this process and have the corrections, but the columns are not applied. I am still confused on the difference between hicConvertMatrix with correction factor set to KR vs. hicCorrectMatrix using KR as the correction method. |
|
Hi Matt, sorry for the delayed answer. With 1.8bill reads and 16 GB and half an hour runtime, there is no way this is not crashing, and if it doesn't I don't what it computes. But with that read number, better go to 128 or maybe even 256 GB of memory. With fewer resources, I would not even try it. Also with the 450 million reads, do not work on a notebook with 16 GB memory. Hi-C (and especially the correction) is very memory-demanding. If you don't have the resources, I recommend using the software-as-a-service approach we offer: hicexplorer.usegalaxy.eu. The difference between The reason why you can't use the normalization and then apply the precomputed correction factors is simply that the data basis changed with the normalization and therefore the correction factors are not the mathematical correct factors anymore. If you apply them, they might look alright, but that doesn't change the fact that they are wrong. Best, Joachim |
|
Thanks so much, Joachim! What you're saying makes total sense. I knew I shouldn't be using 16gb to analyze the data, but wanted to try for some reason lol. I do have an AWS account and I currently have them running now. Ah, the convertmatrix vs. correctmatrix makes sense now. Thanks for the clarification - really, I should have just said it out loud and it would have made more sense! I'll keep you updated with AWS computing (Linux Ubuntu, 196GB of RAM), so can we keep this open for a bit longer in case I run into any more problems? |
|
Hi Joachim, I'll start from the beginning so you can have the full picture.
This code generates an mcool file with all my resolutions (2.5mb, 1mb, 500kb, 250kb, 100kb, 50kb, 25kb, 10kb, 5kb, 1kb) for each dataset. From there, I individually normalize each resolution from each dataset with each other.
From there, I then correct each resolution individually. I did that for each resolution (2.5mb all the way down to 1kb). I did all of this on a AWS EC2 instance - I used ubuntu 20.04, using an m4.16xlarge instance with 64 vCPUs and 256GB of memory and 600GB of EBS storage. Similar to when I was using my laptop, correction took ~1min to ~30mins for the 2.5mb to 1kb resolutions, respectively, which is much shorter than what is on the hicexplorer tutorial page. I then tried the hicexplorer tool using galaxy and got similar results with the 2.5mb resolution matrix - I didn't try 1kb resolution yet because I was getting the same results with the 2.5mb resolution. I don't know if it's something I'm doing wrong in the steps before hicCorrectMatrix or if my dataset is just off or do I need to convert from cool to h5? I also ran some diagnostic plots from 50kb resolution and those are pasted below: And here's diagnostic plots of normalized and then normalized and corrected data, respectively. I also ran a plot for distribution vs. counts as well, which is pasted below as raw, norm, and norm with correction, respectively. Lastly, I made a regional matrix plot on Chromosome 1 from 10-20mb with 50kb resolution.
I'm not sure where all my reads are going! It just makes no sense why I start with 10^5 reads and basically go down to zero. I apologize for the long message and I hope it's something small that I am messing up. |
|
Hi Matt, I confirm this bug with KR correction and cool files. We can't really explain it at the moment, because we have not changed the concerning code section in the last two years. We will come up with a patch as soon as we have identified what goes wrong. In the meantime, the workaround I can offer you is to convert the cool files to h5 as follows:
Using now the KR correction works. A short explanation what is wrong: KR correction normalizes the data in a way that the sum is 1, leading to the small values you have at the moment. This behaviour is normal, and some competitors of us offer it only that way. What we apply after the correction is to scale the values back the original data range. For reasons we need to identify this does not work for cool files. Thanks for reporting this issue, Joachim |
|
Hi Matt, the correction factors have not been scaled back to the original data range as I suspected in the last post. In the development branch, there is a fix for this provided. Feel free to try it. Best, Joachim |
|
Hi Joachim, |
|
Hi Matt, a bit of work is required:
Best, Joachim |
|
Thanks for the info, Joachim! I have since just been using the workaround that you suggested converting to an .h5 file. From there, I normalized and then corrected using hicCorrectMatrix. I am, however, running into another problem. I was recently running hicPCA and used the structure suggested on the hicexplorer page: It seemed to run smoothly, but when I then tried to use those output files for hicCompartmentalization using this command: However, I keep getting an error: It's my understanding that I can just use the obs_exp matrix created during hicPCA just the same as if I created it using hicTransform so I should be able to plug it right into hicCompartmentalization or is that not correct? I am not sure if it's my files or just am inputting commands wrong? I looked at my PCA1.bedgraph file in my text editor and the first few lines look like this: I'm not familiar with the structure, but seems odd that there's letters there? When I look at my .h5 files, they also have letters - so maybe they're supposed to be there? I apologize for the naive questions! |
|
Hi Matt, for PCA: you need to specify the file format: https://github.com/deeptools/HiCExplorer/blob/master/hicexplorer/hicPCA.py#L64 The default is bigwig, and therefore the 'numbers and letters' you see are hexadecimal numbers representing the binary bigwig format. Therefore you get the error in hicCompartmentalization, there we expect bedgraph and not bigwig files. Also, the h5 is a binary format based on hdf5 files, the same is true for the cool files. Binary files have major advantages, also handling Hi-C matrices as regular text files is in my opinion a very naive and error-prone solution - but I know that competing Hi-C analysis software handle it in this way. To open these files you need special software that know how to read this data. If you really want to have a look at the internal structures of h5 or cool files you need either the access via the API or use software like the hdf viewer. Best, Joachim |
|
Hi Joachim, |
|
Hi Joachim,
Thanks so much! |
More or less, the correction factors are directly applied to the data for h5 files; therefore, in the h5-> cool created files, it will be the same.
I can't give you a 'one fits' all solution. The best way is to compute the TADs and plot them with pyGenomeTracks to see if it is matching as well as possible. And simply redo this until you have a good setting. The general issue with TADs is that if you have 20 algorithms, they will give you 20 different solutions... |
Hello,
Thanks so much for the great tool. I'm new to hicexplorer (and bioinformatics, in general) and I had a quick question about normalization and correction.
I've been reading through the docs and found the hicConvertFormat info and I was able to get an .mcool matrix from my .hic.
From there, I read that the doc suggests normalizing first and then correcting, but I'm a bit confused as to if using the command hicConvertFormat with correction_name KR is automatically applied to the matrix or if I need to use hicCorrectMatrix. My original .hic file has corrections in it and when using hicInfo it shows:
"The following columns are available: ['chrom' 'start' 'end' 'KR' 'SCALE' 'VC' 'VC_SQRT']."
I found previous questions regarding this issue:
#661
#570
Where I'm confused is that in issue 661, it is suggested to use hicConvertFormat to get .hic to .cool then using hicConvertFormat again to apply correction_name, but gives no info on normalization.
In issue 570 it is stated to use hicConvertFormat to get .hic to .cool and then normalize using hicNormalize followed by matrix correction via hicConvertFormat only if the correction is stored in the original matrix; if not then you must use hicCorrectMatrix.
I just want to make sure that I am on the right track by:
converting some.hic to some.cool file;
normalizing some.cool to some_normalized.cool;
and lastly correcting some_normalized.cool to some_normalized_corrected.cool via the hicConvertFormat with the correction_name KR option set.
I hope this makes sense and thanks so much for the help!
-Matt
The text was updated successfully, but these errors were encountered: