cooler.coarsen_cooler() is not functioning appropriately?

[KunFang93]

Hi,

I met a situation that cooler.coarsen_cooler() behaviors differently when I use one chromosome and all chromosomes:
My codes is following

h1_clr_res1k = cooler.Cooler(f'{outdir}/raw_data/H1.cool')
h1_chr17_pixels = h1_clr_res1k.pixels().fetch('chr17')
cooler.create_cooler(cool_uri=f'{outdir}/raw_data/H1_res1k_chr17.v2.cool',
                     bins=h1_clr_res1k.bins()[:],
                     pixels=h1_clr_res1k.pixels().fetch('chr17'),
                     ordered=True,
                     dtypes={'count': np.float32})

cooler.coarsen_cooler(f'{outdir}/raw_data/H1_res1k_chr17.v2.cool', f'{outdir}/raw_data/H1_res1000k_chr17.v2.cool', 1000,chunksize=1.5*1e7)
chr = 'chr17'
test = cooler.Cooler(f'{outdir}/raw_data/H1_res1000k_chr17.v2.cool')
f, ax = plt.subplots(1,1)
test_mat = test.matrix(balance=False).fetch(chr)
vmax = np.max([test_mat.max()])
norm = LogNorm(vmax=vmax)
im = ax.matshow(
    test_mat,
    norm=norm,
    cmap='fall',
    extent=(0,test.chromsizes[chr], test.chromsizes[chr],0)
)
ax.set_title(f'HiC {chr}', y=1.08)
ax.set_ylabel('position, Mb')
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04, label='count(log)')
plt.savefig(f'{outdir}/test_norm_chr17.png')
plt.close()


cooler.coarsen_cooler(f'{outdir}/raw_data/H1.v2.cool', f'{outdir}/raw_data/H1_res1000k_allchr.v2.cool', 1000, chunksize=1.5*1e7)
test2 = cooler.Cooler(f'{outdir}/raw_data/H1_res1000k_allchr.v2.cool')
f, ax = plt.subplots(1,1)
test2_mat = test2.matrix(balance=False).fetch(chr)
vmax = np.max([test2_mat.max()])
norm = LogNorm(vmax=vmax)
im = ax.matshow(
    test2_mat,
    norm=norm,
    cmap='fall',
    extent=(0,test2.chromsizes[chr], test2.chromsizes[chr],0)
)
ax.set_title(f'HiC {chr}', y=1.08)
ax.set_ylabel('position, Mb')
plt.colorbar(im, ax=ax, fraction=0.046, pad=0.04, label='count(log)')
plt.savefig(f'{outdir}/test2_norm_chr17.png')
plt.close()

And test_norm_chr17.png looks like this

but test2_norm_chr17.png looks weird

I wondered how could I solve the problem is test2 figure? Thanks for your time and help!

Best,
Kun

[KunFang93]

I found that the problem might be related to the iterator for generating H1.v2.cool but not sure how to solve it TT...
Use the following codes will lead to test2_norm_chr17.png

def chrom_norm_iterator(foo):
    for chr in ref_dict:
        print(f"Processing {chr}")
        ...Processing code...
        all_coo_chr.columns = ['bin1_id','bin2_id','count']
        yield all_coo_chr
def save_cooler(cooler_path,bins_df,chrom_iter):
    cooler.create_cooler(cool_uri=cooler_path,
                         bins=bins_df,
                         pixels=chrom_iter,
                         ordered=True,
                         dtypes={'count': np.float32})
    return None

But the following code without using iterator works

def chrom_norm_iterator(foo):
    all_pixels = []
    for chr in ref_dict:
        print(f"Processing {chr}")
        ...Processing code...
        all_coo_chr.columns = ['bin1_id','bin2_id','count']
        all_pixels.append(all_coo_chr)
     return pd.concat(all_pixels)
def save_cooler(cooler_path,bins_df,all_pixels):
    cooler.create_cooler(cool_uri=cooler_path,
                         bins=bins_df,
                         pixels=all_pixels,
                         ordered=True,
                         dtypes={'count': np.float32})
    return None

Thanks in advance!

[nvictus]

Without a reproducible example, I can't say for sure, but if I had to guess, it looks like in the chr17 case you might be passing a bin table restricted to chr17, which changes the bin IDs (i.e. the first bin of chr17 is now bin 0, etc.). That means you would have to relabel the bin IDs of your pixels accordingly.

Alternatively, use the same global bin table as the full matrix, but only store cis chr17 data in it. Then you don't need to relabel any bins.

[KunFang93]

Thanks for your reply. Yes, you are absolutely correct. It was related to wrong bins index or ID if I remember correctly. I have changed in my codes but somehow forgot to post answer here and close the issue.