Append a new batch of data#
Here, we’ll learn
how to standardize a less well curated dataset
how to append it as a new batch of data to the growing versioned dataset
import lamindb as ln
import lnschema_bionty as lb
import pandas as pd
ln.track()
💡 loaded instance: testuser1/test-scrna (lamindb 0.55.2)
💡 notebook imports: lamindb==0.55.2 lnschema_bionty==0.31.2 pandas==1.5.3
💡 Transform(id='ManDYgmftZ8Cz8', name='Append a new batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-10-10 15:42:50, created_by_id='DzTjkKse')
💡 Run(id='JRscXJ0ZxufwqGUoGmIJ', run_at=2023-10-10 15:42:50, transform_id='ManDYgmftZ8Cz8', created_by_id='DzTjkKse')
Standardize a less-well curated dataset#
Access 
#
Let’s now consider a dataset with less-well curated features:
adata = ln.dev.datasets.anndata_pbmc68k_reduced()
adata
AnnData object with n_obs × n_vars = 70 × 765
obs: 'cell_type', 'n_genes', 'percent_mito', 'louvain'
var: 'n_counts', 'highly_variable'
uns: 'louvain', 'louvain_colors', 'neighbors', 'pca'
obsm: 'X_pca', 'X_umap'
varm: 'PCs'
obsp: 'connectivities', 'distances'
We see that this dataset is indexed by gene symbols. Because we assume that in-house, we index all datasets by Ensembl IDs, we’ll need to re-curate:
adata.var.head()
| n_counts | highly_variable | |
|---|---|---|
| index | ||
| HES4 | 1153.387451 | True |
| TNFRSF4 | 304.358154 | True |
| SSU72 | 2530.272705 | False |
| PARK7 | 7451.664062 | False |
| RBP7 | 272.811035 | True |
We are still working with human data, and can globally instruct bionty to assume human:
lb.settings.species = "human"
Validate 
#
Curate & validate genes#
lb.Gene.validate(adata.var.index, lb.Gene.symbol);
❗ 70 terms (9.20%) are not validated for symbol: ATPIF1, C1orf228, CCBL2, RP11-782C8.1, RP11-277L2.3, RP11-156E8.1, AC079767.4, GPX1, H1FX, SELT, ATP5I, IGJ, CCDC109B, FYB, H2AFY, FAM65B, HIST1H4C, HIST1H1E, ZNRD1, C6orf48, ...
lb.Gene.inspect(adata.var.index, lb.Gene.symbol);
❗ 70 terms (9.20%) are not validated for symbol: ATPIF1, C1orf228, CCBL2, RP11-782C8.1, RP11-277L2.3, RP11-156E8.1, AC079767.4, GPX1, H1FX, SELT, ATP5I, IGJ, CCDC109B, FYB, H2AFY, FAM65B, HIST1H4C, HIST1H1E, ZNRD1, C6orf48, ...
detected 54 terms with synonyms: ATPIF1, C1orf228, CCBL2, AC079767.4, H1FX, SELT, ATP5I, IGJ, CCDC109B, FYB, H2AFY, FAM65B, HIST1H4C, HIST1H1E, ZNRD1, C6orf48, SEPT7, WBSCR22, RSBN1L-AS1, CCDC132, ...
→ standardize terms via .standardize()
detected 5 Gene terms in Bionty for symbol: 'SNORD3B-2', 'GPX1', 'SOD2', 'RN7SL1', 'IGLL5'
→ add records from Bionty to your Gene registry via .from_values()
couldn't validate 11 terms: 'RP11-156E8.1', 'RP11-782C8.1', 'RP3-467N11.1', 'RP11-620J15.3', 'RP11-277L2.3', 'TMBIM4-1', 'RP11-291B21.2', 'CTD-3138B18.5', 'AC084018.1', 'RP11-390E23.6', 'RP11-489E7.4'
→ if you are sure, create new records via ln.Gene() and save to your registry
Standardize symbols and register additional symbols from Bionty:
adata.var.index = lb.Gene.standardize(adata.var.index, lb.Gene.symbol)
gene_records = lb.Gene.from_values(adata.var.index, lb.Gene.symbol)
ln.save(gene_records)
❗ did not create Gene records for 11 non-validated symbols: 'AC084018.1', 'CTD-3138B18.5', 'RP11-156E8.1', 'RP11-277L2.3', 'RP11-291B21.2', 'RP11-390E23.6', 'RP11-489E7.4', 'RP11-620J15.3', 'RP11-782C8.1', 'RP3-467N11.1', 'TMBIM4-1'
We only want to register data with validated genes: data related to other features wouldn’t be useful to us, anyway.
Hence, we subset the AnnData object to the validated genes:
validated = lb.Gene.validate(adata.var.index, lb.Gene.symbol)
adata_validated = adata[:, validated].copy()
❗ 11 terms (1.40%) are not validated for symbol: RP11-782C8.1, RP11-277L2.3, RP11-156E8.1, RP3-467N11.1, RP11-390E23.6, RP11-489E7.4, RP11-291B21.2, RP11-620J15.3, TMBIM4-1, AC084018.1, CTD-3138B18.5
We also subset raw of the anndata object to the validated genes
adata_validated.raw = adata.raw[:, validated].to_adata()
Now, we need to convert gene symbols into ensembl gene ids:
records = lb.Gene.filter(id__in=[record.id for record in gene_records])
mapper = pd.DataFrame(records.values_list("symbol", "ensembl_gene_id")).set_index(0)[1]
adata_validated.var.insert(0, "gene_symbol", adata_validated.var.index)
adata_validated.var.rename(index=mapper, inplace=True)
adata_validated.var.head()
| gene_symbol | n_counts | highly_variable | |
|---|---|---|---|
| ENSG00000188290 | HES4 | 1153.387451 | True |
| ENSG00000186827 | TNFRSF4 | 304.358154 | True |
| ENSG00000160075 | SSU72 | 2530.272705 | False |
| ENSG00000116288 | PARK7 | 7451.664062 | False |
| ENSG00000162444 | RBP7 | 272.811035 | True |
Raw has the same genes, so set them also.
adata_validated.raw.var.index = adata_validated.var.index
Curate & validate cell types#
Inspection shows none of the terms are validated:
inspector = lb.CellType.inspect(adata_validated.obs.cell_type)
❗ received 9 unique terms, 61 empty/duplicated terms are ignored
❗ 9 terms (100.00%) are not validated for name: Dendritic cells, CD19+ B, CD4+/CD45RO+ Memory, CD8+ Cytotoxic T, CD4+/CD25 T Reg, CD14+ Monocytes, CD56+ NK, CD8+/CD45RA+ Naive Cytotoxic, CD34+
couldn't validate 9 terms: 'CD4+/CD25 T Reg', 'CD56+ NK', 'CD19+ B', 'Dendritic cells', 'CD14+ Monocytes', 'CD34+', 'CD8+ Cytotoxic T', 'CD8+/CD45RA+ Naive Cytotoxic', 'CD4+/CD45RO+ Memory'
→ if you are sure, create new records via ln.CellType() and save to your registry
Let us search the cell type names from the public ontology, and add the name value found in the AnnData object as a synonym to the top match found in the public ontology.
bionty = lb.CellType.bionty() # access the public ontology through bionty
name_mapper = {}
for name in adata_validated.obs.cell_type.unique():
ontology_id = (
bionty.search(name).iloc[0].ontology_id
) # search the public ontology and use the ontology id of the top match
record = lb.CellType.from_bionty(
ontology_id=ontology_id
) # create a record by loading the top match from bionty
name_mapper[name] = record.name # map the original name to standardized name
record.save() # save the record
record.add_synonym(
name
) # add the original name as a synonym, so that next time, we can just run .standardize()
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
❗ now recursing through parents: this only happens once, but is much slower than bulk saving
We can now standardize cell type names using the search-based mapper:
adata_validated.obs.cell_type = adata_validated.obs.cell_type.map(name_mapper)
Now, all cell types are validated:
validated = lb.CellType.validate(adata_validated.obs.cell_type)
assert all(validated)
We don’t want to store any of the other metadata columns:
for column in ["n_genes", "percent_mito", "louvain"]:
adata.obs.drop(column, axis=1)
Register 
#
modalities = ln.Modality.lookup()
experimental_factors = lb.ExperimentalFactor.lookup()
species = lb.Species.lookup()
features = ln.Feature.lookup()
file = ln.File.from_anndata(
adata_validated,
description="10x reference adata",
field=lb.Gene.ensembl_gene_id,
modality=modalities.rna,
)
❗ 3 terms (75.00%) are not validated for name: n_genes, percent_mito, louvain
As we do not want to manage the remaining unvalidated terms in registries, we can save the file.
file.save()
file.labels.add(adata_validated.obs.cell_type, features.cell_type)
file.labels.add(species.human, feature=features.species)
file.labels.add(experimental_factors.single_cell_rna_sequencing, feature=features.assay)
file.describe()
File(id='wRXv3wXHrtOF3OihYYya', suffix='.h5ad', accessor='AnnData', description='10x reference adata', size=857752, hash='j6o6e27xPdqHQyT7Em_7MQ', hash_type='md5', updated_at=2023-10-10 15:43:24)
Provenance:
🗃️ storage: Storage(id='YpFBBtr4', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-10-10 15:41:26, created_by_id='DzTjkKse')
💫 transform: Transform(id='ManDYgmftZ8Cz8', name='Append a new batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-10-10 15:42:50, created_by_id='DzTjkKse')
👣 run: Run(id='JRscXJ0ZxufwqGUoGmIJ', run_at=2023-10-10 15:42:50, transform_id='ManDYgmftZ8Cz8', created_by_id='DzTjkKse')
👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-10-10 15:41:26)
Features:
var: FeatureSet(id='PlJKWLv2xNZtMob20tDy', n=754, type='number', registry='bionty.Gene', hash='WMDxN7253SdzGwmznV5d', updated_at=2023-10-10 15:43:23, modality_id='nG6MZ3aj', created_by_id='DzTjkKse')
'NUCB2', 'CD37', 'HPCAL1', 'ZNF22', 'ADISSP', 'COA1', 'DAAM1', 'ADSL', 'BID', 'GZMA', 'SUMO2', 'HLA-DRA', 'LAT', 'LCK', 'NT5C3B', 'JCHAIN', 'PRF1', 'ABHD17A', 'SPCS2', 'CCT6A', ...
obs: FeatureSet(id='nAOCw869x66BnmaHJlOE', n=1, registry='core.Feature', hash='J_-ageYakMRSB70Itj9F', updated_at=2023-10-10 15:43:24, modality_id='NIjDnou1', created_by_id='DzTjkKse')
🔗 cell_type (9, bionty.CellType): 'B cell, CD19-positive', 'dendritic cell', 'CD16-positive, CD56-dim natural killer cell, human', 'CD8-positive, alpha-beta memory T cell', 'central memory CD8-positive, alpha-beta T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'monocyte', 'mature T cell', 'Cd4-negative, CD8_alpha-negative, CD11b-positive dendritic cell'
external: FeatureSet(id='4Wdwh7kBHsamYc1CH642', n=2, registry='core.Feature', hash='gmtVslHb3x-nqoqNZS2_', updated_at=2023-10-10 15:43:24, modality_id='NIjDnou1', created_by_id='DzTjkKse')
🔗 species (1, bionty.Species): 'human'
🔗 assay (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'
Labels:
🏷️ species (1, bionty.Species): 'human'
🏷️ cell_types (9, bionty.CellType): 'B cell, CD19-positive', 'dendritic cell', 'CD16-positive, CD56-dim natural killer cell, human', 'CD8-positive, alpha-beta memory T cell', 'central memory CD8-positive, alpha-beta T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'monocyte', 'mature T cell', 'Cd4-negative, CD8_alpha-negative, CD11b-positive dendritic cell'
🏷️ experimental_factors (1, bionty.ExperimentalFactor): 'single-cell RNA sequencing'
file.view_flow()
Append a file to the growing dataset#
import lamindb as ln
Query the previous dataset and the file we just created:
dataset_v1 = ln.Dataset.filter(name="My versioned scRNA-seq dataset", version="1").one()
new_file = ln.File.filter().order_by("-created_at").first()
Create a new version of the dataset by sharding it across the new file and the file in the previous version of the dataset:
dataset_v2 = ln.Dataset(
[new_file, dataset_v1.file],
is_new_version_of=dataset_v1,
)
dataset_v2.save()
# annotate the dataset
dataset_v2.labels.add_from(new_file)
dataset_v2.labels.add_from(dataset_v1)
Version 2 of the dataset covers significantly more conditions.
dataset_v2.describe()
Dataset(id='oYS8NR45oBcEPgCISfHj', name='My versioned scRNA-seq dataset', version='2', hash='JNjc88f22TLVPJHdgo7X', updated_at=2023-10-10 15:43:32)
Provenance:
💫 transform: Transform(id='ManDYgmftZ8Cz8', name='Append a new batch of data', short_name='scrna2', version='0', type=notebook, updated_at=2023-10-10 15:42:50, created_by_id='DzTjkKse')
👣 run: Run(id='JRscXJ0ZxufwqGUoGmIJ', run_at=2023-10-10 15:42:50, transform_id='ManDYgmftZ8Cz8', created_by_id='DzTjkKse')
🔖 initial_version: Dataset(id='oYS8NR45oBcEPgCISfCm', name='My versioned scRNA-seq dataset', version='1', hash='6Hu1BywwK6bfIU2Dpku2xZ', updated_at=2023-10-10 15:42:41, transform_id='Nv48yAceNSh8z8', run_id='RNgU2xx7TeUrL3d83b4F', file_id='oYS8NR45oBcEPgCISfCm', created_by_id='DzTjkKse')
👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-10-10 15:41:26)
Features:
var: FeatureSet(id='fEe78TJfVJ4Wh3gfU3mT', n=37257, type='number', registry='bionty.Gene', hash='bWfNZ3hy-yGnPj65T3kc', updated_at=2023-10-10 15:43:26, created_by_id='DzTjkKse')
'ILVBL-AS1', 'None', 'None', 'None', 'None', 'TENT5A', 'SPATA20', 'None', 'LINC00484', 'CFLAR', 'LAT', 'MAPKAPK5', 'LINC01377', 'None', 'None', 'LINC01644', 'VPS8', 'None', 'KCTD12', 'None', ...
obs: FeatureSet(id='KrYPEOnuTBTRi4WqoelO', n=4, registry='core.Feature', hash='Z0BvFRBSIr9xpTLjV1nb', updated_at=2023-10-10 15:42:35, modality_id='NIjDnou1', created_by_id='DzTjkKse')
🔗 donor (12, core.ULabel): '582C', 'A36', 'D503', 'A37', 'A29', '640C', 'D496', 'A52', 'A35', 'A31', ...
🔗 tissue (17, bionty.Tissue): 'jejunal epithelium', 'omentum', 'spleen', 'caecum', 'thymus', 'blood', 'transverse colon', 'duodenum', 'ileum', 'sigmoid colon', ...
🔗 assay (4, bionty.ExperimentalFactor): 'single-cell RNA sequencing', '10x 5' v1', '10x 5' v2', '10x 3' v3'
🔗 cell_type (39, bionty.CellType): 'B cell, CD19-positive', 'dendritic cell', 'CD16-positive, CD56-dim natural killer cell, human', 'CD8-positive, alpha-beta memory T cell', 'central memory CD8-positive, alpha-beta T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'monocyte', 'mature T cell', 'Cd4-negative, CD8_alpha-negative, CD11b-positive dendritic cell', 'germinal center B cell', ...
external: FeatureSet(id='4Wdwh7kBHsamYc1CH642', n=2, registry='core.Feature', hash='gmtVslHb3x-nqoqNZS2_', updated_at=2023-10-10 15:43:24, modality_id='NIjDnou1', created_by_id='DzTjkKse')
🔗 species (1, bionty.Species): 'human'
🔗 assay (4, bionty.ExperimentalFactor): 'single-cell RNA sequencing', '10x 5' v1', '10x 5' v2', '10x 3' v3'
Labels:
🏷️ species (1, bionty.Species): 'human'
🏷️ tissues (17, bionty.Tissue): 'jejunal epithelium', 'omentum', 'spleen', 'caecum', 'thymus', 'blood', 'transverse colon', 'duodenum', 'ileum', 'sigmoid colon', ...
🏷️ cell_types (39, bionty.CellType): 'B cell, CD19-positive', 'dendritic cell', 'CD16-positive, CD56-dim natural killer cell, human', 'CD8-positive, alpha-beta memory T cell', 'central memory CD8-positive, alpha-beta T cell', 'CD8-positive, CD25-positive, alpha-beta regulatory T cell', 'monocyte', 'mature T cell', 'Cd4-negative, CD8_alpha-negative, CD11b-positive dendritic cell', 'germinal center B cell', ...
🏷️ experimental_factors (4, bionty.ExperimentalFactor): 'single-cell RNA sequencing', '10x 5' v1', '10x 5' v2', '10x 3' v3'
🏷️ ulabels (12, core.ULabel): '582C', 'A36', 'D503', 'A37', 'A29', '640C', 'D496', 'A52', 'A35', 'A31', ...
View the flow:
dataset_v2.view_flow()
