ComPath¶
An interactive and extensible web application for comparative analysis of pathway databases. ComPath is available at https://compath.scai.fraunhofer.de/.
This package exposes the Bio2BEL pathway packages into a web application containing multiple built-in visualization and analytics tools allowing for their analysis and comparison. By default, this packages wraps the following default packages:
New pathway/gene signatures resources can be added by forking the ComPath Template Repository.
How to Use¶
Install and load the required packages
If you just cloned the repo, you can run the sh script “load_compath.sh” by typing
sh load_compath.sh
in your terminal. This script will first install all packages and later populate the database.If you have already installed the packages, but not loaded the data. First, load Bio2BEL HGNC (see next section). Next, load all individual pathway database packages KEGG, Reactome, WikiPathways, and MSigDB with
python3 -m compath populate
. This command assumes that these packages are already installed in your Python environment. You can check the packages installed by runningpython3 -m compath ls
in your terminal. Alternatively, you can populate each package independently by running:python3 -m bio2bel_kegg populate
,python3 -m bio2bel_reactome populate
,python3 -m bio2bel_wikipathways populate
, orpython3 -m bio2bel_msig populate
.
The final step is to start the web application locally (runs by default on port 5000 -> http://localhost:5000) by running
python3 -m compath web
Mapping across gene/protein identifiers¶
In order to load the gene sets from default packages, ComPath assumes that Bio2BEL HGNC has been already installed and populated. This package is required to perform the mapping from multiple Protein/Gene identifiers to HGNC symbols. The following steps are needed to install Bio2BEL HGNC:
python3 -m pip install bio2bel_hgnc
python3 -m bio2bel_hgnc populate
Repositories¶
The following is an incomplete listing of the Bio2BEL repositories enabled for use with ComPath.
Source |
Version |
Documentation |
Travis |
DOI |
---|---|---|---|---|
ComPath HGNC |
||||
KEGG |
||||
Reactome |
||||
WikiPathways |
||||
MSigDB |
Command Line Interface¶
compath¶
ComPath at sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
compath [OPTIONS] COMMAND [ARGS]...
drop¶
Drop ComPath DB.
compath drop [OPTIONS]
Options
-
-v
,
--debug
¶
Turn on debugging.
-
-y
,
--yes
¶
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
drop-databases¶
Drop all databases.
compath drop-databases [OPTIONS]
Options
-
-v
,
--debug
¶
Turn on debugging.
-
-y
,
--yes
¶
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
load-hierarchies¶
Load pathway databases with hierarchies.
compath load-hierarchies [OPTIONS]
Options
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
-
-e
,
--email
<email>
¶ Default curator: daniel.domingo.fernandez@scai.fraunhofer.de
load-mappings¶
Load mappings from template.
compath load-mappings [OPTIONS]
Options
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
manage¶
Manage the database.
compath manage [OPTIONS] COMMAND [ARGS]...
Options
-
-c
,
--connection
<connection>
¶ Cache connection. Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
populate¶
Populate all registered Bio2BEL pathway packages.
compath populate [OPTIONS]
Options
-
-v
,
--debug
¶
Turn on debugging.
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
-
-d
,
--delete-first
¶
web¶
Run web service.
compath web [OPTIONS]
Options
-
--host
<host>
¶ Flask host. Defaults to 0.0.0.0
-
--port
<port>
¶ Flask port. Defaults to 5000
-
--template-folder
<template_folder>
¶ Template folder. Defaults to ‘templates’
-
--static-folder
<static_folder>
¶ Template folder. Defaults to ‘static’
-
-v
,
--debug
¶
Turn on debugging.
-
-c
,
--connection
<connection>
¶ Defaults to sqlite:////home/docs/.pybel/bio2bel/bio2bel.db
Manager¶
This module the database manager of ComPath.
-
class
compath.manager.
Manager
(engine, session)[source]¶ Database manager.
Init ComPath manager.
-
get_all_mappings
() → List[compath.models.PathwayMapping][source]¶ Get all mappings in the database.
-
get_all_accepted_mappings
() → List[compath.models.PathwayMapping][source]¶ Get all accepted mappings in the database.
-
get_mappings_by_type
(mapping_type: str) → List[compath.models.PathwayMapping][source]¶ Get all mappings in the database.
- Parameters
mapping_type – type of the mapping
-
get_vote_by_id
(vote_id: str) → Optional[compath.models.Vote][source]¶ Get a vote by its id.
- Parameters
vote_id – identifier
-
get_vote
(user: compath.models.User, mapping: compath.models.PathwayMapping) → Optional[compath.models.Vote][source]¶ Get a vote.
- Parameters
user – User instance
mapping – Mapping instance
-
get_user_by_email
(email: str) → Optional[compath.models.User][source]¶ Get a user by their email address.
-
get_mapping
(service_1_name, pathway_1_id, pathway_1_name, service_2_name, pathway_2_id, pathway_2_name, mapping_type)[source]¶ Query mapping in the database.
- Parameters
service_1_name (str) – manager name of the service 1
pathway_1_id (str) – pathway 1 id
pathway_1_name (str) – pathway 1 name
service_2_name (str) – manager name of the service 1
pathway_2_id (str) – pathway 2 id
pathway_2_name (str) – pathway 2 name
mapping_type (str) – mapping type (isPartOf or equivalentTo)
- Return type
Optional[Mapping]
-
get_mapping_by_id
(mapping_id: int) → Optional[compath.models.PathwayMapping][source]¶ Get a mapping by its id.
- Parameters
mapping_id – mapping id
-
get_or_create_vote
(user, mapping, vote_type=True)[source]¶ Get or create vote.
- Parameters
user (User) – User instance
mapping (PathwayMapping) – Mapping instance
vote_type (Optional[Vote.type]) – vote type
- Return type
-
get_or_create_mapping
(service_1_name, pathway_1_id, pathway_1_name, service_2_name, pathway_2_id, pathway_2_name, mapping_type, user)[source]¶ Get or create a mapping.
- Parameters
service_1_name (str) – manager name of the service 1
pathway_1_name (str) – pathway 1 name
pathway_1_id (str) – pathway 1 id
service_2_name (str) – manager name of the service 1
pathway_2_name (str) – pathway 2 name
pathway_2_id (str) – pathway 2 id
mapping_type (str) – type of mapping
user (User) – the user
- Returns
PathwayMapping and boolean indicating if the mapping was created or not
- Return type
-
claim_mapping
(mapping, user)[source]¶ Check if user has already established the mapping, if not claims it.
- Parameters
mapping (PathwayMapping) – Mapping instance
user (User) – User
- Return type
- Returns
if mapping was assigned to user
-
accept_mapping
(mapping_id)[source]¶ Accept established mapping (from user or curator consensus).
- Parameters
mapping_id (int) – mapping id
- Return type
- Returns
mapping and boolean that indicates if transaction was made
-
get_mappings_from_pathway_with_relationship
(type, service_name, pathway_id, pathway_name)[source]¶ Get all mappings matching pathway and service name.
-
get_decendents_mappings_from_pathway_with_is_part_of_relationship
(service_name, pathway_id, pathway_name)[source]¶ Get all mappings matching pathway and service name.
-
get_ancestry_mappings_from_pathway_with_is_part_of_relationship
(service_name, pathway_id, pathway_name)[source]¶ Get all mappings matching pathway and service name.
-
get_all_mappings_from_pathway
(service_name, pathway_id, pathway_name)[source]¶ Get all mappings matching pathway and service name.
- Parameters
- Return type
- Returns
-
Models¶
ComPath database model.
-
class
compath.models.
User
(**kwargs)[source]¶ User table.
A simple constructor that allows initialization from kwargs.
Sets attributes on the constructed instance using the names and values in
kwargs
.Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.
-
property
is_admin
¶ Is this user an administrator?.
-
property
-
class
compath.models.
Role
(**kwargs)[source]¶ Role table.
A simple constructor that allows initialization from kwargs.
Sets attributes on the constructed instance using the names and values in
kwargs
.Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.
-
class
compath.models.
PathwayMapping
(**kwargs)[source]¶ Mapping table.
A simple constructor that allows initialization from kwargs.
Sets attributes on the constructed instance using the names and values in
kwargs
.Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.
-
service_1_name
¶ service name (e.g., KEGG or Reactome
-
service_1_pathway_id
¶ pathway 1 id
-
service_1_pathway_name
¶ pathway 1 name
-
service_2_name
¶ service name (e.g., KEGG or Reactome
-
service_2_pathway_id
¶ pathway 2 id
-
service_2_pathway_name
¶ pathway 2 name
-
type
¶ Type of Mapping (isPartOf or equivalentTo)
-
accepted
¶ accepted mapping by the admin/curator consensus
-
get_complement_mapping_info
(service_name, pathway_id, pathway_name)[source]¶ Return the info corresponding to the other pathway in a mapping.
-
static
has_pathway_tuple
(type, service_name, pathway_id, pathway_name)[source]¶ Return a filter to get all mappings matching type, service and pathway name and id.
-
static
has_descendant_pathway_tuple
(type, service_name, pathway_id, pathway_name)[source]¶ Return a filter to get all the descendants mappings matching a isPartOf relationship (the predicates), service and pathway name and id.
-
static
has_ancestry_pathway_tuple
(type, service_name, pathway_id, pathway_name)[source]¶ Return a filter to get all the ancestries mappings matching a isPartOf relationship (the subjects), service and pathway name and id.
-
static
has_pathway
(service_name, pathway_id, pathway_name)[source]¶ Return a filter to get all mappings matching service and pathway name and id.
-
static
has_database_pathway
(service_name)[source]¶ Return a filter to get all mappings matching service a name.
-
property
is_acceptable
¶ Return true if the mapping has enough votes to be accepted.
- Return type
-
-
class
compath.models.
Vote
(**kwargs)[source]¶ Vote table.
A simple constructor that allows initialization from kwargs.
Sets attributes on the constructed instance using the names and values in
kwargs
.Only keys that are present as attributes of the instance’s class are allowed. These could be, for example, any mapped columns or relationships.
-
type
¶ Type of vote, by default is up-vote
-
Constants¶
Constants for ComPath.
This module contains all the string constants used in ComPath.
-
compath.constants.
REACTOME_GENE_SET
= '/home/docs/checkouts/readthedocs.org/user_builds/compath/checkouts/stable/src/compath/static/resources/excel/kegg_gene_sets.csv'¶ CSV file for Reactome gene set
-
compath.constants.
KEGG_GENE_SET
= '/home/docs/checkouts/readthedocs.org/user_builds/compath/checkouts/stable/src/compath/static/resources/excel/reactome_gene_sets.csv'¶ CSV file for KEGG gene set
-
compath.constants.
WIKIPATHWAYS_GENE_SET
= '/home/docs/checkouts/readthedocs.org/user_builds/compath/checkouts/stable/src/compath/static/resources/excel/wikipathways_gene_sets.csv'¶ CSV file for WikiPathways gene set
-
compath.constants.
MSIG_GENE_SET
= '/home/docs/checkouts/readthedocs.org/user_builds/compath/checkouts/stable/src/compath/static/resources/excel/msig_gene_sets.csv'¶ CSV file for MSigDB gene set
-
compath.constants.
VOTE_ACCEPTANCE
= 3¶ Minimum number of votes to accept a mapping
-
compath.constants.
KEGG
= 'kegg'¶ KEGG
-
compath.constants.
REACTOME
= 'reactome'¶ Reactome
-
compath.constants.
WIKIPATHWAYS
= 'wikipathways'¶ WikiPathways
-
compath.constants.
MSIG
= 'msig'¶ MSigDB
-
compath.constants.
KEGG_URL
= 'http://www.kegg.jp/kegg-bin/show_pathway?map=map{}&show_description=show'¶ REST API to KEGG
-
compath.constants.
REACTOME_URL
= 'https://reactome.org/PathwayBrowser/#/{}'¶ URL to pathways in Reactome
-
compath.constants.
WIKIPATHWAYS_URL
= 'https://www.wikipathways.org/index.php/Pathway:{}'¶ URL to pathways in WikiPathways
-
compath.constants.
HIERARCHY_MANAGERS
= {'reactome'}¶ Managers with hierarchical information
-
compath.constants.
BLACK_LIST
= {'compath_hgnc', 'hgnc'}¶ Managers without pathway knowledge
-
compath.constants.
MAPPING_TYPES
= {'equivalentTo', 'isPartOf'}¶ Possible mapping types between pathways
-
compath.constants.
STYLED_NAMES
= {'compath_neurommsig_ad': 'NeuroMMSig AD', 'compath_neurommsig_pd': 'NeuroMMSig PD', 'kegg': 'KEGG', 'msig': 'MSig', 'reactome': 'Reactome', 'wikipathways': 'WikiPathways'}¶ Mappings for pathway database
-
compath.constants.
EQUIVALENT_TO
= 'equivalentTo'¶ Possible mapping types between pathways
Curation¶
Modules related with the ComPath curation exercise.
This module loads the hierarchical pathway databases into ComPath.
-
compath.curation.hierarchies.
create_hierarchical_mappings
(pathways, compath_manager, pathway_database, curator)[source]¶ Iterate over pathway objects and creates hierarchies if exist.
- Parameters
pathways (list[Pathway]) – list of pathways
compath_manager (compath.manager.Manager) – ComPath Manager
pathway_database (str) – Name of the pathway database
curator (compath.models.User) – Curator user
- Returns
number of created mappings
- Return type
-
compath.curation.hierarchies.
load_hierarchy
(*, connection=None, curator_email=None)[source]¶ Load the hierarchical relationships for the managers containing them.
This module contains all the curation processing methods to integrate curated mappings to ComPath.
-
compath.curation.parser.
get_users_from_emails
(compath_manager, curator_emails)[source]¶ Return a list with curator users.
-
compath.curation.parser.
parse_special_mappings
(path, curator_emails=None, connection=None)[source]¶ Parse special mapping file.
-
compath.curation.parser.
parse_curation_template
(path, reference_pathway_db, compared_pathway_db, curator_emails=None, connection=None)[source]¶ Load the curation template excel sheet into a pandas Dataframe.
- Parameters
path (str) – path of the excel sheet
reference_pathway_db (str) – name of the manager of the reference pathway_db
compared_pathway_db (str) – name of the manager of the compared pathway_db
curator_emails (Optional[list]) – email of the curators. Needs to be already in the database
connection (str) – database connection
This module contains all the utils for curation processing.
-
compath.curation.utils.
remove_star_from_pathway_name
(pathway_name)[source]¶ Remove the star that label the reference pathway in isPartOf statements.
- Parameters
statements (str) – pathway name
-
compath.curation.utils.
get_pathways_from_statement
(mapping_statement, mapping_type)[source]¶ Return the subject, object of the mapping.
-
compath.curation.utils.
is_valid_pathway_name
(manager_dict, resource, pathway_name)[source]¶ Check if pathway name exists in pathway database.
Visualization¶
All methods used to perform the visualizations.
Network visualization¶
Utils to generate the Cytoscape.js network.
-
compath.visualization.cytoscape.
filter_network_by_similarity
(graph, min_similarity)[source]¶ Remove edges with similarity less than the minimum given.
- Parameters
graph (networkx.Graph) – graph
min_similarity (float) – minimum similarity required to keep an edge
-
compath.visualization.cytoscape.
pathways_to_similarity_network
(manager_dict, pathways)[source]¶ Create a graph with the given pathways related by their similarity
-
compath.visualization.cytoscape.
enrich_graph_with_mappings
(graph, mappings)[source]¶ Enrich a graph with the mapping information.
- Parameters
networkx.Graph (graph) –
mappings (iter) –
-
compath.visualization.cytoscape.
networkx_to_cytoscape_js
(graph)[source]¶ Convert a networkx graph to the cytoscape json format.
- Parameters
graph (iter[PathwayMapping]) –
- Return type
Dendrogram visualization¶
Utils to generate the D3.js dendrogram. This module is adapted from https://gist.github.com/mdml/7537455.
-
compath.visualization.d3_dendrogram.
create_similarity_matrix
(gene_sets)[source]¶ Create a similarity matrix for a given pathway-geneset dataset.
- Parameters
gene_sets (dict) – pathway gene set dictionary
- Return type
pandas.DataFrame
- Returns
similarity matrix
-
compath.visualization.d3_dendrogram.
add_node
(node, parent)[source]¶ Create a nested dictionary from the ClusterNode’s returned by SciPy.
- Parameters
node –
parent (dict) –
-
compath.visualization.d3_dendrogram.
label_tree
(id_name_dict, name_manager_dict, cluster_to_x, tree)[source]¶ Label the tree in a recursive way with names, resource and cluster information.
-
compath.visualization.d3_dendrogram.
get_dendrogram_tree
(gene_sets, pathway_manager_dict)[source]¶ Return ready to plot json data.
-
compath.visualization.d3_dendrogram.
get_descendants
(manager, resource, pathway_id, pathway_name)[source]¶ Generate d3 dendrogram structure by using BFS starting from the starting from a parent (root) node to the last descendants.
-
compath.visualization.d3_dendrogram.
get_mapping_dendrogram
(manager, resource, pathway_id, pathway_name)[source]¶ Generate d3 dendrogram structure by using BFS starting from the queried node in both directions of the hierarchy.
Venn Diagram visualization¶
Utils to generate the Venn Diagram.
-
compath.visualization.venn_diagram.
process_overlap_for_venn_diagram
(gene_sets, skip_gene_set_info=False)[source]¶ Calculate gene sets overlaps and process the structure to render venn diagram -> https://github.com/benfred/venn.js/.
Web¶
This module contains the ComPath web application.
Utils¶
This module contains miscellaneous methods.
-
compath.utils.
dict_to_pandas_df
(d)[source]¶ Transform pandas df into a dict.
- Parameters
d (dict) –
- Return type
pandas.DataFrame
- Returns
pandas dataframe
-
compath.utils.
process_form_gene_set
(text)[source]¶ Process the string containing gene symbols and returns a gene set.
-
compath.utils.
calculate_relative_enrichments
(results, total_pathways_by_resource)[source]¶ Calculate relative enrichment of pathways (enriched pathways/total pathways).
-
compath.utils.
count_genes_in_pathway
(pathways_gene_sets, genes)[source]¶ Calculate how many of the genes are associated to each pathway gene set.
-
compath.utils.
simulate_pathway_enrichment
(resource_gene_sets, gene_set_query, runs=200)[source]¶ Simulate pathway enrichment.
-
compath.utils.
get_genes_without_assigned_pathways
(enrichment_results, genes_query)[source]¶ Return the genes without any known pathway assigned.
-
compath.utils.
get_enriched_pathways
(manager_list, gene_set)[source]¶ Return the results of the queries for every registered manager.
-
compath.utils.
get_gene_pathways
(manager_list, gene)[source]¶ Return the pathways associated with a gene for every registered manager.
-
compath.utils.
get_mappings
(compath_manager, only_accepted=True)[source]¶ Return a pandas dataframe with mappings information as an excel sheet file.
- Parameters
compath_manager (compath.manager.Manager) – ComPath Manager
only_accepted (bool) – only accepted (True) or all (False)
-
compath.utils.
get_pathway_model_by_name
(manager_dict, resource, pathway_name)[source]¶ Return the pathway object from the resource manager.
-
compath.utils.
get_pathway_model_by_id
(app, resource, pathway_id)[source]¶ Return the pathway object from the resource manager.
-
compath.utils.
get_gene_sets_from_pathway_names
(app, pathways)[source]¶ Return the gene sets for a given pathway/resource tuple.
-
compath.utils.
get_pathway_info
(app, pathways)[source]¶ Return the gene sets for a given pathway/resource tuple.
-
compath.utils.
get_last_action_in_module
(module_name, action)[source]¶ Return the info about the last action in the given module.
- Parameters
module_name (str) –
- Returns
-
compath.utils.
perform_hypergeometric_test
(gene_set, manager_pathways_dict, gene_universe, apply_threshold=False, threshold=0.05)[source]¶ Perform hypergeometric tests.
- Parameters
- Return type
- Returns
manager_pathways_dict with p value info
-
compath.utils.
calculate_szymkiewicz_simpson_coefficient
(set_1, set_2)[source]¶ Calculate Szymkiewicz-Simpson coefficient between two sets.
-
compath.utils.
calculate_similarity
(name_1, name_2)[source]¶ Calculate the string based similarity between two names.
-
compath.utils.
get_top_matches
(matches, top)[source]¶ Order list of tuples by second value and returns top values.
-
compath.utils.
filter_results
(results, threshold)[source]¶ Filter a tuple based iterator given a threshold.
ComPath License¶
MIT License
Copyright (c) 2018 Fraunhofer-Gesellschaft e. V. Munich, Germany. All rights reserved.
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.