Skip to content

Commit 5d50911

Browse files
mikecormiermikecormier
committed
Add page on private data recipes
1 parent a6bad13 commit 5d50911

378 files changed

Lines changed: 813 additions & 3 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎GGD-CLI.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ <h3>Navigation</h3>
7272
</ul>
7373
</li>
7474
<li class="toctree-l1"><a class="reference internal" href="contribute.html">Contribute</a></li>
75+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
7576
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
7677
</ul>
7778

‎_sources/index.rst.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -441,5 +441,6 @@ Contents:
441441
using-ggd
442442
GGD-CLI
443443
contribute
444+
private_recipes
444445
recipes
445446

‎_sources/private_recipes.rst.txt‎

Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
.. _private-recipes:
2+
3+
Private Recipes
4+
===============
5+
6+
[:ref:`Click here to return to the home page <home-page>`]
7+
8+
9+
Although GGD has been developed to provide reproducible data access and management for public scientific data,
10+
GGD can also be used to reproducibility curate and manage private data.
11+
12+
For one reason or another, a researcher may wish to keep data private temporarily but still wish to use GGD to
13+
document data access, curation, and management. Below are suggested steps that can be followed in order to use
14+
GGD to manage private data on a local machine.
15+
16+
.. note::
17+
18+
Private data management is not the primary use of GGD. GGD is used for public data access and use. It should be
19+
noted that private data recipes **cannot** be added to the ggd-recipes repository on github, they **cannot** be
20+
added to any ggd conda channel, and they **cannot** be added to ggd cloud caching services. All private data
21+
recipes will need to be stored and maintained by creator/user of those data recipes.
22+
23+
24+
1. Create a private github repository to store private data recipes
25+
-------------------------------------------------------------------
26+
27+
We suggest that you create a private github repository to store your private data recipes. This allows for data recipe
28+
versioning and storage. As noted above, private data recipes **cannot** be stored on the ggd repositories.
29+
30+
It is important that these recipes are somehow stored and maintained as they are the infrastructure and manual that ggd will
31+
use. They are key for the reproducibility and provenance aspects of data management.
32+
33+
34+
2. Create a ggd recipe
35+
----------------------
36+
37+
Use :code:`ggd make-recipe` to create the private data recipe(s) from a bash script with the instruction on data access, curations,
38+
and processing.
39+
40+
see :ref:`ggd make-recipe <ggd-make-recipe>` for more information on how to create a data recipe
41+
42+
43+
44+
3. Check and install the data recipe
45+
------------------------------------
46+
47+
Because the private ggd data recipe will not be added to the ggd library, you **won't** be able to use :code:`ggd install` to install the
48+
recipe. Instead you will use :code:`ggd check-recipe` command to install the recipe locally. In order to do this the :code:`-du` argument
49+
must be added to the :code:`ggd check-recipes` command.
50+
51+
For example, if you have a private data recipe called *grch37-de-novo-callset-lark-v1* you would install it using the following command:
52+
53+
:code:`ggd check-recipe grch37-de-novo-callset-lark-v1 -du`
54+
55+
.. warning::
56+
57+
The :code:`ggd check-recipe` command does not have a :code:`--prefix` argument. This means you have to be in the conda environment
58+
where you wish to install the data.
59+
60+
61+
for more information on the :code:`ggd check-recipe` command see :ref:`ggd check-recipe <ggd-check-recipe>`
62+
63+
.. note::
64+
65+
Installing private data recipes will take longer because the recipe goes through the normal check and validation step before it is
66+
installed. Additionally, there is now way to cache private data recipes, so the speedup seen from installing public data recipes is
67+
not available.
68+
69+
70+
4. Add the data recipe to github
71+
---------------------------------
72+
73+
Now that the data recipe has been created, it has been checked, and installed, you can add it to your private github repository.
74+
75+
76+
.. note::
77+
78+
We are suggesting adding the *data recipe* not the actual data to the private github repository
79+
80+
81+
82+
5. How to access installed data from private recipe
83+
----------------------------------------------------
84+
85+
Once a private data recipe has been installed on your system you can access and use it in a few different ways.
86+
87+
A. Using environment variables
88+
89+
A private GGD recipe will come with environment variables as does a non-private ggd recipes. You must be in the same environment where the
90+
data recipe has been installed in order to use them. You must first run :code:`source activate base` before the environment variables are
91+
active.
92+
93+
B. Showing active environment variables
94+
95+
Environment variables for private data recipes can be listed using :code:`ggd show-env`
96+
97+
For more information about the :code:`ggd show-env` command see: :ref:`ggd-show-env`
98+
99+
C. Listing installed private recipes
100+
101+
A list of installed data recipes, including private recipes, can be seen using :code:`ggd list`.
102+
103+
.. note::
104+
105+
:code:`ggd list` has a :code:`--prefix` argument that is used to list installed data recipes in different
106+
conda environments. The :code:`--prefix` argument **can** be used for private recipes. This means you can list
107+
private data recipes that are installed in a different conda environment then the currently active environment
108+
you are using.
109+
110+
For more information about the :code:`ggd list` command see: :ref:`ggd-list`
111+
112+
D. Getting data files for private recipes
113+
114+
As with normal data recipes, you can use the :code:`ggd get-files` command to get data files created by private data recipes.
115+
116+
.. note::
117+
118+
:code:`ggd get-files` has a :code:`--prefix` argument that is used to get installed data files from different conda environments.
119+
This :code:`--prefix` argument **can** be used for private recipes. This means you can get installed data files from private recipes
120+
that are in a different conda environment then the currently active one.
121+
122+
For more information about the :code:`ggd get-files` command see: :ref:`ggd-get-files`
123+
124+
125+
126+
6. GGD commands the won't work with private recipes
127+
----------------------------------------------------
128+
129+
There are a few GGD commands that won't work with private recipes. Those include:
130+
131+
* :code:`ggd search`
132+
* :code:`ggd predict-path`
133+
* :code:`ggd uninstall`
134+
* :code:`ggd pkg-info`
135+
136+
137+
138+
7. Uninstalling a previously installed private data recipe
139+
-----------------------------------------------------------
140+
141+
To uninstall a private data recipe you will run :code:`ggd check-recipes <recipe name>` where *<recipe name>* represents the path to and name of the
142+
data recipe.
143+
144+
145+
.. note::
146+
147+
The :code:`-du` argument is absent from the :code:`ggd check-recipe` command.
148+
149+
150+
151+
Finally
152+
-------
153+
154+
GGD is a data management system built to mange and distribute publicly available scientific data. As this is the main purpose of GGD we encourage
155+
user to add ggd recipes to the public ggd repositories for the scientific community to use. GGD is built to help remove the inconsistencies with
156+
data processing and management that has plagued researchers for year. Therefore, GGD will continue to encourage public data access, management, and
157+
reproducibility. We understand that sometimes data cannot be shared publicly, but that the convenience and power of GGD can assist researchers and
158+
scientist during an analysis. The features on this page are here to assist if you want to use GGD but truly need to retain data privacy.
159+
However, GGD will continue to promote public data sharing whenever possible, and therefore, the GGD features will be maintained to support such
160+
goals.
161+
162+
163+
164+
165+
166+
167+

‎check-recipe.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ <h3>Navigation</h3>
7272
</ul>
7373
</li>
7474
<li class="toctree-l1"><a class="reference internal" href="contribute.html">Contribute</a></li>
75+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
7576
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
7677
</ul>
7778

‎contribute-recipe.html‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@
1717
<script src="_static/language_data.js"></script>
1818
<link rel="index" title="Index" href="genindex.html" />
1919
<link rel="search" title="Search" href="search.html" />
20-
<link rel="next" title="Available Data Packages" href="recipes.html" />
20+
<link rel="next" title="Private Recipes" href="private_recipes.html" />
2121
<link rel="prev" title="Setting up with Github" href="github-setup.html" />
2222

2323
<link href="https://fonts.googleapis.com/css?family=Lato|Raleway" rel="stylesheet">
@@ -64,6 +64,7 @@ <h3>Navigation</h3>
6464
<li class="toctree-l2 current"><a class="current reference internal" href="#">Contributing a ggd recipe</a></li>
6565
</ul>
6666
</li>
67+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
6768
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
6869
</ul>
6970

‎contribute.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ <h3>Navigation</h3>
6464
<li class="toctree-l2"><a class="reference internal" href="contribute-recipe.html">Contributing a ggd recipe</a></li>
6565
</ul>
6666
</li>
67+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
6768
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
6869
</ul>
6970

‎genindex.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,7 @@ <h3>Navigation</h3>
5858
<li class="toctree-l1"><a class="reference internal" href="using-ggd.html">Using GGD</a></li>
5959
<li class="toctree-l1"><a class="reference internal" href="GGD-CLI.html">GGD Commands</a></li>
6060
<li class="toctree-l1"><a class="reference internal" href="contribute.html">Contribute</a></li>
61+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
6162
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
6263
</ul>
6364

‎ggd-search.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -72,6 +72,7 @@ <h3>Navigation</h3>
7272
</ul>
7373
</li>
7474
<li class="toctree-l1"><a class="reference internal" href="contribute.html">Contribute</a></li>
75+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
7576
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
7677
</ul>
7778

‎github-setup.html‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,7 @@ <h3>Navigation</h3>
6464
<li class="toctree-l2"><a class="reference internal" href="contribute-recipe.html">Contributing a ggd recipe</a></li>
6565
</ul>
6666
</li>
67+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
6768
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
6869
</ul>
6970

‎index.html‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,7 @@ <h3>Navigation</h3>
5959
<li class="toctree-l1"><a class="reference internal" href="using-ggd.html">Using GGD</a></li>
6060
<li class="toctree-l1"><a class="reference internal" href="GGD-CLI.html">GGD Commands</a></li>
6161
<li class="toctree-l1"><a class="reference internal" href="contribute.html">Contribute</a></li>
62+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a></li>
6263
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
6364
</ul>
6465

@@ -514,6 +515,17 @@ <h2>Available Data Packages<a class="headerlink" href="#available-data-packages"
514515
<li class="toctree-l2"><a class="reference internal" href="contribute-recipe.html">Contributing a ggd recipe</a></li>
515516
</ul>
516517
</li>
518+
<li class="toctree-l1"><a class="reference internal" href="private_recipes.html">Private Recipes</a><ul>
519+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#create-a-private-github-repository-to-store-private-data-recipes">1. Create a private github repository to store private data recipes</a></li>
520+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#create-a-ggd-recipe">2. Create a ggd recipe</a></li>
521+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#check-and-install-the-data-recipe">3. Check and install the data recipe</a></li>
522+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#add-the-data-recipe-to-github">4. Add the data recipe to github</a></li>
523+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#how-to-access-installed-data-from-private-recipe">5. How to access installed data from private recipe</a></li>
524+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#ggd-commands-the-won-t-work-with-private-recipes">6. GGD commands the won’t work with private recipes</a></li>
525+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#uninstalling-a-previously-installed-private-data-recipe">7. Uninstalling a previously installed private data recipe</a></li>
526+
<li class="toctree-l2"><a class="reference internal" href="private_recipes.html#finally">Finally</a></li>
527+
</ul>
528+
</li>
517529
<li class="toctree-l1"><a class="reference internal" href="recipes.html">Available Data Packages</a></li>
518530
</ul>
519531
</div>

0 commit comments

Comments
 (0)