paulin/bonobo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge pull request #182 from hartym/master

Browse Source 
Update documentation.
This commit is contained in:
Romain Dorgueil
2017-10-08 13:25:57 +02:00
committed by GitHub
parent 736f73d5ce f40de8b386
commit 21a38686f6
24 changed files with 1681 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
config/conda.yml
View File

@ -10,3 +10,5 @@ dependencies:
- psutil ==5.2.2 - psutil ==5.2.2
- requests ==2.13.0 - requests ==2.13.0
- stevedore ==1.21.0 - stevedore ==1.21.0
# for examples
- pycountry ==17.9.23

24
docs/_templates/alabaster/__init__.py vendored Normal file
View File

@ -0,0 +1,24 @@
import os
from alabaster import _version as version
def get_path():
"""
Shortcut for users whose theme is next to their conf.py.
"""
# Theme directory is defined as our parent directory
return os.path.abspath(os.path.dirname(os.path.dirname(__file__)))
def update_context(app, pagename, templatename, context, doctree):
context['alabaster_version'] = version.__version__
def setup(app):
# add_html_theme is new in Sphinx 1.6+
if hasattr(app, 'add_html_theme'):
theme_path = os.path.abspath(os.path.dirname(__file__))
app.add_html_theme('alabaster', theme_path)
app.connect('html-page-context', update_context)
return {'version': version.__version__,
'parallel_read_safe': True}

2
docs/_templates/alabaster/_version.py vendored Normal file
View File

@ -0,0 +1,2 @@
__version_info__ = (0, 7, 10)
__version__ = '.'.join(map(str, __version_info__))

57
docs/_templates/alabaster/about.html vendored Normal file
View File

@ -0,0 +1,57 @@
{% if theme_logo %}
<p class="logo">
<a href="{{ pathto(master_doc) }}">
<img class="logo" src="{{ pathto('_static/' ~ theme_logo, 1) }}" alt="Logo"/>
{% if theme_logo_name|lower == 'true' %}
<h1 class="logo logo-name">{{ project }}</h1>
{% endif %}
</a>
</p>
{% else %}
<h1 class="logo"><a href="{{ pathto(master_doc) }}">{{ project }}</a></h1>
{% endif %}
{% if theme_description %}
<p class="blurb">{{ theme_description }}</p>
{% endif %}
{% if theme_github_user and theme_github_repo %}
{% if theme_github_button|lower == 'true' %}
<p>
<iframe src="https://ghbtns.com/github-btn.html?user={{ theme_github_user }}&repo={{ theme_github_repo }}&type={{ theme_github_type }}&count={{ theme_github_count }}&size=large&v=2"
allowtransparency="true" frameborder="0" scrolling="0" width="200px" height="35px"></iframe>
</p>
{% endif %}
{% endif %}
{% if theme_travis_button|lower != 'false' %}
{% if theme_travis_button|lower == 'true' %}
{% set path = theme_github_user + '/' + theme_github_repo %}
{% else %}
{% set path = theme_travis_button %}
{% endif %}
<p>
<a href="https://travis-ci.org/{{ path }}">
<img
alt="https://secure.travis-ci.org/{{ path }}.svg?branch={{ theme_badge_branch }}"
src="https://secure.travis-ci.org/{{ path }}.svg?branch={{ theme_badge_branch }}"
/>
</a>
</p>
{% endif %}
{% if theme_codecov_button|lower != 'false' %}
{% if theme_codecov_button|lower == 'true' %}
{% set path = theme_github_user + '/' + theme_github_repo %}
{% else %}
{% set path = theme_codecov_button %}
{% endif %}
<p>
<a href="https://codecov.io/github/{{ path }}">
<img
alt="https://codecov.io/github/{{ path }}/coverage.svg?branch={{ theme_badge_branch }}"
src="https://codecov.io/github/{{ path }}/coverage.svg?branch={{ theme_badge_branch }}"
/>
</a>
</p>
{% endif %}

9
docs/_templates/alabaster/donate.html vendored Normal file
View File

@ -0,0 +1,9 @@
{% if theme_gratipay_user or theme_gittip_user %}
<h3>Donate</h3>
<p>
Consider supporting the authors on <a href="https://www.gratipay.com/">Gratipay</a>:
<script data-gratipay-username="{{ theme_gratipay_user or theme_gittip_user }}"
data-gratipay-widget="button"
src="//gttp.co/v1.js"></script>
</p>
{% endif %}

82
docs/_templates/alabaster/layout.html vendored Normal file
View File

@ -0,0 +1,82 @@
{%- extends "basic/layout.html" %}
{%- block extrahead %}
{{ super() }}
<link rel="stylesheet" href="{{ pathto('_static/custom.css', 1) }}" type="text/css" />
{% if theme_touch_icon %}
<link rel="apple-touch-icon" href="{{ pathto('_static/' ~ theme_touch_icon, 1) }}" />
{% endif %}
{% if theme_canonical_url %}
<link rel="canonical" href="{{ theme_canonical_url }}{{ pagename }}.html"/>
{% endif %}
<meta name="viewport" content="width=device-width, initial-scale=0.9, maximum-scale=0.9" />
{% endblock %}
{# Disable base theme's top+bottom related navs; we have our own in sidebar #}
{%- block relbar1 %}{% endblock %}
{%- block relbar2 %}{% endblock %}
{# Nav should appear before content, not after #}
{%- block content %}
{%- if theme_fixed_sidebar|lower == 'true' %}
<div class="document">
{{ sidebar() }}
{%- block document %}
<div class="documentwrapper">
{%- if render_sidebar %}
<div class="bodywrapper">
{%- endif %}
<div class="body" role="main">
{% block body %} {% endblock %}
</div>
{%- if render_sidebar %}
</div>
{%- endif %}
</div>
{%- endblock %}
<div class="clearer"></div>
</div>
{%- else %}
{{ super() }}
{%- endif %}
{%- endblock %}
{%- block footer %}
<div class="footer">
{% if show_copyright %}&copy;{{ copyright }}.{% endif %}
{% if theme_show_powered_by|lower == 'true' %}
{% if show_copyright %}|{% endif %}
Powered by <a href="http://sphinx-doc.org/">Sphinx {{ sphinx_version }}</a>
&amp; <a href="https://github.com/bitprophet/alabaster">Alabaster {{ alabaster_version }}</a>
{% endif %}
{%- if show_source and has_source and sourcename %}
{% if show_copyright or theme_show_powered_by %}|{% endif %}
<a href="{{ pathto('_sources/' + sourcename, true)|e }}"
rel="nofollow">{{ _('Page source') }}</a>
{%- endif %}
</div>
{% if theme_github_banner|lower != 'false' %}
<a href="https://github.com/{{ theme_github_user }}/{{ theme_github_repo }}" class="github">
<img style="position: absolute; top: 0; right: 0; border: 0;" src="{{ pathto('_static/' ~ theme_github_banner, 1) if theme_github_banner|lower != 'true' else 'https://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png' }}" alt="Fork me on GitHub" class="github"/>
</a>
{% endif %}
{% if theme_analytics_id %}
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', '{{ theme_analytics_id }}']);
_gaq.push(['_setDomainName', 'none']);
_gaq.push(['_setAllowLinker', true]);
_gaq.push(['_trackPageview']);
(function() {
var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
})();
</script>
{% endif %}
{%- endblock %}

10
docs/_templates/alabaster/navigation.html vendored Normal file
View File

@ -0,0 +1,10 @@
<h3>{{ _('Navigation') }}</h3>
{{ toctree(includehidden=theme_sidebar_includehidden, collapse=theme_sidebar_collapse) }}
{% if theme_extra_nav_links %}
<hr />
<ul>
{% for text, uri in theme_extra_nav_links.items() %}
<li class="toctree-l1"><a href="{{ uri }}">{{ text }}</a></li>
{% endfor %}
</ul>
{% endif %}

21
docs/_templates/alabaster/relations.html vendored Normal file
View File

@ -0,0 +1,21 @@
<div class="relations">
<h3>Related Topics</h3>
<ul>
<li><a href="{{ pathto(master_doc) }}">Documentation overview</a><ul>
{%- for parent in parents %}
<li><a href="{{ parent.link|e }}">{{ parent.title }}</a><ul>
{%- endfor %}
{%- if prev %}
<li>Previous: <a href="{{ prev.link|e }}" title="{{ _('previous chapter')
}}">{{ prev.title }}</a></li>
{%- endif %}
{%- if next %}
<li>Next: <a href="{{ next.link|e }}" title="{{ _('next chapter')
}}">{{ next.title }}</a></li>
{%- endif %}
{%- for parent in parents %}
</ul></li>
{%- endfor %}
</ul></li>
</ul>
</div>

706
docs/_templates/alabaster/static/alabaster.css_t vendored Normal file
View File

@ -0,0 +1,706 @@
{% set theme_body_bg = theme_body_bg or theme_base_bg %}
{% set theme_code_highlight_bg = theme_code_highlight_bg or theme_body_bg %}
{% set theme_sidebar_header = theme_sidebar_header or theme_gray_1 %}
{% set theme_sidebar_link = theme_sidebar_link or theme_gray_1 %}
{% set theme_anchor_hover_fg = theme_anchor_hover_fg or theme_gray_1 %}
{% set theme_footnote_border = theme_footnote_border or theme_gray_2 %}
{% set theme_pre_bg = theme_pre_bg or theme_gray_2 %}
{# set up admonition styling #}
{# - basic level #}
{% set theme_admonition_xref_bg = theme_admonition_xref_bg or theme_xref_bg %}
{% set theme_admonition_bg = theme_admonition_bg or theme_gray_2 %}
{% set theme_note_bg = theme_note_bg or theme_gray_2 %}
{% set theme_seealso_bg = theme_seealso_bg or theme_gray_2 %}
{# - critical level #}
{% set theme_danger_bg = theme_danger_bg or theme_pink_1 %}
{% set theme_danger_border = theme_danger_border or theme_pink_2 %}
{% set theme_danger_shadow = theme_danger_shadow or theme_pink_3 %}
{% set theme_error_bg = theme_error_bg or theme_pink_1 %}
{% set theme_error_border = theme_error_border or theme_pink_2 %}
{% set theme_error_shadow = theme_error_shadow or theme_pink_3 %}
{# - warning level #}
{% set theme_caution_bg = theme_caution_bg or theme_pink_1 %}
{% set theme_caution_border = theme_caution_border or theme_pink_2 %}
{% set theme_attention_bg = theme_attention_bg or theme_pink_1 %}
{% set theme_attention_border = theme_attention_border or theme_pink_2 %}
{% set theme_warn_bg = theme_warn_bg or theme_pink_1 %}
{% set theme_warn_border = theme_warn_border or theme_pink_2 %}
{# - normal level #}
{% set theme_important_bg = theme_important_bg or theme_gray_2 %}
{% set theme_tip_bg = theme_tip_bg or theme_gray_2 %}
{% set theme_hint_bg = theme_hint_bg or theme_gray_2 %}
{# /set up admonition styling #}
{% set theme_shadow = theme_shadow or theme_gray_2 %}
{% set theme_topic_bg = theme_topic_bg or theme_gray_2 %}
{% set theme_narrow_sidebar_link = theme_narrow_sidebar_link or theme_gray_3 %}
{% set theme_sidebar_hr = theme_sidebar_hr or theme_gray_3 %}
@import url("basic.css");
/* -- page layout ----------------------------------------------------------- */
body {
font-family: {{ theme_font_family }};
font-size: {{ theme_font_size }};
background-color: {{ theme_base_bg }};
color: {{ theme_base_text }};
margin: 0;
padding: 0;
}
div.document {
width: {{ theme_page_width }};
margin: 30px auto 0 auto;
}
div.documentwrapper {
float: left;
width: 100%;
}
div.bodywrapper {
margin: 0 0 0 {{ theme_sidebar_width }};
}
div.sphinxsidebar {
width: {{ theme_sidebar_width }};
font-size: 14px;
line-height: 1.5;
}
hr {
border: 1px solid {{ theme_hr_border }};
}
div.body {
background-color: {{ theme_body_bg }};
color: {{ theme_body_text }};
padding: 0 30px 0 30px;
}
div.body > .section {
text-align: {{ theme_body_text_align }};
}
div.footer {
width: {{ theme_page_width }};
margin: 20px auto 30px auto;
font-size: 14px;
color: {{ theme_footer_text }};
text-align: right;
}
div.footer a {
color: {{ theme_footer_text }};
}
p.caption {
font-family: {{ theme_caption_font_family }};
font-size: {{ theme_caption_font_size }};
}
{% if theme_show_related|lower == 'false' %}
div.relations {
display: none;
}
{% endif %}
div.sphinxsidebar a {
color: {{ theme_sidebar_link }};
text-decoration: none;
border-bottom: 1px dotted {{ theme_sidebar_link_underscore }};
}
div.sphinxsidebar a:hover {
border-bottom: 1px solid {{ theme_sidebar_link_underscore }};
}
div.sphinxsidebarwrapper {
padding: 18px 10px;
}
div.sphinxsidebarwrapper p.logo {
padding: 0;
margin: -10px 0 0 0px;
text-align: center;
}
div.sphinxsidebarwrapper h1.logo {
margin-top: -10px;
text-align: center;
margin-bottom: 5px;
text-align: {{ theme_logo_text_align }};
}
div.sphinxsidebarwrapper h1.logo-name {
margin-top: 0px;
}
div.sphinxsidebarwrapper p.blurb {
margin-top: 0;
font-style: {{ theme_description_font_style }};
}
div.sphinxsidebar h3,
div.sphinxsidebar h4 {
font-family: {{ theme_head_font_family }};
color: {{ theme_sidebar_header }};
font-size: 24px;
font-weight: normal;
margin: 0 0 5px 0;
padding: 0;
}
div.sphinxsidebar h4 {
font-size: 20px;
}
div.sphinxsidebar h3 a {
color: {{ theme_sidebar_link }};
}
div.sphinxsidebar p.logo a,
div.sphinxsidebar h3 a,
div.sphinxsidebar p.logo a:hover,
div.sphinxsidebar h3 a:hover {
border: none;
}
div.sphinxsidebar p {
color: {{ theme_sidebar_text }};
margin: 10px 0;
}
div.sphinxsidebar ul {
margin: 10px 0;
padding: 0;
color: {{ theme_sidebar_list }};
}
div.sphinxsidebar ul li.toctree-l1 > a {
font-size: 120%;
}
div.sphinxsidebar ul li.toctree-l2 > a {
font-size: 110%;
}
div.sphinxsidebar input {
border: 1px solid {{ theme_sidebar_search_button }};
font-family: {{ theme_font_family }};
font-size: 1em;
}
div.sphinxsidebar hr {
border: none;
height: 1px;
color: {{ theme_sidebar_hr }};
background: {{ theme_sidebar_hr }};
text-align: left;
margin-left: 0;
width: 50%;
}
/* -- body styles ----------------------------------------------------------- */
a {
color: {{ theme_link }};
text-decoration: underline;
}
a:hover {
color: {{ theme_link_hover }};
text-decoration: underline;
}
div.body h1,
div.body h2,
div.body h3,
div.body h4,
div.body h5,
div.body h6 {
font-family: {{ theme_head_font_family }};
font-weight: normal;
margin: 30px 0px 10px 0px;
padding: 0;
}
div.body h1 { margin-top: 0; padding-top: 0; font-size: 240%; }
div.body h2 { font-size: 180%; }
div.body h3 { font-size: 150%; }
div.body h4 { font-size: 130%; }
div.body h5 { font-size: 100%; }
div.body h6 { font-size: 100%; }
a.headerlink {
color: {{ theme_anchor }};
padding: 0 4px;
text-decoration: none;
}
a.headerlink:hover {
color: {{ theme_anchor_hover_fg }};
background: {{ theme_anchor_hover_bg }};
}
div.body p, div.body dd, div.body li {
line-height: 1.4em;
}
div.admonition {
margin: 20px 0px;
padding: 10px 30px;
background-color: {{ theme_admonition_bg }};
border: 1px solid {{ theme_admonition_border }};
}
div.admonition tt.xref, div.admonition code.xref, div.admonition a tt {
background-color: {{ theme_admonition_xref_bg }};
border-bottom: 1px solid {{ theme_admonition_xref_border }};
}
div.admonition p.admonition-title {
font-family: {{ theme_head_font_family }};
font-weight: normal;
font-size: 24px;
margin: 0 0 10px 0;
padding: 0;
line-height: 1;
}
div.admonition p.last {
margin-bottom: 0;
}
div.highlight {
background-color: {{ theme_code_highlight_bg }};
}
dt:target, .highlight {
background: {{ theme_highlight_bg }};
}
div.warning {
background-color: {{ theme_warn_bg }};
border: 1px solid {{ theme_warn_border }};
}
div.danger {
background-color: {{ theme_danger_bg }};
border: 1px solid {{ theme_danger_border }};
-moz-box-shadow: 2px 2px 4px {{ theme_danger_shadow }};
-webkit-box-shadow: 2px 2px 4px {{ theme_danger_shadow }};
box-shadow: 2px 2px 4px {{ theme_danger_shadow }};
}
div.error {
background-color: {{ theme_error_bg }};
border: 1px solid {{ theme_error_border }};
-moz-box-shadow: 2px 2px 4px {{ theme_error_shadow }};
-webkit-box-shadow: 2px 2px 4px {{ theme_error_shadow }};
box-shadow: 2px 2px 4px {{ theme_error_shadow }};
}
div.caution {
background-color: {{ theme_caution_bg }};
border: 1px solid {{ theme_caution_border }};
}
div.attention {
background-color: {{ theme_attention_bg }};
border: 1px solid {{ theme_attention_border }};
}
div.important {
background-color: {{ theme_important_bg }};
border: 1px solid {{ theme_important_border }};
}
div.note {
background-color: {{ theme_note_bg }};
border: 1px solid {{ theme_note_border }};
}
div.tip {
background-color: {{ theme_tip_bg }};
border: 1px solid {{ theme_tip_border }};
}
div.hint {
background-color: {{ theme_hint_bg }};
border: 1px solid {{ theme_hint_border }};
}
div.seealso {
background-color: {{ theme_seealso_bg }};
border: 1px solid {{ theme_seealso_border }};
}
div.topic {
background-color: {{ theme_topic_bg }};
}
p.admonition-title {
display: inline;
}
p.admonition-title:after {
content: ":";
}
pre, tt, code {
font-family: {{theme_code_font_family}};
font-size: {{ theme_code_font_size }};
}
.hll {
background-color: {{theme_code_highlight}};
margin: 0 -12px;
padding: 0 12px;
display: block;
}
img.screenshot {
}
tt.descname, tt.descclassname, code.descname, code.descclassname {
font-size: 0.95em;
}
tt.descname, code.descname {
padding-right: 0.08em;
}
img.screenshot {
-moz-box-shadow: 2px 2px 4px {{ theme_shadow }};
-webkit-box-shadow: 2px 2px 4px {{ theme_shadow }};
box-shadow: 2px 2px 4px {{ theme_shadow }};
}
table.docutils {
border: 1px solid {{ theme_table_border }};
-moz-box-shadow: 2px 2px 4px {{ theme_shadow }};
-webkit-box-shadow: 2px 2px 4px {{ theme_shadow }};
box-shadow: 2px 2px 4px {{ theme_shadow }};
}
table.docutils td, table.docutils th {
border: 1px solid {{ theme_table_border }};
padding: 0.25em 0.7em;
}
table.field-list, table.footnote {
border: none;
-moz-box-shadow: none;
-webkit-box-shadow: none;
box-shadow: none;
}
table.footnote {
margin: 15px 0;
width: 100%;
border: 1px solid {{ theme_footnote_border }};
background: {{ theme_footnote_bg }};
font-size: 0.9em;
}
table.footnote + table.footnote {
margin-top: -15px;
border-top: none;
}
table.field-list th {
padding: 0 0.8em 0 0;
}
table.field-list td {
padding: 0;
}
table.field-list p {
margin-bottom: 0.8em;
}
/* Cloned from
* https://github.com/sphinx-doc/sphinx/commit/ef60dbfce09286b20b7385333d63a60321784e68
*/
.field-name {
-moz-hyphens: manual;
-ms-hyphens: manual;
-webkit-hyphens: manual;
hyphens: manual;
}
table.footnote td.label {
width: .1px;
padding: 0.3em 0 0.3em 0.5em;
}
table.footnote td {
padding: 0.3em 0.5em;
}
dl {
margin: 0;
padding: 0;
}
dl dd {
margin-left: 30px;
}
blockquote {
margin: 0 0 0 30px;
padding: 0;
}
ul, ol {
/* Matches the 30px from the narrow-screen "li > ul" selector below */
margin: 10px 0 10px 30px;
padding: 0;
}
pre {
background: {{ theme_pre_bg }};
padding: 7px 30px;
margin: 15px 0px;
line-height: 1.3em;
}
div.viewcode-block:target {
background: {{ theme_viewcode_target_bg }};
}
dl pre, blockquote pre, li pre {
margin-left: 0;
padding-left: 30px;
}
tt, code {
background-color: {{ theme_code_bg }};
color: {{ theme_code_text }};
/* padding: 1px 2px; */
}
tt.xref, code.xref, a tt {
background-color: {{ theme_xref_bg }};
border-bottom: 1px solid {{ theme_xref_border }};
}
a.reference {
text-decoration: none;
border-bottom: 1px dotted {{ theme_link }};
}
/* Don't put an underline on images */
a.image-reference, a.image-reference:hover {
border-bottom: none;
}
a.reference:hover {
border-bottom: 1px solid {{ theme_link_hover }};
}
a.footnote-reference {
text-decoration: none;
font-size: 0.7em;
vertical-align: top;
border-bottom: 1px dotted {{ theme_link }};
}
a.footnote-reference:hover {
border-bottom: 1px solid {{ theme_link_hover }};
}
a:hover tt, a:hover code {
background: {{ theme_code_hover }};
}
@media screen and (max-width: 870px) {
div.sphinxsidebar {
display: none;
}
div.document {
width: 100%;
}
div.documentwrapper {
margin-left: 0;
margin-top: 0;
margin-right: 0;
margin-bottom: 0;
}
div.bodywrapper {
margin-top: 0;
margin-right: 0;
margin-bottom: 0;
margin-left: 0;
}
ul {
margin-left: 0;
}
li > ul {
/* Matches the 30px from the "ul, ol" selector above */
margin-left: 30px;
}
.document {
width: auto;
}
.footer {
width: auto;
}
.bodywrapper {
margin: 0;
}
.footer {
width: auto;
}
.github {
display: none;
}
}
@media screen and (max-width: 875px) {
body {
margin: 0;
padding: 20px 30px;
}
div.documentwrapper {
float: none;
background: {{ theme_base_bg }};
}
div.sphinxsidebar {
display: block;
float: none;
width: 102.5%;
{%- if theme_fixed_sidebar|lower == 'true' %}
margin: -20px -30px 20px -30px;
{%- else %}
margin: 50px -30px -20px -30px;
{%- endif %}
padding: 10px 20px;
background: {{ theme_narrow_sidebar_bg }};
color: {{ theme_narrow_sidebar_fg }};
}
div.sphinxsidebar h3, div.sphinxsidebar h4, div.sphinxsidebar p,
div.sphinxsidebar h3 a {
color: {{ theme_base_bg }};
}
div.sphinxsidebar a {
color: {{ theme_narrow_sidebar_link }};
}
div.sphinxsidebar p.logo {
display: none;
}
div.document {
width: 100%;
margin: 0;
}
div.footer {
display: none;
}
div.bodywrapper {
margin: 0;
}
div.body {
min-height: 0;
padding: 0;
}
.rtd_doc_footer {
display: none;
}
.document {
width: auto;
}
.footer {
width: auto;
}
.footer {
width: auto;
}
.github {
display: none;
}
}
{%- if theme_fixed_sidebar|lower == 'true' %}
@media screen and (min-width: 876px) {
div.sphinxsidebar {
position: fixed;
margin-left: 0;
}
}
{%- endif %}
/* misc. */
.revsys-inline {
display: none!important;
}
/* Make nested-list/multi-paragraph items look better in Releases changelog
* pages. Without this, docutils' magical list fuckery causes inconsistent
* formatting between different release sub-lists.
*/
div#changelog > div.section > ul > li > p:only-child {
margin-bottom: 0;
}
/* Hide fugly table cell borders in ..bibliography:: directive output */
table.docutils.citation, table.docutils.citation td, table.docutils.citation th {
border: none;
/* Below needed in some edge cases; if not applied, bottom shadows appear */
-moz-box-shadow: none;
-webkit-box-shadow: none;
box-shadow: none;
}

1
docs/_templates/alabaster/static/custom.css vendored Normal file
View File

@ -0,0 +1 @@
/* This file intentionally left blank. */

88
docs/_templates/alabaster/support.py vendored Normal file
View File

@ -0,0 +1,88 @@
# flake8: noqa
from pygments.style import Style
from pygments.token import Keyword, Name, Comment, String, Error, \
Number, Operator, Generic, Whitespace, Punctuation, Other, Literal
# Originally based on FlaskyStyle which was based on 'tango'.
class Alabaster(Style):
background_color = "#f8f8f8" # doesn't seem to override CSS 'pre' styling?
default_style = ""
styles = {
# No corresponding class for the following:
#Text: "", # class: ''
Whitespace: "underline #f8f8f8", # class: 'w'
Error: "#a40000 border:#ef2929", # class: 'err'
Other: "#000000", # class 'x'
Comment: "italic #8f5902", # class: 'c'
Comment.Preproc: "noitalic", # class: 'cp'
Keyword: "bold #004461", # class: 'k'
Keyword.Constant: "bold #004461", # class: 'kc'
Keyword.Declaration: "bold #004461", # class: 'kd'
Keyword.Namespace: "bold #004461", # class: 'kn'
Keyword.Pseudo: "bold #004461", # class: 'kp'
Keyword.Reserved: "bold #004461", # class: 'kr'
Keyword.Type: "bold #004461", # class: 'kt'
Operator: "#582800", # class: 'o'
Operator.Word: "bold #004461", # class: 'ow' - like keywords
Punctuation: "bold #000000", # class: 'p'
# because special names such as Name.Class, Name.Function, etc.
# are not recognized as such later in the parsing, we choose them
# to look the same as ordinary variables.
Name: "#000000", # class: 'n'
Name.Attribute: "#c4a000", # class: 'na' - to be revised
Name.Builtin: "#004461", # class: 'nb'
Name.Builtin.Pseudo: "#3465a4", # class: 'bp'
Name.Class: "#000000", # class: 'nc' - to be revised
Name.Constant: "#000000", # class: 'no' - to be revised
Name.Decorator: "#888", # class: 'nd' - to be revised
Name.Entity: "#ce5c00", # class: 'ni'
Name.Exception: "bold #cc0000", # class: 'ne'
Name.Function: "#000000", # class: 'nf'
Name.Property: "#000000", # class: 'py'
Name.Label: "#f57900", # class: 'nl'
Name.Namespace: "#000000", # class: 'nn' - to be revised
Name.Other: "#000000", # class: 'nx'
Name.Tag: "bold #004461", # class: 'nt' - like a keyword
Name.Variable: "#000000", # class: 'nv' - to be revised
Name.Variable.Class: "#000000", # class: 'vc' - to be revised
Name.Variable.Global: "#000000", # class: 'vg' - to be revised
Name.Variable.Instance: "#000000", # class: 'vi' - to be revised
Number: "#990000", # class: 'm'
Literal: "#000000", # class: 'l'
Literal.Date: "#000000", # class: 'ld'
String: "#4e9a06", # class: 's'
String.Backtick: "#4e9a06", # class: 'sb'
String.Char: "#4e9a06", # class: 'sc'
String.Doc: "italic #8f5902", # class: 'sd' - like a comment
String.Double: "#4e9a06", # class: 's2'
String.Escape: "#4e9a06", # class: 'se'
String.Heredoc: "#4e9a06", # class: 'sh'
String.Interpol: "#4e9a06", # class: 'si'
String.Other: "#4e9a06", # class: 'sx'
String.Regex: "#4e9a06", # class: 'sr'
String.Single: "#4e9a06", # class: 's1'
String.Symbol: "#4e9a06", # class: 'ss'
Generic: "#000000", # class: 'g'
Generic.Deleted: "#a40000", # class: 'gd'
Generic.Emph: "italic #000000", # class: 'ge'
Generic.Error: "#ef2929", # class: 'gr'
Generic.Heading: "bold #000080", # class: 'gh'
Generic.Inserted: "#00A000", # class: 'gi'
Generic.Output: "#888", # class: 'go'
Generic.Prompt: "#745334", # class: 'gp'
Generic.Strong: "bold #000000", # class: 'gs'
Generic.Subheading: "bold #800080", # class: 'gu'
Generic.Traceback: "bold #a40000", # class: 'gt'
}

122
docs/_templates/alabaster/theme.conf vendored Normal file
View File

@ -0,0 +1,122 @@
[theme]
inherit = basic
stylesheet = alabaster.css
pygments_style = alabaster.support.Alabaster
[options]
logo =
logo_name = false
logo_text_align = left
description =
description_font_style = normal
github_user =
github_repo =
github_button = true
github_banner = false
github_type = watch
github_count = true
badge_branch = master
travis_button = false
codecov_button = false
gratipay_user =
gittip_user =
analytics_id =
touch_icon =
canonical_url =
extra_nav_links =
sidebar_includehidden = true
sidebar_collapse = true
show_powered_by = true
show_related = false
gray_1 = #444
gray_2 = #EEE
gray_3 = #AAA
pink_1 = #FCC
pink_2 = #FAA
pink_3 = #D52C2C
base_bg = #fff
base_text = #000
hr_border = #B1B4B6
body_bg =
body_text = #3E4349
body_text_align = left
footer_text = #888
link = #004B6B
link_hover = #6D4100
sidebar_header =
sidebar_text = #555
sidebar_link =
sidebar_link_underscore = #999
sidebar_search_button = #CCC
sidebar_list = #000
sidebar_hr =
anchor = #DDD
anchor_hover_fg =
anchor_hover_bg = #EAEAEA
table_border = #888
shadow =
# Admonition options
## basic level
admonition_bg =
admonition_border = #CCC
note_bg =
note_border = #CCC
seealso_bg =
seealso_border = #CCC
## critical level
danger_bg =
danger_border =
danger_shadow =
error_bg =
error_border =
error_shadow =
## normal level
tip_bg =
tip_border = #CCC
hint_bg =
hint_border = #CCC
important_bg =
important_border = #CCC
## warning level
caution_bg =
caution_border =
attention_bg =
attention_border =
warn_bg =
warn_border =
topic_bg =
code_highlight_bg =
highlight_bg = #FAF3E8
xref_border = #fff
xref_bg = #FBFBFB
admonition_xref_border = #fafafa
admonition_xref_bg =
footnote_bg = #FDFDFD
footnote_border =
pre_bg =
narrow_sidebar_bg = #333
narrow_sidebar_fg = #FFF
narrow_sidebar_link =
font_size = 17px
caption_font_size = inherit
viewcode_target_bg = #ffd
code_bg = #ecf0f3
code_text = #222
code_hover = #EEE
code_font_size = 0.9em
code_font_family = 'Consolas', 'Menlo', 'Deja Vu Sans Mono', 'Bitstream Vera Sans Mono', monospace
font_family = 'goudy old style', 'minion pro', 'bell mt', Georgia, 'Hiragino Mincho Pro', serif
head_font_family = 'Garamond', 'Georgia', serif
caption_font_family = inherit
code_highlight = #FFC
page_width = 940px
sidebar_width = 220px
fixed_sidebar = false

62
docs/_templates/base.html vendored Normal file
View File

@ -0,0 +1,62 @@
{%- extends "alabaster/layout.html" %}
{%- block extrahead %}
{{ super() }}
<style>
div.related {
width: 940px;
margin: 30px auto 0 auto;
}
@media screen and (max-width: 875px) {
div.related {
visibility: hidden;
display: none;
}
}
</style>
{% endblock %}
{%- block footer %}
{{ relbar() }}
<div class="footer">
{% if show_copyright %}&copy;{{ copyright }}.{% endif %}
{% if theme_show_powered_by|lower == 'true' %}
{% if show_copyright %}|{% endif %}
Powered by <a href="http://sphinx-doc.org/">Sphinx {{ sphinx_version }}</a>
&amp; <a href="https://github.com/bitprophet/alabaster">Alabaster {{ alabaster_version }}</a>
{% endif %}
{%- if show_source and has_source and sourcename %}
{% if show_copyright or theme_show_powered_by %}|{% endif %}
<a href="{{ pathto('_sources/' + sourcename, true)|e }}"
rel="nofollow">{{ _('Page source') }}</a>
{%- endif %}
</div>
{% if theme_github_banner|lower != 'false' %}
<a href="https://github.com/{{ theme_github_user }}/{{ theme_github_repo }}" class="github">
<img style="position: absolute; top: 0; right: 0; border: 0;"
src="{{ pathto('_static/' ~ theme_github_banner, 1) if theme_github_banner|lower != 'true' else 'https://s3.amazonaws.com/github/ribbons/forkme_right_darkblue_121621.png' }}"
alt="Fork me on GitHub" class="github"/>
</a>
{% endif %}
{% if theme_analytics_id %}
<script type="text/javascript">
var _gaq = _gaq || [];
_gaq.push(['_setAccount', '{{ theme_analytics_id }}']);
_gaq.push(['_setDomainName', 'none']);
_gaq.push(['_setAllowLinker', true]);
_gaq.push(['_trackPageview']);
(function () {
var ga = document.createElement('script');
ga.type = 'text/javascript';
ga.async = true;
ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
var s = document.getElementsByTagName('script')[0];
s.parentNode.insertBefore(ga, s);
})();
</script>
{% endif %}
{%- endblock %}

9
docs/_templates/index.html vendored
View File

@ -1,7 +1,8 @@
{% extends "layout.html" %} {% extends "base.html" %}
{% set title = _('Bonobo — Data processing for humans') %}
{% block body %}
{% set title = _('Bonobo — Data processing for humans') %}
{% block body %}
<h1 style="text-align: center"> <h1 style="text-align: center">
<img class="logo" src="{{ pathto('_static/bonobo.png', 1) }}" title="Bonobo" alt="Bonobo" <img class="logo" src="{{ pathto('_static/bonobo.png', 1) }}" title="Bonobo" alt="Bonobo"
style=" width: 128px; height: 128px;"/> style=" width: 128px; height: 128px;"/>
@ -9,7 +10,7 @@
<p> <p>
{% trans %} {% trans %}
<b>Bonobo</b> is an Extract Transform Load framework for the Python (3.5+) language. <b>Bonobo</b> is an <b>Extract Transform Load</b> (or ETL) framework for the <b>Python (3.5+)</b> language.
{% endtrans %} {% endtrans %}
</p> </p>

7
docs/_templates/layout.html vendored Normal file
View File

@ -0,0 +1,7 @@
{%- extends "base.html" %}
{%- block content %}
{{ relbar() }}
{{ super() }}
{%- endblock %}

6
docs/_templates/sidebarlogo.html vendored
View File

@ -1,10 +1,10 @@
<a href="{{ pathto(master_doc) }}" style="border: none"> <a href="{{ pathto(master_doc) }}" style="border: none">
<h1 style="text-align: center; margin-top: 0;"> <h1 style="text-align: center; margin: 0;">
<img class="logo" src="{{ pathto('_static/bonobo.png', 1) }}" title="Bonobo" style="width: 48px; height: 48px; vertical-align: bottom"/> <img class="logo" src="{{ pathto('_static/bonobo.png', 1) }}" title="Bonobo" style="width: 48px; height: 48px; vertical-align: bottom"/>
Bonobo Bonobo
</h1> </h1>
</a> </a>
<p> <p style="text-align: center">
Data processing for human beings. Data processing for humans.
</p> </p>

6
docs/conf.py
View File

@ -75,9 +75,9 @@ html_theme = 'alabaster'
html_theme_options = { html_theme_options = {
'github_user': 'python-bonobo', 'github_user': 'python-bonobo',
'github_repo': 'bonobo', 'github_repo': 'bonobo',
'github_button': True, 'github_button': 'true',
'show_powered_by': False, 'show_powered_by': 'false',
'show_related': True, 'show_related': 'true',
} }
html_sidebars = { html_sidebars = {

3
docs/genindex.rst Normal file
View File

@ -0,0 +1,3 @@
Full Index
==========

206
docs/guide/graphs.rst
View File

@ -1,11 +1,211 @@
Graphs Graphs
====== ======
Writing graphs Graphs are the glue that ties transformations together. It's the only data-structure bonobo can execute directly. Graphs
:::::::::::::: must be acyclic, and can contain as much nodes as your system can handle. Although this number can be rather high in
theory, extreme practical cases usually do not exceed hundreds of nodes (and this is already extreme, really).
Debugging graphs
Definitions
:::::::::::
Graph
A directed acyclic graph of transformations, that Bonobo can inspect and execute.
Node
A transformation within a graph. The transformations are stateless, and have no idea whether or not they are
included in a graph, multiple graph, or not at all.
Creating a graph
:::::::::::::::: ::::::::::::::::
Graphs should be instances of :class:`bonobo.Graph`. The :func:`bonobo.Graph.add_chain` method can take as many
positional parameters as you want.
.. code-block:: python
import bonobo
graph = bonobo.Graph()
graph.add_chain(a, b, c)
Resulting graph:
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
BEGIN -> "a" -> "b" -> "c";
}
Non-linear graphs
:::::::::::::::::
Divergences / forks
-------------------
To create two or more divergent data streams ("fork"), you should specify `_input` kwarg to `add_chain`.
.. code-block:: python
import bonobo
graph = bonobo.Graph()
graph.add_chain(a, b, c)
graph.add_chain(f, g, _input=b)
Resulting graph:
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
BEGIN -> "a" -> "b" -> "c";
"b" -> "f" -> "g";
}
.. note:: Both branch will receive the same data, at the same time.
Convergences / merges
---------------------
To merge two data streams ("merge"), you can use the `_output` kwarg to `add_chain`, or use named nodes (see below).
.. code-block:: python
import bonobo
graph = bonobo.Graph()
# Here we mark _input to None, so normalize won't get the "begin" impulsion.
graph.add_chain(normalize, store, _input=None)
# Add two different chains
graph.add_chain(a, b, _output=normalize)
graph.add_chain(f, g, _output=normalize)
Resulting graph:
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
BEGIN -> "a" -> "b" -> "normalize";
BEGIN2 [shape="point"];
BEGIN2 -> "f" -> "g" -> "normalize";
"normalize" -> "store"
}
.. note::
This is not a "join" or "cartesian product". Any data that comes from `b` or `g` will go through `normalize`, one at
a time. Think of the graph edges as data flow pipes.
Named nodes
:::::::::::
Using above code to create convergences can lead to hard to read code, because you have to define the "target" stream
before the streams that logically goes to the beginning of the transformation graph. To overcome that, one can use
"named" nodes:
graph.add_chain(x, y, z, _name='zed')
graph.add_chain(f, g, h, _input='zed')
.. code-block:: python
import bonobo
graph = bonobo.Graph()
# Add two different chains
graph.add_chain(a, b, _output="load")
graph.add_chain(f, g, _output="load")
# Here we mark _input to None, so normalize won't get the "begin" impulsion.
graph.add_chain(normalize, store, _input=None, _name="load")
Resulting graph:
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
BEGIN -> "a" -> "b" -> "normalize (load)";
BEGIN2 [shape="point"];
BEGIN2 -> "f" -> "g" -> "normalize (load)";
"normalize (load)" -> "store"
}
Inspecting graphs
:::::::::::::::::
Bonobo is bundled with an "inspector", that can use graphviz to let you visualize your graphs.
Read `How to inspect and visualize your graph <https://www.bonobo-project.org/how-to/inspect-an-etl-jobs-graph>`_.
Executing graphs Executing graphs
:::::::::::::::: ::::::::::::::::
There are two options to execute a graph (which have a similar result, but are targeting different use cases).
* You can use the bonobo command line interface, which is the highest level interface.
* You can use the python API, which is lower level but allows to use bonobo from within your own code (for example, a
django management command).
Executing a graph with the command line interface
-------------------------------------------------
If there is no good reason not to, you should use `bonobo run ...` to run transformation graphs found in your python
source code files.
.. code-block:: shell-session
$ bonobo run file.py
You can also run a python module:
.. code-block:: shell-session
$ bonobo run -m my.own.etlmod
In each case, bonobo's CLI will look for an instance of :class:`bonobo.Graph` in your file/module, create the plumbery
needed to execute it, and run it.
If you're in an interactive terminal context, it will use :class:`bonobo.ext.console.ConsoleOutputPlugin` for display.
If you're in a jupyter notebook context, it will (try to) use :class:`bonobo.ext.jupyter.JupyterOutputPlugin`.
Executing a graph using the internal API
----------------------------------------
To integrate bonobo executions in any other python code, you should use :func:`bonobo.run`. It behaves very similar to
the CLI, and reading the source you should be able to figure out its usage quite easily.

5
docs/guide/index.rst
View File

@ -1,13 +1,14 @@
Guides Guides
====== ======
Here are a few guides and best practices to work with bonobo. This section will guide you through your journey with Bonobo ETL.
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
graphs introduction
transformations transformations
graphs
services services
environment environment
purity purity

106
docs/guide/introduction.rst Normal file
View File

@ -0,0 +1,106 @@
Introduction
============
The first thing you need to understand before you use Bonobo, or not, is what it does and what it does not, so you can
understand if it could be a good fit for your use cases.
How it works?
:::::::::::::
**Bonobo** is an **Extract Transform Load** framework aimed at coders, hackers, or any other person who's at ease with
terminals and source code files.
It is a **data streaming** solution, that treat datasets as ordered collections of independant rows, allowing to process
them "first in, first out" using a set of transformations organized together in a directed graph.
Let's take a few examples:
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
END [shape="none" label="..."];
BEGIN -> "A" -> "B" -> "C" -> "END";
}
One of the simplest, by the book, cases, is an extractor sending to a transformation, itself sending to a loader.
Bonobo will send an "impulsion" to all transformations linked to the little black dot on the left, here `A`.
`A`'s main topic will be to extract data from somewhere (a file, an endpoint, a database...) and generate some output.
As soon as the first row of `A`'s output is available, Bonobo will start asking `B` to process it. As soon as the first
row of `B`'s output is available, Bonobo will start asking `C` to process it.
While `B` and `C` are processing, `A` continues to generate data.
This approach can be efficient, depending on your requirements, because you may rely on a lot of services that may be
long to answer or unreliable, and you don't have to handle optimizations, parallelism or retry logic by yourself.
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
END [shape="none" label="..."];
END2 [shape="none" label="..."];
BEGIN -> "A" -> "B" -> "END";
"A" -> "C" -> "END2";
}
In this case, any output row of `A`, will be **sent to both** `B` and `C` simultaneously. Again, `A` will continue its
processing while `B` and `C` are working.
.. graphviz::
digraph {
rankdir = LR;
stylesheet = "../_static/graphs.css";
BEGIN [shape="point"];
BEGIN2 [shape="point"];
END [shape="none" label="..."];
BEGIN -> "A" -> "C" -> "END";
BEGIN2 -> "B" -> "C";
}
What is it not?
:::::::::::::::
**Bonobo** is not:
* A data science, or statistical analysis tool, which need to treat the dataset as a whole and not as a collection of
independant rows. If this is your need, you probably want to look at `pandas <https://pandas.pydata.org/>`_.
* A workflow or scheduling solution for independant data-engineering tasks. If you're looking to manage your sets of
data processing tasks as a whole, you probably want to look at `airflow <https://airflow.incubator.apache.org/>`_.
Although there is no Bonobo extension yet that handles that, it does make sense to integrate Bonobo jobs in an airflow
(or other similar tool) workflow.
* A big data solution, `as defined by wikipedia <https://en.wikipedia.org/wiki/Big_data>`_. We're aiming at "small
scale" data processing, which can be still quite huge for humans, but not for computers. If you don't know whether or
not this is sufficient for your needs, it probably means you're not in the "big data" land.
Where to jump next?
:::::::::::::::::::
If you did not run through it yet, we highly suggest that you go through the :doc:`tutorial </tutorial/index>` first.
Then, you can jump to the following guides, in no particuliar order:
.. toctree::
:maxdepth: 1
transformations
graphs
services
environment
purity

103
docs/guide/services.rst
View File

@ -1,14 +1,10 @@
Services and dependencies Services and dependencies
========================= =========================
:Last-Modified: 20 may 2017 You'll want to use external systems within your transformations, including databases, HTTP APIs, other web services,
filesystems, etc.
You'll probably want to use external systems within your transformations. Those systems may include databases, apis Hardcoding those services is a good first step, but as your codebase grows, will show limits rather quickly.
(using http, for example), filesystems, etc.
You can start by hardcoding those services. That does the job, at first.
If you're going a little further than that, you'll feel limited, for a few reasons:
* Hardcoded and tightly linked dependencies make your transformations hard to test, and hard to reuse. * Hardcoded and tightly linked dependencies make your transformations hard to test, and hard to reuse.
* Processing data on your laptop is great, but being able to do it on different target systems (or stages), in different * Processing data on your laptop is great, but being able to do it on different target systems (or stages), in different
@ -16,70 +12,77 @@ If you're going a little further than that, you'll feel limited, for a few reaso
pre-production environment, or production system. Maybe you have similar systems for different clients and want to select pre-production environment, or production system. Maybe you have similar systems for different clients and want to select
the system at runtime. Etc. the system at runtime. Etc.
Service injection Definition of service dependencies
::::::::::::::::: ::::::::::::::::::::::::::::::::::
To solve this problem, we introduce a light dependency injection system. It allows to define named dependencies in To solve this problem, we introduce a light dependency injection system. It allows to define **named dependencies** in
your transformations, and provide an implementation at runtime. your transformations, and provide an implementation at runtime.
Class-based transformations For function-based transformations, you can use the :func:`bonobo.config.use` decorator to mark the dependencies. You'll
--------------------------- still be able to call it manually, providing the implementation yourself, but in a bonobo execution context, it will
be resolve and injected automatically, as long as you provided an implementation to the executor (more on that below).
To define a service dependency in a class-based transformation, use :class:`bonobo.config.Service`, a special .. code-block:: python
descriptor (and subclass of :class:`bonobo.config.Option`) that will hold the service names and act as a marker
for runtime resolution of service instances.
Let's define such a transformation: from bonobo.config import use
@use('orders_database')
def select_all(database):
yield from database.query('SELECT * FROM foo;')
For class based transformations, you can use :class:`bonobo.config.Service`, a special descriptor (and subclass of
:class:`bonobo.config.Option`) that will hold the service names and act as a marker for runtime resolution of service
instances.
.. code-block:: python .. code-block:: python
from bonobo.config import Configurable, Service from bonobo.config import Configurable, Service
class JoinDatabaseCategories(Configurable): class JoinDatabaseCategories(Configurable):
database = Service('primary_sql_database') database = Service('orders_database')
def __call__(self, database, row): def call(self, database, row):
return { return {
**row, **row,
'category': database.get_category_name_for_sku(row['sku']) 'category': database.get_category_name_for_sku(row['sku'])
} }
This piece of code tells bonobo that your transformation expect a service called "primary_sql_database", that will be Both pieces of code tells bonobo that your transformation expect a service called "orders_database", that will be
injected to your calls under the parameter name "database". injected to your calls under the parameter name "database".
Function-based transformations Providing implementations at run-time
------------------------------ -------------------------------------
No implementation yet, but expect something similar to CBT API, maybe using a `@Service(...)` decorator. See Bonobo will expect you to provide a dictionary of all service implementations required by your graph.
`issue #70 <https://github.com/python-bonobo/bonobo/issues/70>`_.
Provide implementation at run time
----------------------------------
Let's see how to execute it:
.. code-block:: python .. code-block:: python
import bonobo import bonobo
graph = bonobo.graph( graph = bonobo.graph(...)
*before,
JoinDatabaseCategories(), def get_services():
*after, return {
) 'orders_database': my_database_service,
}
if __name__ == '__main__': if __name__ == '__main__':
bonobo.run( bonobo.run(graph, services=get_services())
graph,
services={
'primary_sql_database': my_database_service, .. note::
}
) A dictionary, or dictionary-like, "services" named argument can be passed to the :func:`bonobo.run` API method.
The "dictionary-like" part is the real keyword here. Bonobo is not a DIC library, and won't become one. So the
A dictionary, or dictionary-like, "services" named argument can be passed to the :func:`bonobo.run` helper. The implementation provided is pretty basic, and feature-less. But you can use much more evolved libraries instead of
"dictionary-like" part is the real keyword here. Bonobo is not a DIC library, and won't become one. So the implementation the provided stub, and as long as it works the same (a.k.a implements a dictionary-like interface), the system will
provided is pretty basic, and feature-less. But you can use much more evolved libraries instead of the provided use it.
stub, and as long as it works the same (a.k.a implements a dictionary-like interface), the system will use it.
Command line interface will look at services in two different places:
* A `get_services()` function present at the same level of your graph definition.
* A `get_services()` function in a `_services.py` file in the same directory as your graph's file, allowing to reuse the
same service implementations for more than one graph.
Solving concurrency problems Solving concurrency problems
---------------------------- ----------------------------
@ -87,7 +90,7 @@ Solving concurrency problems
If a service cannot be used by more than one thread at a time, either because it's just not threadsafe, or because If a service cannot be used by more than one thread at a time, either because it's just not threadsafe, or because
it requires to carefully order the calls made (apis that includes nonces, or work on results returned by previous it requires to carefully order the calls made (apis that includes nonces, or work on results returned by previous
calls are usually good candidates), you can use the :class:`bonobo.config.Exclusive` context processor to lock the calls are usually good candidates), you can use the :class:`bonobo.config.Exclusive` context processor to lock the
use of a dependency for a time period. use of a dependency for the time of the context manager (`with` statement)
.. code-block:: python .. code-block:: python
@ -101,18 +104,10 @@ use of a dependency for a time period.
api.last_call() api.last_call()
Service configuration (to be decided and implemented)
:::::::::::::::::::::::::::::::::::::::::::::::::::::
* There should be a way to configure default service implementation for a python file, a directory, a project ...
* There should be a way to override services when running a transformation.
* There should be a way to use environment for service configuration.
Future and proposals Future and proposals
:::::::::::::::::::: ::::::::::::::::::::
This is the first proposed implementation and it will evolve, but looks a lot like how we used bonobo ancestor in This first implementation and it will evolve. Base concepts will stay, though.
production.
May or may not happen, depending on discussions. May or may not happen, depending on discussions.

149
docs/guide/transformations.rst
View File

@ -1,8 +1,90 @@
Transformations Transformations
=============== ===============
Here is some guidelines on how to write transformations, to avoid the convention-jungle that could happen without Transformations are the smallest building blocks in Bonobo ETL.
a few rules.
They are written using standard python callables (or iterables, if you're writing transformations that have no input,
a.k.a extractors).
Definitions
:::::::::::
Transformation
The base building block of Bonobo, anything you would insert in a graph as a node. Mostly, a callable or an iterable.
Extractor
Special case transformation that use no input. It will be only called once, and its purpose is to generate data,
either by itself or by requesting it from an external service.
Loader
Special case transformation that feed an external service with data. For convenience, it can also yield the data but
a "pure" loader would have no output (although yielding things should have no bad side effect).
Callable
Anything one can call, in python. Can be a function, a python builtin, or anything that implements `__call__`
Iterable
Something we can iterate on, in python, so basically anything you'd be able to use in a `for` loop.
Function based transformations
::::::::::::::::::::::::::::::
The most basic transformations are function-based. Which means that you define a function, and it will be used directly
in a graph.
.. code-block:: python
def get_representation(row):
return repr(row)
graph = bonobo.Graph(
[...],
get_representation,
[...],
)
It does not allow any configuration, but if it's an option, prefer it as it's simpler to write.
Class based transformations
:::::::::::::::::::::::::::
For less basic use cases, you'll want to use classes to define some of your transformations. It's also a better choice
to build reusable blocks, as you'll be able to create parametrizable transformations that the end user will be able to
configure at the last minute.
Configurable
------------
.. autoclass:: bonobo.config.Configurable
Options
-------
.. autoclass:: bonobo.config.Option
Services
--------
.. autoclass:: bonobo.config.Service
Methods
-------
.. autoclass:: bonobo.config.Method
ContextProcessors
-----------------
.. autoclass:: bonobo.config.ContextProcessor
Naming conventions Naming conventions
@ -44,50 +126,35 @@ can be used as a graph node, then use camelcase names:
upper = Apply(str.upper) upper = Apply(str.upper)
Function based transformations Testing
:::::::::::::::::::::::::::::: :::::::
As Bonobo use plain old python objects as transformations, it's very easy to unit test your transformations using your
favourite testing framework. We're using pytest internally for Bonobo, but it's up to you to use the one you prefer.
If you want to test a transformation with the surrounding context provided (for example, service instances injected, and
context processors applied), you can use :class:`bonobo.execution.NodeExecutionContext` as a context processor and have
bonobo send the data to your transformation.
The most basic transformations are function-based. Which means that you define a function, and it will be used directly
in a graph.
.. code-block:: python .. code-block:: python
def get_representation(row): from bonobo.constants import BEGIN, END
return repr(row) from bonobo.execution import NodeExecutionContext
graph = bonobo.Graph( with NodeExecutionContext(
[...], JsonWriter(filename), services={'fs': ...}
get_representation, ) as context:
)
# Write a list of rows, including BEGIN/END control messages.
context.write(
BEGIN,
Bag({'foo': 'bar'}),
Bag({'foo': 'baz'}),
END
)
It does not allow any configuration, but if it's an option, prefer it as it's simpler to write. # Out of the bonobo main loop, we need to call `step` explicitely.
context.step()
context.step()
Class based transformations
:::::::::::::::::::::::::::
A lot of logic is a bit more complex, and you'll want to use classes to define some of your transformations.
The :class:`bonobo.config.Configurable` class gives you a few toys to write configurable transformations.
Options
-------
.. autoclass:: bonobo.config.Option
Services
--------
.. autoclass:: bonobo.config.Service
Methods
-------
.. autoclass:: bonobo.config.Method
ContextProcessors
-----------------
.. autoclass:: bonobo.config.ContextProcessor

6
docs/index.rst
View File

@ -11,6 +11,10 @@ Bonobo
reference/index reference/index
faq faq
contribute/index contribute/index
.. toctree::
:hidden:
genindex genindex
modindex modindex