paulin/bonobo
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'upstream/0.2' into colorama

Browse Source
This commit is contained in:
Romain Dorgueil
2017-04-24 21:37:11 +02:00
parent 9fc2e0ef94 9bf8f5c3a7
commit cff32c1612
31 changed files with 763 additions and 157 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
.travis.yml
View File

@ -10,5 +10,7 @@ install:
- pip install coveralls
script:
- make clean docs test
# - pip install pycountry
# - bin/run_all_examples.sh
after_success:
- coveralls

2
Makefile
View File

@ -1,7 +1,7 @@
# This file has been auto-generated.
# All changes will be lost, see Projectfile.
#
# Updated at 2017-04-24 21:35:29.734614
# Updated at 2017-04-24 21:37:09.094705
PYTHON ?= $(shell which python)
PYTHON_BASENAME ?= $(shell basename $(PYTHON))

2
Projectfile
View File

@ -4,7 +4,7 @@ name = 'bonobo'
description = 'Bonobo'
license = 'Apache License, Version 2.0'
url = 'https://bonobo-project.org/'
url = 'https://www.bonobo-project.org/'
download_url = 'https://github.com/python-bonobo/bonobo/tarball/{version}'
author = 'Romain Dorgueil'

82
README.rst
View File

@ -49,83 +49,13 @@ concepts work.
----
Made with ♥ by `Romain Dorgueil <https://twitter.com/rdorgueil>`_ and `contributors <https://github.com/python-bonobo/bonobo/graphs/contributors>`_.
Issues: https://github.com/python-bonobo/bonobo/issues
Roadmap: https://www.bonobo-project.org/roadmap
Slack: https://bonobo-slack.herokuapp.com/
----
Roadmap (in progress)
:::::::::::::::::::::
Bonobo is young. This roadmap is alive, and will evolve. Its only purpose is to
write down incoming things somewhere.
Version 0.2
-----------
* Changelog
* Migration guide
* Update documentation
* Threaded does not terminate anymore (fixed ?)
* More tests
Bugs:
- KeyboardInterrupt does not work anymore. (fixed ?)
- ThreadPool does not stop anymore. (fiexd ?)
Configuration
.............
* Support for position arguments (options), required options are good candidates.
Context processors
..................
* Be careful with order, especially with python 3.5. (done)
* @contextual decorator is not clean enough. Once the behavior is right, find a
way to use regular inheritance, without meta.
* ValueHolder API not clean. Find a better way.
Random thoughts and things to do
................................
* Class-tree for Graph and Nodes
* Class-tree for execution contexts:
* GraphExecutionContext
* NodeExecutionContext
* PluginExecutionContext
* Class-tree for ExecutionStrategies
* NaiveStrategy
* PoolExecutionStrategy
* ThreadPoolExecutionStrategy
* ProcessPoolExecutionStrategy
* ThreadExecutionStrategy
* ProcessExecutionStrategy
* Class-tree for bags
* Bag
* ErrorBag
* InheritingBag
* Co-routines: for unordered, or even ordered but long io.
* "context processors": replace initialize/finalize by a generator that yields only once
* "execute" function:
.. code-block:: python
def execute(graph: Graph, *, strategy: ExecutionStrategy, plugins: List[Plugin]) -> Execution:
pass
* Handling console. Can we use a queue, and replace stdout / stderr ?
Made with ♥ by `Romain Dorgueil <https://twitter.com/rdorgueil>`_ and `contributors <https://github.com/python-bonobo/bonobo/graphs/contributors>`_.

83
appveyor.yml Normal file
View File

@ -0,0 +1,83 @@
# See https://github.com/ogrisel/python-appveyor-demo
environment:
global:
# SDK v7.0 MSVC Express 2008's SetEnv.cmd script will fail if the
# /E:ON and /V:ON options are not enabled in the batch script intepreter
# See: http://stackoverflow.com/a/13751649/163740
CMD_IN_ENV: "cmd /E:ON /V:ON /C .\\bin\\appveyor\\run_with_env.cmd"
matrix:
- PYTHON: "C:\\Python35"
PYTHON_VERSION: "3.5.3"
PYTHON_ARCH: "32"
- PYTHON: "C:\\Python35-x64"
PYTHON_VERSION: "3.5.3"
PYTHON_ARCH: "64"
- PYTHON: "C:\\Python36"
PYTHON_VERSION: "3.6.1"
PYTHON_ARCH: "32"
- PYTHON: "C:\\Python36-x64"
PYTHON_VERSION: "3.6.1"
PYTHON_ARCH: "64"
install:
# If there is a newer build queued for the same PR, cancel this one.
# The AppVeyor 'rollout builds' option is supposed to serve the same
# purpose but it is problematic because it tends to cancel builds pushed
# directly to master instead of just PR builds (or the converse).
# credits: JuliaLang developers.
- ps: if ($env:APPVEYOR_PULL_REQUEST_NUMBER -and $env:APPVEYOR_BUILD_NUMBER -ne ((Invoke-RestMethod `
https://ci.appveyor.com/api/projects/$env:APPVEYOR_ACCOUNT_NAME/$env:APPVEYOR_PROJECT_SLUG/history?recordsNumber=50).builds | `
Where-Object pullRequestId -eq $env:APPVEYOR_PULL_REQUEST_NUMBER)[0].buildNumber) { `
throw "There are newer queued builds for this pull request, failing early." }
- ECHO "Filesystem root:"
- ps: "ls \"C:/\""
- ECHO "Installed SDKs:"
- ps: "ls \"C:/Program Files/Microsoft SDKs/Windows\""
# Install Python (from the official .msi of http://python.org) and pip when
# not already installed.
- ps: if (-not(Test-Path($env:PYTHON))) { & bin\appveyor\install.ps1 }
# Prepend newly installed Python to the PATH of this build (this cannot be
# done from inside the powershell script as it would require to restart
# the parent CMD process).
- "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
# Check that we have the expected version and architecture for Python
- "python --version"
- "python -c \"import struct; print(struct.calcsize('P') * 8)\""
# Upgrade to the latest version of pip to avoid it displaying warnings
# about it being out of date.
- "pip install --disable-pip-version-check --user --upgrade pip"
# Install the build dependencies of the project. If some dependencies contain
# compiled extensions and are not provided as pre-built wheel packages,
# pip will build them from source using the MSVC compiler matching the
# target Python version and architecture
- "%CMD_IN_ENV% pip install -e ."
test_script:
# Run the project tests
- "%CMD_IN_ENV% pytest --capture=no --cov=bonobo --cov-report html"
after_test:
# If tests are successful, create binary packages for the project.
- "%CMD_IN_ENV% python setup.py bdist_wheel"
- "%CMD_IN_ENV% python setup.py bdist_wininst"
- "%CMD_IN_ENV% python setup.py bdist_msi"
- ps: "ls dist"
artifacts:
# Archive the generated packages in the ci.appveyor.com build report.
- path: dist\*
#on_success:
# - TODO: upload the content of dist/*.whl to a public wheelhouse
#

229
bin/appveyor/install.ps1 Normal file
View File

@ -0,0 +1,229 @@
# Sample script to install Python and pip under Windows
# Authors: Olivier Grisel, Jonathan Helmus, Kyle Kastner, and Alex Willmer
# License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
$MINICONDA_URL = "http://repo.continuum.io/miniconda/"
$BASE_URL = "https://www.python.org/ftp/python/"
$GET_PIP_URL = "https://bootstrap.pypa.io/get-pip.py"
$GET_PIP_PATH = "C:\get-pip.py"
$PYTHON_PRERELEASE_REGEX = @"
(?x)
(?<major>\d+)
\.
(?<minor>\d+)
\.
(?<micro>\d+)
(?<prerelease>[a-z]{1,2}\d+)
"@
function Download ($filename, $url) {
$webclient = New-Object System.Net.WebClient
$basedir = $pwd.Path + "\"
$filepath = $basedir + $filename
if (Test-Path $filename) {
Write-Host "Reusing" $filepath
return $filepath
}
# Download and retry up to 3 times in case of network transient errors.
Write-Host "Downloading" $filename "from" $url
$retry_attempts = 2
for ($i = 0; $i -lt $retry_attempts; $i++) {
try {
$webclient.DownloadFile($url, $filepath)
break
}
Catch [Exception]{
Start-Sleep 1
}
}
if (Test-Path $filepath) {
Write-Host "File saved at" $filepath
} else {
# Retry once to get the error message if any at the last try
$webclient.DownloadFile($url, $filepath)
}
return $filepath
}
function ParsePythonVersion ($python_version) {
if ($python_version -match $PYTHON_PRERELEASE_REGEX) {
return ([int]$matches.major, [int]$matches.minor, [int]$matches.micro,
$matches.prerelease)
}
$version_obj = [version]$python_version
return ($version_obj.major, $version_obj.minor, $version_obj.build, "")
}
function DownloadPython ($python_version, $platform_suffix) {
$major, $minor, $micro, $prerelease = ParsePythonVersion $python_version
if (($major -le 2 -and $micro -eq 0) `
-or ($major -eq 3 -and $minor -le 2 -and $micro -eq 0) `
) {
$dir = "$major.$minor"
$python_version = "$major.$minor$prerelease"
} else {
$dir = "$major.$minor.$micro"
}
if ($prerelease) {
if (($major -le 2) `
-or ($major -eq 3 -and $minor -eq 1) `
-or ($major -eq 3 -and $minor -eq 2) `
-or ($major -eq 3 -and $minor -eq 3) `
) {
$dir = "$dir/prev"
}
}
if (($major -le 2) -or ($major -le 3 -and $minor -le 4)) {
$ext = "msi"
if ($platform_suffix) {
$platform_suffix = ".$platform_suffix"
}
} else {
$ext = "exe"
if ($platform_suffix) {
$platform_suffix = "-$platform_suffix"
}
}
$filename = "python-$python_version$platform_suffix.$ext"
$url = "$BASE_URL$dir/$filename"
$filepath = Download $filename $url
return $filepath
}
function InstallPython ($python_version, $architecture, $python_home) {
Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
if (Test-Path $python_home) {
Write-Host $python_home "already exists, skipping."
return $false
}
if ($architecture -eq "32") {
$platform_suffix = ""
} else {
$platform_suffix = "amd64"
}
$installer_path = DownloadPython $python_version $platform_suffix
$installer_ext = [System.IO.Path]::GetExtension($installer_path)
Write-Host "Installing $installer_path to $python_home"
$install_log = $python_home + ".log"
if ($installer_ext -eq '.msi') {
InstallPythonMSI $installer_path $python_home $install_log
} else {
InstallPythonEXE $installer_path $python_home $install_log
}
if (Test-Path $python_home) {
Write-Host "Python $python_version ($architecture) installation complete"
} else {
Write-Host "Failed to install Python in $python_home"
Get-Content -Path $install_log
Exit 1
}
}
function InstallPythonEXE ($exepath, $python_home, $install_log) {
$install_args = "/quiet InstallAllUsers=1 TargetDir=$python_home"
RunCommand $exepath $install_args
}
function InstallPythonMSI ($msipath, $python_home, $install_log) {
$install_args = "/qn /log $install_log /i $msipath TARGETDIR=$python_home"
$uninstall_args = "/qn /x $msipath"
RunCommand "msiexec.exe" $install_args
if (-not(Test-Path $python_home)) {
Write-Host "Python seems to be installed else-where, reinstalling."
RunCommand "msiexec.exe" $uninstall_args
RunCommand "msiexec.exe" $install_args
}
}
function RunCommand ($command, $command_args) {
Write-Host $command $command_args
Start-Process -FilePath $command -ArgumentList $command_args -Wait -Passthru
}
function InstallPip ($python_home) {
$pip_path = $python_home + "\Scripts\pip.exe"
$python_path = $python_home + "\python.exe"
if (-not(Test-Path $pip_path)) {
Write-Host "Installing pip..."
$webclient = New-Object System.Net.WebClient
$webclient.DownloadFile($GET_PIP_URL, $GET_PIP_PATH)
Write-Host "Executing:" $python_path $GET_PIP_PATH
& $python_path $GET_PIP_PATH
} else {
Write-Host "pip already installed."
}
}
function DownloadMiniconda ($python_version, $platform_suffix) {
if ($python_version -eq "3.4") {
$filename = "Miniconda3-3.5.5-Windows-" + $platform_suffix + ".exe"
} else {
$filename = "Miniconda-3.5.5-Windows-" + $platform_suffix + ".exe"
}
$url = $MINICONDA_URL + $filename
$filepath = Download $filename $url
return $filepath
}
function InstallMiniconda ($python_version, $architecture, $python_home) {
Write-Host "Installing Python" $python_version "for" $architecture "bit architecture to" $python_home
if (Test-Path $python_home) {
Write-Host $python_home "already exists, skipping."
return $false
}
if ($architecture -eq "32") {
$platform_suffix = "x86"
} else {
$platform_suffix = "x86_64"
}
$filepath = DownloadMiniconda $python_version $platform_suffix
Write-Host "Installing" $filepath "to" $python_home
$install_log = $python_home + ".log"
$args = "/S /D=$python_home"
Write-Host $filepath $args
Start-Process -FilePath $filepath -ArgumentList $args -Wait -Passthru
if (Test-Path $python_home) {
Write-Host "Python $python_version ($architecture) installation complete"
} else {
Write-Host "Failed to install Python in $python_home"
Get-Content -Path $install_log
Exit 1
}
}
function InstallMinicondaPip ($python_home) {
$pip_path = $python_home + "\Scripts\pip.exe"
$conda_path = $python_home + "\Scripts\conda.exe"
if (-not(Test-Path $pip_path)) {
Write-Host "Installing pip..."
$args = "install --yes pip"
Write-Host $conda_path $args
Start-Process -FilePath "$conda_path" -ArgumentList $args -Wait -Passthru
} else {
Write-Host "pip already installed."
}
}
function main () {
InstallPython $env:PYTHON_VERSION $env:PYTHON_ARCH $env:PYTHON
InstallPip $env:PYTHON
}
main

88
bin/appveyor/run_with_env.cmd Normal file
View File

@ -0,0 +1,88 @@
:: To build extensions for 64 bit Python 3, we need to configure environment
:: variables to use the MSVC 2010 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 4 (SDK v7.1)
::
:: To build extensions for 64 bit Python 2, we need to configure environment
:: variables to use the MSVC 2008 C++ compilers from GRMSDKX_EN_DVD.iso of:
:: MS Windows SDK for Windows 7 and .NET Framework 3.5 (SDK v7.0)
::
:: 32 bit builds, and 64-bit builds for 3.5 and beyond, do not require specific
:: environment configurations.
::
:: Note: this script needs to be run with the /E:ON and /V:ON flags for the
:: cmd interpreter, at least for (SDK v7.0)
::
:: More details at:
:: https://github.com/cython/cython/wiki/64BitCythonExtensionsOnWindows
:: http://stackoverflow.com/a/13751649/163740
::
:: Author: Olivier Grisel
:: License: CC0 1.0 Universal: http://creativecommons.org/publicdomain/zero/1.0/
::
:: Notes about batch files for Python people:
::
:: Quotes in values are literally part of the values:
:: SET FOO="bar"
:: FOO is now five characters long: " b a r "
:: If you don't want quotes, don't include them on the right-hand side.
::
:: The CALL lines at the end of this file look redundant, but if you move them
:: outside of the IF clauses, they do not run properly in the SET_SDK_64==Y
:: case, I don't know why.
@ECHO OFF
SET COMMAND_TO_RUN=%*
SET WIN_SDK_ROOT=C:\Program Files\Microsoft SDKs\Windows
SET WIN_WDK=c:\Program Files (x86)\Windows Kits\10\Include\wdf
:: Extract the major and minor versions, and allow for the minor version to be
:: more than 9. This requires the version number to have two dots in it.
SET MAJOR_PYTHON_VERSION=%PYTHON_VERSION:~0,1%
IF "%PYTHON_VERSION:~3,1%" == "." (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,1%
) ELSE (
SET MINOR_PYTHON_VERSION=%PYTHON_VERSION:~2,2%
)
:: Based on the Python version, determine what SDK version to use, and whether
:: to set the SDK for 64-bit.
IF %MAJOR_PYTHON_VERSION% == 2 (
SET WINDOWS_SDK_VERSION="v7.0"
SET SET_SDK_64=Y
) ELSE (
IF %MAJOR_PYTHON_VERSION% == 3 (
SET WINDOWS_SDK_VERSION="v7.1"
IF %MINOR_PYTHON_VERSION% LEQ 4 (
SET SET_SDK_64=Y
) ELSE (
SET SET_SDK_64=N
IF EXIST "%WIN_WDK%" (
:: See: https://connect.microsoft.com/VisualStudio/feedback/details/1610302/
REN "%WIN_WDK%" 0wdf
)
)
) ELSE (
ECHO Unsupported Python version: "%MAJOR_PYTHON_VERSION%"
EXIT 1
)
)
IF %PYTHON_ARCH% == 64 (
IF %SET_SDK_64% == Y (
ECHO Configuring Windows SDK %WINDOWS_SDK_VERSION% for Python %MAJOR_PYTHON_VERSION% on a 64 bit architecture
SET DISTUTILS_USE_SDK=1
SET MSSdk=1
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Setup\WindowsSdkVer.exe" -q -version:%WINDOWS_SDK_VERSION%
"%WIN_SDK_ROOT%\%WINDOWS_SDK_VERSION%\Bin\SetEnv.cmd" /x64 /release
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
) ELSE (
ECHO Using default MSVC build environment for 64 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)
) ELSE (
ECHO Using default MSVC build environment for 32 bit architecture
ECHO Executing: %COMMAND_TO_RUN%
call %COMMAND_TO_RUN% || EXIT 1
)

9
bin/run_all_examples.sh Executable file
View File

@ -0,0 +1,9 @@
#! /bin/bash
__PATH__=$(cd $(dirname "$0")/..; pwd)
EXAMPLES=$(cd $__PATH__; find bonobo/examples -name \*.py -not -name __init__.py)
for example in $EXAMPLES; do
echo "===== $example ====="
(cd $__PATH__; time bonobo run $example > /dev/null);
done

25
bonobo/__init__.py
View File

@ -67,17 +67,36 @@ def create_strategy(name=None):
return factory()
def _is_interactive_console():
import sys
return sys.stdout.isatty()
def _is_jupyter_notebook():
try:
return get_ipython().__class__.__name__ == 'ZMQInteractiveShell'
except NameError:
return False
def run(graph, *chain, strategy=None, plugins=None):
strategy = create_strategy(strategy)
if len(chain):
warnings.warn('DEPRECATED. You should pass a Graph instance instead of a chain.')
from bonobo import Graph
graph = Graph(graph, *chain)
return strategy.execute(graph, plugins=plugins)
strategy = create_strategy(strategy)
plugins = []
if _is_interactive_console():
from bonobo.ext.console import ConsoleOutputPlugin
if ConsoleOutputPlugin not in plugins:
plugins.append(ConsoleOutputPlugin)
if _is_jupyter_notebook():
from bonobo.ext.jupyter import JupyterOutputPlugin
if JupyterOutputPlugin not in plugins:
plugins.append(JupyterOutputPlugin)
return strategy.execute(graph, plugins=plugins)
del sys
del warnings

7
bonobo/commands/run.py
View File

@ -1,6 +1,5 @@
import argparse
from bonobo import Graph, run
import bonobo
def execute(file, quiet=False):
@ -22,7 +21,7 @@ def execute(file, quiet=False):
except Exception as exc:
raise
graphs = dict((k, v) for k, v in context.items() if isinstance(v, Graph))
graphs = dict((k, v) for k, v in context.items() if isinstance(v, bonobo.Graph))
assert len(graphs) == 1, (
'Having zero or more than one graph definition in one file is unsupported for now, '
@ -34,7 +33,7 @@ def execute(file, quiet=False):
# todo if console and not quiet, then add the console plugin
# todo when better console plugin, add it if console and just disable display
return run(graph)
return bonobo.run(graph)
def register(parser):

18
bonobo/context/execution.py
View File

@ -76,7 +76,7 @@ class GraphExecutionContext:
def ensure_tuple(tuple_or_mixed):
if isinstance(tuple_or_mixed, tuple):
return tuple_or_mixed
return (tuple_or_mixed, )
return (tuple_or_mixed,)
class LoopingExecutionContext(Wrapper):
@ -170,16 +170,24 @@ class LoopingExecutionContext(Wrapper):
class PluginExecutionContext(LoopingExecutionContext):
PERIOD = 0.5
def __init__(self, wrapped, parent):
LoopingExecutionContext.__init__(self, wrapped, parent)
# Instanciate plugin. This is not yet considered stable, as at some point we may need a way to configure
# plugins, for example if it depends on an external service.
super().__init__(wrapped(self), parent)
def shutdown(self):
self.wrapped.finalize(self)
self.alive = False
try:
self.wrapped.finalize()
except Exception as exc: # pylint: disable=broad-except
self.handle_error(exc, traceback.format_exc())
finally:
self.alive = False
def step(self):
try:
self.wrapped.run(self)
self.wrapped.run()
except Exception as exc: # pylint: disable=broad-except
self.handle_error(exc, traceback.format_exc())

5
bonobo/core/strategies/executor.py
View File

@ -22,7 +22,6 @@ class ExecutorStrategy(Strategy):
def execute(self, graph, *args, plugins=None, **kwargs):
context = self.create_graph_execution_context(graph, plugins=plugins)
context.recv(BEGIN, Bag(), END)
executor = self.create_executor()
@ -30,16 +29,13 @@ class ExecutorStrategy(Strategy):
futures = []
for plugin_context in context.plugins:
def _runner(plugin_context=plugin_context):
plugin_context.start()
plugin_context.loop()
plugin_context.stop()
futures.append(executor.submit(_runner))
for node_context in context.nodes:
def _runner(node_context=node_context):
node_context.start()
node_context.loop()
@ -67,6 +63,7 @@ class ProcessPoolExecutorStrategy(ExecutorStrategy):
class ThreadCollectionStrategy(Strategy):
def execute(self, graph, *args, plugins=None, **kwargs):
print(type(self), 'execute', graph, args, plugins, kwargs)
context = self.create_graph_execution_context(graph, plugins=plugins)
context.recv(BEGIN, Bag(), END)

18
bonobo/examples/tutorials/tutorial_basics_summary.py
View File

@ -1,19 +1,11 @@
from bonobo import Graph, ThreadPoolExecutorStrategy
def yield_from(*args):
yield from args
import bonobo
# Represent our data processor as a simple directed graph of callables.
graph = Graph(
lambda: (x for x in ('foo', 'bar', 'baz')),
graph = bonobo.Graph(
['foo', 'bar', 'baz'],
str.upper,
print,
)
# Use a thread pool.
executor = ThreadPoolExecutorStrategy()
# Run the thing.
executor.execute(graph)
if __name__ == '__main__':
bonobo.run(graph)

28
bonobo/ext/console/plugin.py
View File

@ -14,10 +14,11 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import os
import sys
from functools import lru_cache
import blessings
import blessings
import os
import psutil
from bonobo.core.plugins import Plugin
@ -27,7 +28,7 @@ from bonobo.util import terminal as t
@lru_cache(1)
def memory_usage():
process = psutil.Process(os.getpid())
return process.get_memory_info()[0] / float(2**20)
return process.get_memory_info()[0] / float(2 ** 20)
# @lru_cache(64)
@ -46,10 +47,11 @@ class ConsoleOutputPlugin(Plugin):
"""
def __init__(self, prefix=''):
self.prefix = prefix
def __init__(self, context):
self.context = context
self.prefix = ''
def _write(self, context, rewind):
def _write(self, graph_context, rewind):
profile, debug = False, False
if profile:
append = (
@ -58,18 +60,16 @@ class ConsoleOutputPlugin(Plugin):
)
else:
append = ()
self.write(context, prefix=self.prefix, append=append, debug=debug, profile=profile, rewind=rewind)
self.write(graph_context, prefix=self.prefix, append=append, debug=debug, profile=profile, rewind=rewind)
# self.widget.value = [repr(component) for component in context.parent.components]
def run(self, context):
if t.is_a_tty:
self._write(context.parent, rewind=True)
def run(self):
if sys.stdout.isatty():
self._write(self.context.parent, rewind=True)
else:
pass # not a tty
def finalize(self, context):
self._write(context.parent, rewind=False)
def finalize(self):
self._write(self.context.parent, rewind=False)
@staticmethod
def write(context, prefix='', rewind=True, append=None, debug=False, profile=False):

1
bonobo/ext/jupyter/helpers.py
View File

@ -1 +0,0 @@

1
bonobo/ext/jupyter/js/dist/index.js vendored
View File

@ -70,7 +70,6 @@ define(["jupyter-js-widgets"], function(__WEBPACK_EXTERNAL_MODULE_2__) { return
// When serialiazing entire widget state for embedding, only values different from the
// defaults will be specified.
var BonoboModel = widgets.DOMWidgetModel.extend({
defaults: _.extend({}, widgets.DOMWidgetModel.prototype.defaults, {
_model_name: 'BonoboModel',

2
bonobo/ext/jupyter/js/dist/index.js.map vendored
View File

File diff suppressed because one or more lines are too long

1
bonobo/ext/jupyter/js/src/bonobo.js
View File

@ -9,7 +9,6 @@ var _ = require('underscore');
// When serialiazing entire widget state for embedding, only values different from the
// defaults will be specified.
var BonoboModel = widgets.DOMWidgetModel.extend({
defaults: _.extend({}, widgets.DOMWidgetModel.prototype.defaults, {
_model_name: 'BonoboModel',

17
bonobo/ext/jupyter/plugin.py
View File

@ -14,11 +14,22 @@ except ImportError as e:
class JupyterOutputPlugin(Plugin):
def initialize(self, context):
def __init__(self, context):
self.context = context
self.widget = BonoboWidget()
IPython.core.display.display(self.widget)
def run(self, context):
self.widget.value = [repr(component) for component in context.parent.components]
def run(self):
self.widget.value = [repr(node) for node in self.context.parent.nodes]
finalize = run
"""
TODO JUPYTER WIDGET
###################
# close the widget? what does it do?
https://ipywidgets.readthedocs.io/en/latest/examples/Widget%20Basics.html#Closing-widgets
"""

1
bonobo/ext/jupyter/static/index.js
View File

@ -73,7 +73,6 @@ define(["jupyter-js-widgets"], function(__WEBPACK_EXTERNAL_MODULE_2__) { return
// When serialiazing entire widget state for embedding, only values different from the
// defaults will be specified.
var BonoboModel = widgets.DOMWidgetModel.extend({
defaults: _.extend({}, widgets.DOMWidgetModel.prototype.defaults, {
_model_name: 'BonoboModel',

2
bonobo/ext/jupyter/static/index.js.map
View File

File diff suppressed because one or more lines are too long

3
bonobo/ext/opendatasoft.py
View File

@ -20,11 +20,12 @@ class OpenDataSoftAPI(Configurable):
netloc = Option(str, default='data.opendatasoft.com')
path = Option(path_str, default='/api/records/1.0/search/')
rows = Option(int, default=100)
timezone = Option(str, default='Europe/Paris')
kwargs = Option(dict, default=dict)
@ContextProcessor
def compute_path(self, context):
params = (('dataset', self.dataset), ('rows', self.rows), ) + tuple(sorted(self.kwargs.items()))
params = (('dataset', self.dataset), ('rows', self.rows), ('timezone', self.timezone)) + tuple(sorted(self.kwargs.items()))
yield self.endpoint.format(scheme=self.scheme, netloc=self.netloc, path=self.path) + '?' + urlencode(params)
@ContextProcessor

15
docs/_templates/index.html vendored
View File

@ -4,7 +4,9 @@
<div style="border: 2px solid red; font-weight: bold; margin: 1em; padding: 1em">
Bonobo is currently <strong>ALPHA</strong> software. That means that the doc is not finished, and that
some APIs will change.
some APIs will change.<br>
There are a lot of missing sections, including comparison with other tools. But if you're looking for a
replacement for X, unless X is an ETL, bonobo is probably not what you want.
</div>
<h1 style="text-align: center">
@ -15,14 +17,14 @@
<p>
{% trans %}
<strong>Bonobo</strong> is a line-by-line data-processing toolkit for python 3.5+ emphasizing simple and
atomic data transformations defined using a directed graph of plain old python callables.
atomic data transformations defined using a directed graph of plain old python callables (functions and
generators).
{% endtrans %}
</p>
<p>
{% trans %}
<strong>Bonobo</strong> is a full-featured Extract-Transform-Load library that won't force you to use an
ugly IDE.
<strong>Bonobo</strong> is a extract-transform-load framework that uses python code to define transformations.
{% endtrans %}
</p>
@ -103,6 +105,11 @@
Console, ...) or write your own.
{% endtrans %}
</li>
<li>
{% trans %}
Work in progress: read the <a href="https://www.bonobo-project.org/roadmap">roadmap</a>.
{% endtrans %}
</li>
</ul>
<p>{% trans %}

22
docs/changelog.rst
View File

@ -2,22 +2,30 @@ Changelog
=========
0.9.0
:::::
Incoming...
:::::::::::
* todo migrate doc
* todo migrate tests
* todo migrate transforms ?
Version 0.3
:::::::::::
* Autodetect if within jupyter notebook context, and apply plugin if it's the case.
* Console run should allow console plugin as a command line argument.
* New bonobo.structs package with simple data structures (bags, graphs, tokens).
Initial release
:::::::::::::::
* Migration from rdc.etl.
* New cool name.
* New cool name (ok, that's debatable).
* Only supports python 3.5+, aggressively (which means, we can use async, and we remove all things from python 2/six compat)
* Removes all thing deprecated and/or not really convincing
* We want transforms to be simple callables, so refactoring of the harness mess
* Removes all thing deprecated and/or not really convincing from rdc.etl.
* We want transforms to be simple callables, so refactoring of the harness mess.
* We want to use plain python data structures, so hashes are removed. If you use python 3.6, you may even get sorted dicts.
* Input/output MUX DEMUX removed, maybe no need for that in the real world. May come back, but not in 1.0
* Change dependency policy. We need to include only the very basic requirements (and very required). Everything related to transforms that we may not use (bs, sqla, ...) should be optional dependencies.
* execution strategies !!!
* Change dependency policy. We need to include only the very basic requirements (and very required). Everything related
to transforms that we may not use (bs, sqla, ...) should be optional dependencies.
* Execution strategies, threaded by default.

15
docs/contribute/index.rst
View File

@ -4,15 +4,20 @@ Contributing
Contributing to bonobo is simple. Although we don't have a complete guide on this topic for now, the best way is to fork
the github repository and send pull requests.
Keep the following points in mind:
A few guidelines...
* Although we will ask for 100% backward compatibility starting from 1.0 (following semantic versionning principles),
pre-1.0 versions should do their best to keep compatibility between versions. Wehn in doubt, open a github issue
to discuss things.
* Starting at 1.0, the system needs to be 100% backward compatible. Best way to do so is to ensure the actual expected
behavior is unit tested before making any change. See http://semver.org/.
* There can be changes before 1.0, even backward incompatible changes. There should be a reason for a BC break, but
I think it's best for the speed of development right now.
* The core should stay as light as possible.
* Coding standards are enforced using yapf. That means that you can code the way you want, we just ask you to run
`make format` before committing your changes so everybody follows the same conventions.
* General rule for anything you're not sure about is "open a github issue to discuss the point".
* More formal proposal process will come the day we feel the need for it.
A very drafty roadmap is available in the readme.
Issues: https://github.com/python-bonobo/bonobo/issues
Roadmap: https://www.bonobo-project.org/roadmap
Slack: https://bonobo-slack.herokuapp.com/

112
docs/faq.rst Normal file
View File

@ -0,0 +1,112 @@
F.A.Q.
======
List of questions that went up about the project, in no particuliar order.
Too long; didn't read.
----------------------
Bonobo is an extract-transform-load toolkit for python 3.5+, that use regular python functions, generators and iterators
as input.
By default, it uses a thread pool to execute all functions in parallel, and handle the movement of data rows in the
directed graph using simple fifo queues.
It allows the user to focus on the content of the transformations, and not optimizing blocking or long operations, nor
thinking about threads or subprocesses.
It's lean manufacturing for data.
.. note::
This is NOT a «big data» tool. We process around 5 millions database lines in around 1 hour with rdc.etl, bonobo
ancestor (algorithms are the same, we still need to run a bit of benchmarks).
Can a graph contain another graph?
----------------------------------
No, not for now. There are no tools today in bonobo to insert a graph as a subgraph.
It would be great to allow it, but there is a few design questions behind this, like what node you use as input and
output of the subgraph, etc.
It is something to be seriously considered post 1.0 (probably way post 1.0).
How would one access contextual data from a transformation? Are there parameter injections like pytest's fixtures?
------------------------------------------------------------------------------------------------------------------
There are indeed parameter injections that work much like pytest's fixtures, and it's the way to go for transformation
context.
The API may evolve a bit though, because I feel it's a bit hackish, as it is. The concept will stay the same, but we need
to find a better way to apply it.
To understand how it works today, look at https://github.com/python-bonobo/bonobo/blob/0.2/bonobo/io/csv.py#L63 and class hierarchy.
What is a plugin? Do I need to write one?
-----------------------------------------
Plugins are special classes added to an execution context, used to enhance or change the actual behavior of an execution
in a generic way. You don't need to write plugins to code transformation graphs.
Is there a difference between a transformation node and a regular python function or generator?
-----------------------------------------------------------------------------------------------
No.
Transformation callables are just regular callables, and there is nothing that differentiate it from regular python callables.
You can even use some callables both in an imperative programming context and in a transformation graph, no problem.
Why did you include the word «marketing» in a commit message? Why is there a marketing-automation tag on the project? Isn't marketing evil?
-------------------------------------------------------------------------------------------------------------------------------------------
I do use bonobo for marketing automation tasks. Also, half the job of coding something is explaining the world what
you're actually doing, how to get more informations, and how to use it and that's what I call "marketing" in some
commits. Even documentation is somehow marketing, because it allows a market of potential users to actually understand
your product. Whether the product is open-source, a box of chips or a complex commercial software does not change a
thing.
Marketing may be good or evil, and honestly, it's out of this project topic and I don't care. What I care about is that
there are marketing tasks to automate, and there are some of those cases I can solve with bonobo.
Why not use <some library> instead?
-----------------------------------
I did not find the tasks I had easy to do with the libraries I tried. That may or may not apply for your cases, and that
may or not include some lack of knowledge about some library from me. There is a plan to include comparisons with
major libraries in this documentation, and help from experts of other libraries (python or not) would be very welcome.
See https://github.com/python-bonobo/bonobo/issues/1
Bonobo is not a replacement for pandas, nor dask, nor luigi, nor airflow... It may be a replacement for Pentaho, Talend
or other data integration suites but targets people more comfortable with code as an interface.
All those references to monkeys hurt my head. Bonobos are not monkeys.
----------------------------------------------------------------------
Sorry, my bad. I'll work on this point in the near future, but as an apology, we only have one word that means both
«ape» and «monkey» in french, and I never realised that there was an actual difference. As one question out of two I
got about the project is somehow related to primates taxonomy, I'll make a special effort as soon as I can on this
topic.
Or maybe, I can use one of the comments from reddit as an answer: «Python not only has duck typing; it has the little
known primate typing feature.»
See https://github.com/python-bonobo/bonobo/issues/24
Who is behind this?
-------------------
Me (as an individual), and a few great people that helped me along the way. Not commercially endorsed, or supported.
The code, documentation, and surrounding material is created using spare time and may lack a bit velocity. Feel free
to jump in so we can go faster!
Documentation seriously lacks X, there is a problem in Y...
-----------------------------------------------------------
Yes, and sorry about that. An amazing way to make it better would be to submit a pull request about it. You can read a
bit about how to contribute on page :doc:`contribute/index`.

31
docs/guide/ext/jupyter.rst
View File

@ -1,11 +1,34 @@
Bonobo with Jupyter
==================
There is a builtin plugin that integrates (kind of minimalistically, for now) bonobo within jupyter notebooks, so
you can read the execution status of a graph within a nice (ok not so nice) html/javascript widget.
See https://github.com/jupyter-widgets/widget-cookiecutter for the base template used.
Installation
::::::::::::
Overview
::::::::
To install the widget::
jupyter nbextension enable --py --sys-prefix bonobo.ext.jupyter
Development
:::::::::::
To install the widget for development, make sure you're using an editable install of bonobo (see install document)::
jupyter nbextension install --py --symlink --sys-prefix bonobo.ext.jupyter
jupyter nbextension enable --py --sys-prefix bonobo.ext.jupyter
If you wanna change the javascript, you should run webpack in watch mode in some terminal::
cd bonobo/ext/jupyter/js
npm install
./node_modules/.bin/webpack --watch
To compile the widget into a distributable version (which gets packaged on PyPI when a release is made), just run
webpack::
./node_modules/.bin/webpack
Details
:::::::

1
docs/index.rst
View File

@ -9,6 +9,7 @@ Bonobo
guide/index
reference/index
contribute/index
faq
genindex
modindex

18
docs/install.rst
View File

@ -1,11 +1,6 @@
Installation
============
.. todo::
better install docs, especially on how to use different fork, etc.
Install with pip
::::::::::::::::
@ -32,3 +27,16 @@ If you plan on making patches to Bonobo, you should install it as an "editable"
Note: `-e` is the shorthand version of `--editable`.
Windows support
:::::::::::::::
We had some people report that there are problems on the windows platform, mostly due to terminal features. We're trying
to look into that but we don't have good windows experience, no windows box and not enough energy to provide serious
support there. If you have experience in this domain and you're willing to help, you're more than welcome!
.. todo::
Better install docs, especially on how to use different forks or branches, etc.

78
docs/old-roadmap.rst Normal file
View File

@ -0,0 +1,78 @@
----
Roadmap (in progress)
:::::::::::::::::::::
Bonobo is young. This roadmap is alive, and will evolve. Its only purpose is to
write down incoming things somewhere.
Version 0.2
-----------
* Changelog
* Migration guide
* Update documentation
* Threaded does not terminate anymore (fixed ?)
* More tests
Bugs:
- KeyboardInterrupt does not work anymore. (fixed ?)
- ThreadPool does not stop anymore. (fiexd ?)
Configuration
.............
* Support for position arguments (options), required options are good candidates.
Context processors
..................
* Be careful with order, especially with python 3.5. (done)
* @contextual decorator is not clean enough. Once the behavior is right, find a
way to use regular inheritance, without meta.
* ValueHolder API not clean. Find a better way.
Random thoughts and things to do
................................
* Class-tree for Graph and Nodes
* Class-tree for execution contexts:
* GraphExecutionContext
* NodeExecutionContext
* PluginExecutionContext
* Class-tree for ExecutionStrategies
* NaiveStrategy
* PoolExecutionStrategy
* ThreadPoolExecutionStrategy
* ProcessPoolExecutionStrategy
* ThreadExecutionStrategy
* ProcessExecutionStrategy
* Class-tree for bags
* Bag
* ErrorBag
* InheritingBag
* Co-routines: for unordered, or even ordered but long io.
* "context processors": replace initialize/finalize by a generator that yields only once
* "execute" function:
.. code-block:: python
def execute(graph: Graph, *, strategy: ExecutionStrategy, plugins: List[Plugin]) -> Execution:
pass
* Handling console. Can we use a queue, and replace stdout / stderr ?

2
setup.py
View File

@ -73,7 +73,7 @@ setup(
['bonobo = '
'bonobo.ext.edgy.project.feature:BonoboFeature']
},
url='https://bonobo-project.org/',
url='https://www.bonobo-project.org/',
download_url=
'https://github.com/python-bonobo/bonobo/tarball/{version}'.format(
version=version), )