Jupyter Logo

As you may know we have an Jupyter logo, as an SVG. But as for the Kilogram that as a prototype kilogram that act a reference, we suffer from the fact that the logo does not have an abstract description that could explain how to construct it, which is bad.

So with a little bit of reverse engeniering and Inkscape I was able to extract some geometric primitive to build the jupyter logo.

Still work that need to be done, especailly for the gradient to get the endpoint and the colors.

But this allow to do some nice things like plotting the logo in matplotlib :-)

In [1]:
%matplotlib inline
In [2]:
import numpy
In [3]:
from matplotlib.patches import Circle, Wedge
import matplotlib.pyplot as plt
In [13]:
def Jupyter(w=5, blueprint=False, ax=None, moons=True, steps=None, color='orange'):
    ## x,y center and diameter of primitive circles, 
    ## than to Inkspace.
    xyrs = numpy.array((
            (315, 487, 406),
            (315, 661, 630),
            (315, 315, 630), 
            (178, 262, 32),
            (146, 668, 20),
            (453, 705, 27),
    ))

    center = (315,487)
    xyrs[:,0] -= center[0]
    xyrs[:,1] -= center[1]
    if not ax:
        fig,axes = plt.subplots()
        fig.set_figheight(w)
        fig.set_figwidth(w)
    else:
        axes = ax
    axes.set_axis_bgcolor('white')
    axes.set_aspect('equal')
    axes.set_xlim(-256,256)
    axes.set_ylim(-256,256)
    t=-1
    ec='white'
    if blueprint:
        ec = 'blue'
        
    for xyr in xyrs[:steps]:
        xy,r= xyr[:2],xyr[2]
        if r == 630:
            fill=True;
            c = 'white'
        else:
            fill=True
            c=color

        if r==630 :
            a = 40
            axes.add_artist(Wedge(xy,r/2,a+t*180, 180-a+t*180, fill=fill, color=c, width=145, ec=ec))
            t = t+1
        elif r==406 :
            axes.add_artist(Circle(xy,r/2, fill=fill, color=c, ec=ec))
        else:
            if r<100 and moons:
                axes.add_artist(Circle(xy,r/2, fill=fill, color='gray', ec=ec))
        if blueprint:
            axes.plot(xy[0],xy[1],'+b')
    axes.xaxis.set_tick_params(color='white')
    axes.yaxis.set_tick_params(color='white')
    axes.xaxis.set_ticklabels('')
    axes.yaxis.set_ticklabels('')

    return axes
In [14]:
ax = Jupyter(10, blueprint=False)

And lets make the outline apparent:

In [15]:
ax = Jupyter(10, blueprint=True)

Color and gradients are not right yet, but looks great !

for small graphs you can also remove moons.

In [16]:
fig, axes = plt.subplots(2,3)
for ax in axes.flatten():
    Jupyter(ax=ax, moons=False)

If you run it locally, below you will have an interactive version that will show you the differents step of drawing the logo.

In [17]:
from IPython.html.widgets import interact

cc = (0,1,0.01)

@interact(n=(0,6),r=cc,g=cc, b=cc)
def fun(n=6, outline=False, r=1.0, g=0.5, b=0.0):
    return Jupyter(steps=n, blueprint=outline, color=(r,g,b))

As usual PR welcommed !

NB: I wrote this on my way back from Strata, thanks to Continuum Analytics for sponsoring my ticket, and for the nice conversations we had there !

Autodeploy with travis

TL;DR: After a day fighting with travis, my blog get auto-deployed with Travis using Nikola !

A bit of automation

One of the annoying things when I want to post a blog post like this one, is that I never remember hom to deploy my blog. So, why not completly automatize with a script ?

Well, that one step, but you know what is really good at runnign scripts ? Travis.

Travis have the nice ability to run script in the category after_success , or encrypting file, whice allow a nice deployment setup.

A description of the process.

The first step is to create an ssh key with empty pass passphrase; I like to add it (encrypted) to .travis folder in my repository. Travis have nice doc for that.

Copy the public key to the target github repository deploy key in setting.

In my particular setup the tricky bit where :

To get IPython and nikola master:

  • - pip install -e git+https://github.com/ipython/ipython.git#egg=IPython
  • - pip install -e git+https://github.com/getnikola/nikola.git#egg=nikola

Get the correct layout of folders:

  • blog (gh/carreau/blog)
    • posts (gh/carreau/posts)
    • output (gh/carreau/carreau.github.io)

I had to soft link posts, as this is the repo which trigger travis build, and is by default cloned into ~/posts by travis. carreau/carreau.github.io is clone via https to allow pull request to build (and not push) as the ssh key can only be decrypted on master branch.

Then run nikola build.

In the after_success step (you might want to run unit-test like non-broken link on your blog) you want to check that you are not in a PR, and on master branch before trying to decrypt the ssh key and push the built website:

The following snippet works well.

if [[ $TRAVIS_PULL_REQUEST == false && $TRAVIS_BRANCH == 'master' ]];

Travis CL tool gave you the instruction to decrypt the ssh key, you now "just" have to add it to the keyring.

eval `ssh-agent -s`
chmod 600 .travis/travis_rsa
ssh-add .travis/travis_rsa
cd ../blog/output

And add an ssh remote, which user is git:

git remote add ghpages ssh://git@github.com/Carreau/carreau.github.io

Push, and you are done (don't forget to commit though) !

Extra tips

When testing, do not push on master, push on another branch and open the PR manually to see the diff. Travis env might defier a bit from yours.

Todo

Nikola read metadata file from .meta file, which is annoying. I should patch it to read metadata from notebook Metadata. ALso need a JS extension to make that easy.

Closing thoughts

PR and comments welcommed as usual.

Jupyter

Jupyter

It occured to me recently that we (the IPython team), are relatively bad at communicating. Not that we don't respond to questions, or welcomme pull request, or will not spend a coupel of hours to teach you git even if you just want to update an URL. We are bad for writing good recapitulatif of what as been done. Of course we have weekly meeting on YouTube, plus notes on Hackpad, but it's painfull to listen to, and not alway friendly.

I hence will try to gather in this small post some of the latest questions I think need to be answerd.

What is Jupyter ?

You might have seen Fernando's Talk t SciPy last year that introduce Jupyter. It was a rough overview, a bit hitorical and technical in some point. In even less technical word, it's anything in IPython that could apply to not Python. We are creating this to welcome non-pythonista in the environement and promote interlanguage interoperability.

Can I be part of Jupyter ?

Yes. Right now only IPython team are Jupyter member (we are drafting legal stuff), but if you want to be part of the steering council we'll be happy for you to join. We would especially like you to join if you have a different background than any of us. You are using Jupyter/IPython for teaching a lot ? Or Maintain a Frobulate-lang kernel ? You point of view will be really usefull !

Can my company be part of Jupyter ?

Yes, here again we are still drafting things, but we want more visibility for sponsors and/or company that develop product that are based on JupyterWork with Jupyter and contrbute back. If you have a specific request or question, please contact us, we'd like to know your point of view !

Do I need technical expertise ?

No. You prefer to write blog post ? Do user testing ? Record demo on how to use a software on youtube ? that's helpful. You are designer ? you want to translate things ? That's awesome to. You like doing paperwork and challenging administrative task ? We can probably ask you how to mound a non-profit organisation to organise all that. You don't really know anything but understand a bit who deals with what on te project ? Your help will be helpfull to assign people tasks on the project ! You write a better english than I do ? Great, english is not my first language and I definitively need someone to point out my mistakes.

When will be Jupyter first public release ?

Well technically Jupyter is not only a project but an organisation. We already have a long list of project you might have heard of.

If you are reffering to The Notebook, then current IPython master is what is closer to Jupyter 1.0.

IPython Notebook 4.0 will probably not be as itmight be name Jupyter 1.0.

IPython will disapear ?

No it will not. IPython the console REPL will stay IPython. the IPython kernel will probably become a separate project call IPython-kernel that will be usable in IPython notebook. Basically most of the feature of IPython (traitlets, widgets, config, kernel, qtconsole...) woudl disapear from IPython repo.

You are removing Traitlest/Widgets/Magics.... ?

Not quite, we will make them separate repository, sometime under Jupyter organisation, sometime under IPython.

For exaple the QtConsole is not Python specific (well, it is written in Python), but can be used with Julia, Haskell... Thus it will be transfered to Jupyter org where it will have its own dedicated team and release schedule. This will allow quicker release of versions, and easier entry to contributor to get commits right. the team in charge will also have more flexibility on the feature shipped. Right now we would probably refuse a qtconsole background beeing a pink fluffy unicorn, once split is done the responsible team can decide to do so.

Traitlets in the other hand (that are use from config to widgets) will be under IPython organisation as they are a specificity from IPython kernel. They could be used in matplotlib to configure loads of things. Once split we will probably add things like numpy traitlets, and a lot of other things requested.

IPython will probably start depending on (some of) these subpackages.

Can you describe the future packages that will exists ?

Roughly will have a package/repo for:

  • nbconvert
  • nbformat
  • qtconsole
  • The Notebook
  • Config
  • Traitlets
  • Widgets
    • maybe even 2 packages, 1 for Python side, other for JS Side
  • IPython as a kernel
  • IPython as a CLI REPL.

Splitting this will moslty be the work we will do in betwwen IPython 3.0 and Jupyter 1.0

Can you give a timescale ?

IPython 3.0 beta early February, release end of February, Jupyter 1.0 three Month Later. If we get more help from people, maybe sooner.

What about new features ?

Starting with IPython 3.0, you can have multi-user server with Jupyter-hub. I, in particular, will be working on real-time colaborationand integration with google Drive (think of something like google doc). As it will not be Python specific, it will be under Jupyter umbrella.

Only google Drive ?

Google gave us enough money to hire me for a year. I'll do my best. With more money, more people we could go faster, have more backend. Developpent is also open, and PR, and testing welcommed !

It would be awesome if something like IPython existed for <a language> !

Well, it's not a question but It's called Jupyter. Please don't reinvent the wheel, comme talk to us, we would love to have interoperability or give you that hook you need. Who knows you can help steering the project in a better direction.

Stickers, mugs, tatoo, website, conferences...

Yep all that. But there is only 24h in a day (almost always), and we are only human after all.

Closing thought.

As usual, comment feed back and PR welcommed.

Pip Create

A long time without news

Since Last post lots of things happend, I had 2 pull requests on github to fix some typos on previous posts (now updated). We had yet another Jupyter/IPython developper meeting in Berkeley. By the way, part of IPython will be renamed Jupyter. See Fernando Perez Slides and Talk. I got my PhD on my spare time when not workin on IPython. Went to EuroSciPy, hacked in London In Bloomberg Headquarters for a week-end organized by BloomberLab, got a Post Doc at Berkeley BIDS under the direction of above cited IPython BDFL and is about to move almost 9000 km from my current location to spen a year working on improving your everyday scientific workflow from the West of United-States, I think that's not too bad.

Since I got a little more time than usual to do some python everyday since this is officially my day job I was confronted to a few pyton problems, and also sa some amazing stuff.

Eg, James Powel Ast Expression hack of CPyhton: (original tweet)

>>> e = $(x+y) * w * x
>>> e(x = $z+10, w = 20)(30, 40)
(×, [(×, [(+, [(+, [40, 10]), 30]), 20]), (+, [40, 10])])
>>> _()
80000

Give you unbound python expression that you can manipulate as you like. Pretty awesome, and would allow some neat meta programming. Woudl be nice if he could release his patch so that we could play with it.

A discussion with Olivier Grisel (sikit-learn) raised the point that a good package to deal with deprecation would be nice. Not just deprecation warning, but a decorator/context manager that takes as parameter the version of the software at which point a feature is removed and warn you that you have soem code you can clean.

Using grep to find deprecation warning don't always work, and sometime you would like a flag to just raise instead of printing a deprecation warning, or even do nothing when you are developpin aginst dev library.

And on the code side ?

Jupyter/IPython 3.0 is on its way. Slowly, we are late as usual.Recap will come at the right time.

I am working on integratin with Google Drive, and will try to add real-time synchronisation to Jupyter 1.0/IPython 4.0. IWe plan on addin other Operational Transform library backend, but workin on Google drive first got me my post doc. (Yes Google signed a Check to UC Berkeley)

Of course new project mean more packagin, and as you all know Python packaging is painfull. Especially when starting a new project you need to get all the basic files...etc.

Of course you can use project like audrayr/cookiecutter, but the you stil have the hassle to :

  • set up the GitHub repository,
  • Find a name
  • Hook Travis-CI find the annoyin setting that let you trigger build.
  • Hook up Readthedoc,
  • ... [1]
  • And so on and so forth

[1] add everything that every project should do.

I stronly believe that intelligence shoudl be in the package-manager, not the packager, and my few experiences with julia Pgk, and a small adventure with nodejs npm convinced me that there is really something wrong with Python packagin.

I'm not for a complete rewrite of it, and I completly understand the need for really custom setup script, but the complexity just to create a python package is too high. SO as a week-end hack introducing...

Pip Create

So yes I know that cookiecutter as pre- and post- hooks that might allow you to do that, but that's not the point. I just want a (few) simple command I can remember and which can do most of the heavy liftin for me.

I shamelessly call it PipCreate, because I hope that at some point in the future you will be able to just do a $pip create to enerate a new package.

So to work beyond dependencies, it just need a GitHub Private token (and maybe that you once loin to TravisCI, but haven't tested). The best thin is, if you don't have github token availlble, it will even open the browser for you at the right page. Here are teh step:

$ python3 -m pipcreate # need to create a real executable in future

Now either it grabs the github token from your keychain, or ask you for it and then does all the rest for you:

$ python3 -m pipcreate
will use target dir /Users/bussonniermatthias/eraseme
Comparing "SoftLilly" to other existing package name...
SoftLilly seem to have a sufficiently specific name
Logged in on GitHub as  Matthias Bussonnier
Workin with repository Carreau/SoftLilly
Clonning github repository locally
Cloning into 'SoftLilly'...
warning: You appear to have cloned an empty repository.
Checking connectivity... done.
I am now in  /Users/bussonniermatthias/eraseme/SoftLilly
Travis user: Matthias Bussonnier
syncing Travis with Github, this can take a while...
.....
syncing done
Enabling travis hooks for this repository
Travis hook for this repository are now enabled.
Continuous interation test shoudl be triggerd everytime you push code to github
/Users/bussonniermatthias/eraseme/SoftLilly
[master (root-commit) f4a9a5d] 'initial commit of SoftLilly'
 28 files changed, 1167 insertions(+)
 create mode 100644 .editorconfig
 create mode 100644 .gitignore
 create mode 100644 .travis.yml
 create mode 100644 AUTHORS.rst
 create mode 100644 CONTRIBUTING.rst
 create mode 100644 HISTORY.rst
 create mode 100644 LICENSE
 create mode 100644 MANIFEST.in
 create mode 100644 Makefile
 create mode 100644 README.rst
 create mode 100644 docs/Makefile
 create mode 100644 docs/authors.rst
 create mode 100755 docs/conf.py
 create mode 100644 docs/contributing.rst
 create mode 100644 docs/history.rst
 create mode 100644 docs/index.rst
 create mode 100644 docs/installation.rst
 create mode 100644 docs/make.bat
 create mode 100644 docs/readme.rst
 create mode 100644 docs/usage.rst
 create mode 100644 requirements.txt
 create mode 100644 setup.cfg
 create mode 100755 setup.py
 create mode 100755 softlilly/__init__.py
 create mode 100755 softlilly/softlilly.py
 create mode 100755 tests/__init__.py
 create mode 100755 tests/test_softlilly.py
 create mode 100644 tox.ini
Counting objects: 32, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (25/25), done.
Writing objects: 100% (32/32), 13.41 KiB | 0 bytes/s, done.
Total 32 (delta 0), reused 0 (delta 0)
To git@github.com:Carreau/SoftLilly.git
 * [new branch]      master -> master

It finishes up by openin travis on the test page after a push so that you can see the first test passin after a few seconds.

Hope that will give you soem ideas and patch welcomes.

Anyway, late now, Have fun, PR if you see typos are welcommed too.

PyCon 2014

Pycon 2014

Almost two weeks ago was the beginning of PyCon 2014, I took a break durring the writting of my dissertation to attend, end even if I was a little concerned because it was my first non-scientific-centric Python conference that I attend, I am really happy to have made the choice to go.

I'll write what hopefully will be a (short) recap of (my view) of the conference. Maybe not in a perfect chronological order though.

Getting There

So this year, PyCon Us was... in Montreal Canada, 7 hour flight from where I leave, in a seat narrower than me, and I'm not that wide. Hopefully there was video on demand. I watched Gravity, but I must say I was really desapointed. Beeing a physicist, I saw a lot of mistakes that ruinded the movie. Guys, it is worthless to program an engine that simulate physics if you gave it the wrong rules. From basic physics classes you will learn the following :

  • Angular momentum conservation also apply in space.
  • There is no buoyancy convection, hence no "flame" like the one on earth.
  • Usually the law of physics don't change with time.

Landing in Canada, Custom were more thorough than US custom. They actually searched what PyCon was, and asked me question about it.

Going from Airport to Center Montreal was quite easy, direct by bus 747, finding the right stop was harder has the bus driver was announcing stop number, and not name of the station, and no map was availlable in the bus.

Hotel was a block away from conference center (but not that easy to find when you enter by the given street adress and not the main entrance).

So I was finally settle around 18 hours before the beginning of the tutorials, tweeted to know if anybody was around, meet with Agam, get dinner, sleep.

Tutorials

Durring the first day I was able to attend two tutorials, which I both enjoyed, and allowed me to adjust to the right timezone. 1h lunch is just enough to meet with people and discuss, hopefully you get (some) time to discuss and meet people in the evening.

I catched up with Fernando who was giving the IPython tutorial the next day in the morning.

Btw, do not forget to update your phones app before leaving the country, especially the one you want to use that require textto send you a text when you are abroad and do not get roaming.

The IPython tutorial on thursday went great, except some really weird installation issues where :

  • Anaconda did refuse to install because it is already installed but nowhere on the hardrive, and in the end, you force miniconda install and do all the dependency by hand.

  • Of course you brought Pendrives to avoid user to download the 300Mb install on conference wifi, but you forgot the Win-Xp-32bit installer.

I was really suprised to see people still using IPython 0.12, and modal interface seem to make sens for people having never used IPython.

1h lunch too short, just extend it.

One thig that really make PyCon so great, (and the python comunity in general are the people) I basically missed many tutorial and talk during the week only because of discussing with people. I must that I learned a lot by discussing with Nick Coghlan and Fernando Pérez (ok, as a non native english I might have been really quiet, but learned a lot, and I'm gettign better, I'm starting to hear people accents).

If you want something on python core, you need to be persistant. By default, anything you will propose to the CPython core comunity/PSF will be refused, but you need to perseverate. If you perseverate then you show the community that you are ready to put time into supporting what you are proposing.

Also keep in mind that even if there are obvious issues (haem packaging) people are working on it, you probably don't know the implication :

hal fixing light switch

So please either be patient, or help them with the most basic thing missing : documenting ! Has seen later in the conference the CPython/packaging/docummenting/pip that once were difficult to distinguish becomme separate spaces, where for now people tend to be the same, but things are shifting.

Becomming member of the PSF is now also open to everybody, so you can know get even more involve into Python.

Thursday evening reception

Even after missing one tutorial afternoon can get busy, volunteers packed stuff in bags for the (real) oppenig of the conference the day after. It is also the time were sponsors are starting to set-up their booth and were you can talk to people before the big affluence. Usually the time also to finally see in flesh and blood people you only know by their twitter/github/... handle. Often it even takes you a day or two find out you already know each other.

Unlike other meals, there was (a limited number per person of) free beer, still found a little annoying that you had to pay for soft drink afterward. It is also the right time to look for the different competions around the different booth to win stuff. This goes from Linked-in that has this informal "do this puzzle as fast as you can", to Google were you actually had five minutes chrono to program an AI for your snake in a "Snake Arena" game.

Reception ended pretty early as teh conference had not already started, but if you plan on arriving the day before, it is a must attend if you want to start meeting with people and start adjusting to the timezone when you are flying from further west.

Still empty stomach (except for 3 beers), I met with Jess Hamrick (which describe her version of PyCon here) to eat dinner, before crashing to sleep. I won't repeat was jess has said as she is much more skillful with words than I am.

Core of the conference

I really appreciated the morning during the main conference as the breakfast was served in the convention center, which allow you to socialize early in the morning.

In [15]:
from IPython.display import Image
Image('img/breakfast.jpg', width=600)
Out[15]:

As you can see there is room for plety of people !

I wont spend too much time on the keynotes and talk that were presented, they are already all uploaded on PyVideo / Youtube and I must say that the PyCon team in charge of filming/organising the events did a pretty awesome job !

In general I felt like PyCon was really different from SciPy first by the number of people that are present which is huge :

In [4]:
from IPython.display import Image
Image('img/crowd.jpg')
Out[4]:

Main conference room before keynote (stolen from twitter at some point, if you re-find the author plese tell me so that I could update the post)

Also unlike SciPy, PyCon is much less axed toward science, which both gave me the opportunity to discover lots of project and found out that in the end, IPython is not that used and known. Having been once in Texas and once in Canada, in my own experience conference food during break is alway better in Austin than in Montreal.

In evening we either went to restaurant, or to party organised by Speakers or Sponsors. Note that even if it is great to have open bar (and not cheap alchool, but nice wine) and unlimited cake, I felt like real food was missing.

Sprints

When going to another continent, I usually prefer to optimise and stay as much as I can, hence I also stayed for sprint (I'll write my PhD dissertaion later). In the other hanf Fernando Perez is a Busy man and only was present for the first day and a half. Hence we were not supposed to have a real IPython sprint for the first day.

So we arrived at the sprint on monday morning off the cuff, ready to have only a few people for the sprint. To our surprise after 1h, we were around 20 around the table with our number growing. We were more and more in need for a blackboard to a globab explanation of the IPython internal. Hopefully our hacker spirit lead us to the back of a pycon sign we steal and use the back of the blackboard:

In [8]:
Image('img/fperez-pycon-sign.jpg', width=500)
Out[8]:
In [16]:
Image('img/fperez-whiteboard.JPG')
Out[16]: