# No Pylab Thanks

Please Stop using Pylab¶  thanks to Randy Olson for sending me a PR with grammar and spelling corrections. TL;DR, Please stop advising people to use the pylab flag when using IPython. It is harmful. If you want to help IPython, try to avoid retweeting, and promoting things that use the pylab flag and make the authors aware of the issues. Use explicit import, and %matplotlib inline magic. It is better (and supports switching between inline/not inline.) This was mainly prompted by a day where I came across consecutive issues due to the pylab flag, and people happy to discover it. Pylab, my worst friend¶ When IPython 1.0 was released almost 6 months ago, we had quite a few decisions to make in less than a week and loads of discussions at the last moment. One of those decisions comes and goes often: we want to get rid of this stupid pylab flag. If you look at our stable (1.0) dev doc and examples there shouldn't be a mention of the pylab flag anywhere. If there is, we will quickly remove it. Why? Because it is harmful, first to us, then to new users, to the Python community, and finally to research in general. Do you know about pylab? If you don't, please be extra careful and try to avoid it as much as possible. It is like smoking: it looks cool at first, then after a few years you realise you can live without it, but it makes you sick. You think you know what pylab does? Really? Take few seconds to think what the pylab flag is doing, then read the rest. What is it supposed to do?¶ The pylab package main purpose was to build a transition tool from other languages to Python. As it was more and more common and painful to do the same import in IPython every time when only the IPython shell was around, the --pylab flag was added. Basically, it did the following: import numpy import matplotlib from matplotlib import pylab, mlab, pyplot np = numpy plt = pyplot from IPython.core.pylabtools import figsize, getfigs from pylab import * from numpy import * Did you get it right the first time without cheating ? Are you able to say what has been imported in from pylab import * ? in from numpy import * ? Of course, that is not the only thing it does. But was it your intention to do that the last time you used pylab? It is irreversible¶ Once you activate pylab mode, there is no going back. You cannot unimport things. Of course, with the %pylab magic you can always restart your kernel, but with the flag, all your kernels will start in pylab mode. Are you sure you will not need a non-pylab kernel? Unclear¶ When using the pylab flag, your audience usually has no way of knowing that you used the flag. If I use plot(range(10)), what will happen? Will it pop up a figure ? Inline it? Or throw an error? In [ ]: plot(range(10)) Because I'm mean, I won't execute the cell, so that you don't know whether or not I'm running in pylab mode or not. You might think it's not a big deal, but in teaching and research it is important to communicate exactly what you are doing. It pollutes the namespace¶ In [1]: len(get_ipython().user_ns.keys()) Out[1]: 20 In [2]: %pylab Using matplotlib backend: Agg Populating the interactive namespace from numpy and matplotlib IPython 2.0-dev gives some warning, and will tell you wether it clobbers already non-built-in variables in the user namespace. In [3]: len(get_ipython().user_ns.keys()) Out[3]: 968 Can you really tell me the 948 additional things you now have in your namespace? Not a big deal? Really? It replaces built-ins¶ In [6]: sum,all Out[6]: (, ) Both used to be , before using pylab, and this changes the behavior of your programs! For example, it leads to non-pickable object in some cases: @treycausey on Twitter @ogrisel @myusuf3 Ah, very curious. It loads fine in IPython but not in an IPython notebook. I must have a namespace collision or something. Me: Let me guess: $ipython notebook --pylab? But$ipython alone? Other unrelated actions¶ IPython developed and added the rich display protocol, so we added from IPython.display import display to what import pylab does. Because matplotlib can work with different event loops, pylab added options to activate qt/gtk/osx/etc. event loops, and with the QtConsole and Notebook creation, the ability to select the inline backend, which registers display hooks. The main things users remember, and often the main message that you can read here and there is that: You need to use Pylab to get inline figures. Which is false. Inline pylab mode is a convenient method to activate inline, and set up a display hook with matplotlib. You do not need pylab to have inline images, nor do you need matplotlib. With 1.0 and above the recommended way to set up inline figures would be to use %matplotlib. Side Effects¶ The worst part is it has the side effect of making people use pylab without even knowing it. Later on, they start asking question about how to reuse a graph in matplotlib because no one ever encounters the following code any more: fig,ax = subplots(1,1) ax.plot(...) or why their script suddenly does not work in plain Python but works in IPython. It kills kittens, with a spoon...¶ Maybe not kittens, but it will force developers to explain the same things, again, and again, and again... In [8]: from IPython.display import YouTubeVideo YouTubeVideo('9VDvgL58h_Y',start=4*60+27) Out[8]: There was a problem loading this remote IFrame, have a look as Javascript console Make --pylab disappear.¶ Of course, we will not remove the --pylab flag for compatibility reasons, but please, for 2014, make the following resolutions: Use %matplotlib [inline|qt|osx|gtx] to select the right backend/event hook. Use explicit imports Make people aware of the issues Help by making no new articles/tutorials that mention --pylab If really you only have 10 seconds left, the %pylab magic is fine. As usual, this has been written in an IPython Notebook. You can send me pull request if I made mistakes, or if you have any additional remarks on why you shouldn't use --pylab.

# 07-the-sound-of-hydrogen

The sound Of Hydrogen¶ Inspired by minutephysics, and the explanation do do it in mathematica: The sound of hydrogen. The goal of this notebook is to show how one can play a sound file in notebook, using Html5

# XKCD plots in Matplotlib¶

This notebook originally appeared as a blog post at {{ super() }}

{%- endblock codecell %} Try to look at what Jinja can do, thenlearn about Jinja Filters and imagine they can magically read your config file. For example we provide a filter that highlight by presupposing code is Python. Or one that wraps text at a default length of 80 char... Want a rot13 filter on some codecell when doing exercises for student ? See you next time ! More...¶ One more example from one Pull-Request. In [20]: from IPython.nbconvert.filters.highlight import _pygment_highlight from pygments.formatters import HtmlFormatter from IPython.nbconvert.exporters import HTMLExporter from IPython.config import Config from IPython.nbformat import current as nbformat def my_highlight(source, language='ipython'): formatter = HtmlFormatter(cssclass='highlight-ipynb') return _pygment_highlight(source, formatter, language) c = Config({'CSSHtmlHeaderTransformer': {'enabled':True, 'highlight_class':'highlight-ipynb'}}) exportHtml = HTMLExporter( config=c , filters={'highlight': my_highlight} ) (body,resources) = exportHtml.from_notebook_node(jake_notebook) In [21]: from jinja2 import DictLoader dl = DictLoader({'html_full.tpl': """ {%- extends 'html_basic.tpl' -%} {% block footer %} FOOOOOOOOTEEEEER {% endblock footer %} """}) exportHtml = HTMLExporter( config=None , filters={'highlight': my_highlight}, extra_loaders=[dl] ) (body,resources) = exportHtml.from_notebook_node(jake_notebook) for l in body.split('\n')[-4:]: print l

This post was written entirely in an IPython Notebook: the notebook file is available for download here. For more information on blogging with notebooks in octopress, see my previous post on the subject.

FOOOOOOOOTEEEEER

# 05-YAML Notebook

YAML IPython notebook¶ Little experiment base on the fact that apparently YAML is made to be better readable by Humans than JSON. We've also had some complaint that metadata are not keep in nbconvert when roundtripping through markdown, those two made me think that I could try to see what ipynb files stored as YAML would look like. I'll also use this post to do some experiment for nbviewer future nbviewer features, if you see anything wrong with the css on some device, please tell me. First atempt¶ Apparently Json is a subset of YAML: cp foo.ipynb foo.ipyamlnb Yeah, Mission acomplished ! Second try¶ Install PyYaml, and see what we can do. In [42]: import json import yaml In [43]: from IPython.nbformat import current as nbf In [44]: ls Y*.ipynb YAML Notebook.ipynb In [45]: with open('YAML Notebook.ipynb') as f: nbook = nbf.read( f, 'json') In [46]: nbook.worksheets[0].cells[9] Out[46]: {u'cell_type': u'code', u'collapsed': False, u'input': u'from IPython.nbformat import current as nbf', u'language': u'python', u'metadata': {}, u'outputs': []} I'll skipp the fiddling around with the yaml converter. In short, you have to specify explicitely the part you want to dump in the literal form, otherwise they are exported as list of strings, which is a little painfull to edit afterward. I'm using the safe_dump and safe_load methods (or pass safeLoader and Dumper). Those should be default or otherwise you could unserialise arbitrary object, and have code exucuted. We probably don't want to reproduct the recent file Rail's critical vulnerability that append not so long ago. In [47]: # we'll patch a safe Yaml Dumper sd = yaml.SafeDumper # Dummy class, just to mark the part we want with custom dumping class folded_unicode(unicode): pass class literal_unicode(unicode): pass I know classes should be wit upper case, but we just want to hide the fact that thoses a class to end user. At the same time I define a folded method to use it with markdown cell. when markdown contain really long lines, those will be wrapped in the yaml document. In [48]: def folded_unicode_representer(dumper, data): return dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='>') def literal_unicode_representer(dumper, data): return dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='|') sd.add_representer(folded_unicode, folded_unicode_representer) sd.add_representer(literal_unicode, literal_unicode_representer) with open('YAML Notebook.ipynb') as f: nbjson = json.load(f) now we patch the part of the ipynb file we know we want to be literal or folded In [49]: for tcell in nbjson['worksheets'][0]['cells']: if 'source' in tcell.keys(): tcell['source'] = folded_unicode("".join(tcell['source'])) if 'input' in tcell.keys(): tcell['input'] = literal_unicode("".join(tcell['input'])) In [50]: with open('Yaml.ipymlnb','w') as f: f.write(yaml.dump(nbjson, default_flow_style=False, Dumper=sd)) You can round trip it to json, and it's still a valid ipynb file that can be loaded. Haven't fiddled with it much more. There are just a few gotchas with empty lines as well as trailing whitespace at EOL that can respectively diseapear or make the dumper fall back to a string quoted methods to store values. You can skip down to the end of this notebook to look at how it looks like. It's probably much compact than the current json we emit, in some cases it might be more easy to read, but I don't think it is worth considering using in the format specification. ipynb files are ment to be humanely fixable, and I strongly prefere having a consistent format with simple rules than having to explain what are the meaning of the differents shenigan like : |2+ for literal string. Also support across languages are not consistent, and it would probably be too much of a security burden for all code that will support loading ipynb to take care of sanitazing Yaml. One area where I woudl use it would be to describe the ipynb format at a talk for example, and/or to have metadata editing more human readable/writable. In [51]: !cat Yaml.ipymlnb metadata: name: YAML Notebook nbformat: 3 nbformat_minor: 0 worksheets: - cells: - cell_type: heading level: 1 metadata: {} source: >- YAML IPython notebook - cell_type: markdown metadata: {} source: "Little experiment base on the fact that apparently YAML is made to be\ \ better readable by Humans than JSON.\nWe've also had some complaint that metadata\ \ are not keep in nbconvert when roundtripping through markdown, those two\n\ made me think that I could try to see what ipynb files stored as YAML would\ \ look like. " - cell_type: heading level: 4 metadata: {} source: >- First atempt - cell_type: markdown metadata: {} source: >- Apparently Json is a subset of YAML: - cell_type: markdown metadata: {} source: >2+ cp foo.ipynb foo.ipyamlnb - cell_type: markdown metadata: {} source: >- Yeah, Mission acomplished ! - cell_type: heading level: 4 metadata: {} source: >- Second try - cell_type: markdown metadata: {} source: "Install PyYaml, and see what we can do. " - cell_type: code collapsed: false input: |- import json import yaml language: python metadata: {} outputs: [] - cell_type: code collapsed: false input: |- from IPython.nbformat import current as nbf language: python metadata: {} outputs: [] - cell_type: code collapsed: false input: |- ls Y*.ipynb language: python metadata: {} outputs: [] - cell_type: code collapsed: false input: |- with open('YAML Notebook.ipynb') as f: nbook = nbf.read( f, 'json') language: python metadata: {} outputs: [] - cell_type: code collapsed: false input: |- nbook.worksheets[0].cells[9] language: python metadata: {} outputs: [] - cell_type: markdown metadata: {} source: >- I'll skipp the fiddling around with the yaml converter. In short, you have to specify explicitely the part you want to dump in the literal form, otherwise they are exported as list of strings, which is a little painfull to edit afterward. I'm using the safe_dump and safe_load methods (or pass safeLoader and Dumper). Those should be default or otherwise you could unserialise arbitrary object, and have code exucuted. We probably don't want to reproduct the recent file Rail's critical vulnerability that append not so long ago. - cell_type: code collapsed: false input: |- # we'll patch a safe Yaml Dumper sd = yaml.SafeDumper # Dummy class, just to mark the part we want with custom dumping class folded_unicode(unicode): pass class literal_unicode(unicode): pass language: python metadata: {} outputs: [] - cell_type: markdown metadata: {} source: >- I know classes should be wit upper case, but we just want to hide the fact that thoses a class to end user. At the same time I define a folded method if I want to use it later. - cell_type: code collapsed: false input: |- def folded_unicode_representer(dumper, data): return dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='>') def literal_unicode_representer(dumper, data): return dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='|') sd.add_representer(folded_unicode, folded_unicode_representer) sd.add_representer(literal_unicode, literal_unicode_representer) with open('YAML Notebook.ipynb') as f: nbjson = json.load(f) language: python metadata: {} outputs: [] - cell_type: markdown metadata: {} source: >- now we patch the part of the ipynb file we know we want to be literal or folded - cell_type: code collapsed: false input: |- for tcell in nbjson['worksheets'][0]['cells']: if 'source' in tcell.keys(): tcell['source'] = folded_unicode("".join(tcell['source'])) if 'input' in tcell.keys(): tcell['input'] = literal_unicode("".join(tcell['input'])) language: python metadata: {} outputs: [] - cell_type: code collapsed: false input: |- with open('Yaml.ipymlnb','w') as f: f.write(yaml.dump(nbjson, default_flow_style=False, Dumper=sd)) language: python metadata: {} outputs: [] - cell_type: markdown metadata: {} source: >- You can round trip it to json, and it's still a valid ipynb file that can be loaded. Haven't fiddled with it much more. There are just a few gotchas with empty lines as well as trailing whitespace at EOL that can respectively diseapear or make the dumper fall back to a string quoted methods to store values. One could also try to tiker with folded_unicode in markdown cell that tipically have long lines to play a little more nicely with VCS. - cell_type: markdown metadata: {} source: >- You can skip down to the end of this notebook to loko at how it looks like. It's probably much compact than the current json we emit, in **some** cases it might be more easy to read, but I don't think it is worth considering using in the format specification. ipynb files are ment to be humanely fixable, and I strongly prefere having a consistent format with simple rules than having to explain what are the meaning of the differents shenigan like : |2+ for literal string. Also support across languages are not consistent, and it would probably be too much of a security burden for all code that will support loading ipynb to take care of sanitazing Yaml. One area where I woudl use it would be to describe the ipynb format at a talk for example, and/or to have metadata editing more human readable/writable. - cell_type: code collapsed: false input: |- !cat Yaml.ipymlnb language: python metadata: {} outputs: [] metadata: {}

# 04-initialisation-cell

IPython Notebook Duck-Punching¶ If it walks like a duck and talks like a duck, it’s a duck. So if this duck is not giving you the noise that you want, you’ve got to just punch that duck until it returns what you expect. Small blogpost to answer to one question that has been asked on stackoverflow and on the Issue tracker : When I open a saved IPython Notebook, I need to evaluate all the cells with imports, function definitions etc. to continue working on the session. It is convenient to click Cell > Run All to do this. But what If I do not want to re-evaluate all calculations? Do I need to pick the cells to evaluate by hand each time? [There is] the concept of "initialization cells". You can mark some cells in the notebook as initialization cell, and then perform "evaluate initialization cells" after opening the notebook. This feature should allow certain cells to be marked as Initialization Cells and be evaluated together with the appropriate command. I'll let you get there to read the official answer, but in short : Not in core IPython, this can be done as an extension. Some warning before we start. As long as you don't shut down the IPython webserver, or don't ask for explicit shutdown, a notebook kernel stay alive, meaning that you can leave the page and come back, you wil still have the same active namespace. So you might not want to run those init cell again. Second do not ever, in any case, whatever reason you have, automatically run initialisation cell on notebook load. It is a security risk : If you have such an extension and I send you a notebook with a Initialisation Cell that have rm -rf ~/ in it. You just lost your home folder when openning this notebook. So this will show you how to to that in less than 60 lines of javascript, you might want to take a look at previous post for some info. You know where to find the finished code, as usual it is not perfect, and I wait for PR to fix the edge cases. Let's start. Overview¶ This extension will be in two part even the all thing will fit in one file. So let's decompose what we need. The ability to mark a cell with a certain flag (Initialisation Cell) The ability run all those initialisation cell when needed For me this sound like the need for a custom CellToolbar checkbox, and a Toolbar button 'Run init Cell'. A checkbox because this is typically a Boolean statement, the cell is a Initialisation cell, or not. And the toolbar button is the easiest to reach, and not too complicated to add. The CellToolbar checkbox¶ We will store the boolean of wether of not the cell is an Initialisation Cell in the cell Metadata. To stay clean, we will use a prefixed key to avoid future collision. As this is a draft, I will prefix the name of the key with an underscore to warn future user that directly accessing this key is not supported and that I reserved myself the right to change anything without warning. I will store [true|false|undefined] in cell.metadata._draft.init_cell. I will not forget to check that cell.metadata._draft exist before playing with it. The IPython notebook provide a convenient API to generate checkbox in Cell Toolbar. to use this we need to define a getter and a setter for our metadata. The setter take the cell we act on, and the new value: function(cell, value){ // we check that the _draft namespace exist and create it if needed if (cell.metadata._draft == undefined){cell.metadata._draft = {}} // set the value cell.metadata._draft.init_cell = value } The getter is not much more complicated : function(cell){ var ns = cell.metadata._draft; // if the _draft namespace does not exist return undefined // (will be interpreted as false by checkbox) otherwise // return the value return (ns == undefined)? undefined: ns.init_cell } The api to generate the checkbox has the following signature : CellToolbar.utils.checkbox_ui_generator(label, setter, getter) I can then create my function easily: var CellToolbar= IPython.CellToolbar; var init_cell = CellToolbar.utils.checkbox_ui_generator('Initialisation Cell', // setter function(cell, value){ // we check that the _draft namespace exist and create it if needed if (cell.metadata._draft == undefined){cell.metadata._draft = {}} // set the value cell.metadata._draft.init_cell = value }, //getter function(cell){ var ns = cell.metadata._draft; // if the _draft namespace does not exist return undefined // (will be interpreted as false by checkbox) otherwise // return the value return (ns == undefined)? undefined: ns.init_cell } ); The label will be use in the UI to put a name in front of the checkbox to know its use. So I used a descriptive name. Now we need to register our function, for that we will use CellToolbar.register_callback(name, function);. name should be a string we will use to refer to the function later, in order to use it in multiple place if we wish. Here simply CellToolbar.register_callback('init_cell.chkb', init_cell); And finaly, we use a private method (for now) to generate a CellToolbar preset that can be chosen by the User in the CellToolbar dropdown with : CellToolbar.register_preset(label, ['callback_name','callback_name',....]). This allow to simply mix and match ui elements from different preset for customisation. Here we only have one checkbox so we do: CellToolbar.register_preset('Initialisation Cell', ['init_cell.chkb']); With all the extension I have, I could create a custom CellToolbar simply by adding: CellToolbar.register_preset('My toolbar', ['init_cell.chkb','default.rawedit','slideshow.select']); And you can see below how it looks like In [19]: from IPython.display import Image Image(filename='/Users/bussonniermatthias/Desktop/Ctoolbar.png') Out[19]: Now you just need to select the Initiallisation Cell CellToolbar and check the checkbox you wish. The Toolbar button¶ Now we need a way to run all cells marked as Initialisation Cells. Let's first make a function that loop on all cell and run is they are marked: var run_init = function(){ var cells = IPython.notebook.get_cells(); for(var i in cells){ var cell = cells[i]; var namespace = cell.metadata._draft|| {}; var isInit = namespace.init_cell; // you also need to check that cell is instance of code cell, // but lets keep it short if( isInit === true){ cell.execute(); } } }; Now we use the API to create a function that register a callback on a button click on the main toolbar, we use a descriptive label that shows on hower, on Icon from jQuery UI themeroller, and assign the callback to previous defined function: var add_run_init_button = function(){ IPython.toolbar.add_buttons_group([ { 'label' : 'run init_cell', 'icon' : 'ui-icon-calculator', 'callback': run_init } ]); }; Now we just need to run this function late enough to have effect. We will just listen for the loaded even to trigger this : $([IPython.events]).on('notebook_loaded.Notebook',add_run_init_button); That's it. You just have to put all this stuff in one big file, name it correctly and add the$.getScript in your custom.js. Reload your page/Restart your server (depending on the config you choose). And you should have a new toolbar preset and a button to run your initialisation cell. I've got less than 60 lines of javascript comment and blank lines counted. As this is a separated file, you can easily fork it and add modifications/options. I'm waiting for PRs. I hope this show you that writing extension for the IPython notebook is not that hard, and are easy to share. In [ ]:

# 03-on-notebook-format

IPython Notebook Duck-Punching (II)¶ If it walks like a duck and talks like a duck, it’s a duck. So if this duck is not giving you the noise that you want, you’ve got to just punch that duck until it returns what you expect. Some thoughts on the IPython notebook. For once, this will discuss more what are the possible things than can be done with IPython, with only few really usable code. I hope it will help the reader to better understand in what direction the Core Team whant to drive IPython. This is of course my personnal interpretation of the different points. We discussed since I came on board. But I hope it will help the comunity to feel.concern and participate. Terminology¶ I think one of the first things that should be adressed is the terminology of things. As a non native english speaker I am pretty bad at choosing names for things. And I probably have an horible.english. And I think there is a big problem the Naming in the IPython notebook. The problem is the same as what happend with the terme "gene", at some point in history it was used to explain a specific effect that some phenotype where expressed depending on the genes you had, the problem is that with our current technologie an knowledge on genetics, the boundaries of what a gene is becommes fuzzy. Same apply to the IPython notebook. Let's see. You probably can start the IPyrhon notebook from your command line. You can also ask a friend to send his or her IPython notebook by email. You probably know that shift enter execute a cell in an IPython notebook. But you probably don't send a webserver by email. I haven't seen anyone converting GMail to a power point. I think you see the point. "IPython notebook" represent at the same time: The webserver application that yoi start from the command line, the web frontend that you are used to use, the file format that defines the structures of ipynb files, the actual files you exchanges, and the representations of thoses files into the web applocation. Add on top the fact that kernel can speek to multiple frontend and can talk to non-python kernel, and people get lost. Now you wonder why you should care as you are only using the all bundle? Because you will probably what to share and or reuse what you write, and once you understand the differents things.you will be able to do even more amazing things. The part you don't (really) care about. The webserver. It can mostly be seen as a middleware. The configuration options should be left to the sysadmin. It mostly make the ZMQ/websocket bridge, and is also an endpoint for the web application to access remote filesystem (yes remote, even if localhost). The part you can do without. The kernel. You probably had one or twice a kernel that died In [ ]: