ipython - Python

chapter 2

iPython

Choose Python to see the code.

Choose Python to see the code.

Introduction to Python (https://www.python.org/doc/).

Python


# Python 3: Fibonacci series up to n
 def fib(n):
     a, b = 0, 1
     while a < n:
         print(a, end=' ')
         a, b = b, a+b
     print()
 fib(1000)

IPython-notebook

  • IPython notebook is used to analyse data and for data visualisation.

  • IPython is the component in the toolset that ties everything together; it provides a robust and productive environment for interactive and exploratory computing.

  • ipython noteboook uses a client-server model. This makes it possible to interact with ipython from several different environments. For example, emacs or a web browser.

For more info see: https://ipython.org

Two other key components are Jupyter Notebooks and Anaconda. Jupyter provides Mathematica like notebooks and Anaconda is a package management system.

Jupyter Notebooks, originally called IPython Notebooks,and it commonly used for improving the reproducibility and accessiblity of scientific research.

To activate another kernel environment with python 3 do the following:

First check jupyter-ipython kernel list by typing the following:

Check python kernel list

jupyter kernelspec list

in case you dont see python3 or other python kernel you can create a new one.

To create a new environment do the following:

To create a new environment do the following:

conda create -n py36 python=3.6
source activate py36
python -m ipykernel install --name py36

then type the below to begin an ipython session:

type the below to begin an ipython session:

jupyter notebook

Have fun!!

Other math/science/data oriented Python tools

  • Scikit - machine learning

  • Scikit-image & PIL/Pillow - image processing

  • Blaze - data transformation pipelines & simplified interactions with various data stores

  • Bokeh - Interactive web visualisations

  • Sympy - symbolic algebra (also see Sage)

  • YT - for analysing and visualising volumetric data

  • Numba - a very easy to use JIT compiler (just import it and put @jit annotation on functions you want compiled) and for dealing with genuinely big data there is PySpark and Ibis.

Install ipython on emacs:

First install anaconda: https://www.continuum.io/downloads check your python version in terminal python --version i.e 3.5, and download anaconda3.

After downloaded anaconda open terminal and cd to anacoda3 directory and type:

Choose Shell to see the code

bash Anaconda3-4.3.0-MacOSX-x86_64.sh

press yes for anaconda3 to add the PATH to your .bash_profile

The next step is to:

  • install ipython on emacs. One of the packages is called ein and you can install it through melpa.

copy ein.el and ein.py to your emacs upload directory

  • open .emacs.d and write

Choose emacs-lisp to see the code

(require 'ein)

Start IPython notebook server. Go to terminal and write: jupyter notebook then copy the token and paste it as the password to login to the server.

On emacs hit M-x ein:notebooklist-login and press return to use the localhost:8888, server and use the token (password) to login.

i.e password: 8b6cae64f7dbcfc425a2dsf30cretfdfc7d730dcba9180ab8

Term output example

Choose Shell to see the output


[I 01:49:54.596 NotebookApp] Serving
notebooks from local directory: /Users/usr_name
[I 01:49:54.596 NotebookApp] 0 active kernels
[I 01:49:54.597 NotebookApp] The Jupyter Notebook is running at:
http://localhost:8888/?token8b6cae64f7dbcfc425a2dsf30cretfdfc7d730dcba9180ab8
[I 01:49:54.597 NotebookApp] Use Control-C to stop this server and shut
down all kernels (twice to skip confirmation).
[C 01:49:54.626 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:

  http://localhost:8888/?token=8b6cae64f7dbcfc425a2dsf30cretfdfc7d730dcba9180ab8

If you successfully logged in to the server;

Hit M-x ein:notebooklist-open to open the notebook list. This will open a notebook list buffer.

In the notebook list buffer, you can open notebooks by hitting [Open], [Dir] for directories, create new notebook [New notebook], delete notebook [Delete].

NOTE You can also check ob-python package for source code block ipython in org-mode

You can start testing ipython using these examples: particle-physics-playground-playground-52de62d

CERN examples: particle-physics-playground-playground-52de62d

Sonifying ems (muons) - ipython - SuperCollider in emacs

Editor: Emacs Version 24.5 (9.0)

Ipython package ein on MELPA

SuperCollider 3.7

Data sonification experiment on particle-physics-playground.

For more info see here:

particle-physics-playground

  • Sending OSC messages to other application

to send osc messages to other application install python-osc library

In this case I use SuperCollider port 57120

Choose SuperCollider

// BA 28022017
// Testing osc communication - Receiving data from ipython - 'CMS' (Compact Muon Solenoid) __

s.boot // boot the server
s.record // record
s.stopRecording // stop recording


// create synthdef
(
SynthDef(\ipythontest, {|
	freq = 440, gate = 1,
amp = 0.5, out = 0|
	var env, source;

	env = EnvGen.kr(Env.adsr, 1, doneAction:2);

	//source = SinOsc.ar(freq*2, 0, amp);
	source = SinOsc.ar(SinOsc.ar(freq*2, freq*4, freq*2), 0,  amp);
	// source  = UseWhateverGen.ar();


	Out.ar(out, Pan2.ar(source*env, 0))!2
}).add;

~x=Synth(\ipythontest, [\freq, 440, \amp, 0.5]); // run the synth


// set osc

~a = OSCdef(\oscTest,
	{
			| ... msg | msg.postln;

			~x.set(\freq, msg, \amp, 0.9);
		//~muons = msg [0] [1..];
		//~muons.postln;



		// use the osc messages (msg) for the frequency
	},
	'/print' // OSCmessage name
);
)

ipython notebook

Choose Python


#VA_exp_280217_001


#Import libraries numpy, matplotlib, pythonosc

In [1]
import numpy as np
import matplotlib.pylab as plt

from IPython import get_ipython
get_ipython().run_line_magic('matplotlib', 'inline')

In [2]
#from __future__ import print_function
#from __future__ import division
import sys

sys.path.append("../particle-physics-playground-Sonification-Example_001/tools/")

#from draw_objects3D import *
import cms_tools as cms

In [3]
infile = open('../particle-physics-playground-Sonification-Example_001/data/small_cms_test_file.dat')

collisions = cms.get_collisions(infile)

number_of_collisions = len(collisions)
print ("# of proton-proton collisions: %d" % (number_of_collisions))


# of proton-proton collisions: 10


In [4]
print (collisions[0])


[[[88.9127, 32.9767, -75.1939, 29.541, -1.0], [79.2211, -58.6558, 49.1723, 13.5915, -1.0], [43.313, -5.9129, 40.0892, 12.0431, -1.0], [274.8094, -21.4194, 27.5639, -272.4152, -1.0], [26.6201, 0.5268, -24.7563, -7.4046, 0.0]], [[15.7375, 1.4793, -15.2566, -3.5645, -1]], [], [[52.364, 17.4983, -45.4233, 19.3009], [10.2904, -1.4633, 10.0887, 1.4035]], [44.9415, 0.422]]


In [5]

print (len(collisions[0]))

5


In [6]
METx = collisions[0][4][0]
METy = collisions[0][4][1]

print ("MET x: %f" % (METx))
print ("MET y: %f" % (METy))

MET x: 44.941500
MET y: 0.422000


In [7]
print ("# of jets:      %d" % (len(collisions[0][0])))
print ("# of muons:     %d" % (len(collisions[0][1])))
print ("# of electrons: %d" % (len(collisions[0][2])))
print ("# of photons:   %d" % (len(collisions[0][3])))

# of jets:      5
# of muons:     1
# of electrons: 0
# of photons:   2


In [8]
jets,muons,electrons,photons,met = collisions[0]


In [9]
E,px,py,pz,btag = jets[0]
print ("E:     %8.4f" % (E))
print ("px:    %8.4f" % (px))
print ("py:    %8.4f" % (py))
print ("pz:    %8.4f" % (pz))
print ("btag:  %8.4f" % (btag))

E:      88.9127
px:     32.9767
py:    -75.1939
pz:     29.5410
btag:   -1.0000


In [10]
E,px,py,pz,q = muons[0]
print ("E:  %8.4f" % (E))
print ("px: %8.4f" % (px))
print ("py: %8.4f" % (py))
print ("pz: %8.4f" % (pz))
print ("q:  %8.4f" % (q))

E:   15.7375
px:   1.4793
py: -15.2566
pz:  -3.5645
q:   -1.0000


In [11]
E,px,py,pz = photons[0]
print ("E:  %8.4f" % (E))
print ("px: %8.4f" % (px))
print ("py: %8.4f" % (py))
print ("pz: %8.4f" % (pz))

E:   52.3640
px:  17.4983
py: -45.4233
pz:  19.3009


In [0]
      # Plot the quantities
plt.figure(figsize=(16,4))

plt.subplot(1,3,1)
plt.hist(njets,bins=5,range=(0,5))
plt.xlabel(r'# of jets')
plt.ylabel('# entries')

plt.subplot(1,3,2)
plt.hist(jets_E,bins=25,range=(0,400))
plt.xlabel(r'Jet energy [GeV]')
plt.ylabel('# entries')

plt.subplot(1,3,3)
plt.hist(muons_E,bins=25,range=(0,400))
plt.xlabel(r'Muon energy [GeV]')
plt.ylabel('# entries')

h1:
Watch an example

In [0]
from IPython.display import YouTubeVideo
YouTubeVideo('UfimSbOr9to')

In [13]
infile = open('../particle-physics-playground-Sonification-Example_001/data/mc_dy_1000collisions.dat')


collisions = cms.get_collisions(infile)

# We will use these to store the quantities that we will be plotting later.
njets = []
jets_E = []
muons_E = []
photons_E = []

for collision in collisions:

    jets,muons,electrons,photons,met = collision

    njets.append(len(jets))

    for jet in jets:
        E,px,py,pz,btag = jet
        jets_E.append(px)

    for muon in muons:
        E,px,py,pz,q = muon
        muons_E.append(E)

    for photon in photons:
        E,px,py,pz = photon
        photons_E.append(E)

In [18]
import time

infile = open('../particle-physics-playground-Sonification-Example_001/data/mc_dy_1000collisions.dat')


collisions = cms.get_collisions(infile)

# We will use these to store the quantities that we will be plotting later.
njets = []
jets_E = []
muons_E = []
photons_E = []

for collision in collisions:

    jets,muons,electrons,photons,met = collision

    njets.append(len(jets))

    for jet in jets:
        E,px,py,pz,btag = jet
        jets_E.append(E )

    for muon in muons:
        E,px,py,pz,q = muon
        muons_E.append(E)

    for photon in photons:
        E,px,py,pz = photon
        photons_E.append(E)


             # Set up OSC here

from pythonosc import osc_message_builder
from pythonosc import udp_client

# The port for SuperCollider is '57120'

client = udp_client.SimpleUDPClient("127.0.0.1", 57120)


#client.send_message("/print", muons_E)

# now we can print them out too

for i in muons_E:
      print ("muon was: %d" % i)
      client.send_message("/print", i)
      time.sleep(0.015)

for i in jets_E:
      print ("jet was: %d" % i)
      client.send_message("/print", i)
      time.sleep(0.015)

for i in photons_E:
      print ("photon was: %d" % i)
      client.send_message("/print", i)
      time.sleep(0.015)
# # Plot the quantities

YT visualisation

An example with enzo data

Choose Python


import os

os.chdir('/Users/experiments/yt_pics')

import yt

ds = yt.load("/Users/experiments/Enzo_64/DD0043/data0043")

sc = yt.create_scene(ds, lens_type='perspective')

# Get a reference to the VolumeSource associated with this scene
# It is the first source associated with the scene, so we can refer to it
# using index 0.
source = sc[0]

# Set the bounds of the transfer function
source.tfh.set_bounds((3e-31, 5e-27))

# set that the transfer function should be evaluated in log space
source.tfh.set_log(True)

# Make underdense regions appear opaque
source.tfh.grey_opacity = True

# Plot the transfer function, along with the CDF of the density field to
# see how the transfer function corresponds to structure in the CDF
source.tfh.plot('transfer_function.png', profile_field='density')

# save the image, flooring especially bright pixels for better contrast
sc.save('rendering2.png', sigma_clip=6.0)

For 3D modeling yt see here:

Data Visualisation 3D

Ipython - realtime data

Watching the number of flights on your emacs:

This experiment tested on python 3.5 and emacs - ipython notebook (ein).

For ipython notebook installation see this webpage ipython.

to update to python 3.6 open shel and run the below command:

conda create –name py36 python=3.6

To run this example you need to install some external modes

requests and BeautifulSoup

If you use pip (recommended) open the terminal and type

Choose Shell

$ pip install requests
$ pip install beautifulsoup4

Go to the web page to scrape the number of flights

https://www.flightradar24.com/56.16,-49.51/7

The number is updated every 8 seconds.

To be able to collect the number of flights in real time, go and find the .js file in the webpage. To find the js file go to: Chrome - more tools- developer tools - network - there you'll find the requests under the name feed.js.

Now, run the below code in you ipython notebook. (code taken from here)

Choose Python

import requests
from bs4 import BeautifulSoup
import time

def get_count():
    url = "https://data-live.flightradar24.com/zones/fcgi/feed.js?bounds=59.09,52.64,-58.77,-47.71&faa=1&mlat=1&flarm=1&adsb=1&gnd=1&air=1&vehicles=1&estimated
=1&maxage=7200&gliders=1&stats=1"

    # Request with fake header, otherwise you will get an 403 HTTP error
    r = requests.get(url, headers={'User-Agent': 'Mozilla/5.0'})

    # Parse the JSON
    data = r.json()
    counter = 0

    # Iterate over the elements to get the number of total flights
    for element in data["stats"]["total"]:
        counter += data["stats"]["total"][element]

    return counter

while True:
    print(get_count())
    time.sleep(8)

Watch here a screen capture