Python
Pythonic way of programming :
Idiomatic Python: EAFP versus LBYL
EAFP: “it’s easier to ask for forgiveness than permission”. Quickly, EAFP means that you should just do what you expect to work and if an exception might be thrown from the operation then catch it and deal with that fact. What people are traditionally used to is LBYL: “look before you leap”.
Increasing python speed with Cython variables definition
Python weird "pass reference by value".
Feature flags are a way to deal with dependencies and version compatibility with distributed services and APIs
Packaging :
Create a package Package auto discovery Pyproject.toml configuration Creating entry points (CLI interfaces) Argsparse for parsing CLI arguments Twine documentation
Why using pdm for better packaging : https://chriswarrick.com/blog/2023/01/15/how-to-improve-python-packaging/
python notebook in a pdm environement in vscode
CI/CD
CI/CD collective on stack overflow
Packaging with CI/CD :
Adding a trusted publisher on Pypi
Github CI
Github actions yaml syntax Publish packages from github CI straight to pypi package repository for CI documentation : Do not forget to turn on read and write permissions
Gitlab CI
Gitlab CI yaml syntax Predefined variables in CI/CD Secrets variables for a Group here Publish packages from gitlab CI straight to pypi package repository
Objects :
Reference counting in python : source
Variables in python source
Unit testing :
With pytest
Modules :
Built-ins :
Arguments types specification in functions and classes : def function( value : int = defaultvalue ) :
List : Sort by parameter class instance list
Built in exception types : List of them
Bytes :
Examples :
bytes([0x7C]) == b'\x7C' # IS TRUE
bytes([0xFD,0xFF,0xAE]) == b'\xFd\xff\xae' # IS TRUE
0xFF == 255 # IS TRUE
bytes(2) == b'\x00\x00' == bytes(0x02) # IS TRUE
0x7C == '|' # IS FALSE : even though | hex value is 0x7C. 0x7C written is that way is understood by python as it's unsigned integer decimal value : 124
0x7C ==== '|'.encode() #IS FALSE : '|'/encode returns a byte() type object that cannot be equal to decimals or strings/chars
bytes(0x02) == b'\x02' # IS FALSE : bytes(0x02) gives 2 bytes of 0x00 value. 0xFF gives 255 butes of 0 value
In [1]:>>> 0xFF << 8
65280
In [1]:>>> (0xFF << 8).to_bytes(2, byteorder='big')
b'\xff\x00'
In [1]:>>> int.from_bytes(b'\x00\x01', byteorder='big', signed=False)
65280
map :
map(function, iterable[, iterable1, iterable2,..., iterableN])
Pour le premier argument de fonction, nous pouvons soit passer une fonction définie par l'utilisateur, soit utiliser des fonctions
lambda, en particulier lorsque l'expression est moins complexe.
map(lambda item: item[] expression, iterable)
map()appliesfunctionto each item initerablein a loop and returns a new iterator that yields transformed items on demand.functioncan be any Python function that takes a number of arguments equal to the number of iterables you pass tomap().
def square(number):
return number ** 2
numbers = [1, 2, 3, 4, 5]
squared = map(square, numbers)
list(squared)
https://realpython.com/python-map-function/
(un)nested for loop :
for x, y in ((a, b) for a in range(arr.shape[0]) for b in range(arr.shape[1])):
print(x,y)
is equal (but faster than) :
for x in range(arr.shape[0]):
for y in range(arr.shape[1])):
print(x,y)
generators :
with :
favorite_numbers = [6, 57, 4, 7, 68, 95]
We can create
def square_all(numbers):
for n in numbers:
yield n**2
squares = square_all(favorite_numbers)
and
squares = (n**2 for n in favorite_numbers)
which are exactly the same.
The first one is called a generator function and the second one is called a generator expression (also called genex).
Generator expression are like list comprehensions but encapsulated with ( ) parenthesis and not [ ] brackets and produce a generator and not a list.
To "consume the generator" one can call list(squares)
logging and warnings:
subprocess :
If using subprocess.popen() with a list as cmd argument on must specify shell=True for it to work.
shutil :
async :
A simplified tutorial to async
asyncio :
asyncio is a library to write concurrent code using the async/await syntax.
sniffio :
You're writing a library. You've decided to be ambitious, and support multiple async I/O packages, like Trio, and asyncio, and ... You've written a bunch of clever code to handle all the differences. But... how do you know which piece of clever code to run?
This is a tiny package whose only purpose is to let you detect which async library your code is running under.
Cython :
conda install -c anaconda cython
pip install Cython --install-option="--no-cython-compile"
Import and execute .pyx in library and in jupyter C compiled primitives files.
.pxd files explanation
Collections :
deque for queueing collections of python objects
Deeplablcut :
Integrating externally labeled data into deeplabcut 2.0
Sockets :
Nice implementation of sending objects with hash checks over python with sockets
Pickle :
Pickle issue solving when switching from function definitions in main or in subpackages
Pickle extensive explanation with interactions with other packages
Move to specific byte location in a pickle file ?
Shelve :
A “shelf” is a persistent, dictionary-like object. The difference with “dbm” databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects — anything that the pickle module can handle. This includes most class instances, recursive data types, and objects containing lots of shared sub-objects. The keys are ordinary strings.
Matplotlib :
Data visualisation
Make matplotlib output text as text in plots, not as curves - 2d link
When using matplotlib qt5agg ( embeded in qt app ) and in the same notebook, call a plot later, use %matplotlib inline inbetween !!!!!!!
Creating alpha containing colormaps :
Magnitude spectrum ( fft ) quick vizualisation function : ax.magnitude_spectrum( signal , Fs= sampling freq, scale='dB')
Switching backends from qt5 to inline in python (no %magic)
Save plots to numpy arrays directly
-
-
Pcolormesh (for time series (histogram based averaging)
-
-
Seaborn :
-
Setting figure size :
sns.set(rc={'figure.figsize':(11.7,8.27)})
-
-
Plotly :
-
Scipy :
-
Bokeh :
Live data visualisation (interactive)
https://dash.plotly.com/layout - Another one
Multiline color and visualisation (better than calling several times the line module)
HoverTool bokeh with example and wiki on plot tools (drag pinch etc..)
-
using animations in jupyter notebook / lab based on webGL and Ipython widgets. (ipywidgets)
Gizeh :
Libray for vector graphics (svg, ai)
Shapely :
Simplify a shape : Simplify polygon
Algos :
Retain critical points (Douglas-Peucker)
Retain critical bends (Wang-Müller)
Retain weighted effective areas (Zhou-Jones)
Retain effective areas (Visvalingam-Whyatt)
Smood a N-D curve scipy.interpolate.splprep
Pandas:
ERREUR TRES CHIANTE
settingwithcopywarning . Peut 'Corrompre un dataframe'...
Ne pas utiliser le setter
dataframe["column"] = defaultvalue
dataframe.loc[: , "column"] = defaultvalue
Adding a row to a dataframe - complete guide
Numpy:
Numpy uses SIMD for vectorized operations, hence it's much quicker and some serial looped operations.
Crosscorrelation : scipy correlation
Subclassing a numpy array : source
Why numpy is using an origin convetion diffrent than matplotlib :
Your graphic is not of a matrix, it is of a Cartesian coordinate system. As you say, in the Cartesian coordinate system, the first coordinate give the horizontal position, the second coordinate gives the vertical position, and going up the page means the second coordinate increases. The location
(0, 0)is at the lower left of many graphics, but it can be adjust to be anywhere, since each coordinate may be negative.In a traditional matrix, however, the first coordinate is the row, or vertical position, and the second is the column, horizontal position. Going up the page means a decreasing first coordinate. Location
(1, 1)is the upper left corner of any matrix, since each coordinate must be a positive integer. Here is a Wikipedia image for a matrix:
A readonly view on the original array with the given shape. It is typically not contiguous. (no memory duplication theoretically)
>>> x = np.array([1, 2, 3]) >>> np.broadcast_to(x, (3, 3)) array([[1, 2, 3], [1, 2, 3], [1, 2, 3]])The term broadcasting describes how numpy treats arrays with different shapes during arithmetic operations. Subject to certain constraints, the smaller array is “broadcast” across the larger array so that they have compatible shapes. Broadcasting provides a means of vectorizing array operations so that looping occurs in C instead of Python.
Avoiding nested for loops on numpy arrays :
ndenumerate : Return an iterator yielding pairs of array coordinates and values.
>>> a = np.array([[1, 2], [3, 4]])
>>> for index, x in np.ndenumerate(a):
>>> print(index, x)
(0, 0) 1
(0, 1) 2
(1, 0) 3
(1, 1) 4
np.mgrid[0:5,0:5]
array([[[0, 0, 0, 0, 0],
[1, 1, 1, 1, 1],
[2, 2, 2, 2, 2],
[3, 3, 3, 3, 3],
[4, 4, 4, 4, 4]],
[[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4]]])
Types in numpy:
Numpy uses class objects to encapsule all types of ints ( regardless of bit size, signed or not ) and floats into :
```python
int: np.integer, float: np.floating
```
Similarly :
```python
bool: np.bool, str: np.str
```
With isinstance() built in function, the second parameter can be a tuple , and if any type inside correcponds to the one on the first parameter: the function returns true : thus we can test any int type with :
isinstance( value , ( int , np.integer ))
Another tool : np.issubdtype( ) is made to test dtypes and determine if one descend from another ( signed is important here, but not bitsize )
Ellipsis = . . . An object that is used to say 'an arbitrary number of dimensions' - link
Example : slice = myarray[:,:,i] or slice = myarray[:,:,:,:,:,i] both equals : slice = myarray[..., i]
find indices of max values in a multi dimensionnal array :
indices = np.unravel_index(array.argmax(), array.shape)
Remove dimensions = 1 : np.squeze
PyQt5:
Adding and removing widgets programatically
How to dynamically add and remove widgets and layouts in PyQt5
PyQt parents widgets, memory leaks, and delete on close (forum)
Scipy :
When fitting : primary issue is the function (number of variables, degrees of adjustment, correctly written etc), then the initial values, and if not working, one can adjust the maxfev .source
2D fitting a curve :
source (used, 2D gaussian function py3 compatible, with offset) source (unused, no offset, fit doesn't converge properly)
PyInstaller :
Create Executable from Python Script using Pyinstaller
Online modules :
pyTube :
https://pytube.io/en/latest/api.html#pytube.Stream.download
Imports :
Rename module during import to preserve back compatibility
Check class type of an item without having to import it's library :
(value.__class__.__module__, value.__class__.__name__) == ('numpy', 'ndarray')
From inside __init__ :
sys.path.append(os.path.dirname(__file__))
Necessary because by default, inside init, the sys.path() only contains the path to parent folder (if is a subpackage), and thus, can't import modules inside it's corresponding subpackage.
Environments :
If issues with "visual C++ xx.x required" :
pip install --upgrade setuptools
Export :
conda env export > environment.yml
Import :
Documentation:
Pydoc : automatically creates python documentation files in html. with architecture and links. The dream. Just needs to state """ triple quotemarks """ documentation at the top of the function or calsses to explain their usage
call it this way :
cd <path to the library parent folder>
pydoc -w <path to the library parent folder>
Sourcetrail - Another better flavor for Visualizing and creating graphs of function calls
Visualizing and creating graphs of function calls (to create docs)
-
Numpydoc to Markdown and serve directly (live changes updates):
https://github.com/greenape/mktheapidocs
Pdoc : kind of the same to pydoc.
YAPF : code formatter (best than autopep8 as it reformat your code even if no violation is found)
Google python style guide with docstrings infos.
def fetch_smalltable_rows(table_handle: smalltable.Table,
keys: Sequence[Union[bytes, str]],
require_all_keys: bool = False,
) -> Mapping[bytes, Tuple[str]]:
"""Fetches rows from a Smalltable.
Retrieves rows pertaining to the given keys from the Table instance
represented by table_handle. String keys will be UTF-8 encoded.
Args:
table_handle:
An open smalltable.Table instance.
keys:
A sequence of strings representing the key of each table row to
fetch. String keys will be UTF-8 encoded.
require_all_keys:
Optional; If require_all_keys is True only rows with values set
for all keys will be returned.
Returns:
A dict mapping keys to the corresponding table row data
fetched. Each row is represented as a tuple of strings. For
example:
{b'Serak': ('Rigel VII', 'Preparer'),
b'Zim': ('Irk', 'Invader'),
b'Lrrr': ('Omicron Persei 8', 'Emperor')}
Returned keys are always bytes. If a key from the keys argument is
missing from the dictionary, then that row was not found in the
table (and require_all_keys must have been False).
Raises:
IOError: An error occurred accessing the smalltable.
"""
Function arguments default values and type hints syntax :
def foo(opts : dict = {}) -> str:
return "foobar"
A tool that allows to generate html from md formatted text files.
String fomatting :
The best way (simpler and faster exec at runtime !) to format strings (if python > 3.5)
To format numerical values inside f strings :
In [1]: # format floating points
val = 12.3
print(f'{val:.2f}')
Out[1]: 12.30
In [1]: # format datetime values
import datetime
now = datetime.datetime.now()
print(f'{now:%Y-%m-%d %H:%M}')
Out[1]: 2019-05-11 22:39
In [1]: # justify to the left (default) and add non significant 0
x = 3
print(f'{x:02} {x*x:3}')
x = 4
print(f'{x:02} {x*x:3}')
Out[2]: 03 9
04 16
In [1]: # justify to the right
s2 = 'ab'
s3 = 'abc'
print(f'{s2:>10}')
print(f'{s3:>10}')
Out[1]: ab
abc
In [1]: # hexadecimal
a = 255
print(f"{a:x}")
Out[1]: ff
str and repr peskyness
f"{new_comedian}" #- using the __str__ function of new_comedian
>>>'Eric Idle is 74.'
f"{new_comedian!r}" #- using the __repr__ function of new_comedian
>>>'Eric Idle is 74. Surprise!'
Debug :
Warnings (for example for deprecation) instead of errors raised
Compilation JIT (just in time) with Numba in python (only for function with positionnal aguments)
Performance :
Profiling :
In Ipython notebook, .ipy files or line commands :
%timeit with a space, before a method or satement : time and prints the time taken
Speeding up python :
When using multi processing or multi threading
"Never use for loop again (a bit extreme tho)"
Classes specificities :
Iterator creation for a class :
Iterator class with __iter__ and __next__
Overloading basics, operators , etc
Initialize parent class in child with super()
Explanation of Super
MRO (method resolution order ) for classes methods
To reload a class in Jupyter lab :
import importlib
importlib.reload(module)
Defining a method as a property : @property before the method definition ( can only take one argument then : self )
Making a class able to print a custom layout : use __repr__ and __str__ (source)
Get a class bale to supscript itself : use __getitem__ and __setitem__ (source)
To use leballed supricpt : use getattr(self, name) ( calls an method / property named as variable name )
Slices are returned with this function : https://www.programiz.com/python-programming/methods/built-in/slice where slice is start stop step. If None None : returns all span. Is step none : interpreted as 1
dir( class object ) returns all methods and properties of the given object. Interesting to learn and make smarter code
Calling a parent class from child : super()
Custom Iterating over a slice :
def __getitem__( self, key ) :
if isinstance( key, slice ) :
#Get the start, stop, and step from the slice
return [self[ii] for ii in xrange(*key.indices(len(self)))]
elif isinstance( key, int ) :
if key < 0 : #Handle negative indices
key += len( self )
if key < 0 or key >= len( self ) :
raise IndexError, "The index (%d) is out of range."%key
return self.getData(key) #Get the data from elsewhere
else:
raise TypeError, "Invalid argument type."
@staticmethod # To not use self in a class method arguments
@classmethod # To call 'factory' class before the main class and thus be able to define init parameters from the class itself :
#https://www.programiz.com/python-programming/methods/built-in/classmethod
@property # To call a method like a property
@wraps # To get docstrings even if you call a function decorator somewhere
# https://stackoverflow.com/questions/308999/what-does-functools-wraps-do
@timeit # to profile a function by making a given number of loops on it and printing the mean and sem
List of awsome decorators : Awesome python decorators
Some other usefull decorators ( like add to class decorator ):
%%cython
Make a class callable : implement method ref
__class__
IDE :
-
Spyder
-
Pycharm
-
Jupyter notebook and jupyter lab
-
Atom ( with hydrogen as runtime execution addon)
Specific Applications :
Capture specific area screen with python
Image :
Image enhancement :
Edge contrained curve fitting in the threshold decomposition domain (paper)
Superresolution with DeepNeuralNetworks in opencv.
Image rotation & translation done right :
Shape matching and image calculation attributes :
PDF on image an shape analysis and measurement
Moments invariants de Hu (online course)
SIFT technique for image matching
Machine vision :
Topic abount basic openCV fucntion for machine vision (and Haar cascade classifiers)
Object detection algorithms (convolutionnal neural networks)
Structural similarity index (SSIM)
Others :
Mathematical Calculations :
Graph theory calculation with python : usind lists tuples and dictionnaries to define custom class (there may be some fully implemented libraries online but this is better as a tutorial not only to classes def but also to graphs)
Curvature calculation :
Curvature calculation (from a github user, taking advantage of Rusinkiewicz's paper)
Interestig mathematical discussion around curvature
Setting write acess to files python
Conda error "require conda init" chen calling conda activate : answer : add Anaconda3/library/bin to path