ArcosWiki

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Python development standards ======
This page documents the coding standards, tools and procedure that the lab applies to its python projects.

Coordinator: [[people:jperalta|Javier Peralta]]

Contributors: [[people:amora|Ariel Mora]], [[people:dgarcia|Daniel García Vaglio]]

**Note:** Most of these sections (specially the tools ones) are meant as a small show case and generally have link to the relevant documentation. Please read this links since the information presented here is not exhaustive nor complete. 
===== Code Styling guide  =====
Python projects should strictly follow [[https://www.python.org/dev/peps/pep-0008/|PEP8]].


==== Tools ====
The following tools can aid you to keep your styling on point:

=== Flake8 ===

[[http://flake8.pycqa.org/en/latest/|Flake8]] is a code checker, it will check python files and raise errors and warnings when code breaks PEP8 or if there is a syntax error. 

To install:
<code bash>
apt-get install flake8
</code>

Usage:
<code bash>
flake8 path/to/code/
</code>

=== yapf ===
[[https://github.com/google/yapf|Yet another python formatter (yapf)]] is a code formatter, a program that reformat code to comply with a style guide (like PEP8 which is the default style). Running this over your python project will clear most PEP8 compliance errors.

To install:
<code bash>
apt-get install yapf3
</code>

Usage:
<code bash>
yapf3 -ir path/to/code_dir
</code>
or
<code bash>
yapf3 -i path/to/code/file.py
</code>

==== Logging ====

Logging is a very important part of every software project, because it is the best way to deliver valuable information to our users. Logs are used to report errors, warnings, or just general information of how the program is executing. Good logs help us to discover why the program fails some times, they help us diagnose problems with our programs and help us plan strategies for improvement. It is crucial to maintain good and healthy logging practices so that you can increase the maintainability of your code.

Python developers, knowing that logs are so important, have developed a module just for this: the [[https://docs.python.org/3.7/library/logging.html|Logging]] module. It is very important that we use this tool and **Not plain prints**. Also, take into consideration that logging helpers are going to be removed from ''arcospyu'' very soon. There are 5 default levels of logging, ordered from most important to less important are: ''CRITICAL'', ''Error'', ''Warning'', ''Info'', ''Debug''.

=== Logging levels ===

''CRITICAL'' logs are messages that are shown to the user that report that something **very bad** happened. The program is not able to continue its execution after that failure and it may cause colateral damage (affect a database, a network, files...).

''Error'' logs are used when something goes wrong, but only a part is affected. Most failures can be logged as ''Error''.

The ''Warning'' level is used to report errors from which your application is able to recover autonomously. Your system won't crash because of a ''Warning'', but you are alerting the user that something is not OK. ''Warning'' logs can also be used in libraries when you want to tell your users that there are going to be some API changes. For example, if you want to change the name of a function, or if a method will be deprecated, the best thing to do is to create a ''Warning'' to announce that before doing the changes.

''Info'' logs are for general information, there is nothing going wrong with your program, but you want to inform the user about events that might be of interest. Messages like "The file has been opened", "Connection established" are common ''Info'' messages.

Finally ''Debug'' logs are used only during development. Let's say that you are not sure why your program is behaving in a certain way, and you believe that there might be a problem with the value of a certain variable. So you can create a ''Debug'' log to print that variable's value, to see if there really is a problem.

When you define a logging level, you are telling Python which messages you want to receive. If you define a level //A//, you will receive all //A// messages and all messages with more priority. If you say that you want a ''Error'' level, so you will receive ''Error'' and ''CRITICAL'' logs only. If you define ''Debug'' level, you will receive all logs. For most cases you will want to set ''Info'' or ''Warning'' levels.

=== Loggers ===

In Python, the [[https://docs.python.org/3.7/library/logging.html#logging.Logger|logger]] is the object in charge of handling your logs. This are the objects that you as a programmer will use to create the logs. In order to get a logger you just have to call the module level function [[https://docs.python.org/3.7/library/logging.html#logging.getLogger|getLogger()]]. This function will return a logger object that you can use. Loggers have the ''setLevel'' method. This method sets the level of your logger as explained above. Then you have the ''debug'', ''info'', ''warning'', ''error'', and ''critical'' methods which create a message of their respective level. Note that if you set the ''Warning'' level, and create a ''Debug'' message, it won't be shown to the user. That is exactly the idea of defining these levels.

The following is an example,

<code python>
# myapp.py
import logging
import math

def main():
    # Instead of printing the message send it to a file named 'filename'
    # Set the INFO level
    logging.basicConfig(filename="myapp.log", level=logging.INFO)
    
    # log some info
    logging.info("Started")
    total = 0
    for number in range(-2, 100):
        logging.debug(number)
        try:
            math.sqrt(number)
        except ValueError:
            # report that there was an error, but we handled it.
            logging.warning("Recovering from a negative sqrt")
            number = 0
       
        total += number
       
    if total < 0:
        # If after adding possive numbers you got a negative, then there is something really wrong
        logging.error("Negative sum of positive numbers")
        raise ValueError
    
    # If everything went OK, report it
    logging.info('Finished')

if __name__ == '__main__':
    main()
</code>


=== Colors ===

It is also possible to add colors to your logs. For that we need to deal with a [[https://docs.python.org/3.7/library/logging.html#handler-objects| Handler]] which is the object in charge of sending your logs to the apropiate destination. The default is to send them to the standard output, but you can also send them to a log file as we did in the example above. 

If you want to add colors to your logs you will need a package named `colorlog`. You can install it from PyPi with:

<code bash>
pip install colorlog
</code>

If you do not want to use ''pip'', there are also [[https://packages.debian.org/jessie/python-colorlog| Debian]], [[https://launchpad.net/python-colorlog| Ubuntu]], and [[https://aur.archlinux.org/packages/python-colorlog/| AUR]] packages. For example for Debian, you can install with:

<code bash>
sudo apt install python-colorlog python3-colorlog
</code>

Here you have an example of how to use `colorlog`:

<code python>
import colorlog

# create the format object with colors.
formatter = colorlog.ColoredFormatter(
	"%(asctime)s%(log_color)s%(levelname)-8s%(reset)s %(blue)s%(message)s",
	datefmt=None,
	reset=True,
	log_colors={
		'DEBUG':    'cyan',
		'INFO':     'green',
		'WARNING':  'yellow',
		'ERROR':    'red',
		'CRITICAL': 'red,bg_white',
	},
	secondary_log_colors={},
	style='%'
)

# Create the handler
handler = colorlog.StreamHandler()
# Add the formatter we created earlier
handler.setFormatter(formatter)

# Get a logger called "example". (It is created if it doesn't exist)
logger = colorlog.getLogger("example")
# Add the colored formatter to the logger we just created
logger.addHandler(handler)

logger.error("This is an error")
logger.warning("This is a warning")
logger.info("This is an info (that you will not see)")
logger.debug("This a debug (that you will not see)")
</code>
===== Project Layout =====
The recommended project layout for a python projected named <code>ptoject_name</code> is as follows:
  
<code>
project_name
├── AUTHORS.rst
├── docs
├── examples
│  └── example1.py
├── LICENSE
├── Makefile
├── MANIFEST.in
├── README.rst
├── requirements.dev.txt
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│  └── package_name
│     ├── __init__.py
│     └── module1.py
├── tests
└── tox.ini
</code>


==== cookiecutter ====
[[http://example.com|cookiecutter]] is a command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.

To install:
<code>
apt-get install cookiecutter
</code>

Arcos template for python is hosted in [[https://github.com/arcoslab/arcos-python-cookiecutter|this repo]], to create a new project run:
<code>
cookiecutter https://github.com/arcoslab/arcos-python-cookiecutter
</code>

after the initial clone you can other new repos refering to the cookicutter by name:
<code>
cookiecutter arcos-python-cookiecutter
</code>

===== Python3 Migration =====
The lab have some Python 2 repos that need to migrated to Python3. The process will be as follow:

  - Migrate all repos one by one to support both python 2 and 3 from the same file
  - After all repos are migrated we will drop python 2 support for good

For each repo the following process will be applied:

  - An issue to track the repo's migration progress will be opened
  - Open a new py3stage1 branch
  - Using [[http://python-future.org/futurize.html|futurize]] apply stage1 changes to all files
  - Open a pull request and have another member check the changes.
  - Tag the pre-stage1 release so people can easily use that if an issue raises
  - Merge the changes to master, rebase all other branches from master and resolve merge conflicts
  - Test the changes, this is probably going to be manual since most repos have no unit tests. This may be a good time to write some.
  - Open a new py3stage2 branch.
  - Repeat the process for stage1, testing both for python2 and python3
  - Check everything twice
  - Close the issue

==== Issues ====

Any problems encountered due to migration should be reported with a GitLab issue on the corresponding repo.
==== Progress tracking ====

To see progress please take a look to the [[https://github.com/orgs/arcoslab/projects/1|following project tracker on github]]