This page documents the coding standards, tools and procedure that the lab applies to its python projects.

Coordinator: Javier Peralta

Contributors: Ariel Mora, Daniel García Vaglio

Note: Most of these sections (specially the tools ones) are meant as a small show case and generally have link to the relevant documentation. Please read this links since the information presented here is not exhaustive nor complete.

Python projects should strictly follow PEP8.

The following tools can aid you to keep your styling on point:

Flake8 is a code checker, it will check python files and raise errors and warnings when code breaks PEP8 or if there is a syntax error.

To install:

apt-get install flake8

Usage:

flake8 path/to/code/

Yet another python formatter (yapf) is a code formatter, a program that reformat code to comply with a style guide (like PEP8 which is the default style). Running this over your python project will clear most PEP8 compliance errors.

To install:

apt-get install yapf3

Usage:

yapf3 -ir path/to/code_dir

or

yapf3 -i path/to/code/file.py

Logging is a very important part of every software project, because it is the best way to deliver valuable information to our users. Logs are used to report errors, warnings, or just general information of how the program is executing. Good logs help us to discover why the program fails some times, they help us diagnose problems with our programs and help us plan strategies for improvement. It is crucial to maintain good and healthy logging practices so that you can increase the maintainability of your code.

Python developers, knowing that logs are so important, have developed a module just for this: the Logging module. It is very important that we use this tool and Not plain prints. Also, take into consideration that logging helpers are going to be removed from arcospyu very soon. There are 5 default levels of logging, ordered from most important to less important are: CRITICAL, Error, Warning, Info, Debug.

CRITICAL logs are messages that are shown to the user that report that something very bad happened. The program is not able to continue its execution after that failure and it may cause colateral damage (affect a database, a network, files…).

Error logs are used when something goes wrong, but only a part is affected. Most failures can be logged as Error.

The Warning level is used to report errors from which your application is able to recover autonomously. Your system won't crash because of a Warning, but you are alerting the user that something is not OK. Warning logs can also be used in libraries when you want to tell your users that there are going to be some API changes. For example, if you want to change the name of a function, or if a method will be deprecated, the best thing to do is to create a Warning to announce that before doing the changes.

Info logs are for general information, there is nothing going wrong with your program, but you want to inform the user about events that might be of interest. Messages like “The file has been opened”, “Connection established” are common Info messages.

Finally Debug logs are used only during development. Let's say that you are not sure why your program is behaving in a certain way, and you believe that there might be a problem with the value of a certain variable. So you can create a Debug log to print that variable's value, to see if there really is a problem.

When you define a logging level, you are telling Python which messages you want to receive. If you define a level A, you will receive all A messages and all messages with more priority. If you say that you want a Error level, so you will receive Error and CRITICAL logs only. If you define Debug level, you will receive all logs. For most cases you will want to set Info or Warning levels.

In Python, the logger is the object in charge of handling your logs. This are the objects that you as a programmer will use to create the logs. In order to get a logger you just have to call the module level function getLogger(). This function will return a logger object that you can use. Loggers have the setLevel method. This method sets the level of your logger as explained above. Then you have the debug, info, warning, error, and critical methods which create a message of their respective level. Note that if you set the Warning level, and create a Debug message, it won't be shown to the user. That is exactly the idea of defining these levels.

The following is an example,

# myapp.py
import logging
import math
 
def main():
    # Instead of printing the message send it to a file named 'filename'
    # Set the INFO level
    logging.basicConfig(filename="myapp.log", level=logging.INFO)
 
    # log some info
    logging.info("Started")
    total = 0
    for number in range(-2, 100):
        logging.debug(number)
        try:
            math.sqrt(number)
        except ValueError:
            # report that there was an error, but we handled it.
            logging.warning("Recovering from a negative sqrt")
            number = 0
 
        total += number
 
    if total < 0:
        # If after adding possive numbers you got a negative, then there is something really wrong
        logging.error("Negative sum of positive numbers")
        raise ValueError
 
    # If everything went OK, report it
    logging.info('Finished')
 
if __name__ == '__main__':
    main()

It is also possible to add colors to your logs. For that we need to deal with a Handler which is the object in charge of sending your logs to the apropiate destination. The default is to send them to the standard output, but you can also send them to a log file as we did in the example above.

If you want to add colors to your logs you will need a package named `colorlog`. You can install it from PyPi with:

pip install colorlog

If you do not want to use pip, there are also Debian, Ubuntu, and AUR packages. For example for Debian, you can install with:

sudo apt install python-colorlog python3-colorlog

Here you have an example of how to use `colorlog`:

import colorlog
 
# create the format object with colors.
formatter = colorlog.ColoredFormatter(
	"%(asctime)s%(log_color)s%(levelname)-8s%(reset)s %(blue)s%(message)s",
	datefmt=None,
	reset=True,
	log_colors={
		'DEBUG':    'cyan',
		'INFO':     'green',
		'WARNING':  'yellow',
		'ERROR':    'red',
		'CRITICAL': 'red,bg_white',
	},
	secondary_log_colors={},
	style='%'
)
 
# Create the handler
handler = colorlog.StreamHandler()
# Add the formatter we created earlier
handler.setFormatter(formatter)
 
# Get a logger called "example". (It is created if it doesn't exist)
logger = colorlog.getLogger("example")
# Add the colored formatter to the logger we just created
logger.addHandler(handler)
 
logger.error("This is an error")
logger.warning("This is a warning")
logger.info("This is an info (that you will not see)")
logger.debug("This a debug (that you will not see)")

The recommended project layout for a python projected named

ptoject_name

is as follows:

project_name
├── AUTHORS.rst
├── docs
├── examples
│  └── example1.py
├── LICENSE
├── Makefile
├── MANIFEST.in
├── README.rst
├── requirements.dev.txt
├── requirements.txt
├── setup.cfg
├── setup.py
├── src
│  └── package_name
│     ├── __init__.py
│     └── module1.py
├── tests
└── tox.ini

cookiecutter is a command-line utility that creates projects from cookiecutters (project templates). E.g. Python package projects, jQuery plugin projects.

To install:

apt-get install cookiecutter

Arcos template for python is hosted in this repo, to create a new project run:

cookiecutter https://github.com/arcoslab/arcos-python-cookiecutter

after the initial clone you can other new repos refering to the cookicutter by name:

cookiecutter arcos-python-cookiecutter

The lab have some Python 2 repos that need to migrated to Python3. The process will be as follow:

1. Migrate all repos one by one to support both python 2 and 3 from the same file
2. After all repos are migrated we will drop python 2 support for good

For each repo the following process will be applied:

1. An issue to track the repo's migration progress will be opened
2. Open a new py3stage1 branch
3. Using futurize apply stage1 changes to all files
4. Open a pull request and have another member check the changes.
5. Tag the pre-stage1 release so people can easily use that if an issue raises
6. Merge the changes to master, rebase all other branches from master and resolve merge conflicts
7. Test the changes, this is probably going to be manual since most repos have no unit tests. This may be a good time to write some.
8. Open a new py3stage2 branch.
9. Repeat the process for stage1, testing both for python2 and python3
10. Check everything twice
11. Close the issue

Any problems encountered due to migration should be reported with a GitLab issue on the corresponding repo.

To see progress please take a look to the following project tracker on github