Why Refactoring? How to Restructure Python Package?

Start Refactoring — Reorganizing / Restructuring

In this section, I will walk you through an example config.py by reorganizing the structure, merging duplicated methods, decomposing and writing test code to ensure backward compatibility.

config.py looks like this:

"""
Package: utils.config
before restructuring
"""
CONFIG_NAME = {
"ENABLE_LOGGING": "enable_logging",
"LOGGING_LEVEL": "logging_level",
}
def get_logging_level():
pass
class ConfigHelper:
def get(self, config_name, default=None):
pass
def set(self, config_name, value):
pass
def _get_settings_helper(self):
pass
def get_logging_level():
pass
def is_logging_enabled():
pass
class LOGGING_LEVEL:
VERBOSE = "verbose"
STANDARD = "standard"

Step 1: Write Backward Compatible Code

This step is crucial. Before refactoring our code, test cases MUST be in place. In this case, we write backward compatible code to ensure all references to the classes/functions/constants are still working.

In __init__.py, we shall redefine the class/method signatures:

"""
in __init__.py
This is where backward compatibility code lives.
This is to ensure the refactored package supports
old way of import.
This is incomplete, we will revisit __init__.py later
"""
CONFIG_NAME = {}
def get_logging_level(*args, **kwargs):
pass
class ConfigHelper:
def get(self, *args, **kwargs):
pass
    def set(self, *args, **kwargs):
pass

def _get_settings_helper(self):
pass

    def get_logging_level(self):
pass

def is_logging_enabled(self):
pass


class LOGGING_LEVEL:
pass

The __init__.py is incomplete for now. We will revisit the file later. 
Next, we write a test case to make sure we can still import the package as if we are importing the old package.

"""
in tests.py
Simple backward compatibility test case
"""
class ConfigHelperCompatibilityTestCase(unittest.TestCase):
def test_backward_compatibility(self):
try:
from .config import CONFIG_NAME, LOGGING_LEVEL
from .config import get_logging_level
from .config import ConfigHelper
except ImportError as e:
self.fail(e.message)

This is a simple test case, you may notice some backward compatibility issues are not caught in the test case.

Step 2: Reorganizing Package Structure

This section gives you an idea on how you can reorganize your Python package. Let’s revisit the config.py we have:

"""
Package: utils.config
before restructuring
"""
CONFIG_NAME = {
"ENABLE_LOGGING": "enable_logging",
"LOGGING_LEVEL": "logging_level",
}

def get_logging_level():
pass

class ConfigHelper:
def get(self, config_name, default=None):
pass
    def set(self, config_name, value):
pass
    def _get_settings_helper(self):
pass
    def get_logging_level():
pass
    def is_logging_enabled():
pass

class LOGGING_LEVEL:
VERBOSE = "verbose"
STANDARD = "standard"

Can you spot what’s wrong here? It is messy, there are constants, helpers, duplicated codes in a single file. When the code in config.py grows larger, it will become increasing difficult to navigate within. With this messy structure, you are breeding a spot for circular dependency, hidden coupling and refining the recipe for the tastiest spaghetti code.

How can you reorganize config.py? To me, separation of concerns comes across my mind. The following structure is often considered a good practice to structure Python package (this structure is used in Django as well).

config/
├── abstracts.py # All the abstract classes should live here
├── constants.py # All the constants should live here
├── exceptions.py # All custom exceptions should live here
├── helpers.py # All helpers should live here
├── __init__.py # All backward compatible code in here
├── mixins.py # All mixins goes to here
├── serializers.py # All common serializers goes to here
└── tests.py # All `config` related tests should live here

Let’s revisit our config.py before refactoring and identify where the individual piece of code should reside.

"""
Package: utils.config
before restructuring
"""
# This looks like belongs to utils.config.constants
CONFIG_NAME = {
"ENABLE_LOGGING": "enable_logging",
"LOGGING_LEVEL": "logging_level",
}
# This looks like a helper function, goes to utils.config.helpers
def get_logging_level():
# This looks like a duplicate method
pass
# This looks like a helper class, goes to utils.config.helpers
class ConfigHelper:
def get(self, config_name, default=None):
pass
    def set(self, config_name, value):
pass
    def _get_settings_helper(self):
pass
    def get_logging_level():
# This looks like a duplicate method
pass
    def is_logging_enabled():
pass
# This looks like another constant, goes to utils.config.constants
class LOGGING_LEVEL:
VERBOSE = "verbose"
STANDARD = "standard"

After refactoring, config.py should become a Python package config with a __init__.py in it.

utils/
├──config.py # To be removed
└──config/
├── constants.py
├── helpers.py
├── __init__.py
└── tests.py

In utils.config.constants :

"""
Package: utils.config.constants
after restructuring
"""
# Inconsistent programming construct
CONFIG_NAME = {
"ENABLE_LOGGING": "enable_logging",
"LOGGING_LEVEL": "logging_level",
}
# Inconsistent programming construct
class LOGGING_LEVEL:
VERBOSE = "verbose"
STANDARD = "standard"

In utils.config.helpers :

"""
Package: utils.config.constants
after restructuring
"""
def get_logging_level():
# This is duplicate, removing this
pass
class ConfigHelper:
def get(self, config_name, default=None):
pass
    def set(self, config_name, value):
pass
    def _get_settings_helper(self):
pass
    def get_logging_level():
pass
    def is_logging_enabled():
pass

Step 3: Eliminate and Merging Duplicates

In utils.config.helpers , there are 2 similar methods/functions get_logging_level() and ConfigHelper()._get_logging_level() . Assuming both implementations are identical, it means we have to find a best place to host the function.

In this case, I remove the standalone get_logging_level() and keep the one in ConfigHelper.

"""
Package: utils.config.constants
after removing duplicates
"""
class ConfigHelper:
def get(self, config_name, default=None):
pass
    def set(self, config_name, value):
pass
    def _get_settings_helper(self):
pass
    def get_logging_level():
pass
    def is_logging_enabled():
pass

Step 4: Decomposing

Personally, I’m a fan of decomposition. Instead of having a single class ConfigHelper, we can further decompose ConfigHelper into a hierarchy of classes and mixins.

Hierarchy of Decomposed ConfigHelper

We host our AbstractBaseConfigHelper in abstracts.py:

"""
in abstracts.py
"""
from abc import ABCMeta
class AbstractBaseConfigHelper:
__metaclass__ = ABCMeta

def get(self, config_name):
pass

    def set(self, config_name, value):
pass
    def _get_settings_helper(self):
pass

In mixins.py :

"""
in mixins.py
"""
class LoggingConfigMixin:
def is_logging_enabled():
pass
    def get_logging_level():
pass

In helpers.py :

"""
in helpers.py
"""
class ConfigHelper(
AbstractBaseConfigHelper,
LoggingConfigMixin
):
pass

ConfigHelper is now decomposed into multiple classes and mixins.

Step 5: Complete Our Backward Compatibility Code

In Step 1, we added some code in __init__.py. However, it is largely incomplete. Let’s revisit the file:

"""
in __init__.py
This is where backward compatibility code lives.
This is to ensure the refactored package supports
old way of import.
This is incomplete, we will revisit __init__.py later
"""
CONFIG_NAME = {}
def get_logging_level(*args, **kwargs):
pass
class ConfigHelper:
def get(self, *args, **kwargs):
pass
    def set(self, *args, **kwargs):
pass

def _get_settings_helper(self):
pass

    def get_logging_level(self):
pass

def is_logging_enabled(self):
pass

class LOGGING_LEVEL:
pass

Notice that the bridge between the code above and our newly organized config package is still missing. To establish the bridge, we edit our __init__.py into:

"""
in __init__.py
This is where backward compatibility code lives.
This is to ensure the refactored package supports
old way of import.
"""
from .constants import CONFIG_NAME, LOGGING_LEVEL
from .helpers import ConfigHelper as _ConfigHelper
def get_logging_level(*args, **kwargs):
return _ConfigHelper().get_logging_level()
class ConfigHelper(_ConfigHelper):
pass

Step 6: Notify The Developer

Up to Step 5, our config is properly refactored. However, we need to keep the developers notified about the change. Is there any straightforward way? Yes. We can emit a warning message whenever a developer is trying to import an obsolete function/class/method. For example, we annotate the old functions/classes/methods with decorators:

"""
decorators.py
"""
def refactored_class(message):
def cls_wrapper(cls):
class Wrapped(cls, object):
def __init__(self, *args, **kwargs):
warnings.warn(message, FutureWarning)
super(Wrapped, self).__init__(*args, **kwargs)
return Wrapped
return cls_wrapper

def refactored(message):
def decorator(func):
def emit_warning(*args, **kwargs):
warnings.warn(message, FutureWarning)
return func(*args, **kwargs)
return emit_warning
return decorator

In our __init__.py , we add decorator like this:

"""
in __init__.py
This is where backward compatibility code lives.
This is to ensure the refactored package supports
old way of import.
"""
from .constants import CONFIG_NAME, LOGGING_LEVEL
from .helpers import ConfigHelper as _ConfigHelper
@refactored('get_logging_level() is refactored and deprecated.')
def get_logging_level(*args, **kwargs):
return _ConfigHelper().get_logging_level()
@refactored_class('config.ConfigHelper is refactored and deprecated. Please use config.helpers.ConfigHelper')
class ConfigHelper(_ConfigHelper):
pass

read original article here