The Philosophy of Decorators
When writing programs, we often encounter situations where we have written a lot of functions with similar functionality, but we suddenly realize that we need to perform additional operations on these functions. Of course, we can write these additional operations in each function, but doing so would make the code redundant and difficult to maintain. In this case, we can use decorators to solve this problem.
The essence of a decorator is actually a function that can "decorate" other functions, that is, it can add additional functionality to them or monitor the execution of functions without changing the original function code.
Using Decorators
Decorators in Python are used with the @ symbol and come in two basic forms, with or without arguments.
# Without arguments
@decorator
def func():
pass
# With arguments
@decorator(arg1, arg2)
def func():
pass
Python has many built-in decorators, such as @dataclass to automatically implement the __init__ method when creating a class, or @property to implement getter and setter methods for properties. There are also decorators like @staticmethod and @classmethod to implement static methods and class methods.
Decorators with arguments include @app.route('/path') in the popular Flask web framework, which binds a function to a URL so that the function is executed when the user visits that URL.
Building Decorators
When I first encountered these decorators, I thought they were magical - just add a hat to your own function and you can achieve specific logic. Now that I have been learning this language for some time, I think I can explain the principles behind these decorators in my own words. I have always believed that using "Hello World" to explain the principles of a programming language is not enough, because it is too simple and people cannot understand the convenience that this syntax structure can bring in real use.
Data Download and Reading Requirements
For example, I have a complex function that retrieves data from a network database, performs some formatting, and passes the results to the next function for further processing. The code for this function is as follows:
def get_data(url, start_date, end_date, label_name):
"""Get data from network database
Args:
url (str): Data API endpoint
start_date (str): Start date, e.g. 2023-03-11
end_date (str): End date, e.g. 2023-03-12
label_name (str): Label field name
Returns:
list: Data list
"""
# Retrieve data from network database
data = requests.get(
url,
params={
'start_date': start_date,
'end_date': end_date,
'label_name': label_name
}
)
# Convert to pandas DataFrame format
data = pd.DataFrame(data)
return data
Because I'm working on a data analysis project, my analysis code is constantly being improved, so I have to rerun this function every time I rewrite the analysis steps. Sometimes the data I'm retrieving is very large and the internet connection is slow, so it can take a long time to download the data.
My first thought was to modify the function to save the downloaded data to my local machine. If you've learned pandas before, you might be familiar with this method:
data = pd.to_csv('data.csv')
Then, when I need to call the data later, I can write a separate function to read the data:
def read_data():
data = pd.read_csv('data.csv')
# People who are familiar with pandas may know that
# to_csv and read_csv are not inverse operations
# That is, the data read by read_csv is not exactly
# the same as the original DataFrame before saving
# So some formatting may be needed.
return data
This method temporarily solved my problem. However, as my project progressed, I found that I had to retrieve more and more data with different parameters. For example, for the get_data function mentioned above, the url and date range can change, and there may be other parameters as well. Do I have to write a separate read_data function for each data retrieval function?
Caching Mechanism
At this point, I thought of finding a way to abstract the logic of data storage and retrieval into a function, and then have each data retrieval function execute the storage and retrieval logic at the same time. This is the caching mechanism used by web browsers - when you visit a webpage, the browser first checks if there is a cache on the local device, and if so, it uses the cache directly. If not, it downloads it from the server. This reduces network requests and improves webpage loading speed.
In my data analysis project, I can do the same thing. When I pull the data for the first time, I save it locally, and the next time I pull the same data, I retrieve it directly from the local cache instead of pulling it from the network again. This reduces network requests and improves data retrieval speed.
There are some details to consider here. In the caching mechanism of a web browser, when the browser visits a URL, it first asks the server if the data corresponding to the URL has been updated. If it has, it will be re-downloaded, otherwise, it will use the local cache. This means that the browser first makes a small request to get information on whether the data has been updated. However, in my data retrieval process, I don't have access to the URL access process mechanism, so I have to find a way to determine whether the data has been downloaded. This is simple - I just save each set of parameters used in data retrieval and compare them to the previous set of parameters used. If they are the same, I retrieve the data from the local cache, otherwise, I re-download the data.
This method is not foolproof, as there may be cases where data is updated without any change to the parameters used to access it. However, for my current project, this method is sufficient. The logic has been thought out, and now it's time to implement it. For functions like this, the convenience of decorators is particularly evident.
Implementation
Storage and Retrieval
In Python, we can use pickle to store and retrieve data. pickle is a built-in serialization module in Python that can convert Python objects into byte streams, and then convert the byte stream back into Python objects.
import pickle
# store data
with open('data.pkl', 'wb') as f:
pickle.dump(data, f)
# retrieve data
with open('data.pkl', 'rb') as f:
data = pickle.load(f)
The advantage of pickle is that storage and retrieval are inverse operations, which means that we can store any Python object locally and then retrieve it, without worrying about format conversion issues. Of course, the disadvantage is that it does not store data in a text format, so we cannot view it directly like we can with CSV files.
File Name Hashing
Sometimes, our parameters may contain special characters, or the function may contain unserializable objects, making it impossible to save the parameters directly in the file name. In this case, we can convert the parameters into a hash value. Hashing is a one-way encryption algorithm, which means that we can convert any string into a hash value, but we cannot reverse-engineer the original string from the hash value.
This type of conversion is widely used for password storage, as we can convert a user's password into a hash value and then store the hash value in a database, even if the database is hacked, the hacker cannot reverse-engineer the user's password from the hash value.
We use this conversion method in the caching logic to generate valid and non-repeating file names.
I want to emphasize here that this algorithm is not perfect, which means that different data may generate the same hash value, a situation known as hash collision. However, the probability of hash collisions is extremely small, and for my project, it can be ignored.
Here, I will write a simple hash function to convert the parameters into a hash value.
import hashlib
def params_to_hash(*args, **kwargs):
"""Converts the function's arguments into a 32-character hash value.
Args:
*args: positional arguments
**kwargs: keyword arguments
Returns:
str: The first 32 characters of the hash value of the arguments.
"""
# Concatenate the arguments with '_' as separator
args_str = '_'.join([str(arg) for arg in args])
kwargs_str = '_'.join([f'{key}:{value}' for key, value in kwargs.items()])
# Concatenate the two parts with '.' as separator
total_str = '.'.join([args_str, kwargs_str])
# Use sha256 algorithm to compute the hash value
result = hashlib.sha256(total_str.encode('utf-8')).hexdigest()[:32]
return result
Writing Decorators
Let's start with a decorator without any arguments.
from functools import wraps
def cache(func):
# Use the built-in functools.wraps decorator
# to preserve the original function's metadata.
# This allows us to access the function name via func.__name__.
# Otherwise, func.__name__ would be 'wrapper'.
@wraps(func)
def wrapper(*args, **kwargs):
# Save directory
cache_dir = 'data'
# File name
cache_file = f'{cache_dir}/{func.__name__}/{params_to_hash(*args, **kwargs)}.pkl'
# Logic for saving and reading data
def read():
# Read data and return it directly
with open(cache_file, 'rb') as f:
return pickle.load(f)
def exec_and_save():
# Execute function and get result
result = func(*args, **kwargs)
# Create directory for saving if it doesn't exist
os.makedirs(os.path.dirname(cache_file), exist_ok=True)
# Save result
with open(cache_file, 'wb') as f:
pickle.dump(result, f)
# Return result
return result
# If the file exists, read the file. Otherwise, execute the function and save the result.
if os.path.exists(cache_file):
return read()
else:
return exec_and_save()
return wrapper
The usage of this decorator is like this:
@cache
def get_data(url, start_date, end_date, label_name):
...
After running the function once, we can find the cache data in the data/get_data directory in the working directory.
Now I have higher requirements. I want to specify the folder name for saving data, so we need to add arguments to the decorator. Let's modify the decorator we just wrote.
def cache(cache_dir='data'):
def inner(func):
@wraps(func)
def wrapper(*args, **kwargs):
cache_file = f'{cache_dir}/{func.__name__}/{params_to_hash(*args, **kwargs)}.pkl'
def read():
with open(cache_file, 'rb') as f:
return pickle.load(f)
def exec_and_save():
result = func(*args, **kwargs)
os.makedirs(os.path.dirname(cache_file), exist_ok=True)
with open(cache_file, 'wb') as f:
pickle.dump(result, f)
return result
if os.path.exists(cache_file):
return read()
else:
return exec_and_save()
return wrapper
return inner
Now we can specify the cache directory like this:
# Use the default cache directory
@cache()
def get_data(url, start_date, end_date, label_name):
...
Or
# Change the directory name to 'api'
@cache(cache_dir='api')
def get_data(url, start_date, end_date, label_name):
...
"@" is Syntactic Sugar
Lastly, the @ symbol is a syntactic sugar in Python. Its purpose is to put the call to the decorator above the function definition, so that the original function doesn't have to be wrapped every time the function is called.
The following code is equivalent to using @ above the function definition:
# For decorators without arguments
get_data = cache(get_data)
# For decorators with arguments
get_data = cache(cache_dir='api')(get_data)
腻害👍