How to Make Python Faster

Why is Python slow

Since Python is a high-level dynamic language, its code is compiled when it is running, and a lot of complicated work is done during the compilation process. For example, Python's syntax greatly simplifies variable declarations, and it does not require specifying the types of the variables, so the interpreter needs to infer at runtime, which will take more time.

In short, the reason why python makes it easy for you to write is because it does a lot of work behind the scenes.

Besides using a low-level language, is there anything I can try first?

When you find that you use a for loop and the number of iterations is large, you will start to think, can this process be replaced by matrix operations?

Python beginners or linear algebra beginners may not know much about matrices. To put it bluntly, it is actually arranging numbers into a square matrix, and then performing operations in the same way at the same time, and these operations have a lot of fancy names, which is linear algebra. category. But here, no knowledge of linear algebra is required. Numpy is a very powerful Python package that can help you implement matrix operations quickly. The reason why it is fast is that it has done a lot of optimizations in the C language at the bottom, the most important of which is that it can perform parallel computing and give full play to the performance of the machine, while the for loop we directly use in Python is serial, and also It is to wait for the previous calculation to complete before starting the next one. When the amount of data is huge, this speed is simply catastrophic.

Of course, not all loop logic can be replaced by matrix operations. For example, when using a for loop to request different APIs, the loop logic contains operations other than numerical calculations, so it cannot be replaced by matrix operations. However, if you can abstract a loop of pure numerical computation, like in this example, if you can make the API return all the data at once, then you can write the data into a matrix, and you can use matrix operations to process the data in parallel.

An example: displaying 4D data with a 3D coordinate system

I used to be very interested in four-dimensional space. Mathematically, four-dimensional space does exist, but I know that we can't directly display four-dimensional objects in three-dimensional space, so I thought, at least we can think of something, just Representing data so that we can better understand the existence of four-dimensional space. A good approach is to use gradient colors to represent the data in the fourth dimension in a 3D image.

First look at the for loop method

As you can see, I even wrote a generator oscillate_range to optimize this process, trying to reduce some redundant loops. I am using the tqdm module to display a progress bar for the outermost loop.

As you can see, the first few loops are not fast but acceptable, but towards the end of the loop, the speed is suffocating. This is actually because under the same sampling accuracy, the smaller the radius, the fewer data points in the sphere. On the contrary, the larger the radius, the more data points in the sphere, and the longer the operation time. Looking back, when I first ran this code, I just created a sphere, and I didn't finish running it when I went to the large toilet. This is the first time that the performance of python has really affected my mood. So I think there must be a better way to achieve it, so I have the following story.

Test the running time with the timeit module

It can be seen that only one meshgrid function completes all the work of the above nested for loops. Moreover, the speed of parallel operation is significantly faster, and there is no need to care about redundant calculations at all.

Summary

Although the matrix version of the code looks simpler, the meshgrid function does a lot of work behind the scenes, which is far more complicated than the simple and intuitive nested loop logic I wrote.

In addition, I believe that many Python beginners must have heard about the slow speed of Python, so when they started writing code, they were worried about how to optimize and speed up, but in fact, only when you experience the pain once, you start to become hungry to find a solution, and you will naturally find the right direction for optimization. Now that you have chosen Python, you only need to grasp the first principle of writing code that is "if it runs, just relax."

The Forgotten 4D Sphere

This is not a real 4D sphere, just a 3D sphere with a color scale to mark the distance from the position of the data point in the fourth dimension to the center of the sphere. It can also be said that this three-dimensional sphere is the projection of the four-dimensional sphere in our three-dimensional space. It can be seen from the figure that the closer to the center of the sphere, the larger the value of the fourth dimension. Therefore, it is reasonable to imagine that when a four-dimensional sphere is uniformly squeezed into our three-dimensional space, assuming that the extrusion does not cause the three-dimensional sphere to deform, the sphere will become denser from the center. Then if it eventually deforms, in three dimensions, the matter will expand outward from the center of the sphere. In the words of "The Three-Body Problem", just as the "dual-vector foil" makes the three-dimensional space fall to the two-dimensional space, the two-dimensional space will spread around the plane. Our "triple-vector brick" can make the four-dimensional space fall into the three-dimensional space, and the matter will spread in every direction. By the way, just like the description of the four-dimensional space in the "The Three-Body Problem", from the perspective of the four-dimensional, all the structures are unfolded. Inside our three-dimensional sphere model, many substances in the fourth dimension appear in our three-dimensional space just like they come out from nothing, which leads to the expansion of our sphere.