I’m up to my elbows in refactoring my project. It’s based on Django, PostgreSQL, some math/science/statistics libraries, interfaces with Discord, and collects data from a 3rd party API. All running the old fashioned way on a Debian VPS.

Every part of this was due for a refresh as I had pinned all the important parts.

This required refactoring was started and burned out a few times, but with a deadline for the introduction of this new 3rd party API. The current production system scrapes from a different data source. The schema has changed drastically.

This is what I call version 4 of the project.

So I’m clearly daydreaming about what I would do if I did a complete rewrite instead of this refactoring. I’m going to toss these intrusive thoughts out of the way here so I can focus on the immediate task at hand and come back to these notes once I get V4 out the door, all embedded and “done”.

1. Third-party data requests should be cached

Compress the “rarely used” json files, leave the test/dev data raw. This way, if needed, the entire database can be rebuilt from this collection of data instead of having to redo all the API calls.

2. The output generated in response to a user request must be cached

For a certain percentage (subquest: calculate the ballpark) of the content generated in response to user requests will not change over time. All of these should return cached output.

3. Break down more things into a single functional process, feed them from a task queue

Currently my system has a few big “apps” that run a complete process from start to finish. This slows down testing and debugging.

4. as many raw SQL queries as possible

Use a framework and it’s ORM where it makes sense (model definition, migrations, external API endpoints, user settings, etc.), but for the core data storage/retrieval; just do SQL. Of course with a nice connection/cursor -> data structure wrapper.

5. test suite to capture the known foot weapons

Many of the above points can also be combined to improve testing. For example, once the outputs of user commands are cached; it is easy to replay a command and compare outputs as a simple automated regression testing process. Similarly, splitting up the functions of today’s large “standalone” processes makes testing much simpler.

6. Monitor the key “steps” in the flow from user to system and back to user

Set appropriate calls to action in case there are any errors.

7. Check incoming data from third-party API for schema changes

Deal gracefully with as many edge cases as you can think of.

So yeah, summary is: I really want to use Elixir and Phoenix for version 5 🙂