accotron devlog for 2022W48
Started writing already as I’m developing this week to keep up to date and think out loud.
I didn’t end up getting the document models and loading in last week (despite it taking about an hour and a half) as I was too deep into Foundation.
Document data
Many resources besides journals do not have to be codified in the relational database in the same way that journals have to be. Journals and their lines are used in various reports and having them encoded into rows helps a lot with building (fast and efficient) queries.
In turn, many of the weaker links such as information on an invoice or the chart of accounts can be stored easily without including the data schema in the database.
The way these resources are now stored as is with this layout:
(id, String, primary_key)
(tenant_id, ForeignKey(xero_tenant.id), primary_key)
(endpoint, Enum(xero.XeroDocumentEndpoint))
(data, JSON)
This is in pseudo-SQLAlchemy; basically a table which has two primary keys (some resources such as currencies and tax types do not have UUIDs and should be unique by (ID, tenant ID) instead of just ID, although this is more relevant for tax types). I forgot about tenant_id in last week’s blog, but that is definitely necessary too.
PostgreSQL has amazing JSON support and it’s just a pleasure to use, in turn, I don’t mind having the data column just as a plain JSON. The alternative would be encoding the schema in the database for each resource type, and I can’t think of a good reason why as a read-only API-consumer (unlike with journal lines).
Chain report
Chain report now interleaves the history entries for a resource (if available) with its journal entries. For now, I decided to not save history entries in the database as the chain reports loads only one resource at a time and it’s not necessary to cache the history. Impact on report performance is minimal.
Also added a missing filter on tenant_id, have to be a lot more diligent with integration tests. Accidentally discovered this with the new tenant-switcher, which replaces the tenant_id in the URI but retains other parameters. I thought it’d be useful if you’re repeatedly checking the same report for multiple tenants while filing sales tax returns, for example.
OAuth2 sign up
As per last week’s note, I added a registered_at column and now check this during the OAuth2 sign up flow to ensure that partially initialised users cannot occur.
Merge xero.py and storage.py classes
Decided not to go ahead with this as despite the overlap, avoiding application logic in storage.py is more reasonable. Besides, it also feels too early to make a decision on this considering the refactor necessary for this.
Journal annotations
One helpful feature of the journal trail report will of course be the visualisation aspect of it. VCS is helpful in the review process because it highlights the changes, to guide your focus better.
The basic purposes of a journal (in an immutable ledger) can be categorised as: additions, reversals, modifications. A modification is a special case of an addition which makes a nominal impact in the books over a previously-reversed journal. A deletion is a special case of a reversal which ends the trail and leaves no nominal impact whatsoever in the books for the underlying resource (e.g. an invoice).
Knowing these basic categories, and using certain heuristics, it is easy to label journals in O(n) time.
Mechanically, this annotation process is now only ran when you request a trail report, but one of the goals for the next week is to encode these annotations and record them in the database, such that they can be retrieved to enhance other reports.
For the next week
I wanted to get done with some more UI amenities such as summaries of the underlying resources (like a preview of the current state of the invoice or manual journal) on top of the trail report, and wrap up the facelift on the sales tax comparison report, but these are left for the next week!