Repository navigation
Simplify internals to prep for 1.0 release #601
Description
Activity
@jbogaardt apologies if you've discussed this at meeting that I missed/spaced out on. by arrow backend, do you mean we'll entirely abandon the 5D numpy structure into a 2D columnar structure?
Yes, that was the initial thought - the idea being arrow has a much more active community than numpy/sparse. That said, there are some aspects of the models that are much easier to compute using arrays. Bootstrap resampling comes to mind, but a lot of triangle work is vector/array arithmetic which is better with arrays. So arrow is not a perfect fit and I think it requires more thought. It would be nice if there was a modern array library with dense and sparse support beyond 2D that capitalizes on some of the latest trends in compression/serialization. I think pydata/sparse remains the best option at this point.
yo, I think 2D is the solution to all our problems. I'm so sick of all the grain issues. i'm one more grain issue away from rewriting the triangle internals to force storing at OMDM, and only regraining for estimation and display. In 2D, we can keep claim-level accident date and dates of transactions. then only map them to o and d grain at run time and aggregate for estimation or display. basically 2D enables sparse (if I'm understanding the purpose of sparse correctly) support by default.
That's a more lazy (as opposed to eager) approach and has merit. It would certainly simplify the internals. It could also improve or degrade performance issues on larger datasets. The package was written to manage all data in RAM and
sparseis used to minimize the RAM footprint. Consider this example:import chainladder as cl prism_sparse = cl.load_sample('prism') prism_dense = prism_sparse.set_backend('numpy') print('Sparse Array Mb Usage:', prism_sparse.values.nbytes) print('Dense Array Mb Usage:', prism_dense.values.nbytes) print('Size Ratio:', prism_sparse.values.nbytes / prism_dense.values.nbytes)
Sparse Array Mb Usage: 2908272 Dense Array Mb Usage: 15779635200 Size Ratio: 0.0001843054014328544
Both numpy and sparse give the same API for matrix arithmetic (extremely useful for developing actuarial methods), but when throwing around in-memory data at very a granular level, a sparse representation is far more scalable. The competing forces we have are:
- Data management - I believe Arrow is superior and your lazy evaluation will keep data sparse
- Reserving Algorithms - Array-based math is superior. I think developing actuarial techniques using dataframe-style arithmetic is much less elegant.
There are a ton of ways to architect a better solution including hybrid approaches - I'm just not sure we fully explored the merits of different approaches holistically enough.
We can use caching to improve lazy evaluation speed.
On your comment regarding array math, maybe we can think of it as an opportunity for stricter API adherence. If I'm understanding your point correctly, you are talking about how easy it is to just plop down a tri.values[,,,1:] / tri.values[,,,:-1] versus clumsy window functions in columnar representation. But we can work around that with a dedicated get_sub_triangle() function in the API. then tri.get_sub_triangle(development, [1:]) / tri.get_sub_triangle(development, [:-1]) can be just as convenient as array math yet also allow an arrow implementation.
For array math, I was thinking more from the internals . Things like:
- Estimating development factors and their statistics.
- Estimating the hat matrix for the bootstrap method.
These are just a couple of higher complexity examples, but linear algebra operations on triangles are abundant in the internals and in research papers. Will DataFrame-style arithmetic be performant? Will we be trading lower code complexity in the Triangle class for higher code complexity in our estimators - and is this trade-off worth it?
I don't really know. The Triangle class is not perfect, but its relatively mature and mostly there. Being built on arrays seems to set us up for easier expansion of our list of estimators. I have a nagging feeling that chainladder should aspire to be more of a suite of actuarial techniques than a data management tools.
Alas we don't have to choose between actuarial techniques or data management. Here is an excerpt from Clark 2003, which we reference in the package but is also on the exam 7 syllabus currently. According to him, Arrow is integral to more techniques.
Let me think about how we can do some performance testing next.
I need close out a bunch of browser windows full of ideas so I'm recording them here:
Have any of you explored JAX? It looks like an all-in-one library with CPU, GPU, parallel processing, and sparse support, with a numpy-compatible API, which I thought could cut down on trying to maintain 4 separate backends. The main downside I see is that their sparse support is still listed as experimental. I think we can keep an eye on this one to see where it goes.
Since Polars is a dataframe and not a multidimensional array library, I suppose it's off the table as a backend option. I think there might be some performance improvements in select areas, such as I/O like when data are ingested into a DataFrame during the
Triangle__init__, or when they are output as DataFrames.26 remaining items
will vdims etc. go away immediately upon #1303 landing?
In the current version of #1303, yes: vdims and kdims are removed directly from Triangle (raising AttributeError), with setstate migrating legacy pickled instances into _axes and emitting a UserWarning. The initial draft had FutureWarning property shims, but we dropped them during review since they were internal implementation details rather than intended public API.
In #1304, odims and ddims currently keep FutureWarning property shims pointing to origin and development. If the team prefers having consistent FutureWarning shims for vdims and kdims across 0.11 leading up to 1.0 rather than dropping them immediately, restoring those property shims on Triangle is straightforward. Open to whichever deprecation path you prefer.
i see. i would prefer a consistent approach. but it sounds like you guys came across implementation challenges that resulted diverging approaches.
how much are you gonna be putting in to the
TriangleAxisAPI? as in, how much are we expecting users change how they work with the existing public axis properties? my default mental model would be- if users don't need to change how they currently work with
index,developmentetc., we can just put a more descriptive error message into all the xdims shims - if users need to change how they currently work, we probably should offer working shims with warnings as a crutch for a release or two.
- if users don't need to change how they currently work with
They shouldn't need to change how they currently work, as in, we're keeping the public API as-is (we are keeping index -> DataFrame, for example). If you try to even search for an xdim term in the docs you turn up with nothing, so I think justifiably we can just drop them, they were never meant to be part of the public API and not naming them with a preceding underscore was a mistake.
Metadata
Metadata
Type
Fields
Priority
Effort
Scope
On the path towards 1.0 there are several refactors worth considering that would simplify the code base and bring it more into software development best practices.
Backend support
To migrate to an arrow backend, we should minimize existing backend support to just numpy/sparse.
cupy and dask were always elective and not well maintained. They add unnecessary complexity to the project and should be removed.
Clarify Triangle public API
The Estimators occasionally manipulate the private internals of the Triangle class. This makes the Estimator code heavily coupled to the internals of the Triangle making refactors of Triangle internals difficult. We need to force etimators to only use the public API. This may include adding additional support properties and methods to the Triangle class that the estimators can call on as opposed to direct manipulation of the underlying triangle structure.
kdims, vdims, odims, ddims
These are internal aliases for index, columns, origin, development. They exist to bypass some of the consistency checks on these axes. These certainly shouldn't be in the public API, but even internal to a Triangle, they are difficult to manage. They should be refactored out of the Triangle with no changes to the public API.
None of these should change any of the exiting public API of the package and all test should continue to pass.