RFC: first draft of insertion API

[Thomas A Caswell (tacaswell)]

Don't merge this, I want to add a bit more before it goes in.

[Thomas A Caswell (tacaswell)]

So, this works:

In [54]: ac = AnalysisClient()

In [55]: a = ac.create_header('foobar', baz=123)

In [56]: a.add_result(dd=dd)
Out[56]: 
{'data': {'dd': '/home/tcaswell/.cache/ab/7c117203-0397-488d-a4cd-019ee9e9246e.csv'},
 'descriptor': {'auto_gen': True,
  'data_keys': {'dd': {'columns': ['bar', 'foo'],
    'dtype': 'table',
    'external': 'FILEPATH:csv',
    'shape': (5, 2)}},
  'header': {'baz': 123,
   'date': 1450925755.5471232,
   'name': 'foobar',
   'uid': '1f5bd54c-0589-4d31-8f2b-aca289638dc8'},
  'uid': '9c1ec35b-385d-4758-8202-89b9a4dcb956'},
 'uid': '788f21bd-486b-4a03-ab5c-2ac1bc9724b4'}

In [57]: r = _

In [58]: open_file(r.data['dd'], r.descriptor.data_keys['dd'])
Out[62]: 
   bar  foo
0    4    0
1    3    1
2    2    2
3    1    3
4    0    4

In [63]: dd
Out[63]: 
   bar  foo
0    4    0
1    3    1
2    2    2
3    1    3
4    0    4

In [64]: 

That is pretty cool.

[Dan Allan (danielballan)]

Neat. I don't immediately see why open_file needs the second arg.
On Wed, Dec 23, 2015 at 10:02 PM Thomas A Caswell notifications@github.com
wrote:

So, this works:

In [54]: ac = AnalysisClient()

In [55]: a = ac.create_header('foobar', baz=123)

In [56]: a.add_result(dd=dd)
Out[56]:
{'data': {'dd': '/home/tcaswell/.cache/ab/7c117203-0397-488d-a4cd-019ee9e9246e.csv'},
'descriptor': {'auto_gen': True,
'data_keys': {'dd': {'columns': ['bar', 'foo'],
'dtype': 'table',
'external': 'FILEPATH:csv',
'shape': (5, 2)}},
'header': {'baz': 123,
'date': 1450925755.5471232,
'name': 'foobar',
'uid': '1f5bd54c-0589-4d31-8f2b-aca289638dc8'},
'uid': '9c1ec35b-385d-4758-8202-89b9a4dcb956'},
'uid': '788f21bd-486b-4a03-ab5c-2ac1bc9724b4'}

In [57]: r = _

In [58]: open_file(r.data['dd'], r.descriptor.data_keys['dd'])
Out[62]:
bar foo0 4 01 3 12 2 23 1 34 0 4

In [63]: dd
Out[63]:
bar foo0 4 01 3 12 2 23 1 34 0 4

In [64]:

That is pretty cool.

—
Reply to this email directly or view it on GitHub
#3 (comment).

[Deleted user (ghost)]

I agree with Dan Allan (@danielballan) if descriptor will always be part of data and there won't be too many data_keys

[Thomas A Caswell (tacaswell)]

still todo (not all in this PR)

• make LPy compatible
• back with FS instead of ad-hoc writing to ~/.cache
• enforce the schemas
• talk to an database

[Thomas A Caswell (tacaswell)]

also use pip caching on travis (see https://github.com/matplotlib/basemap/pull/248/files) so we can get night running again.

[Thomas A Caswell (tacaswell)]

open_file takes in a str + a data_key entry, it needs the second arg to know how to open the file. This is a super light weight version of FS where you get

1. 1 datum per resource
2. no resource kwargs
3. the handler to be used is embedded after the : in the external field of the data key
4. you get one arg into the handler

Look at https://github.com/NSLS-II/analysisstore/pull/3/files#diff-0d61030312bd6e18e7b51877ec248cf6R215 for how I imagine open_file will be used.

[Eric Dill (ericdill)]

Why did this get renamed to analysisbucket?

[Eric Dill (ericdill)]

Does assert res.data['arr'] == arr work with py.test and numpy arrays?

[Thomas A Caswell (tacaswell)]

No, you get an exception that the truth value of an array is ambiguous.

[Dan Allan (danielballan)]

You could use all(), I guess, but the advantage of numpy tests is it checks
dtype and so on.
On Sat, Dec 26, 2015 at 1:44 PM Thomas A Caswell notifications@github.com
wrote:

In analysisbucket/tests/test_client.py
#3 (comment):

• md = {k: ord(k) for k in 'abcdef'}

• h = ac.create_header('testing', uid='fixed', time=0, **md)
• h = h.Document()

• tab = pd.DataFrame({'a': range(15), 'b': 'a'*15})
• arr = np.arange(15)

• res = ac.add_result(h, tab=tab, arr=arr)

• for d in [res['data'], res.descriptor.data_keys]:

•    for k in ['tab', 'arr']:
  

•        assert k in d
  

• res = fill_result(res)

• assert_array_equal(res.data['arr'], arr)

No, you get an exception that the truth value of an array is ambiguous.

—
Reply to this email directly or view it on GitHub
https://github.com/NSLS-II/analysisstore/pull/3/files#r48448981.

[Thomas A Caswell (tacaswell)]

Would it be possible to use these tests again? As with amostra, I really like the idea of having a 'trivial' reference implementation.

[Deleted user (ghost)]

Yes, when I separate the server/client. You beat me to asking you to open this PR when we have the asclient (or w/e we call it) repo

[Max Rakitin (mrakitin)]

Thomas A Caswell (@tacaswell), do you think it can be reused? The PR is quite old (travis config should be switched to GHA, of course), but the rest can be potentially useful.