How Do I Merge A 2d Numpy Array With A Structured Array

I would like to know how to merge a 2d numpy array (n rows by m columns) with an existing structured array (with n rows). This sample code shows what I am trying to do: D = np.arr

Solution 1:

I couldn't find np.lib.recfunctions.merge_arrays. Maybe we have different numpy versions. Rather than dig around for it, I'll do the merger myself, using methods that I've seen in other rec functions.

Your samples:

In [2]: D = np.array([('a', 12), ('b', 14), ('c', 10)],
    dtype=np.dtype([('label','a2'),  ('ht', 'i2')]))    
In [3]: xxx =  np.array([[1,2,3],[5,6,7],[78,88,98]])

In [6]: D.dtype
Out[6]: dtype([('label', 'S2'), ('ht', '<i2')])

Define a new dtype for E. This could be built from D.dtype, but I'll just type it in.

In [9]: dt=np.dtype([('label', 'S2'), ('ht', '<i2'),
            ('xxx', xxx.dtype,xxx.shape[1])])

And a new blank array of the right shape and dtype:

In [11]: E=np.zeros(D.shape, dtype=dt)

In [12]: E
Out[12]: 
array([('', 0, [0, 0, 0]), ('', 0, [0, 0, 0]), ('', 0, [0, 0, 0])], 
      dtype=[('label', 'S2'), ('ht', '<i2'), ('xxx', '<i4', (3,))])

Now copy fields from D and xxx. This kind of is common in the rec functions. There isn't anything more streamlined for compound dtypes.

In [13]: for name in D.dtype.names:
   ....:     E[name] = D[name]    
In [14]: E['xxx']=xxx

In [15]: E
Out[15]: 
array([('a', 12, [1, 2, 3]), ('b', 14, [5, 6, 7]), ('c', 10, [78, 88, 98])], 
      dtype=[('label', 'S2'), ('ht', '<i2'), ('xxx', '<i4', (3,))])

That's your target, right?

---

For some reason I have to import recfunctions separately. That's not usual for numpy.

from numpy.libimport recfunctions

When mentioning merge_arrays and append_field, you really should describe what didn't work, whether it was an error, or you didn't like the result.

merge_array does work:

In [35]: e=recfunctions.merge_arrays((D,xxx))
Out[35]: 
array([(('a', 12), 1), (('b', 14), 2), (('c', 10), 3), (('-1', -1), 5),
       (('-1', -1), 6), (('-1', -1), 7), (('-1', -1), 78),
       (('-1', -1), 88), (('-1', -1), 98)], 
      dtype=[('f0', [('label', 'S2'), ('ht', '<i2')]), ('f1', '<i4')])

It's just that the 2 input arrays are separate fields:

In[38]: e['f0']Out[38]: 
array([('a', 12), ('b', 14), ('c', 10), ('-1', -1), ('-1', -1), ('-1', -1),('-1', -1), ('-1', -1), ('-1', -1)], 
      dtype=[('label', 'S2'), ('ht', '<i2')])

In[39]: e['f1']Out[39]: array([ 1,  2,  3,  5,  6,  7, 78, 88, 98])

append_fields works for a simple addition, but not for xxx with 3 columns

In [57]: recfunctions.append_fields(D,'xxx',[1,2,3],usemask=False)
Out[57]: 
array([('a', 12, 1), ('b', 14, 2), ('c', 10, 3)], 
      dtype=[('label', 'S2'), ('ht', '<i2'), ('xxx', '<i4')])

Or I could turn xxx into a structured array, and append that:

In [60]: x1=np.zeros((3,),dtype=[('xxx',int,(3,))])
In [61]: x1['xxx']=xxx
In [62]: x1
Out[62]: 
array([([1, 2, 3],), ([5, 6, 7],), ([78, 88, 98],)], 
      dtype=[('xxx', '<i4', (3,))])

In [63]: recfunctions.append_fields(D,'xxx',x1,usemask=False)
Out[63]: 
array([('a', 12, ([1, 2, 3],)), ('b', 14, ([5, 6, 7],)),
       ('c', 10, ([78, 88, 98],))], 
      dtype=[('label', 'S2'), ('ht', '<i2'), ('xxx', [('xxx', '<i4', (3,))])])

Closer, but not quite right. I could keep playing with these functions, but is it worth it?

---

These are equivalent, though recursive_fill handles more complex dtypes:

recfunctions.recursive_fill_fields(D,E)
fornamein D.dtype.names: E[name] = D[name]

Here's a way of building E.dtype with recfunctions, but there's got to be something simpler:

dt2=list(recfunctions.flatten_descr(np.dtype(recfunctions.zip_descr([D,x1]))))

Solution 2:

[(d,list(x)) for x,d in zip(xxx,D)]

gives

[(('a', 12), [1, 2, 3]), (('b', 14), [5, 6, 7]), (('c', 10), [78, 88, 98])]

and

np.array([(d,list(x)) for x,d in zip(xxx,D)])

gives

array([[('a', 12), [1, 2, 3]],
       [('b', 14), [5, 6, 7]],
       [('c', 10), [78, 88, 98]]], dtype=object)