Groupby.mode() - feature request

[j-musial]

Feature Request

Hi,

I use pandas a lot in my projects and I got stack with a problem of running the "mode" function (most common element) on a huge groupby object. There are few solutions available using aggregate and scipy.stats.mode, but they are unbearably slow in comparison to e.g. groupby.mean(). Thus, I would like to make a feature request to add cytonized version of groupby.mode() operator.

Thanks in advance!

Jan Musial

[chris-b1]

xref #19165 (master tracker)

A PR would be welcome! As you note the key thing would be implementing a single-pass group mode function in cython, can look at others for examples.

pandas/pandas/_libs/groupby_helper.pxi.in

Line 213 in aa9e002

def group_mean_{{name}}(ndarray[{{dest_type2}}, ndim=2] out,

[jreback]

how is this different that .value_counts() ?

[j-musial]

Hi Jeff, As you can see below .value_counts() does not apply to groupby object. Furthermore, returning sorted values and counts within thousands/millions of groups gives huge overheads, whereas all you want is the most frequent value. Best, Jan

>>

pd.DataFrame({'a':['a','a','a','a','b','b','b','b'],'b':[1,1,2,3,1,2,2,3]}).groupby('a').value_counts() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib/python2.7/dist-packages/pandas/core/groupby.py", line 529, in __getattr__ (type(self).__name__, attr)) AttributeError: 'DataFrameGroupBy' object has no attribute 'value_counts'

>> pd.__version__

'0.19.2'

…

On Tue, Jan 16, 2018 at 12:35 AM, Jeff Reback ***@***.***> wrote: how is this different that .value_counts() ? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AX63CvX5BIOlK47m-61a26oBEGAvrJTOks5tK-DBgaJpZM4Re3QD> .

[jreback]

.value_counts() is a series method. I guess mode would simply give back the max per group.

In [32]: pd.options.display.max_rows=12

In [33]: ngroups = 100; N = 100000; np.random.seed(1234)

In [34]: df = pd.DataFrame({'key': np.random.randint(0, ngroups, size=N), 'value': np.random.randint(0, 10000, size=N)})

In [35]: df.groupby('key').value.value_counts()
Out[35]: 
key  value
0    5799     3
     7358     3
     8860     3
     185      2
     583      2
     872      2
             ..
99   9904     1
     9916     1
     9922     1
     9932     1
     9935     1
     9936     1
Name: value, Length: 95112, dtype: int64

[j-musial]

Thanks Jeff! Unlike .value_counts() the .mode() would act like a reduction function returning just one value. Returning entire histogram just to get the most frequent value is huge waste of computing resources. Best, Jan

…

On Tue, Jan 16, 2018 at 12:14 PM, Jeff Reback ***@***.***> wrote: .value_counts() is a series method. I guess mode would simply give back the max per group. In [32]: pd.options.display.max_rows=12 In [33]: ngroups = 100; N = 100000; np.random.seed(1234) In [34]: df = pd.DataFrame({'key': np.random.randint(0, ngroups, size=N), 'value': np.random.randint(0, 10000, size=N)}) In [35]: df.groupby('key').value.value_counts() Out[35]: key value 0 5799 3 7358 3 8860 3 185 2 583 2 872 2 .. 99 9904 1 9916 1 9922 1 9932 1 9935 1 9936 1 Name: value, Length: 95112, dtype: int64 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#19254 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AX63CnSbVx2jD-KDExPJYtY0eLmF_Qp6ks5tLIR7gaJpZM4Re3QD> .

[bfarrer]

If I have a DataFrame that includes a column of cities and I want to know the most frequent city to occur on the list. I'm pretty new to python and pandas, so maybe there's an easy alternative, but I'm not aware of one.

[BrittonWinterrose]

I am interested in this feature as well. +1