Fix binding difference between callables and callback protocols

[hauntsaninja]

Fixes #15926 (comment), fixes #16200

[ilevkivskyi]

Hm, it looks like all new errors are technically correct. NumPy stubs take into account current (inconsistent) mypy's behavior. I am not sure what is the best way forward: even if we fix the NumPy stubs, everyone who doesn't pin mypy will get errors until they update NumPy stubs, and vice versa.

[RonnyPfannschmidt]

would it be feasible to have a per module configuration option that helps to annotate whether or not to do the check,

that way one could opt-in to the new way and the numpy specs could gradually fix their types

ideally this would warn in a way that helps people figure what flags to set at either definition or usage site

[RonnyPfannschmidt]

im working around this using a Concatenate and mypy_extensions.Arg for now

[RonnyPfannschmidt]

i thik i now understand the underlying issue

its structurally incorrect to do the first parameters strop off

instead its necessary to be aware of the method descriptor protocol and applying it

so any callable protocol would need a __get__ that returns the correct method version for the callable

and for callables its also a bug to pretend they bind, in particular when __call__ using objects would be valid pass ins that do in fact not run any __get__

i'd like if there was a typing helper for this so one could spell Binding/nonBinding locations for those

[RonnyPfannschmidt]

i learned that the workaround with mypy_extensions no longer works

[ilevkivskyi]

The relevant definitions in numpy: https://github.com/search?q=repo%3Anumpy%2Fnumpy%20_SignedIntOp&type=code. Possibly the only way forward is to add some heavy hard-coded special-casing while numpy definitions are being fixed.

cc @jorenham It may be good to finally fix this little inconsistency in mypy, see (#15993 (comment) for some more details).

[jorenham]

Yea that part of the numpy stubs has been bothering me for a while now. I've made some progress in fixing those "methods" over the last year, but there's indeed still work to be done.

Keep in mind that if I fix this, and I do so by the book, then it'll be in numpy 2.4, which will be released in 5 months or so. If we also take the SPEC 0 support window into account (which we should, because in general numpy users often use older versions), which means that we should add another two years to this.

What I think I'll do is fix this, and if mypy_primer remains silent, then I'll try to sneak it in as a 2.3 patch release. It's a bit risky, but it'll save half a year of incorrect behavior, so it's probably worth risk. That still leaves the 2 year-long support window though...

Special casing numpy also doesn't sound very attractive to me either. But I also don't want to be the reason that mypy can't proceed with this. Maybe we should wait until there is at least one numpy version where this will work correctly (i.e. ignore the support window), so that we can tell numpy users that are affected by this that they should upgrade to the latest numpy release? In the worst case, that will be 5 months, and in the best case that'll be one month.

Either way, thanks for the ping; I really appreciate it :)

[jorenham]

The NumPy issues have been fixed (on main) and will be included in 2.3.4 or 2.4.0, whichever comes first.

[jorenham]

The NumPy issues have been fixed (on main) and will be included in 2.3.4 or 2.4.0, whichever comes first.

The 2.3.4 release is planned for this weekend

[ilevkivskyi]

@jorenham Great, thanks for the update!

[jorenham]

The NumPy issues have been fixed (on main) and will be included in 2.3.4 or 2.4.0, whichever comes first.

The 2.3.4 release is planned for this weekend

There was a bit of a delay apparently, but the 2.3.4 wheels have just been uploaded to PyPI: https://pypi.org/project/numpy/

[github-actions]

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅