[ENH] Consistent 3D output for single-target point predictions in TimeXer v1.

[PranavBhatP]

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Small change in the code to provide a 3d output tensor for point predictions with 3rd dimension set to 1. With this change the output contract for TimeXer is

• Point predictions: (batch_size, predictions, 1) where the 3rd dimension indicates a single target.
• Quantile predicitons: (batch_size, predictions, num_quantiles) where the 3rd dimension indicates the number of quantiles for which the output is generated.

What should a reviewer concentrate their feedback on?

PR checklist

• The PR title starts with either [ENH], [MNT], [DOC], or [BUG]. [BUG] - bugfix, [MNT] - CI, test framework, [ENH] - adding or improving code, [DOC] - writing or improving documentation or docstrings.
• Added/modified tests
• Used pre-commit hooks when committing to ensure that code is compliant with hooks. Install hooks with pre-commit install.
  To run hooks independent of commit, execute pre-commit run --all-files

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@f87bb56). Learn more about missing BASE report.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1936   +/-   ##
=======================================
  Coverage        ?   87.07%           
=======================================
  Files           ?      136           
  Lines           ?     8612           
  Branches        ?        0           
=======================================
  Hits            ?     7499           
  Misses          ?     1113           
  Partials        ?        0           

Flag	Coverage Δ
cpu	87.07% <100.00%> (?)
pytest	87.07% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[fkiraly]

I am worried that we are removing tests but not adding tests.

If we are removing an API stepout, then we should remove a stepout in the testes to match, no?

[PranavBhatP]

it was an unintended removal, happened by mistake, sorry. The code stays the same in test_timexer.py

[fkiraly]

ok, so it was a bug! Good that we do reviews.

Related question: is there a stepout we can remove now?

[PranavBhatP]

is there a stepout we can remove now?

with that question in mind, I tried tweaking with a few of the tensor shapes, and I was able to reduce the number of stepouts. now, the default return is a 3d tensor - (batch_size, prediction,_length, n_quantiles). For point predictions, n_quantiles = 1 by default, and for explicit quantiles we equate the n_quantiles=len(loss.quantiles) where loss has to be QuantileLoss

[fkiraly]

You mention you reduced the number of stepouts, but I do not see a change in tests - did you forget to push them?

[PranavBhatP]

no i meant stepouts in the actual model code. We were handling quantile and non-quantile predictions separately inside the model. But with this PR, I have unified the handling to a single format for all types of metrics used (quantile or point).

The code for the tests ultimately remains the same, since we are testing the output shape of predict API for TimeXer, which is not the concern of this PR.

If there is a misunderstanding, please let me know.

[fkiraly]

Yes, I meant tests.

I think thta one of the following must be true:

• there was a stepout in the tests that we can now remove
• the tests were not complete or were faulty (e.g., they are not covering the API this was a stepout from)

[agobbifbk]

Seems that all the shape are well managed. Here you are using the self.n_quantiles as indicator of the number of output per channel. What if I want to use a distribution loss here? Should we rename it in a more general way?

[fkiraly]

Repeating my review request:

• if there was a stepout for the forecaster in tests, please remove the stepout
• if there was no stepout, please review the tests why there was no stepout and/or why they still passed

[PranavBhatP]

Here you are using the self.n_quantiles as indicator of the number of output per channel. What if I want to use a distribution loss here? Should we rename it in a more general way?

I see a possibility of using DistributionLoss metrics with TimeXer, and it might need a few tweaks to get it working. I don't think it works in the current sense. I think we can maybe park this as an item for a separate PR, since we have to again add tests for its usage with distribution losses.

[PranavBhatP]

Repeating my review request:
if there was a stepout for the forecaster in tests, please remove the stepout
if there was no stepout, please review the tests why there was no stepout and/or why they still passed

I looked at the code, and I think there is missing test. Until now we were only testing the output for model.predict. The reason why the tests were passing was because the output of predict remains consistent, independent of the input format as it is dependent on the metric used in the model. However there is nothing to test the output of the model from its forward method. This is pending work and was missed previously, imo. I will add the test in this PR.