Skip to content

[BUG] inspect command does not work #894

@uralik

Description

@uralik

Describe the bug

lighteval inspect does not work due to some import and runtime errors

To Reproduce

lighteval tasks inspect "lighteval|gsm8k|0|0" --num-samples 1

PLEASE NOTE: this PR fixes it: #887

Expected behavior

Expected output:

---------- lighteval|gsm8k|0 ----------
---------- SAMPLES
-- sample 0 --
{ 'choices': [ ' Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\n'
               'She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s '
               'market.\n'
               '#### 18'],
  'fewshot_samples': [],
  'fewshot_sorting_class': None,
  'generation_grammar': None,
  'generation_size': None,
  'gold_index': 0,
  'id': '0',
  'images': None,
  'instruction': None,
  'num_asked_few_shots': 0,
  'num_effective_few_shots': 0,
  'num_samples': 1,
  'original_query': None,
  'query': 'Question: Janet’s ducks lay 16 eggs per day. She eats three for '
           'breakfast every morning and bakes muffins for her friends every '
           "day with four. She sells the remainder at the farmers' market "
           'daily for $2 per fresh duck egg. How much in dollars does she make '
           "every day at the farmers' market?\n"
           'Answer:',
  'sampling_methods': [],
  'specific': None,
  'stop_sequences': None,
  'task_name': 'gsm8k',
  'unconditioned_query': None,
  'use_logits': False}

Version info

lighteval installed from main branch

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions