-
Notifications
You must be signed in to change notification settings - Fork 319
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Describe the bug
lighteval inspect does not work due to some import and runtime errors
To Reproduce
lighteval tasks inspect "lighteval|gsm8k|0|0" --num-samples 1
PLEASE NOTE: this PR fixes it: #887
Expected behavior
Expected output:
---------- lighteval|gsm8k|0 ----------
---------- SAMPLES
-- sample 0 --
{ 'choices': [ ' Janet sells 16 - 3 - 4 = <<16-3-4=9>>9 duck eggs a day.\n'
'She makes 9 * 2 = $<<9*2=18>>18 every day at the farmer’s '
'market.\n'
'#### 18'],
'fewshot_samples': [],
'fewshot_sorting_class': None,
'generation_grammar': None,
'generation_size': None,
'gold_index': 0,
'id': '0',
'images': None,
'instruction': None,
'num_asked_few_shots': 0,
'num_effective_few_shots': 0,
'num_samples': 1,
'original_query': None,
'query': 'Question: Janet’s ducks lay 16 eggs per day. She eats three for '
'breakfast every morning and bakes muffins for her friends every '
"day with four. She sells the remainder at the farmers' market "
'daily for $2 per fresh duck egg. How much in dollars does she make '
"every day at the farmers' market?\n"
'Answer:',
'sampling_methods': [],
'specific': None,
'stop_sequences': None,
'task_name': 'gsm8k',
'unconditioned_query': None,
'use_logits': False}
Version info
lighteval installed from main branch
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working