Segmentation fault in debug mode

[ufownl]

When I built and ran in debug mode, I got a warning in allocator.h followed by a segmentation fault:

  __ _  ___ _ __ ___  _ __ ___   __ _   ___ _ __  _ __
 / _` |/ _ \ '_ ` _ \| '_ ` _ \ / _` | / __| '_ \| '_ \
| (_| |  __/ | | | | | | | | | | (_| || (__| |_) | |_) |
 \__, |\___|_| |_| |_|_| |_| |_|\__,_(_)___| .__/| .__/
  __/ |                                    | |   | |
 |___/                                     |_|   |_|


tokenizer                     : [no path specified]
weights                       : gemma2-2b-it-sfp-single.sbs
compressed_weights            : [no path specified]
model                         : 
weight_type                   : sfp
max_generated_tokens          : 2048
prefill_tbatch                : 64
decode_qbatch                 : 16
multiturn                     : 1
image_file                    : [no path specified]

*Usage*
  Enter an instruction and press enter (%C resets conversation, %Q quits).

*Examples*
  - Write an email to grandma thanking her for the cookies.
  - What are some historical attractions to visit around Massachusetts?
  - Compute the nth fibonacci number in javascript.
  - Write a standup comedy bit about GPU programming.

> Hey, man!

[ Reading prompt ] Warn at allocator.h:237: Check why RowPtr stride=2048 < StrideForCyclicOffsets(cols=2048), T=4; this forces us to disable cyclic offsets.
Segmentation fault (core dumped)

Maybe there are hidden problems introduced.

[jan-wassenberg]

Thank you for raising the issue. I just ran an (internal, not CMake) asan build and also see the warning (this is basically a TODO until we soon have support for padded rows), but no crash on gemma2 9B and gemma1 2B.
What kind of debug build did you mean?

[ufownl]

What kind of debug build did you mean?

Build it with the flag -DCMAKE_BUILD_TYPE=Debug using CMake.

I also tested gemma1 2B today and performed the same as above, crashed after that warning. There's no crash if set the flag -DCMAKE_BUILD_TYPE=Release.

One more thing, I built it on macOS with this option and the linker complained with undefined symbol error:

Undefined symbols for architecture x86_64:
  "___truncsfbf2", referenced from:
      hwy::bfloat16_t hwy::ConvertScalarTo<hwy::bfloat16_t, float, (void*)0, (void*)0, (void*)0>(float) in libgemma.a[12](common.cc.o)
      hwy::BF16FromF32(float) in libgemma.a[12](common.cc.o)
      hwy::bfloat16_t::FromBits(unsigned short) in libgemma.a[12](common.cc.o)
      __ZN3hwy10bfloat16_tC1EDF16b in libgemma.a[12](common.cc.o)
      __ZN3hwy10bfloat16_tC1EDF16b in libgemma.a[12](common.cc.o)
      __ZN3hwy10bfloat16_tC2EDF16b in libgemma.a[12](common.cc.o)
      hwy::bfloat16_t hwy::ConvertScalarTo<hwy::bfloat16_t, float, (void*)0, (void*)0, (void*)0>(float) in libgemma.a[15](bf16.cc.o)
      ...
ld: symbol(s) not found for architecture x86_64

[jan-wassenberg]

Thanks. We haven't yet investigated the debug build, though I'm curious as to why they used/useful? (We very often run asan/msan, but I'd think debug builds are too slow.)

On ___truncsfbf2, this is an unfortunate issue introduced in LLVM, where the compiler-rt library does not match what the compiler requires. See also JuliaLang/julia#52067. I think it can be fixed by updating the compiler, or at least getting clang and compiler-rt from the same place.

[ufownl]

I used the debug build to locate the crash in my project and eventually discovered through enabled assertions that it was because Allocator::Init was not called. :)

[jan-wassenberg]

Sorry about that. It's not an ideal interface, but there are low-level helper functions that depend on Allocator and it's not clear how to better/more safely handle this.

Do I understand correctly that the crash was caused by not calling Init, and we're good? Or is the crash still happening?
(FYI the particular warning mentioned should not go away after calling Init, it is related to the incomplete handling of tensor strides, which is being updated shortly.)

[ufownl]

Do I understand correctly that the crash was caused by not calling Init, and we're good? Or is the crash still happening?

The crash due to not calling Allocator::Init was another matter and has been solved in my project. The crash after that warning will still occur in debug mode, but it does not affect usage in release mode.

[ufownl]

@jan-wassenberg I tried running the executable built in debug mode using gdb on Linux, and it complained with the following error:

Thread 1 "P0 X00 C005" received signal SIGSEGV, Segmentation fault.
0x000055555561c731 in _mm256_load_pd (__P=0x7ffffffe3db0) at /usr/lib/gcc/x86_64-linux-gnu/9/include/avxintrin.h:862
862	  return *(__m256d *)__P;

Backtrace:

#0  0x000055555561c731 in _mm256_load_pd (__P=0x7ffffffe3db0) at /usr/lib/gcc/x86_64-linux-gnu/9/include/avxintrin.h:862
#1  hwy::N_AVX2::Load<hwy::N_AVX2::Simd<double, 4, 0> > (aligned=0x7ffffffe3db0) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/ops/x86_256-inl.h:3471
#2  0x000055555576df93 in gcpp::N_AVX2::MMAddHorizontalSumsIntoPartial<2ul, gcpp::N_AVX2::MMSetPartial>::MaybeLoad<0ul, hwy::N_AVX2::Simd<double, 4ul, 0>, hwy::N_AVX2::Vec256<double> > (d4=..., N=4, 
    buf=0x7ffffffe3db0) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:313
#3  0x0000555555735b48 in gcpp::N_AVX2::MMAddHorizontalSumsIntoPartial<2ul, gcpp::N_AVX2::MMSetPartial>::operator()<hwy::N_AVX2::Simd<float, 8ul, 0>, hwy::N_AVX2::Vec256<float> > (this=0x7ffffffe44e0, 
    df=..., F00=..., F01=..., F02=..., F03=..., F10=..., F11=..., F12=..., F13=..., F20=..., F21=..., F22=..., F23=..., F30=..., F31=..., F32=..., F33=..., row_c=0, col_c=1760, partial=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:267
#4  0x00005555556f1f1c in gcpp::N_AVX2::MMKernel::LoopKC<2ul, gcpp::N_AVX2::MMSetPartial, float> (A_view=..., B_view=..., row_ac=0, imc=0, col_c=1760, kc=1280, tag=..., args=..., C=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:624
#5  0x00005555556b7af2 in gcpp::N_AVX2::MMKernel::A2C0<gcpp::N_AVX2::MMSetPartial, float> (A_view=..., B_view=..., mr=2, range_mc=..., row_b=1760, kc=1280, tag=..., args=..., C=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:380
#6  0x0000555555983e04 in gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(hwy::bfloat16_t*, gcpp::IndexRange const&, gcpp::IndexRange const&, auto:1)#1}::operator()<gcpp::N_AVX2::MMSetPartial>(hwy::bfloat16_t*, gcpp::IndexRange const&, gcpp::IndexRange const&, gcpp::N_AVX2::MMSetPartial) const (
    this=0x7fffffff9cd0, B_storage=0x7ffffffe4760, range_kc=..., range_nc=..., out_tag=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:984
#7  0x0000555555983c17 in gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1}::operator()(gcpp::IndexRange const&) const (this=0x7ffffffe4740, range_kc=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:997
#8  0x0000555555992edf in gcpp::IndexRangePartition::VisitFirst<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1}>(gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const::{lambda(gcpp::IndexRange const&)#1} const&) const
    (this=0x7fffffff9d68, func=...) at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:247
#9  0x0000555555983b83 in gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}::operator()(gcpp::IndexRange const&) const (this=0x7fffffff9cd0, range_nc=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:996
#10 0x0000555555992fb2 in gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&)::{lambda(gcpp::IndexRange const&, unsigned long)#1}::operator()(gcpp::IndexRange const&, unsigned long) const
    (this=0x7fffffff9b30, worker_range=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:99
#11 0x000055555599d6f8 in _ZZN4gcpp19ParallelizeOneRangeIZNS_10MMParallel5ForNPIZNKS_6N_AVX212MMPerPackage6DoNT_KINS_9SfpStreamEfEEvRKNS_8ConstMatIT_EEmRKNS_6RowPtrIT0_EEEUlRKNS_10IndexRangeEE0_EEvSJ_mmmRKS8_EUlSJ_mE_EEvRKNS_19IndexRangePartitionERN3hwy10ThreadPoolESM_ENKUlmmE_clEmm (this=0x7fffffff9ab0, task=5, thread=5) at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:297
#12 0x00005555559b8b18 in _ZN3hwy4pool5Tasks11CallClosureIZN4gcpp19ParallelizeOneRangeIZNS3_10MMParallel5ForNPIZNKS3_6N_AVX212MMPerPackage6DoNT_KINS3_9SfpStreamEfEEvRKNS3_8ConstMatIT_EEmRKNS3_6RowPtrIT0_EEEUlRKNS3_10IndexRangeEE0_EEvSN_mmmRKSC_EUlSN_mE_EEvRKNS3_19IndexRangePartitionERNS_10ThreadPoolESQ_EUlmmE_EEvPKvmm (opaque=0x7fffffff9ab0, task=5, worker=5)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:540
#13 0x0000555555579a08 in hwy::pool::Tasks::WorkerRunSingle (this=0x555556df15b0, worker=5) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:485
#14 0x00005555555798fd in hwy::pool::Tasks::WorkerRun (this=0x555556df15b0, worker=0x555556df1980) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:469
#15 0x0000555555582ed7 in hwy::pool::MainAdapter::operator()<hwy::SpinPause, hwy::pool::WaitSpin1, hwy::pool::BarrierCounter1> (this=0x555556df1598, spin=..., wait=..., barrier=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:834
#16 0x00005555555807ef in hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&>::operator()<hwy::SpinPause, hwy::pool::WaitSpin1> (this=0x7fffffff99e0, spin=..., wait=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:770
#17 0x000055555557ea2d in hwy::pool::FunctorAddWait<hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&> >::operator()<hwy::SpinPause> (this=0x7fffffff99f0, spin=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:730
#18 0x000055555557d36b in hwy::CallWithSpin<hwy::pool::FunctorAddWait<hwy::pool::FunctorAddBarrier<hwy::pool::MainAdapter&> > > (spin_type=hwy::SpinType::kPause, func=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/spin.h:321
#19 0x000055555557bc08 in hwy::pool::CallWithConfig<hwy::pool::MainAdapter&> (config=..., func=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:788
#20 0x00005555559aab15 in _ZN3hwy10ThreadPool3RunIZN4gcpp19ParallelizeOneRangeIZNS2_10MMParallel5ForNPIZNKS2_6N_AVX212MMPerPackage6DoNT_KINS2_9SfpStreamEfEEvRKNS2_8ConstMatIT_EEmRKNS2_6RowPtrIT0_EEEUlRKNS2_10IndexRangeEE0_EEvSM_mmmRKSB_EUlSM_mE_EEvRKNS2_19IndexRangePartitionERS0_SP_EUlmmE_EEvmmSP_ (this=0x555556df1500, begin=0, end=6, closure=...)
    at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:1065
#21 0x000055555599d76f in gcpp::ParallelizeOneRange<gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&)::{lambda(gcpp::IndexRange const&, unsigned long)#1}> (get1=..., pool=..., func=...)
    at /home/ufownl/Softwares/gemma.cpp/./util/threading.h:295
#22 0x00005555559931f3 in gcpp::MMParallel::ForNP<gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2}>(gcpp::IndexRange const&, unsigned long, unsigned long, unsigned long, gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float>(gcpp::ConstMat<gcpp::SfpStream> const&, unsigned long, gcpp::RowPtr<float> const&) const::{lambda(gcpp::IndexRange const&)#2} const&) (this=0x7fffffffd788, range_np=..., nx_multiple=16, inner_tasks=1, pkg_idx=0, func=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:96
#23 0x000055555598427e in gcpp::N_AVX2::MMPerPackage::DoNT_K<gcpp::SfpStream, float> (this=0x7fffffff9cd0, B=..., num_packed_B=5242880, C=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:989
#24 0x00005555559755d9 in gcpp::N_AVX2::MMPerPackage::operator()<gcpp::SfpStream, float> (this=0x7fffffff9cd0, B=..., C=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:898
#25 0x0000555555968ed8 in gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}::operator()(unsigned long) const (this=0x7fffffff9ef0, pkg_idx=0) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:1282
#26 0x0000555555975783 in gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&)::{lambda(unsigned long, unsigned long)#1}::operator()(unsigned long, unsigned long) const (this=0x7fffffff9e80, task=0, pkg_idx=0) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:79
#27 0x00005555559851a6 in hwy::ThreadPool::Run<gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&)::{lambda(unsigned long, unsigned long)#1}> (
    this=0x555556dec480, begin=0, end=1, closure=...) at /home/ufownl/Softwares/gemma.cpp/build/_deps/highway-src/hwy/contrib/thread_pool/thread_pool.h:1044
#28 0x000055555597580c in gcpp::MMParallel::ForPkg<gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1}>(unsigned long, gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float>(gcpp::ConstMat<float> const&, gcpp::ConstMat<gcpp::SfpStream> const&, gcpp::RowPtr<float> const&, gcpp::MMArgs const&, gcpp::MMConfig const&)::{lambda(unsigned long)#1} const&) (this=0x7fffffffd788, max_packages=1, func=...)
    at /home/ufownl/Softwares/gemma.cpp/./ops/matmul.h:75
#29 0x0000555555968f8e in gcpp::N_AVX2::MMImpl::DoMatMul<float, gcpp::SfpStream, float> (A=..., B=..., C=..., args=..., config=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:1279
#30 0x0000555555955a5e in gcpp::N_AVX2::MatMul<float, gcpp::SfpStream, float> (A=..., B=..., add=0x0, env=..., C=...) at /home/ufownl/Softwares/gemma.cpp/./ops/matmul-inl.h:1360
#31 0x000055555596792e in gcpp::N_AVX2::GemmaAttention<gcpp::SfpStream>::ComputeQKV (this=0x7fffffffae90, num_interleaved=12) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:267
#32 0x0000555555953a9b in gcpp::N_AVX2::GemmaAttention<gcpp::SfpStream>::operator() (this=0x7fffffffae90) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:511
#33 0x0000555555942b2a in gcpp::N_AVX2::Attention<gcpp::SfpStream> (type=gcpp::LayerAttentionType::kGemma, queries_pos=..., queries_prefix_end=..., num_tokens=12, layer=0, activations=..., 
    layer_weights=0x555779b80000, div_seq_len=..., kv_caches=...) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:576
#34 0x000055555593d59f in gcpp::N_AVX2::TransformerLayer<gcpp::SfpStream> (queries_pos=..., queries_prefix_end=..., num_tokens=12, cache_layer_idx=0, layer_weights=0x555779b80000, activations=..., 
    div_seq_len=..., kv_caches=...) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:968
#35 0x0000555555935f41 in gcpp::N_AVX2::Prefill<gcpp::SfpStream> (queries_prompt=..., queries_pos=..., queries_prefix_end=..., query_idx_start=0, weights=..., activations=..., runtime_config=..., 
    div_seq_len=..., kv_caches=...) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:1121
#36 0x0000555555930600 in gcpp::N_AVX2::GenerateT<gcpp::SfpStream> (model=..., activations=..., runtime_config=..., queries_prompt=..., queries_pos_in=..., queries_prefix_end=..., query_idx_start=0, 
    kv_caches=..., timing_info=...) at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:1457
#37 0x000055555592b055 in gcpp::N_AVX2::GenerateSingleT<gcpp::SfpStream> (model=..., runtime_config=..., prompt=..., pos=0, prefix_end=0, kv_cache=..., env=0x7fffffffd780, timing_info=...)
    at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:1517
#38 0x000055555590aa3d in gcpp::GenerateSingle (model=..., runtime_config=..., prompt=..., pos=0, prefix_end=0, kv_cache=..., env=0x7fffffffd780, timing_info=...)
    at /home/ufownl/Softwares/gemma.cpp/./gemma/gemma-inl.h:1591
#39 0x00005555555fb72c in gcpp::GenerateSingleT<gcpp::SfpStream>::operator() (this=0x7fffffffbd87, model=..., runtime_config=..., prompt=..., pos=0, prefix_end=0, kv_cache=..., env=0x7fffffffd780, 
    timing_info=...) at /home/ufownl/Softwares/gemma.cpp/gemma/gemma.cc:98
#40 0x00005555555f9a7d in gcpp::ModelWeightsStorage::CallForModelWeight<gcpp::GenerateSingleT, gcpp::RuntimeConfig const&, hwy::Span<int const> const&, unsigned long&, unsigned long&, gcpp::KVCache&, gcpp::MatMulEnv*, gcpp::TimingInfo&> (this=0x7fffffffd990) at /home/ufownl/Softwares/gemma.cpp/./gemma/weights.h:597
#41 0x00005555555f5c02 in gcpp::Gemma::Generate (this=0x7fffffffd8f0, runtime_config=..., prompt=..., pos=0, prefix_end=0, kv_cache=..., timing_info=...)
    at /home/ufownl/Softwares/gemma.cpp/gemma/gemma.cc:132
#42 0x0000555555577f0f in gcpp::ReplGemma(gcpp::Gemma&, gcpp::KVCache&, gcpp::AppArgs const&, gcpp::InferenceArgs const&, std::function<bool (int, float)> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) (model=..., kv_cache=..., app=..., args=..., accept_token=..., eot_line="") at /home/ufownl/Softwares/gemma.cpp/gemma/run.cc:202
#43 0x00005555555784ce in gcpp::Run (loader=..., inference=..., app=...) at /home/ufownl/Softwares/gemma.cpp/gemma/run.cc:265
#44 0x00005555555788a7 in main (argc=9, argv=0x7fffffffddc8) at /home/ufownl/Softwares/gemma.cpp/gemma/run.cc:298

I also tried to run the executable built in release mode on macOS, when I used Ctrl+D to end the conversation, it complained with the following error (even though I just typed Ctrl+D directly after loading the weights):

gemma(69902,0x7ff847af0b00) malloc: Incorrect checksum for freed object 0x7fd1a8704868: probably modified after being freed.
Corrupt value: 0x2000000000000030
gemma(69902,0x7ff847af0b00) malloc: *** set a breakpoint in malloc_error_break to debug
[1]    69902 abort      ./gemma --tokenizer tokenizer.spm --weights 2.0-2b-it-sfp.sbs --model   1

[jan-wassenberg]

Thank you for sharing the debugger output, this is very helpful. The issue is an aligned load, I forgot to add HWY_ALIGN in front of "buf[" in matmul-inl.h. There are two such occurrences. Fixing shortly :)

[ufownl]

@jan-wassenberg the crash on GCC should be fixed by #532 . However, the core dump on macOS does not seem to be caused by this, and it still exists.

I set a breakpoint in malloc_error_break as the hint above. Sometimes, it will hit at the beginning of running, and has the following backtrace:

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007ff804bdcc5d libsystem_malloc.dylib`malloc_error_break
    frame #1: 0x00007ff804bbb2b5 libsystem_malloc.dylib`malloc_vreport + 730
    frame #2: 0x00007ff804be332a libsystem_malloc.dylib`malloc_zone_error + 183
    frame #3: 0x00007ff804bb1240 libsystem_malloc.dylib`tiny_malloc_from_free_list + 2045
    frame #4: 0x00007ff804bb0472 libsystem_malloc.dylib`tiny_malloc_should_clear + 320
    frame #5: 0x00007ff804baf272 libsystem_malloc.dylib`szone_malloc_should_clear + 61
    frame #6: 0x00007ff804d56324 libc++abi.dylib`operator new(unsigned long) + 52
    frame #7: 0x0000000100772dd8 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] void* std::__1::__libcpp_operator_new[abi:ne180100]<unsigned long>(__args=336) at new:303:10 [opt]
    frame #8: 0x0000000100772dd0 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] std::__1::__libcpp_allocate[abi:ne180100](__size=336, __align=8) at new:327:10 [opt]
    frame #9: 0x0000000100772dd0 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] std::__1::allocator<gcpp::BoundedTopology::Cluster>::allocate[abi:ne180100](this=<unavailable>, __n=<unavailable>) at allocator.h:125:32 [opt]
    frame #10: 0x0000000100772dc9 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] std::__1::__allocation_result<std::__1::allocator_traits<std::__1::allocator<gcpp::BoundedTopology::Cluster>>::pointer> std::__1::__allocate_at_least[abi:ne180100]<std::__1::allocator<gcpp::BoundedTopology::Cluster>>(__alloc=<unavailable>, __n=<unavailable>) at allocate_at_least.h:55:19 [opt]
    frame #11: 0x0000000100772dc9 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] std::__1::__split_buffer<gcpp::BoundedTopology::Cluster, std::__1::allocator<gcpp::BoundedTopology::Cluster>&>::__split_buffer(this=<unavailable>, __cap=<unavailable>, __start=0, __a=0x00007ff7bfefea20) at __split_buffer:343:25 [opt]
    frame #12: 0x0000000100772dc9 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) [inlined] std::__1::__split_buffer<gcpp::BoundedTopology::Cluster, std::__1::allocator<gcpp::BoundedTopology::Cluster>&>::__split_buffer(this=<unavailable>, __cap=<unavailable>, __start=0, __a=0x00007ff7bfefea20) at __split_buffer:339:32 [opt]
    frame #13: 0x0000000100772dc9 gemma`gcpp::BoundedTopology::Package::Package(hwy::BitSet4096<1024ul> const&, hwy::Topology const&, unsigned long, gcpp::BoundedSlice) at vector:1425:49 [opt]
    frame #14: 0x0000000100772da6 gemma`gcpp::BoundedTopology::Package::Package(this=0x00007ff7bfefea10, enabled_lps=0x00007ff7bfefeb60, topology=0x00007ff7bfeff030, pkg_idx=0, cluster_slice=(skip_ = 0, max_ = 0)) at topology.cc:189:12 [opt]
    frame #15: 0x0000000100771c53 gemma`gcpp::BoundedTopology::InitFromTopology(hwy::BitSet4096<1024ul> const&, gcpp::BoundedSlice, gcpp::BoundedSlice) [inlined] gcpp::BoundedTopology::Package::Package(this=0x00007ff7bfefea10, enabled_lps=0x00007ff7bfefeb60, topology=0x00007ff7bfeff030, pkg_idx=0, cluster_slice=(skip_ = 0, max_ = 0)) at topology.cc:184:63 [opt]
    frame #16: 0x0000000100771c2c gemma`gcpp::BoundedTopology::InitFromTopology(hwy::BitSet4096<1024ul> const&, gcpp::BoundedSlice, gcpp::BoundedSlice) [inlined] gcpp::BoundedTopology::InitFromTopology(hwy::BitSet4096<1024ul> const&, gcpp::BoundedSlice, gcpp::BoundedSlice)::$_0::operator()(this=<unavailable>, pkg_idx=0) const at topology.cc:282:17 [opt]
    frame #17: 0x0000000100771c2c gemma`gcpp::BoundedTopology::InitFromTopology(hwy::BitSet4096<1024ul> const&, gcpp::BoundedSlice, gcpp::BoundedSlice) at topology.h:62:7 [opt]
    frame #18: 0x0000000100771be1 gemma`gcpp::BoundedTopology::InitFromTopology(this=<unavailable>, enabled_lps=0x00007ff7bfefeb60, package_slice=(skip_ = 0, max_ = 0), cluster_slice=(skip_ = 0, max_ = 0)) at topology.cc:280:17 [opt]
    frame #19: 0x00000001007714ce gemma`gcpp::BoundedTopology::BoundedTopology(this=0x00007ff7bfeff030, package_slice=(skip_ = 0, max_ = 0), cluster_slice=(skip_ = 0, max_ = 0), lp_slice=(skip_ = 0, max_ = 0)) at topology.cc:91:5 [opt]
    frame #20: 0x0000000100002c5e gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] gcpp::CreateTopology(app=0x00007ff7bfeff1a0) at app.h:121:10 [opt]
    frame #21: 0x0000000100002c2b gemma`gcpp::Run(loader=0x00007ff7bfeff200, inference=0x00007ff7bfeff288, app=0x00007ff7bfeff1a0) at run.cc:235:36 [opt]
    frame #22: 0x0000000100003721 gemma`main(argc=<unavailable>, argv=<unavailable>) at run.cc:303:5 [opt]
    frame #23: 0x00007ff804a0d2cd dyld`start + 1805

And the hit after typing Ctrl+D, there is the following traceback:

* thread #1, name = 'P0 X01 C000', queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x00007ff804bdcc5d libsystem_malloc.dylib`malloc_error_break
    frame #1: 0x00007ff804bbb2b5 libsystem_malloc.dylib`malloc_vreport + 730
    frame #2: 0x00007ff804be332a libsystem_malloc.dylib`malloc_zone_error + 183
    frame #3: 0x00007ff804bb3d5b libsystem_malloc.dylib`tiny_free_list_remove_ptr + 578
    frame #4: 0x00007ff804bb311c libsystem_malloc.dylib`tiny_free_no_lock + 650
    frame #5: 0x00007ff804bb2d67 libsystem_malloc.dylib`free_tiny + 526
    frame #6: 0x000000010000320d gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] void std::__1::__libcpp_operator_delete[abi:ne180100]<void*>(__args=<unavailable>) at new:312:3 [opt]
    frame #7: 0x0000000100003208 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] void std::__1::__do_deallocate_handle_size[abi:ne180100]<>(__ptr=<unavailable>, __size=<unavailable>) at new:334:10 [opt]
    frame #8: 0x0000000100003208 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::__libcpp_deallocate[abi:ne180100](__ptr=<unavailable>, __size=<unavailable>, __align=8) at new:349:12 [opt]
    frame #9: 0x0000000100003208 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::allocator<hwy::Topology::Cluster>::deallocate[abi:ne180100](this=0x00006000037fc040, __p=<unavailable>, __n=<unavailable>) at allocator.h:139:7 [opt]
    frame #10: 0x0000000100003208 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::allocator_traits<std::__1::allocator<hwy::Topology::Cluster>>::deallocate[abi:ne180100](__a=0x00006000037fc040, __p=<unavailable>, __n=<unavailable>) at allocator_traits.h:289:9 [opt]
    frame #11: 0x0000000100003208 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Cluster, std::__1::allocator<hwy::Topology::Cluster>>::__destroy_vector::operator()[abi:ne180100](this=<unavailable>) at vector:492:9 [opt]
    frame #12: 0x00000001000031fb gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Cluster, std::__1::allocator<hwy::Topology::Cluster>>::~vector[abi:ne180100](this=<unavailable>) at vector:501:67 [opt]
    frame #13: 0x00000001000031fb gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Cluster, std::__1::allocator<hwy::Topology::Cluster>>::~vector[abi:ne180100](this=<unavailable>) at vector:501:65 [opt]
    frame #14: 0x00000001000031fb gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] hwy::Topology::Package::~Package(this=<unavailable>) at topology.h:84:10 [opt]
    frame #15: 0x00000001000031e9 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] hwy::Topology::Package::~Package(this=<unavailable>) at topology.h:84:10 [opt]
    frame #16: 0x00000001000031e9 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::allocator<hwy::Topology::Package>::destroy[abi:ne180100](this=0x00007ff7bfeff050, __p=<unavailable>) at allocator.h:176:87 [opt]
    frame #17: 0x00000001000031e9 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] void std::__1::allocator_traits<std::__1::allocator<hwy::Topology::Package>>::destroy[abi:ne180100]<hwy::Topology::Package, void>(__a=0x00007ff7bfeff050, __p=<unavailable>) at allocator_traits.h:311:9 [opt]
    frame #18: 0x00000001000031e9 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Package, std::__1::allocator<hwy::Topology::Package>>::__base_destruct_at_end[abi:ne180100](this=0x00007ff7bfeff040 size=1, __new_last=0x00006000037fc000) at vector:926:7 [opt]
    frame #19: 0x00000001000031c4 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Package, std::__1::allocator<hwy::Topology::Package>>::__clear[abi:ne180100](this=0x00007ff7bfeff040 size=1) at vector:920:5 [opt]
    frame #20: 0x00000001000031c4 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Package, std::__1::allocator<hwy::Topology::Package>>::__destroy_vector::operator()[abi:ne180100](this=<unavailable>) at vector:490:16 [opt]
    frame #21: 0x00000001000031b8 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Package, std::__1::allocator<hwy::Topology::Package>>::~vector[abi:ne180100](this=0x00007ff7bfeff040 size=1) at vector:501:67 [opt]
    frame #22: 0x00000001000031b8 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] std::__1::vector<hwy::Topology::Package, std::__1::allocator<hwy::Topology::Package>>::~vector[abi:ne180100](this=0x00007ff7bfeff040 size=1) at vector:501:65 [opt]
    frame #23: 0x00000001000031b8 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] hwy::Topology::~Topology(this=0x00007ff7bfeff040) at topology.h:63:8 [opt]
    frame #24: 0x00000001000031a0 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] hwy::Topology::~Topology(this=0x00007ff7bfeff040) at topology.h:63:8 [opt]
    frame #25: 0x00000001000031a0 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] gcpp::BoundedTopology::~BoundedTopology(this=0x00007ff7bfeff040) at topology.h:83:7 [opt]
    frame #26: 0x0000000100003144 gemma`gcpp::Run(gcpp::LoaderArgs&, gcpp::InferenceArgs&, gcpp::AppArgs&) [inlined] gcpp::BoundedTopology::~BoundedTopology(this=0x00007ff7bfeff040) at topology.h:83:7 [opt]
    frame #27: 0x0000000100003144 gemma`gcpp::Run(loader=0x00007ff7bfeff210, inference=0x00007ff7bfeff298, app=<unavailable>) at run.cc:271:1 [opt]
    frame #28: 0x0000000100003721 gemma`main(argc=<unavailable>, argv=<unavailable>) at run.cc:303:5 [opt]
    frame #29: 0x00007ff804a0d2cd dyld`start + 1805

More information: Disabling topology by setting the compiler flag -DGEMMA_DISABLE_TOPOLOGY=1 could avoid the core dump, which seems like caused by some components in highway library.

[jan-wassenberg]

hm, interesting. Looks like we may have memory corruption early during init, and I agree it's likely in Highway. Are you able to run with -fsanitize=address and =memory? Those often are able to catch the issue. We don't see any issue on Linux, so it's probably in the Apple-specific code.