[report bug] Encountered when inferencing with Mistral models #124
Description
I met the following error when inferencing with base_model Mistral-7b-Instruct-v0.2
File "~/Medusa/medusa/model/modeling_mistral_kv.py", line 74, in _make_sliding_window_causal_mask
mask = torch.triu(mask, diagonal=-sliding_window)
^^^^^^^^^^^^^^^
TypeError: bad operand type for unary -: 'NoneType'
I think I have found the reason: the sliding_window value is null for both Mistral-7b-Instruct-v0.2 and Mistral-7b-Instruct-v0.3, which triggers the bug above.
To fix it, I suggest the author add a line to file "medusa/model/modeling_mistral_kv.py", line 74, as follows:
switch
mask = torch.triu(mask, diagonal=-sliding_window)
into
if sliding_window is not None:
mask = torch.triu(mask, diagonal=-sliding_window)