Codestral (25.01)
A cutting-edge model specifically designed for code generation, including fill-in-the-middle and code completion.Codestral (24.12) is explicitly designed for code generation tasks. It helps developers write and interact with code through a shared instruction and completion API endpoint. As it masters code and can also converse in a variety of languages, it can be used to design advanced AI applications for software developers.
Characteristics | Codestral (24.12) | Mistral Large (24.11) | Mistral Nemo |
---|---|---|---|
Preferred use-cases | Code-specific tasks to enhance developer productivity (e.g. autocompletion, automated code review, test suite generation) | Complex tasks requiring advanced reasoning abilities or a high level of specialization (e.g. creative writing, agentic workflows, code generation) | Streamlined tasks that one can do in bulk (e.g. classification, customer support, text generation) |
Enable the Vertex AI API.
Authenticate with one of the standard mechanisms documented here.
Execute the following commands/script in Cloud Shell or a local terminal window with the gcloud CLI installed. Authenticate and replace PROJECT_ID with your Google Cloud project ID. You can find the supported regions here.
You can send a POST request to the specified API endpoint to get a response from the Mistral model. Find here, Mistral's API documentation and more info on parameter settings such as temperature, top_p, max_tokens, etc.
Please be aware that safe_prompt
is the only parameter from Mistral API not supported for now. This will be supported soon. You will be able to set safe_prompt:true
to enable the optional system prompt in order to enforce guardrails on top of Mistral models for chat completion.
You will need to install the httpx and the google-auth packages in your virtual environment. You will also need to set the GOOGLE_REGION and GOOGLE_PROJECT_ID environment variables respectively to the target region and project ids to use. You can find the supported regions here.
Model | Context length | HumanEval | MBPP | CruxEval | LiveCodeBench | RepoBench | Spider | CanItEdit | HumanEval (average) | HumanEvalFIM (average) |
---|---|---|---|---|---|---|---|---|---|---|
Codestral-2501 | 256k | 86.6% | 80.2% | 55.5% | 37.9% | 38.0% | 66.5% | 50.5% | 71.4% | 85.9% |
Codestral-2405 22B | 32k | 81.1% | 78.2% | 51.3% | 31.5% | 34.0% | 63.5% | 50.5% | 65,6% | 82.1% |
Codellama 70B instruct | 4k | 67.1% | 70.8% | 47.3% | 20.0% | 11.4% | 37.0% | 29.5% | 55.3% | - |
DeepSeek Coder 33B instruct | 16k | 77.4% | 80.2% | 49.5% | 27.0% | 28.4% | 60.0% | 47.6% | 65.1% | 85.3% |
DeepSeek Coder V2 lite | 128k | 83.5% | 83.2% | 49.7% | 28.1 | 20.0% | 72.0% | 41.0% | 65.9% | 84.1% |
Model | HumanEval Python | HumanEval C++ | HumanEval Java | HumanEval Javascript | HumanEval Bash | HumanEval TypeScript | HumanEval C# | HumanEval (average) |
---|---|---|---|---|---|---|---|---|
Codestral-2501 | 86.6% | 78.9% | 72.8% | 82.6% | 43.0% | 82.4% | 53.2% | 71.4% |
Codestral-2405 22B | 81.1% | 68.9% | 78.5% | 71.4% | 40.5% | 74.8% | 43.7% | 65,6% |
Codellama 70B instruct | 67.1% | 56.5% | 60.8% | 62.7% | 32.3% | 61.0% | 46.8% | 55.3% |
DeepSeek Coder 33B instruct | 77.4% | 65.8% | 73.4% | 73.3% | 39.2% | 77.4% | 49.4% | 65.1% |
DeepSeek Coder V2 lite | 83.5% | 68.3% | 65.2% | 80.8% | 34.2% | 82.4% | 46.8% | 65.9% |
Model | HumanEvalFIM Python | HumanEvalFIM Java | HumanEvalFIM JS | HumanEvalFIM (average) |
---|---|---|---|---|
Codestral-2501 | 80.2% | 89.6% | 87.96% | 85.89% |
Codestral-2405 22B | 77.0% | 83.2% | 86.08% | 82.07% |
OpenAI FIM API* | 80.0% | 84.8% | 86.5% | 83.7% |
DeepSeek Chat API | 78.8% | 89.2% | 85.78% | 84.63% |
DeepSeek Coder V2 lite | 78.7% | 87.8% | 85.90% | 84.13% |
DeepSeek Coder 33B instruct | 80.1% | 89.0% | 86.80% | 85.3% |
Model | HumanEvalFIM Python | HumanEvalFIM Java | HumanEvalFIM JS | HumanEvalFIM (average) |
---|---|---|---|---|
**Codestral-2501 ** | ** 92.5%** | 97.1% | 96.1% | 95.3% |
Codestral-2405 22B | 90.2% | 90.1% | 95.0% | 91.8% |
OpenAI FIM API* | 91.1% | 91.8% | 95.2% | 92.7% |
DeepSeek Chat API | 91.7% | 96.1% | 95.3% | 94.4% |
Instruct example
Sample Input (POST request payload)
Sample Output
Fill-in-the-middle example
Sample Input (POST request payload)
Sample Output
Fill-in-the-middle mode fills the void between the prompt and the suffix provided by the user. It can also be used in a “completion” mode by setting the suffix to an empty string (see example here).
To avoid having a model being too verbose or to prevent infinite generation issues, you can specify an optional stop token by adding the stop value to the payload (see example here).
Resource ID | Release date | Release stage | Description |
---|---|---|---|
codestral-2501 | 2025-01-13 | General Availability |
Codestral blog post and Model Card
Mistral’s documentation: Visit Mistral's documentation for a wealth of resources on model capabilities, prompting techniques, use case guidelines, and more.
Mistral Cookbook repository: Check out example code for a variety of tasks.
Google Cloud Console has failed to load JavaScript sources from www.gstatic.com.
Possible reasons are: