# GraLoRA

**Repository Path**: jackmacoder/GraLoRA

## Basic Information

- **Project Name**: GraLoRA
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: Not specified
- **Default Branch**: main
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2025-10-29
- **Last Updated**: 2025-10-29

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# [NeurIPS2025 Spotlight🌟] GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning

This repository contains the official implementation of **GraLoRA**.

![GraLoRA](./figure/gralora_overview.png)

---

## Environment Setup

To install the required dependencies:

```bash
conda create -n gralora python=3.10 -y
conda activate gralora
pip3 install -r requirements.txt
pip3 install -r additional_requirements.txt
pip3 install -e ./peft --config-settings editable_mode=compat
pip3 install -e ./lm-evaluation-harness --config-settings editable_mode=compat
pip3 install -e ./bigcode-evaluation-harness --config-settings editable_mode=compat
```
For a stable setup, it is recommended to work on top of the `nvcr.io/nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04` Docker image.

---

## Download Datasets

### Training Datasets

1. **Code Generation Training Dataset**
    ```bash
    python3 scripts/tools/dl_dataset.py ise-uiuc/Magicoder-Evol-Instruct-110K data/ise-uiuc/Magicoder-Evol-Instruct-110K
    ```

2. **Commonsense Reasoning Training Dataset**
    ```bash
    python3 scripts/tools/dl_dataset.py zwhe99/commonsense_170k data/zwhe99/commonsense_170k
    ```

### Evaluation Datasets

1. **Code Generation**: This dataset is automatically downloaded when running the evaluation script.

2. **Commonsense Reasoning**:  
   Download manually from [LLM-Adapters Dataset](https://github.com/AGI-Edgerunners/LLM-Adapters/tree/main/dataset)  
   and store the dataset in `./data/`.

---

## Training Scripts

Training scripts for all main experiments are available in `./scripts/train`.

To reproduce a specific experiment, run:

```bash
./scripts/train/$TARGET_TRAIN_SCRIPT
```

A detailed explanation of how to run the scripts can be found in the [README_TRAIN.md](./scripts/train/README_TRAIN.md) file.

We provide separate scripts for each task. While their behavior is nearly identical, the **instruction formats** for Alpaca-chat differ slightly. We adopted the chat formats used by the original repositories:  
- [RaSA](https://github.com/zwhe99/RaSA) for Code Generation  
- [LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters) for Commonsense Reasoning

---

## Evaluation Scripts

Evaluation scripts for all tasks are located in `./scripts/eval`.

To run an evaluation:

```bash
./scripts/eval/$TARGET_EVAL_SCRIPT
```

A detailed explanation of how to run the scripts can be found in the [README_EVAL.md](./scripts/eval/README_EVAL.md) file.

### Code Generation

#### 1. Generate Responses

```bash
EXP_DIR=path_to_this_experiment
python3 code_evaluation.py \
    --model $PATH_TO_BASE_MODEL \
    --peft_model $PATH_TO_PEFT_MODEL \
    --tasks humanevalsynthesize-python \
    --prompt alpaca-chat \
    --do_sample True \
    --temperature 0.2 \
    --n_samples 50 \
    --batch_size 20 \
    --max_length 2048 \
    --allow_code_execution \
    --precision bf16 \
    --metric_output_path $EXP_DIR/metric_output_path.json \
    --save_generations \
    --save_generations_path $EXP_DIR/save_generations_path.json \
    --generation_only
```

#### 2. Compute pass@1, pass@5, and pass@10

```bash
python3 code_evaluation.py \
    --tasks humanevalplus \
    --n_samples 50 \
    --num_workers 48 \
    --timeout 20 \
    --k 1 5 10 \
    --allow_code_execution \
    --metric_output_path $EXP_DIR/metric_output_path.json \
    --load_generations_path $EXP_DIR/save_generations_path_humanevalsynthesize-python.json \
    --results_path $EXP_DIR/results.json
```

The final result will be saved at `EXP_DIR/results.json`.

---

### Commonsense Reasoning

```bash
EXP_DIR=path_to_this_experiment
TASK_NAME=target_task_name
python3 commonsense_evaluate.py \
    --base_model $PATH_TO_BASE_MODEL \
    --peft_model $PATH_TO_PEFT_MODEL \
    --dataset $TASK_NAME \
    --bf16 | tee -a ${EXP_DIR}/${TASK_NAME}.txt
```

The final result will be saved at `EXP_DIR/${TASK_NAME}.txt`.

---

## Results

GraLoRA achieves superior performance on the HumanEval+ task:

![Result](./figure/code_generation_result.png)


## 🔥 News 🔥

- [2025/09] GraLoRA is accepted by NeurIPS 2025 as a **spotlight**🌟
- [2025/05] GraLoRA is open-sourced!

## 🔗 BibTeX 🔗

```bibtex
@article{jung2025gralora,
      title={GraLoRA: Granular Low-Rank Adaptation for Parameter-Efficient Fine-Tuning}, 
      author={Yeonjoon Jung and Daehyun Ahn and Hyungjun Kim and Taesu Kim and Eunhyeok Park},
      year={2025},
      eprint={2505.20355},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/2505.20355}, 
}
```

## 🤝 Acknowledgements

This work builds upon the contributions of the following repositories:

- [PEFT (Hugging Face)](https://github.com/huggingface/peft)
- [LLM-Adapters](https://github.com/AGI-Edgerunners/LLM-Adapters)
- [RaSA](https://github.com/zwhe99/RaSA)